A guardrail segmentation method based on convolutional neural network

Abstract

The present invention discloses a guardrail segmentation method based on a convolutional neural network, comprising: obtaining a picture containing a guardrail and marking a fence area; constructing a segmentation network framework of deeplabV3+ based on a weighted loss function; inputting the marked guardrail picture into the segmentation network framework of deeplabV3+ based on a weighted loss function to train a segmentation network model of deeplabV3+ based on a weighted loss function, until the segmentation network model of deeplabV3+ based on a weighted loss function outputs a binary image that meets a preset resolution threshold and only contains the guardrail; after the picture to be detected is manually annotated, the annotated picture is input into the trained network model of deeplabV3+ based on a weighted loss function to output the segmented binary image of the picture to be detected. The generated binary image is fitted into a rectangle through the marked 9 points of the drawn guardrail area by a quadratic linear interpolation method. By comparing the generated rectangles, it is determined whether there is a vacancy and whether there is a bend, so as to determine whether the guardrail is missing or bent.

Description

Technical Field

[0001] The present invention relates to the technical field of guardrail detection, and in particular to a guardrail segmentation method based on a convolutional neural network. Background Art

[0002] Guardrails are common traffic facilities for vehicles on urban roads. Detecting roadside guardrails is extremely important for highly automated driving. The presence of roadside guardrails provides physical restrictions on the free space for lateral movement of vehicles. Therefore, guardrail detection is very important.

[0003] At present, there are three main ways to detect guardrails: alarm detection, GPS positioning detection, and straight line detection through images. Among them, guardrail detection based on alarms usually uses a certain detection device with alarm function to detect the condition of the guardrail. If the guardrail is damaged or not in a straight line, the warning function is realized through warning lights, speakers and other alarm devices. GPS positioning detection usually includes a mobile platform, a displacement sensor, a GPS module, a full network module and an alarm device. When in use, the device is close to the column to be detected, fixed, and the force-applying device is moved to a specific height and distance close to the column through the three-dimensional mobile platform in the detection device. Then the force-applying device is started to apply force to the column. After the column is balanced, the displacement sensor measures the displacement of the column. By comparing the measurement result with the displacement standard value, it is confirmed whether the guardrail is defective. For straight line detection based on images, the image straight line detection method is usually used to detect the collected pictures. The specific process is to perform straight line fitting after gray value processing and binary processing on the input image. Many straight line areas are obtained, and rectangular areas are formed by expanding the straight line areas, and the rectangular areas are classified and judged. Screen out the squares that are not guardrails, select the guardrails, cut the guardrails, and classify and judge each part to determine whether there is any missing.

[0004] The above two methods of detecting guardrails based on alarms and GPS positioning detection require the modification of guardrails, which is relatively complicated to construct. Moreover, if there is rain or snow for a long time, sensors such as GPS and alarms are prone to failure, which is not conducive to detection. The method based on straight line detection is greatly affected by interference in the image, such as rain or snow, which may cause the guardrail to fail to be detected and lead to false alarms, and has poor robustness. The image-based straight line detection method is greatly affected by interference in the image, such as rain or snow, which may cause the guardrail to fail to be detected and lead to false alarms, and has poor robustness. Summary of the invention

[0005] 1. Technical problems to be solved:

[0006] In response to the above technical problems, the present invention provides a guardrail segmentation method based on a convolutional neural network, which is highly robust and does not require any objects other than electronic probes on site, thus avoiding the phenomenon of failure to detect guardrail defects due to damage to electronic devices. Compared with the straight line detection method, the method proposed in this patent is more robust and can successfully detect guardrails in harsh environments and alarm for missing and bent parts.

[0007] 2. Technical solution:

[0008] A guardrail segmentation method based on a convolutional neural network is characterized by comprising the following steps:

[0009] Step 1: First, use a crawler tool to crawl images containing guardrails on the Internet, and manually mark the guardrail areas in the images.

[0010] Step 2: Perform image enhancement on the image generated in step 1 to form a guardrail dataset.

[0011] Step 3: Construct a segmentation network framework of deeplabV3+ based on the weighted loss function.

[0012] Step 4: Input the above-labeled guardrail image and its labeled true value data into the segmentation network framework of deeplabV3+ based on the weighted loss function to train the segmentation network model of deeplabV3+ based on the weighted loss function until the segmentation network model of deeplabV3+ based on the weighted loss function outputs a binary image that meets the preset resolution threshold and only contains the guardrail.

[0013] Step 5: After manually marking 9 points in the guardrail area of ​​the image to be detected, the marked image is input into the trained deeplabV3+ network model based on the weighted loss function to output the segmented binary image of the image to be detected.

[0014] Step 6: Use the quadratic linear interpolation method to fit the drawn guardrail area into a rectangle through the 9 marked points using the binary image generated in the previous step.

[0015] Step 7: Compare the rectangle generated in step 6 to determine whether the sides of the rectangle are empty or bent, so as to determine whether the guardrail is missing or bent.

[0016] Furthermore, the segmentation network architecture of deeplabV3+ based on the weighted loss function in step three includes a backbone network, an encoder, a decoder, and a loss function.

[0017] Furthermore, the backbone network includes 5 convolution modules, the first convolution module consists of 32 3x3 filters with a step size of 2, the second convolution module consists of 4 1*1 filters plus 32 3*3 filters with a step size of 2 plus 64 filters with a step size of 1; the third convolution module consists of 128 1*1 filters plus 64 3*3 filters with a step size of 2 plus 128 1*·1 filters with a step size of 1; the fourth convolution module first performs upsampling and then passes through 128 1*1 filters plus 64 3*3 filters; the fifth convolution layer first performs deconvolution and then passes through 64 1*1 filters plus 32 3*3 filters.

[0018] Furthermore, the encoder passes the features of the last layer in the backbone network through 32 1*1 filters, 32 3*3 filters, 32 3*3 dilated convolution maximum pooling, and mean pooling, and concatenates the five processed feature maps; then passes them through a 1*1 filter of size 128.

[0019] Furthermore, the decoder first passes the features output by the backbone network through 32 1*1 filters, then upsamples the output features in the encoder by 2 times and connects them to the features, and finally generates an output image through 32 3*3 filters, upsampling and 1 3*3 filter, and finally outputs an output image of size H*W*1.

[0020] Furthermore, the loss function is trained using a weighted loss function, and the loss function formula is as follows:

[0021]

[0022] In the above formula, L is the size of the loss, n is the number of input data, w1 and w2 are weighted values ​​based on importance. Usually w1 is set to 100 and w2 is set to 1. yi is the probability that the current pixel in the output layer is classified as a guardrail.

[0023] Furthermore, the picture to be detected is a picture obtained by a network camera or a picture collected by a mobile device.

[0024] 3. Beneficial effects:

[0025] (1) A convolutional neural network-based guardrail segmentation method proposed in the present invention captures a highway with a camera, and sends the captured image to a deeplabV3+ segmentation network model constructed based on a weighted loss function, and determines whether the guardrail is missing or bent based on the output result.

[0026] (2) The present invention realizes the segmentation of guardrail and non-guardrail areas through a segmentation network framework based on a weighted loss function.

[0027] (3) The present invention generates a binary area from the image captured by the camera, generates a rectangle by selecting a drawing area, and performs a threshold comparison on the rectangular area to determine whether the guardrail is missing or bent.

[0028] In summary, the guardrail detection method provided by the present invention only needs to obtain the images collected on site, adopts non-contact detection technology, does not need to break the road to block traffic for construction, the equipment is stable and not easy to damage, the construction cost is low, and the maintenance is convenient. The equipment collects high-definition images and uses the images for guardrail detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The binary image of the image to be detected that only contains the guardrail output in step 5 of the present invention;

[0030] Figure 2 It is the overall flow chart of the present invention;

[0031] Figure 3 A flowchart of establishing a network model of deeplabV3+ based on a weighted loss function in the present invention;

[0032] Figure 4 A schematic diagram of an encoder and a decoder of a segmentation network architecture of deeplabV3+ based on a weighted loss function in the present invention;

[0033] Figure 5 is a binary image when the guardrail is missing in a specific embodiment;

[0034] Figure 6 A binary image of a guardrail when it is bent in a specific embodiment. DETAILED DESCRIPTION

[0035] The present invention will be described in detail below with reference to the accompanying drawings.

[0036] A guardrail segmentation method based on a convolutional neural network is characterized by comprising the following steps:

[0037] Step 1: First, use a crawler tool to crawl images containing guardrails on the Internet, and manually mark the guardrail areas in the images.

[0038] Step 2: Perform image enhancement on the image generated in step 1 to form a guardrail dataset.

[0039] Step 3: Construct a segmentation network framework of deeplabV3+ based on the weighted loss function.

[0040] Step 4: Input the above-labeled guardrail image and its labeled true value data into the segmentation network framework of deeplabV3+ based on the weighted loss function to train the segmentation network model of deeplabV3+ based on the weighted loss function until the segmentation network model of deeplabV3+ based on the weighted loss function outputs a binary image that meets the preset resolution threshold and only contains the guardrail.

[0041] Step 5: After manually marking 9 points in the guardrail area of ​​the image to be detected, the marked image is input into the trained deeplabV3+ network model based on the weighted loss function to output the segmented binary image of the image to be detected.

[0042] Step 6: Use the quadratic linear interpolation method to fit the drawn guardrail area into a rectangle through the 9 marked points using the binary image generated in the previous step.

[0043] Step 7: Compare the rectangle generated in step 6 to determine whether the sides of the rectangle are empty or bent, so as to determine whether the guardrail is missing or bent.

[0044] Furthermore, the segmentation network architecture of deeplabV3+ based on the weighted loss function in step three includes a backbone network, an encoder, a decoder, and a loss function.

[0045] Furthermore, the backbone network includes 5 convolution modules, the first convolution module consists of 32 3x3 filters with a step size of 2, the second convolution module consists of 4 1*1 filters plus 32 3*3 filters with a step size of 2 plus 64 filters with a step size of 1; the second convolution module consists of 128 1*1 filters plus 64 3*3 filters with a step size of 2 plus 128 1*·1 filters with a step size of 1; the fourth convolution module first performs upsampling and then passes through 128 1*1 filters plus 64 3*3 filters; the fifth convolution layer first performs deconvolution and then passes through 64 1*1 filters plus 32 3*3 filters;

[0046] Furthermore, the encoder passes the features of the last layer in the backbone network through 32 1*1 filters, 32 3*3 filters, 32 3*3 dilated convolution maximum pooling, and mean pooling, and performs a splicing operation on the processed five feature maps; then passes them through a 1*1 filter of size 128;

[0047] Furthermore, the decoder first passes the features output by the backbone network through 32 1*1 filters, then upsamples the output features in the encoder by 2 times and connects them to the features, and finally generates an output image through 32 3*3 filters, upsampling and 1 3*3 filter, and finally outputs an output image of size H*W*1.

[0048] Furthermore, the loss function is trained using a weighted loss function, and the loss function formula is as follows:

[0049]

[0050] In the above formula, L is the size of the loss, n is the number of input data, w1 and w2 are weighted values ​​based on importance. Usually w1 is set to 100 and w2 is set to 1. yi is the probability that the current pixel in the output layer is classified as a guardrail.

[0051] Furthermore, the picture to be detected is a picture obtained by a network camera or a picture collected by a mobile device. Specific embodiment:

[0053] As attached Figure 1 The figure shows a binary image of a picture to be detected which only contains guardrails in the present invention, and the white part in the figure is the guardrail area.

[0054] Attached Figure 2 This is the overall flow chart of the present invention. As shown in the figure, the network model of deeplabV3+ based on the weighted loss function is first trained, and then the guardrail binary image conversion is performed on the image to be detected. Finally, the guardrail area of ​​the converted image is detected to see if there is any missing or curved image.

[0055] As attached Figure 3 As shown, it is a flow chart of establishing a network model of deeplabV3+ based on the weighted loss function in the present invention. After establishing a guardrail database, manually marking the guardrail area in the picture, and strengthening the picture, it is input into the network architecture of deeplabV3+ of the weighted loss function for training, and a binary image that meets the requirements is trained.

[0056] As attached Figure 4 As shown in the figure, it is a schematic diagram of the backbone network and encoder of the deeplabV3+ segmentation network architecture based on the weighted loss function in the present invention; the network architecture includes the backbone network, the encoder, and the decoder. The backbone network adopts the Xception network architecture based on deep separable. Xception first implements downsampling through a 3x3 convolutional network with a step size of 2 and a 3x3x64 convolutional network. The subsequent network passes through 3 identical deep separable convolutional layers. The deep separable network layer is as follows Figure 4As shown in the attached Figure 5 As shown in the figure, encoder: The encoder passes the feature extraction layer information through 1*1 convolution layer, 3*3 convolution layer, 3*3 hole convolution, maximum pooling, mean pooling, and concatenates the information obtained through these layers and then performs dimensionality reduction operation through 1*1 convolution layer. The decoder uses 1x1 convolution layer to reduce the dimensionality of the information obtained from the feature extraction layer, and concatenates this layer with the four-fold downsampling output of the encoder, and finally outputs it.

[0057] As attached Figure 5 It is a binary image when the guardrail is judged to be missing during the specific implementation process, as shown in the attached figure. Figure 6 It is a binary image when the guardrail is judged to be bent during the specific implementation process.

[0058] Although the present invention has been disclosed as above in terms of preferred embodiments, they are not intended to limit the present invention. Anyone skilled in the art can make various changes or modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection defined by the claims of this application.

Claims

1. A guardrail segmentation method based on convolutional neural network, characterized by: The following steps are involved: Step 1: First, use a crawler tool to crawl images containing guardrails on the Internet, and manually mark the guardrail areas in the images; Step 2: Perform image enhancement on the images generated in step 1 to form a guardrail dataset Step 3: Construct a segmentation network framework of deeplabV3+ based on the weighted loss function; Step 4: Input the marked guardrail image and its marked true value data into the segmentation network framework of deeplabV3+ based on the weighted loss function to train the segmentation network model of deeplabV3+ based on the weighted loss function, until the segmentation network model of deeplabV3+ based on the weighted loss function outputs a binary image that meets the preset resolution threshold and only contains the guardrail; Step 5: After manually marking 9 points in the guardrail area of ​​the image to be tested, the marked image is input into the trained deeplabV3+ network model based on the weighted loss function to output the segmented binary image of the image to be tested; Step 6: Use the quadratic linear interpolation method to fit the drawn guardrail area into a rectangle through the 9 marked points using the binary image generated in the previous step; Step 7: Compare the rectangle generated in step 6 to determine whether the sides of the rectangle are empty or bent, so as to determine whether the guardrail is missing or bent; The deeplabV3+ segmentation network architecture based on the weighted loss function in step 3 includes a backbone network, an encoder, a decoder, and a loss function; The backbone network includes 5 convolution modules, the first convolution module consists of 32 3x3 filters with a step size of 2, the second convolution module consists of 4 1*1 filters plus 32 3*3 filters with a step size of 2 plus 64 filters with a step size of 1; the second convolution module consists of 128 1*1 filters plus 64 3*3 filters with a step size of 2 plus 128 1*1 filters with a step size of 1; the fourth convolution module first performs upsampling and then passes through 128 1*1 filters plus 64 3*3 filters; the fifth convolution layer first performs deconvolution and then passes through 64 1*1 filters plus 32 3*3 filters; The encoder passes the features of the last layer in the backbone network through 32 1*1 filters, 32 3*3 filters, 32 3*3 dilated convolution maximum pooling, and mean pooling, and performs a splicing operation on the processed five feature maps; then passes them through a 1*1 filter of size 128; The decoder first passes the features output by the backbone network through 32 1*1 filters, then upsamples the output features in the encoder by 2 times and connects them to it, and finally generates an output image through 32 3*3 filters, upsampling and 1 3*3 filter, and finally outputs an output image of size H*W*1; The loss function is trained using a weighted loss function, and the loss function formula is as follows: In the above formula, L is the size of the loss, n is the number of input data, w1 and w2 are weighted values ​​based on importance. Usually w1 is set to 100 and w2 is set to 1. yi is the probability that the current pixel in the output layer is classified as a guardrail.

2. The guardrail segmentation method based on convolutional neural network according to claim 1, characterized in that: The image to be detected is an image acquired through a network camera or an image collected by a mobile device.

Images