Road obstacle detection method based on spatio-temporal context

Abstract

The application relates to a kind of road obstacle detection methods based on space-time context. First, the spatial distribution of obstacle and road is derived using data-driven method to establish scene layout model using spatial context information, so as to eliminate misdetected targets outside the distribution of road scene and enhance the confidence of obstacle targets in road area;Then, an obstacle tracking method based on optical flow and object region selection is proposed to track targets using time context information, so as to eliminate the missed detection caused by environmental changes in the detection process;Finally, the above two kinds of context information are fused to detect the road surface obstacle. Experimental results show that the method of the application effectively improves the detection accuracy of road surface obstacle.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a road obstacle detection method based on space-time context. BACKGROUND

[0002] With the rapid development of economy, cars have become a part of people's daily travel, and the resulting traffic safety problem has become a global concern. Among them, the traffic order is not ideal, and the defects or obstacles on the road are important factors affecting traffic safety. In the process of high-speed driving of vehicles, some unpredictable factors can cause traffic accidents, and the obstacles in the road that are not cleaned in time are one of the important factors. Vehicle skidding, loss of control and scratching caused by obstacles occur frequently, which seriously affects traffic safety. Therefore, the detection of obstacles in the road is an effective means to ensure driving safety, reduce the number of traffic accidents and reduce casualties. At the same time, road obstacle detection is an important content in the current international intelligent transportation system research field. Intelligent vehicles use machine vision technology to sense road environment information, analyze and understand the information in the driving environment to quickly locate the position and detect the size of the obstacle, and feedback to the driver in time, so as to realize the safe auxiliary driving of intelligent vehicles.

[0003] Road obstacle detection aims to locate the obstacle target in front of the vehicle from the driving monitoring image, such as Figure 1 The current road obstacle detection methods mainly include infrared detection, radar detection, machine vision-based detection and multi-sensor fusion detection. However, the radar and infrared sensors are often disturbed by the external environment and cannot work normally. The machine vision-based detection method uses a camera to replace the human eye to obtain the surrounding environment information, which has the characteristics of strong anti-interference, fast sampling speed, rich information acquisition and low cost. Compared with laser, radar and ultrasonic signals, image signals have great advantages in describing the integrity, breadth and human cognitive habits of target information. Therefore, the present application mainly uses the machine vision-based detection method to detect the road obstacles.

[0004] Current object detection methods are mainly based on convolutional neural networks (CNN). Broadly speaking, these algorithms can be divided into two-stage object detection algorithms and one-stage object detection algorithms. The most representative of two-stage object detection algorithms is the R-CNN series, which mainly generates candidate regions first, and then further classifies and adjusts the bounding box according to the generated candidate regions. The original R-CNN uses CNNs for classification and positioning of objects in candidate regions, but this requires repeated calculation of convolutional features for each candidate region, which is too slow for real-time applications. Fast R-CNN is faster than R-CNN because it extracts region features from shared features generated from the entire image to save redundant calculations. Faster R-CNN [1] combines object proposal and classification into a unified model, thereby improving the efficiency of the network and allowing end-to-end training. On the other hand, the YOLO series is a classic example of one-stage object detection algorithms. It treats object detection as a regression problem, and YOLO can directly predict bounding boxes and target classes without generating proposals and region refinement, making it more suitable for real-time applications. In the present method, we choose Faster RCNN [1] and YOLOv5 [2] as our benchmark object detectors and explore how to fuse spatio-temporal context information to improve their results. SUMMARY

[0005] The present application aims to overcome the problem that existing road obstacle detection methods generally lack the guidance of context information, and when there are motion blur, light changes or partial occlusion between video frames during the detection process, some false positives and missed detections often occur, thereby affecting the detection accuracy. A road obstacle detection method based on spatio-temporal context is provided, which effectively improves the detection accuracy of road obstacles.

[0006] To achieve the above-mentioned purpose, the technical scheme of the present application is: a road obstacle detection method based on spatio-temporal context, first, a data-driven method is used to derive the spatial distribution of obstacles and roads to establish a scene layout model using spatial context information, thereby eliminating false positives outside the road scene distribution and enhancing the confidence of obstacle targets in the road area; then, an obstacle tracking method based on optical flow and object region selection is proposed to track targets using temporal context information, thereby eliminating missed detections caused by environmental changes during the detection process; finally, the two kinds of context information are fused to detect the target and remove background noise.

[0007] Compared with the prior art, the method has the following beneficial effects: the method first uses a data-driven method to derive the spatial distribution of obstacles and roads to establish a scene layout model using spatial context information, thereby eliminating false detection targets outside the road scene distribution and enhancing the confidence of obstacle targets in the road area. Then a method of obstacle tracking based on optical flow and object region selection is proposed to track the target using temporal context information, thereby eliminating the missed detection caused by environmental changes in the detection process. Finally, the above two context information is fused to detect the target. The method effectively improves the detection accuracy of road obstacles. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 An example image of a road obstacle.

[0009] Figure 2 A flowchart of the method of the present application.

[0010] Figure 3 A scene layout construction flowchart.

[0011] Figure 4 A road distribution construction flowchart.

[0012] Figure 5 An optical flow tracking schematic diagram.

[0013] Figure 6 An example detection result of the SOD dataset test set. GT: input image with labeled real frame. Faster RCNN: detection result using Faster RCNN. Faster RCNN+SL+OF: detection result using Faster RCNN assisted by scene layout and optical flow. YOLOv5: detection using YOLOv5. YOLOv5+SL+OF: detection result using YOLOv5 assisted by scene layout and optical flow. Black frame: false positive, white frame: true positive.

[0014] Figure 7 An example of detection results of the Lost and Found dataset test set. GT: input image with labeled real frame. Faster RCNN: detection result using Faster RCNN. Faster RCNN+SL: detection result of Faster RCNN plus scene layout. Black frame: false positive, white frame: true positive. DETAILED DESCRIPTION

[0015] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings.

[0016] The application discloses a road obstacle detection method based on space-time context.

[0017] The following is the specific implementation process of the application.

[0018] In the application, a general framework is proposed to integrate space-time context into target detection, which can work with any target detection algorithm outputting bounding boxes. First, the target detection result is obtained through a detector, and then a scene layout model is used to calculate the space context score of each detection box. Then, the score is combined with the detection result to suppress the false detection box in the background. Finally, the object is tracked through the optical flow so as to provide additional support for target detection in frames with weak visual clues. The execution flow of the method of the application is shown in Figure 2 .

[0019] More specifically, it is assumed that at time t, we have an image I t as input. Let the object hypotheses be x e X, where X is the object pose space. To simplify the notation, we assume that each object is x = (x c ,b s ,b r ,o), where x c = (b x ,b y ) is the image coordinate position of the object center, b s = (b w ,b h ) is a scale, b r is an aspect ratio, and o e O represents a target class. At the same time, each x also represents a target bounding box. The object detection algorithm defines a score function S d (x) for each valid object hypothesis x. For example, in FasterRCNN, the score is usually obtained through multi-class softmax scoring on the convolution feature map. For any given object hypothesis x, a scene layout score S L (x) is proposed. The final detection score is the weighted sum of the two scores:

[0020] S(x) = S D (x) + theta S L (x) (1)

[0021] where θ is a hyper-parameter of the relative importance between two terms. Moreover, since the image of the current frame is I t , we define the image of the next frame as I (t+δ) . Similarly, for a given object hypothesis x in the current frame, the bounding box generated by the optical flow tracking in the next frame is denoted as x (t+δ) . And for the bounding box generated by the optical flow, we redefine a scoring function as shown in equation (4).

[0022] 1. Scene layout construction

[0023] 1.1. Obstacle layout construction

[0024] In real driving scenarios, the spatial distribution of obstacles has a strong regularity. We usually focus on the obstacles in front of the vehicle that may affect driving safety. At the same time, due to the distance of these obstacles, they can be very small, and due to motion blur, lighting changes or partial occlusion, their visual information can be very weak. Therefore, the detector usually has low confidence in these objects, and in practice, it is impossible to reliably detect these objects. To solve the above problems, we statistically analyze the distribution of obstacles in the training set. We first obtain the real bounding box of all obstacle targets in the training set, and take all their center points to obtain the two-dimensional spatial distribution in the form of a heat map. The result is shown in Fig. Figure 3 (a), which shows that the distribution of obstacles in the image has a strong regularity, mainly concentrated in the center of the image. We can use this distribution as prior information to regularize the target detection results.

[0025] 1.2. Road layout construction

[0026] At the same time, we further obtain the road distribution to assist the construction of the scene layout model, so as to effectively detect the obstacles in the road and appropriately ignore the misdetected targets outside the road. We notice that the road segmentation annotations are provided in the Small Obstacle Detection dataset and the Lost and Found dataset, so we can easily obtain the road profile in the training set. In order to obtain the final road profile, we take several points along the road profile in the annotation of each image to obtain the average road profile in multiple images. Since the obstacle distribution we obtained in the previous step may not be uniformly distributed in the road area, we propose a method to map the obstacle distribution to the entire road area. Specifically, according to the distribution of obstacles, different weights are given in the horizontal and vertical directions of the road area, as shown in Fig. Figure 4 It can be seen that for our daily driving scenarios, the road gradually opens from far to near, roughly in the shape of a pyramid.

[0027] 1.3, Obstacle detection based on scene layout

[0028] The obstacle distribution and road distribution are fused and then normalized to obtain the final road scene layout, as shown in Figure 3 (c). After that, we can generate a score S L (x) for each object hypothesis x by the scene layout. Then we can combine the object hypothesis score x obtained from the object detector S D (x) with the scene layout score S L (x) to obtain the final score S(x). The definition of S L (x) is as follows:

[0029]

[0030] Here S L (x) is the final score of object x in the scene layout, M(x) is the position score of object x in the scene layout, a is a variable parameter to adjust the scene layout score, and b is a fixed bias value of the scene layout score.

[0031] According to formula 2, we assume that when the score M(x) < 0.15, the target is located outside the road distribution, which will be considered as a false detection. When the score is 0.15 < M(x) < 0.6, we think that the target is located within the road area but close to the boundary, so the detection score does not change. When the score M(x) > 0.6, it is considered that the object appears in the position where we are most likely to detect the obstacle, and the score is increased accordingly.

[0032] 2, Object tracking based on optical flow field

[0033] We use the Lucas Kanade (LK) method to calculate the optical flow between two consecutive frames. We consider two frames from time t to t+δ here to realize the tracking of the detected object. The LK method is generally based on the following three assumptions: the brightness of the part of the object being tracked in the scene remains basically unchanged; the movement is relatively slow relative to the frame rate; adjacent points on the same surface in the scene should have similar motion. In addition, we also introduce an image pyramid to improve its performance. By reducing the size of the image, the movement speed of the object in the image is relatively reduced, so that the object with faster movement speed can be better tracked.

[0034] When detecting and tracking road obstacles, the above assumptions may not be met due to constantly changing motion, occlusion, etc. Therefore, instead of calculating the optical flow of the entire scene, we focus on the vicinity of the detected object. Therefore, we can eliminate unnecessary interference in the scene, as shown in Figure 5 The specific steps are as follows:

[0035] (1) The detector is used to detect obstacles, obtain the detection results of the previous frame and the next frame, and select the previous frame score S D (x) > 0.3 detection.

[0036] (2) In order to determine whether the subsequent frame is missed, we define a search area A r by expanding the detection bounding box of the detection object in the subsequent frame. r The definition of search area A

[0037] A r (x) = (b x , b y , yb w + t, b h + t) (3)

[0038] Where y and b are the adjustment coefficients of the selected area width and height, and t represents the initial size of the area. Here, a > 1, b > 1, t > 0, because we only enlarge the detection box to define the search area.

[0039] (3) If the obstacle is missed in the next frame (i.e. there is no bounding box of the same obstacle class inside A r (x)), we crop the search area given in step 2 in the previous frame and the next frame, and use the Shi-Tomasi corner detection algorithm to obtain the image corner points in the cropped area, and select the feature corner points located on the detection object.

[0040] (4) Then use the Lucas Kanade optical flow method to track the feature corner points in the cropped area to obtain the offset of the feature corner points in the previous and next two frames.

[0041] (5) We transfer the detection bounding box in the previous frame to the next frame by the obtained offset.

[0042] 3. Object detection based on spatio-temporal perception

[0043] On the basis of the above spatio-temporal context modeling, we combine it into the final inference process. First, the detector detects the obstacles in the input image, and then uses the scene layout model to process the detection results to eliminate false detections outside the road scene distribution and enhance the credibility of small obstacles in the road area. Second, through obstacle tracking and frame transfer, the missed bounding box in the road area is recovered. Finally, in order to avoid recovering false detection boxes, we will combine the scene layout model to calculate the score of the recovered bounding box. Our final score function is written as follows:

[0044] S(x (t+δ) ) = S(x) - (l log M(x (t+δ) ) 2+b) (4)

[0045] where S(x (t+δ) ) is the score function of the object x (t+δ) generated by the optical flow tracking. M(x (t+δ) ) is the position score of x (t+δ) in the scene layout. λ is a variable parameter to adjust the score of the scene layout model. b is the bias to provide a fixed initial score for the scene layout model. If the score S(x (t+δ) ) < 0.3, we consider the recovered bounding box as invalid and remove it.

[0046] To evaluate the performance of the road obstacle detection algorithm, we conduct experiments on the Small Obstacle Detection dataset and the Lost and Found dataset. The Small Obstacle Detection dataset contains 2927 images, with 1937, 530, and 460 images in the training set, validation set, and test set, respectively. It is composed of 15 video sequences, which utilize a set of different small obstacle instances, and different road scenes and obstacle sets are used in the training set, validation set, and test set, respectively. The Lost and Found dataset contains images of small items (such as cargo, wooden sticks, and toys, etc.) randomly scattered in front of the vehicle. The training set and test set contain 1036 and 1068 images, respectively. The proposed algorithm is compared with the current advanced target detection algorithm in terms of detection accuracy. We use the average precision (AP) to evaluate the detection algorithm. Specifically, true positive (TP) represents the number of correctly detected positive objects, false positive (FP) represents the number of background regions incorrectly labeled as objects, and false negative (FN) represents the number of positive objects that are not detected. The recall and precision are calculated according to the following two equations, respectively.

[0047]

[0048]

[0049] Given the recall and precision at different detection score thresholds, we can draw the precision-recall curve of the test results, and the average precision (AP) represents the area between the curve and the coordinate axis. In addition, we evaluate the detection results at different IoU thresholds, where the IoU threshold represents the intersection over union between the detection box and the true box, and when it is greater than a certain threshold, it means that the obstacle target is detected. We use the AP value at the IoU threshold of 50% and 75% and the average AP value at the IoU threshold from 50% to 95% (increasing by 5% intervals) as evaluation indicators. The higher the AP value represents the better detection effect.

[0050] 1. Parameter selection

[0051] In this work, we use Faster RCNN and YOLOv5 object detection algorithms as the reference comparison. We use their latest implementation in PyTorch without any changes. In the Faster RCNN algorithm, we use ResNet-50 as the backbone network, SGD as the optimizer, and also use the ImageNet pre-trained model for network initialization. In addition, in order to learn better multi-scale feature representation, we use the feature pyramid network (FPN). In the YOLOv5 algorithm, we use yolov5l6 as the network structure, and use the MS-COCO pre-trained model to initialize the network weight; train for 100 epochs, and select the best epoch in the validation result for evaluation.

[0052] 2. Experimental results

[0053] In order to qualitatively compare the effectiveness of our method, we conduct an ablation experiment. Taking Faster RCNN and YOLOv5 as the reference, the experimental results obtained by adding the scene layout model (SL), the optical flow target tracking (OF), and the combination of the two methods are shown in Table 1.

[0054] Table 1 Ablation experiment results on SOD dataset

[0055]

[0056] Taking AP50 as an example, the scene layout (SL) model and the optical flow target tracking (OF) method are combined with the Faster RCNN algorithm on the SOD dataset, and the average precision is increased by 1.6% and 1.8% respectively, which proves the effectiveness of the two methods. Then the combination of the two methods can be significantly improved by 4.0%. For the YOLOv5 algorithm, the scene layout model provides an improvement of 0.5%. However, the use of optical flow for target tracking can improve 1.1%. Finally, the combination of the two methods can provide a maximum improvement of 1.6%. We also give some qualitative results in Figure 6 which clearly show that our method can eliminate false positives while detecting distant obstacles in weak visual information.

[0057] In addition, since the time interval between adjacent image frames in the Lost and Found dataset is large, the time smoothness between adjacent two frames is poor. Therefore, in this dataset, we only use the scene layout model. Table 2 summarizes the quantitative results we obtained. As can be seen, using the scene layout model, AP50 increases by 0.9%. At the same time, we give some qualitative results in Figure 7Some qualitative results are shown in the middle to demonstrate that our method can eliminate some false positive cases while improving the obstacle score of the Faster RCNN algorithm within the scene layout.

[0058] Table 2 Experimental results on the Lost and Found dataset

[0059]

[0060] Our method is also very efficient in terms of computation. Take the Faster RCNN algorithm for example in the SOD dataset, and we only use an RTX3080Ti graphics card for detection. The average time spent per inference is about 61 ms. After adding the scene layout model, each inference still only takes 61 ms. Because the parameters in the scene layout model can be calculated in advance and directly used in the inference stage. Then add the optical flow tracking method, the time consumption increases to 89 ms. Because we use a region selection-based optical flow calculation method to eliminate unnecessary interference in the scene, greatly simplifying the calculation, only increasing the time consumption by 28 ms.

[0061] Based on the above experimental analysis, the two methods proposed in this paper, whether used alone or in combination, can improve the final detection results and are an effective method for road obstacle detection.

[0062] References:

[0063] [1] Ren, S., He, K., Girshick, R. & Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015).

[0064] [2] Jocher, G. et al. ultralytics / yolov5: v5.0-YOLOv5-P6 1280 models. AWS, Supervise.ly and YouTube integrations 10 (2021).

[0065] The above is the preferred embodiment of the present application, any changes made according to the technical solutions of the present application, as long as the function generated does not exceed the scope of the technical solutions of the present application, belongs to the protection scope of the present application.

Claims

1. A spatio-temporal context based road obstacle detection method characterized by, Comprise: Step S1, using data-driven method to derive the spatial distribution of obstacles and roads, to establish the scene layout model with spatial context information; the specific implementation is as follows: 1) Obstacle layout construction: data statistics on the distribution of obstacles in the obstacle training set in the real driving scene; first, get the real bounding box of all obstacle targets in the training set, and take all their center points to obtain the two-dimensional spatial distribution in the form of heat map, the distribution of obstacles in the image shows strong regularity, which is mainly concentrated in the center of the image; 2) Road layout construction: a method of mapping obstacle distribution to the entire road area is proposed, that is, according to the distribution of obstacles, different weights are given in the road area along the transverse and longitudinal directions; 3) The obstacle distribution and road distribution are fused and normalized to obtain the final road scene layout; After that, a score S is generated for each object hypothesis x by the scene layout L (x); then, the object hypothesis score obtained from the object detector S D (x) is combined with the scene layout score S L (x) to obtain a final score S(x); S L (x) is defined as follows: Here S L (x) is the final score of object hypothesis x in the scene layout, M(x) is the location score of object hypothesis x in the scene layout, a is a variable parameter for adjusting the score of the scene layout, and b is a fixed bias value for the score of the scene layout. According to the above formula, it is assumed that when the score M(x) <0.15, the target is located outside the road distribution, which will be considered as a false detection; when the score is 0.15 <M(x) <0.6, it is considered that the target is located in the road area but close to the boundary, so the detection score does not change; when the score M(x) >0.6, it is considered that the object appears in the most likely position to detect obstacles, and the score increases accordingly; Step S2, a kind of obstacle tracking method based on optical flow and object region selection is proposed; Step S3, fuse two kinds of context information to detect the target and remove background noise; The specific implementation of the proposed obstacle tracking method based on optical flow and object region selection is as follows: (1) obtain the obstacle detection results of the previous frame and the next frame, and select the previous frame score S D (x) detection of >0.3 (2) To determine whether a subsequent frame is missed, by expanding the detection boundary box of the detection object, a search area A is defined in the subsequent frame r ; the definition of search area A r is as follows A r (x) = (b x ,b y ,γb w +τ,βb h +τ) where γ and β are adjustment factors for the width and height of the selected region, respectively, and τ represents the initial size of the region;(b x ,b y ) is the image coordinate position of the object center; (3) If the obstacle A is missed in the next frame r (x) There is no bounding box of the same obstacle category in the inside, crop the search region given in step (2) in the previous frame and the next frame, and use the Shi-Tomasi corner detection algorithm to obtain the image corner points in the cropped region, and select the feature corner points located on the detected object; (4) Then use LucasKanade optical flow method to track the feature corner points in the cropped area to obtain the offset of the feature corner points in the previous and next two frames; (5) The detection bounding box in the previous frame is transferred to the next frame by the obtained offset.

2. The spatio-temporal context-based road obstacle detection method of claim 1, wherein, The specific implementation of the fusion of two kinds of context information for target detection and background noise removal is as follows: First, detect the obstacles in the input image, use the scene layout model to process the detection results to eliminate the false detection outside the road scene distribution, and enhance the reliability of small obstacles in the road area; Secondly, through obstacle tracking and frame transfer, the missed frame of the road area is recovered; finally, in order to avoid the recovery of false detection frame, the score of the recovered frame is calculated combined with the scene layout model, and the final score function is written as follows: S(x (t+δ) ) = S(x) - (λ log M(x (t+δ) ) 2 +b) S(x) = S D (x) + θS L (x) where S(x (t+δ) ) is the score function of the object x (t+δ) generated by the optical flow tracking; M(x (t+δ) ) is the position score of x (t+δ) in the scene layout; λ is a variable parameter for adjusting the score of the scene layout model; if the score S(x (t+δ) ) < 0.3, the recovered bounding box is considered invalid and removed.

Images