A method for cone and bucket detection and recognition in a computing resource constrained situation

Abstract

The present application relates to a kind of cone bucket detection and identification method under the condition of computing resource limitation, belong to the field of autonomous driving, including image acquisition and image preprocessing;Image is detected, such as dim, then directly artificial feature detection module;Otherwise, synchronous into neural network module and artificial feature detection module;The existence of cone bucket on the picture is determined by neural network module, and the position of surrounding frame;Using artificial feature method, determine the existence of red, blue cone bucket on the picture, and the position of surrounding frame;The results are fused by IOU fusion module;Output identification result and cone bucket pixel coordinates.The present application has the continuity of identification and higher accuracy, can be suitable for detection under different illumination conditions, can effectively improve the accuracy of cone bucket detection, improve the robustness of cone bucket detection algorithm, enhance the environmental perception ability of autonomous driving under the condition of computing resource limitation, especially for the identification ability of road cone bucket.

Claims

1. A method for cone and bucket detection and identification in a computing resource constrained situation, characterized by, The method comprises the following steps: (1) image acquisition and image preprocessing; (2) light detection is performed on the picture, if the light detection result shows that the picture is dark, step (3) is skipped and step (4), i.e. the artificial feature detection module, is directly entered; Otherwise, steps (3) and (4) are performed synchronously, and are respectively transmitted into a neural network module and an artificial feature detection module; (3) the existence of red and blue cones and the position of the surrounding frame on the picture are determined through the YOLO in the Dark neural network module; (4) the artificial feature detection module: the existence and position of the cone are judged by using color and shape recognition based on the HSV space; (5) the results of steps (3) and (4) are fused through an IOU fusion module; (6) the recognition result and the pixel coordinates of the cone are output.

2. The cone and bucket detection and identification method in a computing resource constrained situation according to claim 1, wherein, The step (1) comprises the following steps: 1.1: using an intrinsic calibration functional package, calibrating the camera intrinsic parameters and eliminating distortion; 1.2: sampling video frames in time: the video is actually the result of sequentially playing a plurality of pictures, and one picture is used every several frames for subsequent processing to reduce the operation pressure; 1.3: down-sampling the image in space to reduce the size: the image pyramid is composed of pictures gradually reduced in size, and some features can be more easily extracted at different sizes after 1 / 2 sampling. The Gaussian pyramid technology is used to reduce the image size and reduce the operation pressure; Given the number n of pictures in the pyramid, the row number h and the column number w of the original image, and the expected value t of the number of layers of the pyramid, there is the following relationship: n = log2(min(h, w)) - t, where t ∈ [0, log2(min(h, w))] min(h, w) refers to the minimum value of the row number h and the column number w.

3. The cone and bucket detection and identification method in a computing resource constrained situation according to claim 2, wherein, In step 1.1, first, the camera_calibration functional package is obtained from the ROS community, the camera driver file is run, and whether there is an image topic published is observed; Then, a 9x7 chessboard is printed, and the square length needs to be consistent with 0.108 in the command. If not, it needs to be manually modified to be consistent. During the running process, the chessboard is translated, rotated and flipped in front of the camera. When the progress bar of the calibration window is all green, the calibration is completed, and the calibration parameters can be saved by clicking "save" and saved to the corresponding folder of the camera parameters.

4. The cone and bucket detection and identification method in a computing resource constrained situation according to claim 1, wherein, In the step (2), when the light detection is performed on the picture, the brightness of the picture is defined as: Wherein, the meaning of # is the number, max(R, G, B) is actually the pixel brightness in the HSV color space, threshold is a value of 0-255, #all pixels represents the number of all pixels, and the formula represents the ratio of the pixels with the pixel brightness greater than the threshold threshold to all pixels, and the value ranges from 0 to 1. The luminance threshold is A, and A is 0.

25. If the luminance of the picture is less than A, the picture is considered to be dark, step (3) is skipped, and step (4) is directly entered. Otherwise, steps (3) and (4) are performed synchronously.

5. The cone and bucket detection and identification method in a compute resource constrained situation according to claim 1, wherein, The step (3) specifically comprises the following steps: 3.1: making an RGB image dataset of the cone Image shooting is performed on the camera used in the vehicle-mounted computer. The shooting location is randomly selected, and the shooting needs to cover a variety of lighting conditions, mainly including dim and dark lighting conditions. The number of data sets is more than 1000; The label text of the coordinates of the cone position is made by using a labeling software; each picture is labeled by an open source software labelme to generate a label file containing the pixel coordinates (x, y) of all the traffic cone rectangular bounding boxes in the picture, height, width and category information; wherein, the pixel coordinate information of the rectangular bounding box, the height and the width information are automatically generated by the labeling software, and the category information is divided into red and blue cones; the picture is randomly divided into a training set, a verification set and a test set according to a fixed proportion; 3.2: Data enhancement is performed on the picture, including horizontal flipping of the picture, random cropping of part of the edge area and restoration of the size; random position cutout; changing the brightness and contrast of the picture; Wherein, the horizontal flipping refers to the symmetric flipping of the picture in the horizontal direction; the random cropping refers to the random cropping of part of the area of the original picture; the cutput refers to the deletion of a small range of data of the original picture, and a small area "hole” without data appears; 3.3: Network training The YOLO in the Dark neural network is used to pre-train the network parameters on the SID data set, and the network parameters are fine-tuned on the cone data set, that is, a pre-trained model with strong generalization ability is trained on an existing large data set, and then the pre-trained model is used as the basis for the specific cone recognition task, and the model is retrained using back propagation to adapt the network parameters to the cone recognition task, so as to achieve good recognition accuracy; The mAP and network parameters are saved during the training process, and after the training process is completed, the network parameters corresponding to the highest mAP are selected as the final parameters of the YOLO in the Dark neural network; when the number of training reaches a predetermined number, the training stops; 3.4: The final parameters obtained by training in step 3.3 are loaded, detection and recognition are performed, and the detection bounding box of the target is output.

6. The cone and bucket detection and identification method in a computing resource constrained situation according to claim 5, wherein, The step (4) uses the color and shape recognition based on the HSV space to judge the existence and position of the cone, specifically including: 4.1: Convert the picture to HSV color space and take the mask: convert the RGB image to HSV color space; extract the red and blue pixel regions located in the fixed HSV three-component interval, and further binarize to form red and blue masks; The HSV color space includes three components: hue, saturation and lightness, and the color model is represented by an inverted cone: the main axis represents the lightness, the distance from the main axis represents the saturation, and the angle around the main axis represents the hue; 4.2: Data filtering processing 4.2.1: Morphological opening operation, that is, erosion operation and dilation operation The erosion operation takes the local minimum value in the kernel range, and the dilation operation takes the local maximum value in the kernel range, which expands the area of the cone region in the mask image, so that it has a similar effect to the flood fill; 4.2.2: Median filtering The median filtering selects the median value of the pixels in the convolution kernel to approximate the actual value to remove isolated noise by using the sorting method; 4.3: Calculate the boundary contour The Canny operator is used to take the boundary contour, the Douglas-Peucker method is used to convert the continuous smooth curve into a broken line, the polygon fitting is performed on the conical barrel contour points, and then the optimized boundary contour is obtained from the boundary that may be the conical barrel. 4.4: Feature extraction Since the conical barrel shape is simple, the aspect ratio of the boundary, the area, and whether it has a triangular feature are further limited; in order to detect the triangular feature, the Hough transform is used to extract the straight line information, and the obtained straight line is matched in the vicinity of the mask corresponding region, specifically: since the conical barrel recognition result is not an accurate complete triangle, the angle of the two oblique sides of the conical barrel obtained by the Hough transform recognition is calculated, if the angle is within [25°, 65°], it is considered as a conical barrel, otherwise it is considered as not a conical barrel. 4.5: Frame the conical barrel On the basis of the boundary contour calculated in step 4.3, the conical barrel bounding box is calculated, and the specific method is: first, exclude the boundary curve that does not meet the angle interval in step 4.4, then find the leftmost, rightmost, uppermost and lowermost points of the boundary curve in the boundary curve that meets the condition, and combine them to obtain the smallest rectangular frame that can enclose the boundary curve.

7. The cone and bucket detection and identification method in a computing resource constrained situation according to claim 6, characterized in that, In step (5), only step (4) is performed, and step (5) directly outputs the category and conical barrel corresponding pixel coordinates; In the case of synchronously performing steps (3) and (4), steps 5.1 and 5.2 are performed respectively to obtain the fused conical barrel category and pixel coordinate output; 5.1: Align artificial features with neural network output The recognition results are aligned so that the recognition results of steps (3) and (4) correspond to the same conical barrel; In the red conical barrel and blue conical barrel categories, the intersection over union of the bounding boxes of the artificial feature method and the neural network recognition results is calculated respectively, that is, the intersection area of the bounding boxes is divided by the union area, and the intersection over union greater than the threshold value and the largest among all the intersection over unions is considered as the same color of the same conical barrel; otherwise, it is considered as unreliable data and discarded. The threshold value of the intersection over union is 0.

6. 5.2: Output fusion For the aligned recognition results of the two methods, the average value of the pixel horizontal coordinate x, the pixel vertical coordinate y, the height and the width is taken as the output, that is, the vertex of the new bounding box is located at the midpoint of the corresponding vertex of the original bounding box.

Images