A visual-based marking vehicle detection method

Abstract

The application belongs to the technical field of automatic parking, and particularly relates to a visual parking space detection method. A marking point detection network structure adopted by the application comprises two parallel branches. In the first branch, in addition to the prediction of the position offset and confidence of the marking point, the angle prediction of the parking space edge line corresponding to the marking point is added. In the second branch, a parking space type classification branch is added, and the parking space is classified into three types, i.e. a vertical parking space, a parallel parking space and an inclined parking space. The post-processing derives more accurate parking space positions and orientations according to the model prediction of the paired marking points, the type of the parking space corresponding to each marking point in the paired marking points and the angle of the corresponding edge line. The application adds more constraints to the output results, so that the noise in the detection results can be significantly reduced, and the output parking space is more accurate and reasonable.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic parking, and particularly relates to a visual parking space detection method. BACKGROUND

[0002] Visual parking space detection can be roughly divided into two categories, one based on traditional image processing algorithms and the other based on deep learning algorithms. The visual parking space detection algorithm based on deep learning has obvious performance improvement compared with the traditional image processing algorithm, and can be simply divided into three categories. One category predicts the marking points of the parking space first, and then infers the parking space. Another category predicts the entrance line or entrance area of the parking space first, and then infers the parking space. The third category directly predicts the parking space or other. Whether the marking points of the parking space are predicted first or the entrance line or entrance area of the parking space is predicted first, complex post-processing is needed to determine the adjacency relationship between the marking points and deduce the final parking space, so that an incorrect detection result is easily caused. The document Attentional Graph Neural Network for Parking-slot Detection. International Conference on Robotics and Automation 2021. (denoted as AGN) takes the features of the marking points of the parking space in the surround view as graph-structrued data, and uses a graph structure network to aggregate the adjacency relationship between the marking points, so that the parking space information can be output in an end-to-end manner. The original marking point detection network structure of the AGN algorithm is shown in Figure 1 As shown in the figure, the input is the image feature output by the image feature extraction network, and the output is a 16*16*3-dimensional vector. Wherein 16*16 is the number of grid cells, each grid cell will predict a marking point; 3 is the feature length of the marking point, corresponding to the position offset (x offset and y offset) of the marking point coordinates relative to the grid cell, and the confidence of the marking point. The surround view image is input, and the paired parking space marking points are directly predicted and output, and then the entrance line of the parking space can be obtained according to the paired marking points. Denote the two output paired marking points as P1 and P2, the model defaults P1 and P2 as the two close marking points in the counterclockwise direction of the parking space, and assumes that the parking space orientation is perpendicular to the entrance line formed by P1P2, so that the algorithm can only predict right-angle parking spaces (perpendicular parking spaces and parallel parking spaces), and cannot predict inclined parking spaces, and the default parking space depth is a fixed value. SUMMARY

[0003] The purpose of the application is to improve the practicability and positioning accuracy of parking space detection, and provide a visual line parking space detection method.

[0004] The technical scheme of the present application is: a visual-based marking line parking space detection method, comprising the following steps:

[0005] Obtain front-view, rear-view, left-view and right-view fisheye distortion images from four ring-view fisheye cameras on the vehicle, and obtain a ring-view spliced image with a resolution of 512*512 pixels through distortion correction and splicing fusion.

[0006] Input the ring-view spliced image into an image feature extraction network, and output high-dimensional features Figure 1 One path is sent to a marker point detection network for marker point detection, and the other path is sent to a marker point feature encoder for marker point feature extraction; the marker point detection network includes two parallel branches, wherein branch one is used to predict the position offset, confidence, and angle of the parking space edge line corresponding to the marker point represented by cosθ / sinθ, and outputs a 16*16*5-dimensional vector; branch two is used to predict the parking space type, which is divided into three types: vertical parking space, parallel parking space and diagonal parking space, and outputs a 16*16*3-dimensional vector.

[0007] According to the marker point information output by the marker point detection network, the marker point features are sampled from the features output by the marker point feature encoder, and the marker point coordinates output by the marker point detection network are input into a marker point position encoder for position encoding, and are added to the marker point features to achieve information enhancement effect.

[0008] The enhanced marker point features are input into a graph feature aggregation network to learn the adjacency information of the marker points.

[0009] The entrance line discriminator outputs the possibility of adjacency between different marker points, i.e. the possibility of forming a reasonable parking space, according to the adjacency information of the marker points learned by the graph feature aggregation network; it is considered that different marker points can form a reasonable parking space if the adjacency possibility between different marker points is greater than 0.5.

[0010] The post-processing derives more accurate parking space positions and orientations according to the pairs of marker points predicted by the marker point detection network, and the type of the parking space corresponding to each marker point in the pairs of marker points and the angle of the corresponding edge line.

[0011] In the above scheme, branch one of the marker point detection network adopts a mean square error loss as shown in formula (1), and branch two of the marker point detection network adopts a cross-entropy loss as shown in formula (2);

[0012]

[0013]

[0014] Wherein: N represents the number of batch samples, M represents the dimension of the regression quantity, K represents the number of parking space types, y represents the true value, This represents the predicted value.

[0015] In the above scheme, the post-processing method is as follows:

[0016] Determine whether the angle and type of the input pair of marker points are the same.

[0017] If the paired markers have the same angle and type, further determine whether they are both right angles or both diagonal. If they are both right angles, output the angle of perpendicularity, parallelism, or parking space as 90°. If they are both diagonal, further determine whether the absolute difference of the angles is greater than 10°. If it is greater, output the diagonal and the angle with higher confidence. If it is less, output the diagonal and the average of the two angles.

[0018] If the angle and type of the paired markers are the same, then it is further determined whether there is a diagonal, right-angled, vertical, or parallel situation. In this case, the type and angle with higher confidence are output.

[0019] Beneficial effects: This method adds more constraints to the output by incorporating the direction (angle) of parking space edges and the prediction of parking space type into the marker detection network. This significantly reduces noise in the detection results, making the output parking spaces more accurate and reasonable. Extensive testing in real-world scenarios has shown a substantial reduction in false alarms and misreporting. Attached Figure Description

[0020] Figure 1 The original marker detection network structure of the AGN algorithm in the background technology;

[0021] Figure 2 This is a flowchart of the present invention;

[0022] Figure 3 This is the marker detection network structure used in this invention;

[0023] Figure 4 This is a flowchart of the post-processing of the present invention;

[0024] Figure 5 This is a schematic diagram of the included angle of the edge lines caused by distortion. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0026] See appendix Figure 2The application discloses a visual-based marking vehicle detection method, which is based on the AGN algorithm in the background art, and adds a marking point corresponding vehicle orientation (angle) and type prediction and a post-processing step, so that the practicability and positioning accuracy of the whole algorithm are greatly improved.

[0027] The visual-based marking vehicle detection method is realized in the following way:

[0028] Front-view, rear-view, left-view and right-view fisheye distortion images are obtained from four fisheye cameras, and a fisheye splicing image with a resolution of 512*512 pixels is obtained through distortion correction and splicing fusion.

[0029] The fisheye splicing image is input into an image feature extraction network, and high-dimensional features output by the image feature extraction network are obtained. Figure 1 One path is sent into a marking point detection network for marking point detection, and the other path is sent into a marking point feature encoder for marking point feature extraction.

[0030] The marking point detection network used in the method is shown in the following figure: Figure 3 The marking point detection network includes two parallel branches, wherein branch one is used for predicting the position offset, confidence and angle of the marking point corresponding to the vehicle edge line expressed by cosθ / sinθ, and outputs a 16*16*5-dimensional vector; and branch two is used for predicting the vehicle type, which is divided into three types of vertical vehicle, parallel vehicle and diagonal vehicle, and outputs a 16*16*3-dimensional vector.

[0031] The branch one of the marking point detection network adopts a mean square error loss shown in formula (1), and the branch two of the marking point detection network adopts a cross-entropy loss shown in formula (2).

[0032]

[0033]

[0034] Wherein, N represents the number of batch samples, M represents the dimension of the regression quantity, K represents the number of vehicle types, y represents the true value, and x represents the predicted value.

[0035] According to the marking point information output by the marking point detection network, the marking point features are sampled from the features output by the marking point feature encoder, and the marking point coordinates output by the marking point detection network are input into a marking point position encoder for position encoding, and then the marking point features are added to the marking point position encoder to realize information enhancement effect.

[0036] The enhanced marking point features are input into a graph feature aggregation network to learn the adjacent information of the marking points.

[0037] The entrance line discriminator outputs the probability of adjacency between different markers based on the adjacency information learned by the graph feature aggregation network, i.e. the probability of forming a reasonable parking space; if the adjacency probability between different markers is greater than 0.5, it is considered that a reasonable parking space can be formed.

[0038] The post-processing derives a more accurate parking space location and orientation based on the paired markers predicted by the marker detection network, as well as the type of parking space corresponding to each marker in the pair and the angle of the corresponding edge line.

[0039] like Figure 4 As shown: The post-processing method is as follows:

[0040] Determine whether the angles and types of the input pair of markers are the same.

[0041] If the paired markers have the same angle and type, further determine whether they are both right angles or both diagonal. If they are both right angles, output the angle of perpendicularity, parallelism, or parking space as 90°. If they are both diagonal, further determine whether the absolute difference of the angles is greater than 10°. If it is greater, output the diagonal and the angle with higher confidence. If it is less, output the diagonal and the average of the two angles.

[0042] If the angle and type of the paired markers are the same, then it is further determined that there is a diagonal, right-angled, or vertical and parallel situation. In this case, the type and angle with higher confidence are output.

[0043] In actual image annotation, the direction of the marker point can only be marked along the edge line in the image. However, if the ground is uneven or other factors cause the vehicle to undulate, obvious distortion may be retained in the panoramic stitched image, which will cause the annotation result to be inconsistent with the actual situation.

[0044] by Figure 5 For example, the two markers for the parallel parking spaces on the right, along with their corresponding entrance lines and two side lines, exhibit significant distortion. However, in actual annotation, the directions of the two markers can only be marked along the two white side lines, resulting in an angle between the directions of the two markers (the angles corresponding to the two markers are not equal). By adding parking space type prediction in this method, the predicted parking spaces for both markers are now parallel parking spaces (both at right angles). Therefore, the difference in predicted angles can be ignored, and the angle of the parking space can be directly output as 90°.

[0045] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A vision-based marking vehicle detection method, characterized in that, The method comprises the following steps: obtaining front-view, rear-view, left-view and right-view fisheye distortion images from four fisheye cameras mounted on a vehicle, and obtaining a surround-view spliced image with a resolution of 512*512 pixels through distortion correction and splicing fusion; inputting the surround-view spliced image into an image feature extraction network, sending the output high-dimensional feature map to a marker point detection network for marker point detection and to a marker point feature encoder for marker point feature extraction; the marker point detection network comprises two parallel branches, wherein branch one is used for predicting the position offset, confidence and angle of the marker point corresponding to the parking space boundary expressed in cosθ / sinθ, and outputs a 16*16*5-dimensional vector; branch two is used for predicting the parking space type, which is divided into three types: perpendicular parking space, parallel parking space and diagonal parking space, and outputs a 16*16*3-dimensional vector; sampling the marker point features from the features output by the marker point feature encoder according to the marker point information output by the marker point detection network, and inputting the marker point coordinates output by the marker point detection network into a marker point position encoder for position encoding and adding to the marker point features to achieve information enhancement effect; inputting the enhanced marker point features into a graph feature aggregation network to learn the adjacency information of the marker points; the entry line discriminator outputs the possibility of adjacency between different marker points, i.e. the possibility of forming a reasonable parking space, according to the adjacency information of the marker points learned by the graph feature aggregation network; it is considered that a reasonable parking space can be formed if the adjacency possibility between different marker points is greater than 0.5; the post-processing derives more accurate parking space positions and orientations according to the pairs of marker points predicted by the marker point detection network, the type of the parking space corresponding to each marker point in the pair of marker points and the angle of the corresponding boundary line.

2. The vision-based lane-marking vehicle detection method of claim 1, wherein, branch one of the marker point detection network adopts a mean square error loss as shown in formula (1), and branch two of the marker point detection network adopts a cross-entropy loss as shown in formula (2); wherein: N represents the number of batch samples, M represents the dimension of regression quantity, K represents the number of parking space types, y represents the true value, represents the predicted value.

3. A vision-based lane-marking vehicle detection method according to claim 1 or 2, characterized in that, the processing method of the post-processing is: judging whether the angle and type of the input pair of marker points are the same; if the angle and type of the pair of marker points are the same, further determining whether they are both right angles or diagonal columns; in the case of being both right angles, outputting the perpendicular or parallel or the angle of the parking space being 90°; in the case of being both diagonal columns, further determining whether the absolute difference of the angle is >10°, if yes, outputting the diagonal column and the angle with high confidence, if no, outputting the mean value of the angles of the two; if the angle and type of the pair of marker points are the same, further determining whether there is a diagonal column, a right angle or a perpendicular or parallel situation, and outputting the type and angle with high confidence in this situation.

Images