A method, device, storage medium and equipment for vehicle positioning based on vision

Abstract

The embodiments of the present application disclose a method, device, storage medium and equipment for vehicle positioning based on vision, and relate to the field of autonomous driving technology. The present application realizes vehicle positioning based on vision through image data, avoids the use of high-cost laser radar equipment, and uses different visual features and different positioning algorithms to perform positioning calculations for different road traffic scenes at non-intersections and intersections according to the collected image data. In non-intersection scenes, the traffic semantic element features and ORB features extracted by the two algorithms are used to output the positioning posture in real time, and the traffic semantic element features are used to calculate the posture in the intersection scene, which can not only meet the real-time positioning needs, but also eliminate the accumulated errors in non-intersection scenes at the intersection, improve the positioning accuracy, and avoid the use of loop detection and global optimization algorithms with large computational complexity, and the computing power requirements for the chip are relatively low, which further saves costs while meeting the actual use needs.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and specifically to a vision-based vehicle positioning method, device, storage medium and equipment. Background Art

[0002] Positioning technology is an important part of the field of autonomous driving. It is used to obtain information such as the vehicle's location and posture. Currently, positioning technology is mainly divided into two categories: laser positioning and visual positioning.

[0003] The existing technical solutions for laser positioning mainly include LOAM (Lidar Odometry and Mapping in Real-time), LeGO-LOAM (Lightweight and Ground-Optimized Lidar Odometry and Mapping), LIO-SAM (Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping), etc. The implementation principle is mainly to extract the point, line and surface features in the laser point cloud, and then calculate the pose based on the matching features of each frame to achieve positioning.

[0004] The existing technical solutions for visual positioning mainly include VINS-Mono, ORB-SLAM series (ORB-SLAM1, ORB-SLAM2, ORB-SLAM3), etc. The implementation principle of VINS-Mono is mainly based on the method of optical flow tracking to achieve positioning. The back-end uses the bag of words model (BOW) to detect loops to optimize the pose and eliminate cumulative errors. The implementation principle of the ORB-SLAM series is mainly to detect and extract the ORB (Oriented FAST and Rotated BRIEF) features in the scene, and then perform tracking and matching to calculate the optimized pose, thereby achieving positioning. The back-end uses the bag of words model (BOW) to detect loops to optimize the pose and eliminate cumulative errors.

[0005] At present, existing technologies can hardly meet actual needs. Although laser positioning technology is relatively mature, its main disadvantage is the high cost of laser radar hardware. Visual positioning is based on low-cost hardware such as cameras, but the technology and stability are still relatively immature, and the application scenarios are relatively limited.

[0006] For example, VINS-Mono does not extract feature descriptors and uses the optical flow method for tracking and matching. Once the picture is blurred or the image is lost, the positioning information will be lost. The application scenarios are also relatively limited. For weak textures and environments with few key points, the stability and robustness are also poor. The front-end accumulated errors are corrected by the back-end loop detection, which requires a lot of calculations. The ORB-SLAM series of visual positioning methods have poor effects or even fail in outdoor traffic scenes with open roads. The main reason is that the geometric texture information of such scenes is not rich enough, and relatively few ORB features can be extracted. It is sensitive to noise, resulting in poor stability and robustness. At the same time, it is easy to lose frames when rotating. In addition, the back-end loop detection corrects the accumulated errors at the front end, which requires a lot of calculations and high chip computing power. Summary of the invention

[0007] The main purpose of this application is to provide a vision-based vehicle positioning method, device, storage medium and equipment, aiming to solve the problem that the existing technology is difficult to meet actual needs.

[0008] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0009] In a first aspect, an embodiment of the present application provides a vehicle positioning method based on vision, comprising the following steps:

[0010] Acquire image data of the road traffic scene where the target vehicle is located;

[0011] According to the judgment result of whether the target vehicle is located at the intersection scene, an image feature extraction algorithm is determined; wherein, when the target vehicle is located at the intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on deep learning, and when the target vehicle is not located at the intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on image processing and an extraction algorithm based on deep learning, the extraction algorithm based on deep learning is used to extract traffic semantic element features, and the extraction algorithm based on image processing is used to extract ORB features;

[0012] Extract target features from image data based on image feature extraction algorithm;

[0013] Tracking and matching are performed based on target features to obtain the positioning posture of the target vehicle.

[0014] In a possible implementation of the first aspect, when the target vehicle is not located at an intersection scene, extracting target features in image data based on an image feature extraction algorithm includes:

[0015] Extract the ORB features of image data based on the extraction algorithm of image processing;

[0016] Based on the deep learning extraction algorithm, traffic semantic element features of image data are extracted; wherein the traffic semantic element features include lane line features.

[0017] In a possible implementation of the first aspect, tracking and matching are performed based on target features to obtain a positioning posture of the target vehicle, including:

[0018] Tracking and matching based on the ORB features extracted from the current frame and the ORB features extracted from the previous frame;

[0019] Tracking and matching of lane line features extracted from the current frame with offline local semantic maps;

[0020] The tracked and matched lane line features are used as constraints to obtain the first positioning posture sub-parameters;

[0021] The ORB features of the tracking match are used as constraints to obtain the second positioning pose sub-parameters;

[0022] The positioning posture of the target vehicle is obtained according to the first positioning posture sub-parameter and the second positioning posture sub-parameter.

[0023] In a possible implementation of the first aspect, when the target vehicle is located at an intersection, extracting target features from image data based on an image feature extraction algorithm includes:

[0024] Based on the deep learning extraction algorithm, the traffic semantic element features of the image data are extracted; wherein the traffic semantic element features include stop line features, traffic light features and traffic sign features.

[0025] In a possible implementation of the first aspect, tracking and matching are performed based on target features to obtain a positioning posture of the target vehicle, including:

[0026] Tracking and matching of traffic semantic element features extracted from the current frame with the offline local semantic map;

[0027] The tracked and matched traffic semantic element features are used as constraints to obtain the positioning pose of the target vehicle.

[0028] In a possible implementation of the first aspect, tracking and matching traffic semantic element features extracted based on the current frame with an offline local semantic map includes:

[0029] Based on the traffic semantic element features extracted from the current frame, obtain the two-dimensional feature points of the stop line feature in the image data, the central two-dimensional point of the detection frame of the traffic light feature in the image data, and the central two-dimensional point of the detection frame of the traffic sign feature in the image data;

[0030] The offline local semantic map is loaded, and through the Euclidean distance constraint, the two-dimensional feature points of the stop line feature in the image data, the central two-dimensional point of the detection frame of the traffic light feature in the image data, and the central two-dimensional point of the detection frame of the traffic sign feature in the image data are obtained with the corresponding semantic elements in the offline local semantic map for tracking and matching.

[0031] In a possible implementation of the first aspect, acquiring image data of a road traffic scene in which a target vehicle is located includes:

[0032] Image data of the road traffic scene where the target vehicle is located is obtained based on multiple cameras.

[0033] In a second aspect, an embodiment of the present application provides a vision-based vehicle positioning device, comprising:

[0034] An acquisition module, the acquisition module is used to acquire image data of a road traffic scene where a target vehicle is located;

[0035] A determination module, the determination module is used to determine an image feature extraction algorithm according to a judgment result of whether the target vehicle is located at an intersection scene; wherein, when the target vehicle is located at an intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on deep learning, and when the target vehicle is not located at an intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on image processing and an extraction algorithm based on deep learning, the extraction algorithm based on deep learning is used to extract traffic semantic element features, and the extraction algorithm based on image processing is used to extract ORB features;

[0036] An extraction module, which is used to extract target features from image data based on an image feature extraction algorithm;

[0037] Positioning module,The positioning module is used to track and match based on target features to obtain the positioning posture of the target vehicle.

[0038] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, the vision-based vehicle positioning method provided in any one of the first aspects above is implemented.

[0039] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein:

[0040] The memory is used to store computer programs;

[0041] The processor is used to load and execute a computer program so that the electronic device executes the vision-based vehicle positioning method provided in any one of the first aspects above.

[0042] Compared with the prior art, the beneficial effects of this application are:

[0043] The embodiments of the present application propose a vision-based vehicle positioning method, device, storage medium and equipment, the method comprising: acquiring image data of a road traffic scene where a target vehicle is located; determining an image feature extraction algorithm based on a judgment result of whether the target vehicle is located at an intersection scene; wherein, when the target vehicle is located at an intersection scene, the image feature extraction algorithm comprises: an extraction algorithm based on deep learning, and when the target vehicle is not located at an intersection scene, the image feature extraction algorithm comprises: an extraction algorithm based on image processing and an extraction algorithm based on deep learning, the extraction algorithm based on deep learning is used to extract traffic semantic element features, and the extraction algorithm based on image processing is used to extract ORB features; based on the image feature extraction algorithm, extracting target features from the image data; performing tracking and matching based on the target features to obtain the positioning posture of the target vehicle. This application realizes vision-based vehicle positioning by collecting image data, avoiding the use of high-cost LiDAR equipment. On this basis, different visual features and different positioning algorithms are used for positioning calculations according to the collected image data for different road traffic scenes at non-intersections and intersections. In non-intersection scenarios, the traffic semantic element features and ORB features extracted by the two algorithms are used to output the positioning posture in real time, and the traffic semantic element features are used to calculate the posture in intersection scenarios. This not only meets the real-time positioning needs, but also eliminates the accumulated errors in non-intersection scenarios at intersections, thereby improving the positioning accuracy. At the same time, it avoids the use of loop detection and global optimization algorithms with large computational complexity, and has relatively low requirements on the computing power of the chip, further saving costs while meeting actual usage needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the structure of an electronic device of a hardware operating environment involved in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a process flow of a vision-based vehicle positioning method provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the camera settings in the vision-based vehicle positioning method provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of a flow chart of a vision-based vehicle positioning method provided in an embodiment of the present application in one implementation manner;

[0048] Figure 5 A schematic diagram of a module of a vision-based vehicle positioning device provided in an embodiment of the present application;

[0049] Markings in the figure: 101 - processor, 102 - communication bus, 103 - network interface, 104 - user interface, 105 - memory. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] See attached Figure 1 , attached Figure 1 This is a schematic diagram of the structure of an electronic device of the hardware operating environment involved in the embodiment of the present application. The electronic device may include: a processor 101, such as a central processing unit (CPU), a communication bus 102, a user interface 104, a network interface 103, and a memory 105. Among them, the communication bus 102 is used to realize the connection and communication between these components. The user interface 104 may include a display (Display), an input unit such as a keyboard (Keyboard), and the user interface 104 may also include a standard wired interface and a wireless interface. The network interface 103 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 105 may optionally be a storage device independent of the aforementioned processor 101. The memory 105 may be a high-speed random access memory (RAM) memory, or it may be a stable non-volatile memory (NVM), such as at least one disk storage. The processor 101 may be a general-purpose processor, including a central processing unit, a network processor, etc., or it may be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component.

[0052] Those skilled in the art will appreciate that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0053] As attached Figure 1 As shown, the memory 105 as a storage medium may include an operating system, a network communication module, a user interface module, and a vision-based vehicle positioning device.

[0054] In the attached Figure 1In the electronic device shown, the network interface 103 is mainly used for data communication with the network server; the user interface 104 is mainly used for data interaction with the user; the processor 101 and the memory 105 in the present application can be set in the electronic device, and the electronic device calls the vision-based vehicle positioning device stored in the memory 105 through the processor 101, and executes the vision-based vehicle positioning method provided in the embodiment of the present application.

[0055] See attached Figure 2 Based on the hardware device of the aforementioned embodiment, the embodiment of the present application provides a vehicle positioning method based on vision, comprising the following steps:

[0056] S10: Acquire image data of the road traffic scene where the target vehicle is located.

[0057] In the specific implementation process, the target vehicle is a vehicle that needs to be positioned, usually an unmanned vehicle, but of course it can also be a manned vehicle with intelligent driving mode. In order to achieve vision-based positioning, the image data of the road traffic scene where it is located is collected and fed back. The collection can be carried out through the camera installed on the vehicle. Since ORB features and traffic semantic element features will be used later, the location of each feature in the actual road traffic scene is not convenient for extraction from the image collected by one camera, so multiple cameras can be set for collection, that is, to obtain the image data of the road traffic scene where the target vehicle is located, including:

[0058] Image data of the road traffic scene where the target vehicle is located is obtained based on multiple cameras.

[0059] As attached Figure 3 As shown in the figure, two cameras are used for acquisition, and one camera is installed with a viewing angle facing the oblique upward area, as shown in the attached figure. Figure 3 The camera 1 in the figure is mainly used to collect the traffic light, traffic sign semantic elements, and surrounding environment ORB features in the traffic semantic elements; the other camera is installed with a viewing angle facing the oblique downward area, such as the attached Figure 3 The camera 2 in the image is mainly used to collect semantic elements such as ground lane lines and stop lines in traffic semantic elements. After the image data is collected, the image data can be pre-processed to improve the quality and facilitate subsequent processing. The pre-processing means include scaling and enhancement.

[0060] S20: Determine an image feature extraction algorithm based on a judgment result of whether the target vehicle is located at an intersection scene; wherein, when the target vehicle is located at an intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on deep learning; when the target vehicle is not located at an intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on image processing and an extraction algorithm based on deep learning; the extraction algorithm based on deep learning is used to extract traffic semantic element features, and the extraction algorithm based on image processing is used to extract ORB features.

[0061] In the specific implementation process, in the existing visual positioning-based solutions, in the visual SLAM process, the estimation of posture is often a recursive process, that is, the posture of the previous frame solves the posture of the current frame. Therefore, the posture produces cumulative errors as it is transmitted frame by frame, and loop detection is an effective method to eliminate the cumulative errors. SLAM refers to real-time positioning and map construction, or concurrent map construction and positioning. This application only addresses positioning problems and does not involve map construction. The offline local semantic map used is based on laser SLAM and manually drawn high-precision maps. Loop detection refers to whether the moving target has returned to the previous position. If a loop is detected, the information will be passed to the back end for optimization processing, so that the positioning result is more accurate. Loop detection generally has a large amount of calculation, and how to judge whether returning to the previous position to produce a loop is also a current difficulty.

[0062] Taking the above difficulties into consideration, the embodiments of the present application provide two image feature extraction algorithms for processing. One is an algorithm based on deep learning, which extracts traffic semantic element features such as lane lines, stop lines, traffic lights, and traffic signs in road traffic scenes; the other is an algorithm based on traditional image processing, which extracts ORB features in road traffic scenes. Positioning is divided into intersection scenes and non-intersection scenes, and different methods are used for vehicle positioning. In non-intersection scenes, the traffic semantic element features and ORB features extracted by the two algorithms are used to output the positioning posture in real time, and the traffic semantic element features are used to calculate the posture in the intersection scene. This can not only meet the real-time positioning needs, but also eliminate the accumulated errors in the non-intersection scene at the intersection.

[0063] In road traffic scenarios, deep learning-based methods are used to detect and extract traffic semantic elements such as lane lines, stop lines, traffic lights, and traffic signs. Among them, the UFLD (Ultra Fast Lane Detection) deep learning algorithm can be used to detect and extract lane lines and stop lines, and the YOLO deep learning algorithm can be used to detect and extract traffic lights and traffic signs. Specifically:

[0064] Based on the UFLD algorithm, the lane line detection idea is as follows:

[0065] a. Assume that the image size of a lane line is HxW, and HxW classification problems need to be processed;

[0066] b. Lane lines are selected for direction. Assuming that the selection is made on h rows, we only need to deal with the classification problem on h rows, but the classification problem on each row is W-dimensional.

[0067] c. The original HxW classification problem is simplified to only h classification problems, and since the rows for positioning can be set manually, the size of h can be set as needed, but generally h is much smaller than the image height H;

[0068] d. In this way, the number of categories is directly reduced from HxW to h, and h is much smaller than H, not to mention that h is much smaller than HxW, which greatly speeds up the lane detection algorithm and meets the real-time detection requirements;

[0069] Based on the UFLD algorithm, the detection idea of ​​the stop line is similar to that of the lane line, except that the selection is changed from row direction to column direction.

[0070] Traffic lights and traffic signs in traffic scenes are detected based on the YOLO (You Only Look Once) algorithm. The implementation principle of YOLO is to input the entire image into the neural network to directly predict the target position and the corresponding category. Considering the real-time requirements of online positioning, this application replaces the YOLO backbone network Darknet-53 with the DenseNet network. Under the same convolutional layer, relatively few training parameters are required. While ensuring the detection accuracy, the computing power requirements are relatively low. Since each layer of the DenseNet network is directly connected to all the previous layers, the features that have been learned by the previous layers do not need to be repeatedly trained and learned. Therefore, the deep learning model established based on DensetNet has relatively high detection performance and can meet the detection efficiency requirements of traffic lights and traffic signs in intersection scenes.

[0071] The feature points on the image extracted by the ORB algorithm are the ORB features. The ORB algorithm is divided into two parts, namely feature point extraction and feature point description. In the ORB feature extraction algorithm, the FAST feature point detection algorithm is used to extract feature points, and the BRIEF feature descriptor algorithm is used to describe feature points. The two are combined to make an improved and optimized algorithm based on the original basis of the two.

[0072] Feature point extraction part of the ORB algorithm: FAST feature points are a type of corner point, which mainly detects places where the grayscale of local pixels changes significantly, and is known for its fast speed. The core idea is: if a pixel is very different from its neighboring pixels (too bright or too dark), it is more likely to be a corner point. The detection process is as follows:

[0073] a. Select pixel p in the image and assume its brightness is Ip;

[0074] b. Set a threshold T (for example, 20% of IP);

[0075] c. With pixel p as the center, select 16 pixels on a circle with a radius of 3;

[0076] d. If there are N consecutive points on the selected circle whose brightness is greater than Ip+T or less than Ip-T, then pixel p can be considered as a feature point (N is usually 12, that is, FAST-12, and other commonly used N values ​​are 9 and 11, which are called FAST-9 and FAST-11 respectively);

[0077] e. Loop through the above four steps and perform the same operation on each pixel.

[0078] The feature point description part of the ORB algorithm: BRIEF is a binary descriptor, and its description vector consists of many 0s and 1s. The 0s and 1s here encode the size relationship between two pixels (for example, p and q) near the key point: if p is larger than q, then 1 is taken, otherwise 0 is taken. If 128 such p and q are taken, a 128-dimensional vector consisting of 0 and 1 is finally obtained. The selection of p and q is generally based on a certain probability distribution, and the positions of p and q are randomly selected. The original BRIEF descriptor is not rotationally invariant, so it is easy to lose when the image is rotated. ORB calculates the direction of the key point in the FAST feature point extraction stage, so the direction information can be used to calculate the "Steer BRIEF" feature after rotation, making the ORB descriptor have better rotation invariance.

[0079] S30: Extract target features from the image data based on an image feature extraction algorithm.

[0080] In the specific implementation process, the implementation process of the image feature extraction algorithm has been described in detail in the above embodiments. In the actual application stage, it is determined whether the vehicle is currently in an intersection or a non-intersection scene, and the corresponding image feature extraction algorithm is used to extract the target features in the image data.

[0081] On the one hand, in non-intersection scenarios, that is, when the target vehicle is not at an intersection, the target features in the image data are extracted based on the image feature extraction algorithm, including:

[0082] Extract the ORB features of image data based on the extraction algorithm of image processing;

[0083] Based on the deep learning extraction algorithm, traffic semantic element features of image data are extracted; wherein the traffic semantic element features include lane line features.

[0084] On the other hand, in the intersection scenario, that is, when the target vehicle is located at the intersection, the target features in the image data are extracted based on the image feature extraction algorithm, including:

[0085] Based on the deep learning extraction algorithm, the traffic semantic element features of the image data are extracted; wherein the traffic semantic element features include stop line features, traffic light features and traffic sign features.

[0086] In the specific implementation process, two algorithms are used to extract ORB features and traffic semantic element features of image data respectively in non-intersection scenarios. The traffic semantic element features required at this time include lane line features. In intersection scenarios, the surrounding environment has richer features that can be used. At this time, the traffic semantic element features in the target features extracted from the collected images include stop line features, traffic light features, and traffic sign features.

[0087] S40: Tracking and matching are performed based on the target features to obtain the positioning posture of the target vehicle.

[0088] In the specific implementation process, the idea of ​​locating the target vehicle is roughly the same, and it is achieved through tracking and matching, so as to calculate the positioning posture of the vehicle. Positioning refers to obtaining the posture of the observed target in the specified coordinate system, and the positioning posture includes position (translation parameters: x, y, z) and posture (rotation parameters: yaw, pitch, roll). In the field of unmanned driving, positioning refers to calculating the posture of the vehicle body target in the map coordinate system during driving, including translation parameters x, y, z and rotation parameters yaw, pitch, roll. In the embodiment of this application, it is assumed that the vehicle body coordinate system is: the origin is the center of the rear axle of the vehicle, x points to the front of the vehicle, y points to the left of the vehicle, and z points to the sky.

[0089] In non-intersection scenarios, the tracking, matching and positioning calculation of vehicles are explained as follows:

[0090] Tracking and matching are performed based on target features to obtain the positioning posture of the target vehicle, including:

[0091] Tracking and matching based on the ORB features extracted from the current frame and the ORB features extracted from the previous frame;

[0092] Tracking and matching of lane line features extracted from the current frame with offline local semantic maps;

[0093] The tracked and matched lane line features are used as constraints to obtain the first positioning posture sub-parameters;

[0094] The ORB features of the tracking match are used as constraints to obtain the second positioning pose sub-parameters;

[0095] The positioning posture of the target vehicle is obtained according to the first positioning posture sub-parameter and the second positioning posture sub-parameter.

[0096] In the specific implementation process, in the previous stage, the deep learning-based extraction algorithm has extracted the lane lines of the current frame, and then tracked and matched the extracted lane line semantic features with the offline local semantic map. The specific matching process is as follows:

[0097] a. Extract the two-dimensional feature points of different lane lines in the image in the current frame based on the deep learning extraction algorithm , m=1,2..., where k represents the number of different lane lines, and m represents the number of feature points detected for each lane line;

[0098] b. Load the offline local semantic map, and then use the Euclidean distance constraint to obtain the lane line detection points in the image and the corresponding lane lines in the offline local semantic map to complete the matching.

[0099] In the previous stage, the extraction algorithm based on image processing has extracted the ORB features of the current frame and the previous frame, and then matched the feature points between frames through the special descriptor. The specific matching process is as follows:

[0100] a. Assume that feature points are extracted in the current frame , m=1, 2..., feature points are extracted in the previous frame , n=1,2..., where t represents the current frame, t-1 represents the previous frame, and m and n represent the number of feature points detected and extracted in each frame image;

[0101] b. Based on the Fast Approximate Nearest Neighbor (FLANN) algorithm, calculate the Hamming distance (HammingDistance) with the corresponding descriptor ;

[0102] c. Sort by size and take the nearest feature point as the matching point to complete the matching of feature points.

[0103] In non-intersection road scenes, the matched lane line features are first used as constraints to calculate the two parameters y and yaw, which are the first positioning sub-parameters. Then, the matched ORB features are used as constraints to calculate the four parameters x, z, pitch, and roll, which are the second positioning sub-parameters. Specifically, in non-intersection road scenes, a two-stage method is used to calculate the positioning posture:

[0104] First, the matched lane line features are used as constraints to calculate the two parameters y and yaw. The specific calculation process is as follows:

[0105] a. By matching the lane lines in the previous stage with the offline local semantic map, the 3D-2D correspondence constraints between the spatial points of the lane lines in the map and the detection points in the image are obtained;

[0106] b. Based on the PnP (Perspective-n-Point) algorithm, the 3D spatial points of the lane lines in the map are projected into the image, and the line equation expression of the lane line projection points is obtained by least squares fitting;

[0107] c. Then, the BA (Bundle Adjustment) algorithm is used to iteratively optimize the reprojection error between the 2D lane line detection points in the image and the 3D lane line in the map.

[0108] d. Based on the constraints of the lane line, only the two parameters y and yaw are calculated and optimized. The optimization goal is to minimize the distance d from the 2D detection point to the 3D lane line projection line. When d is less than the set threshold, the iterative optimization is completed;

[0109] Secondly, the matched ORB features are used as constraints to calculate the four parameters x, z, pitch, and roll. The specific calculation process is as follows:

[0110] a. By matching the ORB features of the current frame and the previous frame in the previous stage, the 3D-3D correspondence constraints of the ORB feature points between frames are obtained;

[0111] b. Based on the epipolar constraint algorithm, the matching point pairs are used as constraints to calculate and optimize the four parameters x, z, pitch, and roll.

[0112] In the intersection scenario, the tracking, matching and positioning calculation of the vehicle are explained:

[0113] Tracking and matching are performed based on target features to obtain the positioning posture of the target vehicle, including:

[0114] Tracking and matching of traffic semantic element features extracted from the current frame with the offline local semantic map;

[0115] The tracked and matched traffic semantic element features are used as constraints to obtain the positioning pose of the target vehicle.

[0116] More specifically: Tracking and matching of traffic semantic element features extracted from the current frame with the offline local semantic map, including:

[0117] Based on the traffic semantic element features extracted from the current frame, obtain the two-dimensional feature points of the stop line feature in the image data, the central two-dimensional point of the detection frame of the traffic light feature in the image data, and the central two-dimensional point of the detection frame of the traffic sign feature in the image data;

[0118] The offline local semantic map is loaded, and through the Euclidean distance constraint, the two-dimensional feature points of the stop line feature in the image data, the central two-dimensional point of the detection frame of the traffic light feature in the image data, and the central two-dimensional point of the detection frame of the traffic sign feature in the image data are obtained with the corresponding semantic elements in the offline local semantic map for tracking and matching.

[0119] In the specific implementation process, in the intersection scene, the deep learning-based extraction algorithm has extracted the stop line, traffic light, and traffic sign of the current frame, and then tracked and matched the extracted stop line, traffic light, and traffic sign semantic features with the offline local semantic map. The specific matching process is as follows:

[0120] a. Extract the two-dimensional feature points of the stop line in the image in the current frame based on the deep learning extraction algorithm , m=1, 2..., extract the detection frame of the traffic light in the image in the current frame, and calculate the central two-dimensional point of the detection frame , n=1, 2..., extract the detection frame of the traffic sign in the image in the current frame, and calculate the central two-dimensional point of the detection frame , p=1,2..., where t represents the current frame, m represents the number of feature points of the stop line detected in the current frame, n represents the number of center points of the traffic light detection frame detected in the current frame, and p represents the number of center points of the traffic sign detection frame detected in the current frame;

[0121] b. Load the offline local semantic map, and then use the Euclidean distance constraint to obtain the stop line detection point, traffic light detection frame center point, traffic sign detection frame center point in the image and the corresponding semantic elements in the offline local semantic map to complete the matching.

[0122] In the intersection road scene, the matching stop line, traffic light, and traffic sign are used as constraints to further optimize the six positioning parameters of x, y, z, yaw, pitch, and roll to eliminate the accumulated positioning errors in non-intersection scenes. Specifically, the process is as follows:

[0123] a. Through the matching of the stop line, traffic light, and traffic sign in the previous stage with the offline local semantic map, the corresponding constraints between the 3D points of the semantic elements of the stop line, traffic light, and traffic sign in the map and the 2D points detected in the image are obtained;

[0124] b. Based on the PnP (Perspective-n-Point) algorithm, the 3D points of the stop line in the map are projected into the image, and the line equation expression of the stop line projection point is obtained by least squares fitting. The 3D points of the traffic lights in the map are projected into 2D points in the image, and the 3D points of the traffic signs in the map are projected into 2D points in the image.

[0125] c. Then, through the BA (Bundle Adjustment) algorithm, the reprojection error between the stop line 2D detection point in the image and the stop line 3D point in the map, the reprojection error between the traffic light 2D detection point in the image and the traffic light 3D point in the map, and the reprojection error between the traffic sign 2D detection point in the image and the traffic sign 3D point in the map are iteratively optimized;

[0126] d. Based on the constraints of stop lines, traffic lights, and traffic signs, the six positioning parameters x, y, z, yaw, pitch, and roll are further optimized. The optimization goal is to minimize the distance d1 from the stop line 2D image detection point to the 3D stop line projection fitting line in the map, minimize the distance d2 from the traffic light 2D image detection center point to the 3D traffic light projection point in the map, and minimize the distance d3 from the traffic sign 2D image detection center point to the 3D traffic sign projection point in the map. When d1, d2, and d3 are less than the set threshold, the iterative optimization is completed.

[0127] The embodiment of the present application integrates traffic semantic elements and traditional ORB features, and the calculated pose is relatively reliable and robust, which solves the limitation that ORB features alone cannot extract rich ORB corner features in open sections. The positioning pose accuracy is further optimized by the above method, which not only eliminates the cumulative error caused by non-intersection road scenes, but also achieves positioning accuracy within 10cm. At the same time, the loop detection and global optimization modules with large computational complexity are removed, and the chip computing power requirements are low.

[0128] In this embodiment, the present application realizes vision-based vehicle positioning by collecting image data, avoiding the use of high-cost lidar equipment. On this basis, according to the collected image data, different visual features and different positioning algorithms are used to perform positioning calculations for different road traffic scenes at non-intersections and intersections. In non-intersection scenarios, the traffic semantic element features and ORB features extracted by the two algorithms are used to output the positioning posture in real time, and the traffic semantic element features are used to calculate the posture in intersection scenarios. This can not only meet the real-time positioning needs, but also eliminate the accumulated errors in non-intersection scenarios at intersections, thereby improving the positioning accuracy. At the same time, it avoids the use of loop detection and global optimization algorithms with large computational complexity, and has relatively low requirements on the computing power of the chip, thereby further saving costs while meeting actual usage needs.

[0129] See attached Figure 4 , in the attached Figure 4 The present application is further described below in the following embodiments:

[0130] First, perform positioning initialization, use the on-board positioning device to provide the initial position and posture, and obtain the initial position and posture when the unmanned vehicle starts. The on-board positioning device is a positioning device such as GPS or integrated navigation.

[0131] Secondly, image data collection is carried out. The on-board camera is used to collect road traffic scene data. These data are saved as image data and can be subjected to image preprocessing operations such as scaling and enhancement.

[0132] After image acquisition, image feature extraction is performed. On the one hand, ORB features are extracted based on image processing methods, and on the other hand, traffic semantic elements such as lane lines, stop lines, traffic lights, and traffic signs are extracted based on deep learning methods.

[0133] Then, the scene is divided into intersection scene and non-intersection scene for feature tracking and matching, and the positioning pose is calculated and optimized based on the tracking and matching results. In the non-intersection road scene, the lane line extracted from the current frame is tracked and matched with the offline local semantic map, and the ORB feature extracted from the current frame is tracked and matched with the previous frame; in the intersection road scene, the extracted stop line, traffic light, and traffic sign are tracked and matched with the offline local semantic map; in the non-intersection road scene, the matched lane line feature is first used as a constraint to calculate the two parameters y and yaw, and then the matched ORB feature is used as a constraint to calculate the four parameters x, z, pitch, and roll; in the intersection road scene, the matched stop line, traffic light, and traffic sign are used as constraints to further optimize the six positioning parameters x, y, z, yaw, pitch, and roll;

[0134] The six parameters required for final positioning of the vehicle have been obtained, and the positioning of the vehicle is completed. The positioning posture (x, y, z, yaw, pitch, roll) can be output to the downstream modules such as perception, regulation and control of the autonomous driving system for use.

[0135] See attached Figure 5 Based on the same inventive concept as in the above-mentioned embodiment, the embodiment of the present application further provides a vehicle positioning device based on vision, comprising:

[0136] An acquisition module, the acquisition module is used to acquire image data of a road traffic scene where a target vehicle is located;

[0137] A determination module, the determination module is used to determine an image feature extraction algorithm according to a judgment result of whether the target vehicle is located at an intersection scene; wherein, when the target vehicle is located at an intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on deep learning, and when the target vehicle is not located at an intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on image processing and an extraction algorithm based on deep learning, the extraction algorithm based on deep learning is used to extract traffic semantic element features, and the extraction algorithm based on image processing is used to extract ORB features;

[0138] An extraction module, which is used to extract target features from image data based on an image feature extraction algorithm;

[0139] Positioning module,The positioning module is used to track and match based on target features to obtain the positioning posture of the target vehicle.

[0140] Those skilled in the art should understand that the division of the various modules in the embodiment is only a division of logical functions, and can be fully or partially integrated into one or more actual carriers in actual application, and these modules can be implemented entirely in the form of software called by a processing unit, or entirely in the form of hardware, or in the form of a combination of software and hardware. It should be noted that each module in the vision-based vehicle positioning device in this embodiment corresponds one-to-one to each step in the vision-based vehicle positioning method in the aforementioned embodiment. Therefore, the specific implementation of this embodiment can refer to the aforementioned implementation, which will not be repeated here.

[0141] Based on the same inventive concept as in the aforementioned embodiment, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when loaded and executed by a processor, implements a vision-based vehicle positioning method as provided in the embodiment of the present application.

[0142] Based on the same inventive concept as in the above-mentioned embodiment, an embodiment of the present application further provides an electronic device, including a processor and a memory, wherein:

[0143] The memory is used to store computer programs;

[0144] The processor is used to load and execute a computer program so that the electronic device executes a vision-based vehicle positioning method as provided in the embodiment of the present application.

[0145] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0146] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0147] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0148] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0149] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0150] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0152] In summary, the present application provides a vision-based vehicle positioning method, device, storage medium and equipment, the method comprising: obtaining image data of a road traffic scene in which a target vehicle is located; determining an image feature extraction algorithm based on a judgment result of whether the target vehicle is located in an intersection scene; wherein, when the target vehicle is located in an intersection scene, the image feature extraction algorithm comprises: an extraction algorithm based on deep learning, and when the target vehicle is not located in an intersection scene, the image feature extraction algorithm comprises: an extraction algorithm based on image processing and an extraction algorithm based on deep learning, the extraction algorithm based on deep learning is used to extract traffic semantic element features, and the extraction algorithm based on image processing is used to extract ORB features; based on the image feature extraction algorithm, extract target features in the image data; perform tracking and matching based on the target features to obtain the positioning posture of the target vehicle. This application realizes vision-based vehicle positioning by collecting image data, avoiding the use of high-cost LiDAR equipment. On this basis, different visual features and different positioning algorithms are used for positioning calculations according to the collected image data for different road traffic scenes at non-intersections and intersections. In non-intersection scenarios, the traffic semantic element features and ORB features extracted by the two algorithms are used to output the positioning posture in real time, and the traffic semantic element features are used to calculate the posture in intersection scenarios. This not only meets the real-time positioning needs, but also eliminates the accumulated errors in non-intersection scenarios at intersections, thereby improving the positioning accuracy. At the same time, it avoids the use of loop detection and global optimization algorithms with large computational complexity, and has relatively low requirements on the computing power of the chip, further saving costs while meeting actual usage needs.

[0153] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A vehicle positioning method based on vision, characterized in that: The following steps are involved: Acquiring image data of a road traffic scene where a target vehicle is located; the acquiring image data of a road traffic scene where a target vehicle is located includes: Based on multiple cameras, image data of a road traffic scene where a target vehicle is located is obtained; the cameras include two cameras, one camera is installed with a viewing angle facing an oblique upward area, and is used to collect traffic light features, traffic sign features, and surrounding environment ORB features in traffic semantic element features; the other camera is installed with a viewing angle facing an oblique downward area, and is used to collect ground lane line features and stop line features in traffic semantic element features; Determine an image feature extraction algorithm according to a result of determining whether the target vehicle is located at an intersection; wherein, when the target vehicle is located at an intersection, the image feature extraction algorithm includes: an extraction algorithm based on deep learning; and when the target vehicle is not located at an intersection, the image feature extraction algorithm includes: an extraction algorithm based on image processing and an extraction algorithm based on deep learning, wherein the extraction algorithm based on deep learning is used to extract traffic semantic element features, and the extraction algorithm based on image processing is used to extract ORB features; Extracting target features from the image data based on the image feature extraction algorithm; Tracking and matching are performed based on the target features to obtain the positioning posture of the target vehicle.

2. The method for vehicle positioning based on vision according to claim 1, characterized in that: In the case where the target vehicle is not located at the intersection scene, extracting the target features in the image data based on the image feature extraction algorithm includes: Extracting ORB features of the image data based on the image processing-based extraction algorithm; Based on the deep learning-based extraction algorithm, traffic semantic element features of the image data are extracted; wherein the traffic semantic element features include lane line features.

3. The method for vehicle positioning based on vision according to claim 2, characterized in that: The tracking and matching based on the target features to obtain the positioning posture of the target vehicle includes: Tracking and matching the ORB feature extracted based on the current frame and the ORB feature extracted from the previous frame; Tracking and matching the lane line features extracted based on the current frame with the offline local semantic map; Using the tracked and matched lane line features as constraints to obtain first positioning posture sub-parameters; Using the ORB features tracked and matched as constraints, obtaining second positioning posture sub-parameters; The positioning posture of the target vehicle is obtained according to the first positioning posture sub-parameter and the second positioning posture sub-parameter.

4. The method for vehicle positioning based on vision according to claim 1, characterized in that: In the case where the target vehicle is located at an intersection, extracting target features from the image data based on the image feature extraction algorithm includes: Based on the deep learning-based extraction algorithm, traffic semantic element features of the image data are extracted; wherein the traffic semantic element features include stop line features, traffic light features, and traffic sign features.

5. The method for vehicle positioning based on vision according to claim 4, characterized in that: The tracking and matching based on the target features to obtain the positioning posture of the target vehicle includes: Tracking and matching the traffic semantic element features extracted based on the current frame with the offline local semantic map; The tracked and matched traffic semantic element features are used as constraints to obtain the positioning posture of the target vehicle.

6. The method for vehicle positioning based on vision according to claim 5, characterized in that: The step of tracking and matching the traffic semantic element features extracted based on the current frame with the offline local semantic map includes: Based on the traffic semantic element features extracted from the current frame, obtain the two-dimensional feature points of the stop line feature in the image data, the central two-dimensional point of the detection frame of the traffic light feature in the image data, and the central two-dimensional point of the detection frame of the traffic sign feature in the image data; Load the offline local semantic map, and obtain the two-dimensional feature points of the stop line feature in the image data, the central two-dimensional point of the detection frame of the traffic light feature in the image data, and the central two-dimensional point of the detection frame of the traffic sign feature in the image data and the corresponding semantic elements in the offline local semantic map through Euclidean distance constraints for tracking and matching.

7. A vehicle positioning device based on vision, characterized in that: include: An acquisition module, the acquisition module is used to acquire image data of a road traffic scene where a target vehicle is located; The step of acquiring image data of a road traffic scene where a target vehicle is located includes: Based on multiple cameras, image data of a road traffic scene where a target vehicle is located is obtained; the cameras include two cameras, one camera is installed with a viewing angle facing an oblique upward area, and is used to collect traffic light features, traffic sign features, and surrounding environment ORB features in traffic semantic element features; the other camera is installed with a viewing angle facing an oblique downward area, and is used to collect ground lane line features and stop line features in traffic semantic element features; A determination module, the determination module is used to determine an image feature extraction algorithm according to a judgment result of whether the target vehicle is located at an intersection scene; wherein, when the target vehicle is located at an intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on deep learning, and when the target vehicle is not located at an intersection scene, the image feature extraction algorithm includes: an extraction algorithm based on image processing and an extraction algorithm based on deep learning, the extraction algorithm based on deep learning is used to extract traffic semantic element features, and the extraction algorithm based on image processing is used to extract ORB features; An extraction module, the extraction module is used to extract target features in the image data based on the image feature extraction algorithm; A positioning module is used to perform tracking and matching based on the target features to obtain the positioning posture of the target vehicle.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is loaded and executed by a processor, the vision-based vehicle positioning method as described in any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: comprising a processor and a memory, wherein: The memory is used to store computer programs; The processor is used to load and execute the computer program so that the electronic device executes the vision-based vehicle positioning method as described in any one of claims 1-6.

Images