Vehicle positioning method, device, electronic device, and computer-readable storage medium

Abstract

The present application discloses a vehicle positioning method, device, electronic device, and computer-readable storage medium. The method includes: obtaining a road image collected by a vehicle, performing semantic segmentation on the road image using a preset semantic segmentation model to obtain a semantic segmentation image; preprocessing the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image; matching the road sign contour information with the road sign data in the corresponding high-precision vector map; constructing a nonlinear optimization model based on the road sign matching results; determining positioning correction information through the nonlinear optimization model, and correcting the initial positioning information of the vehicle using the positioning correction information to obtain the final positioning result of the vehicle. The present application uses data from a high-precision vector map to correct the initial positioning information of the vehicle, thereby improving the positioning accuracy and positioning stability of the vehicle, and greatly reducing the positioning cost compared to the laser radar positioning method.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a vehicle positioning method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In the field of autonomous driving, accurate and reliable positioning technology is essential for autonomous driving and safe obstacle avoidance. Currently, common vehicle positioning solutions include: positioning methods based on GPS (Global Positioning System) or RTK (Real Time Kinematic), positioning methods based on LiDAR, and traditional pure vision positioning methods.

[0003] In the above solutions, GPS and RTK positioning equipment have high deployment costs and poor positioning performance when communications are interfered with. LiDAR solutions have high hardware costs and rely on diverse scene structures, resulting in large positioning errors in scenes with similar structures. While purely visual solutions are low-cost, they are susceptible to environmental interference, such as lighting, and cannot guarantee positioning accuracy and stability. Summary of the Invention

[0004] The embodiments of the present application provide a vehicle positioning method, device, electronic device, and computer-readable storage medium to improve the positioning accuracy of autonomous driving vehicles and reduce positioning costs.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a vehicle positioning method, wherein the method includes:

[0007] Acquire a road image collected by a vehicle, and perform semantic segmentation on the road image using a preset semantic segmentation model to obtain a semantic segmentation image;

[0008] Preprocessing the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image;

[0009] Matching the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result;

[0010] constructing a nonlinear optimization model based on the road sign matching results;

[0011] Positioning correction information is determined by the nonlinear optimization model, and the initial positioning information of the vehicle is corrected by the positioning correction information to obtain a final positioning result of the vehicle.

[0012] Optionally, preprocessing the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image includes:

[0013] Searching for road sign contours in the semantic segmentation image, and recording contour points in the road sign contours and corresponding road sign types;

[0014] Merging the found road sign contours to obtain a merged road sign contour;

[0015] filtering the merged road sign contour according to the number of contour points in the merged road sign contour to obtain a filtered road sign contour;

[0016] The filtered road sign contours, contour points in the filtered road sign contours, and corresponding road sign types are used as road sign contour information corresponding to the semantic segmentation image.

[0017] Optionally, matching the road sign contour information with road sign data in a corresponding high-precision vector map to obtain a road sign matching result includes:

[0018] Obtaining initial positioning information of the vehicle;

[0019] Based on the initial positioning information, obtaining road sign data in the corresponding high-precision vector map;

[0020] The road sign contour information is matched with the road sign data in the high-precision vector map to obtain the road sign matching result.

[0021] Optionally, matching the road sign contour information with road sign data in a corresponding high-precision vector map to obtain a road sign matching result includes:

[0022] Based on the camera extrinsic parameters of the vehicle, the road sign contour information is converted into a vehicle body coordinate system through an inverse perspective transformation to obtain a 3D road sign contour in the vehicle body coordinate system;

[0023] Based on the initial positioning information of the vehicle, converting the road marking data in the high-precision vector map into the vehicle body coordinate system to obtain the road marking data in the vehicle body coordinate system;

[0024] The 3D road sign outline in the vehicle body coordinate system is matched with the road sign data in the vehicle body coordinate system to obtain the road sign matching result.

[0025] Optionally, the 3D road sign outline includes a linear road sign outline and a non-linear road sign outline, and converting the road sign outline information into a vehicle body coordinate system through an inverse perspective transformation based on the vehicle's camera extrinsic parameters to obtain the 3D road sign outline in the vehicle body coordinate system includes:

[0026] For the linear road sign outline, retaining 3D contour points in the linear road sign outline;

[0027] For the non-linear road sign outline, a circumscribed rectangle of the non-linear road sign outline is determined, and each vertex of the circumscribed rectangle is retained.

[0028] Optionally, the road sign outline information includes a road sign outline, and matching the road sign outline information with road sign data in a corresponding high-precision vector map to obtain a road sign matching result includes:

[0029] According to the road sign type corresponding to the road sign outline, the road sign outline is matched with each road sign data corresponding to the same road sign type in the high-precision vector map to obtain a plurality of matching results corresponding to the road sign outline;

[0030] The road sign matching result is determined according to a plurality of matching results corresponding to the road sign contour.

[0031] Optionally, the road sign matching result includes the distance between the contour point in the non-linear road sign contour and the matching road sign point in the road sign data, and the distance between the contour point in the linear road sign contour and the matching road sign line in the road sign data;

[0032] The constructing of a nonlinear optimization model according to the road sign matching result includes:

[0033] Constructing a matching point distance constraint condition based on the distance between the contour point in the non-linear road sign contour and the matching road sign point in the road sign data;

[0034] Constructing a point-to-straight-line distance constraint condition based on the distance between the contour point in the linear road sign contour and the road sign line matched in the road sign data;

[0035] The nonlinear optimization model is constructed according to the matching point distance constraint, the point-to-straight line distance constraint, and the initial positioning information constraint.

[0036] In a second aspect, an embodiment of the present application further provides a vehicle positioning device, wherein the device includes:

[0037] an acquisition unit, configured to acquire a road image collected by a vehicle, and perform semantic segmentation on the road image using a preset semantic segmentation model to obtain a semantically segmented image;

[0038] a preprocessing unit, configured to preprocess the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image;

[0039] a matching unit, configured to match the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result;

[0040] A construction unit, configured to construct a nonlinear optimization model according to the road sign matching result;

[0041] A positioning correction unit is used to determine positioning correction information through the nonlinear optimization model, and correct the initial positioning information of the vehicle through the positioning correction information to obtain a final positioning result of the vehicle.

[0042] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0043] processor; and

[0044] A memory arranged to store computer executable instructions, which when executed cause the processor to perform any of the methods described above.

[0045] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes any of the aforementioned methods.

[0046] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the vehicle positioning method of the embodiments of the present application first obtains a road image collected by the vehicle, and uses a preset semantic segmentation model to semantically segment the road image to obtain a semantic segmentation image; then, according to a preset preprocessing strategy, the semantic segmentation image is preprocessed to obtain the road sign contour information corresponding to the semantic segmentation image; then, the road sign contour information is matched with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result; then, a nonlinear optimization model is constructed based on the road sign matching result; finally, the positioning correction information is determined by the nonlinear optimization model, and the initial positioning information of the vehicle is corrected by the positioning correction information to obtain the final positioning result of the vehicle. The embodiments of the present application use the data of the high-precision vector map to correct the initial positioning information of the vehicle, and solve the optimal positioning correction information by constructing a nonlinear optimization model, which further improves the positioning accuracy and positioning stability of the vehicle when the GPS or RTK positioning signal is poor, and compared with the positioning method using positioning equipment such as lidar, it greatly reduces the positioning cost and improves the positioning effect of scenes with similar structures. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 A schematic flow chart of a vehicle positioning method according to an embodiment of the present application;

[0049] Figure 2 This is a structural diagram of a vehicle positioning device in an embodiment of the present application;

[0050] Figure 3 This is a schematic structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0053] The embodiment of the present application provides a vehicle positioning method, such as Figure 1As shown, a flow chart of a vehicle positioning method in an embodiment of the present application is provided, wherein the method at least includes the following steps S110 to S150:

[0054] Step S110 , obtaining a road image captured by a vehicle, and performing semantic segmentation on the road image using a preset semantic segmentation model to obtain a semantic segmentation image.

[0055] When performing vehicle positioning in the embodiment of the present application, a road image captured by a vehicle-mounted camera pre-installed on the vehicle is first obtained. The road image may refer to an image obtained by the vehicle-mounted camera taking pictures of the road surface and vehicles on the road surface within the surrounding visible range while the vehicle is driving.

[0056] After obtaining the road image, semantic segmentation is further performed on the road image using a pre-trained semantic segmentation model, resulting in a semantically segmented image. Each pixel value in the semantically segmented image corresponds to a segmentation type, which can specifically include road sign types, including but not limited to lane lines, arrows, stop signs, and sidewalks. This semantic segmentation model can be trained based on an existing convolutional neural network, such as the LaneNet model. The specific form of semantic segmentation model employed is flexibly selected by those skilled in the art based on practical needs and is not specifically defined herein.

[0057] Step S120 : preprocessing the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image.

[0058] After obtaining the semantic segmentation model, it is necessary to perform certain preprocessing operations on the semantic segmentation model. The purpose of preprocessing is to determine the road sign contour information included in the semantic segmentation image. For example, each specific road sign will correspond to a road sign contour, which can be determined based on the road sign type corresponding to each pixel value in the semantic segmentation image and the distance between adjacent pixels.

[0059] Step S130 : Matching the road sign outline information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result.

[0060] The road sign contour information obtained based on the semantic segmentation model can be regarded as positioning information obtained based on deep learning. Although the accuracy and reliability of the data have been improved compared to traditional vision technology, relying solely on this data is still not enough to meet the requirements for positioning accuracy and positioning stability in autonomous driving scenarios.

[0061] Based on this, the embodiment of the present application uses a pre-constructed high-precision vector map to further obtain high-precision road sign data in the high-precision vector map. By matching the road sign contour information obtained above with the road sign data in the high-precision vector map, the road sign data that matches the road sign contour information is determined in the high-precision vector map, and then the road sign matching result is obtained.

[0062] The road sign data in the high-precision vector map can specifically include information such as the road sign type, geometric description, and lane location. As mentioned above, road sign types include but are not limited to: lane lines, arrows, stop lines, sidewalks, etc. The geometric description is used to describe the points, lines, polygons, etc. that constitute the road sign.

[0063] Step S140: constructing a nonlinear optimization model based on the road sign matching result.

[0064] The road sign matching results represent the matching between the road sign contour information in the image obtained based on the semantic segmentation model and the road sign data in the high-precision vector map. However, in actual situations, it is difficult to ensure a complete match between the two, that is, there will be a certain matching error. Therefore, the embodiment of the present application constructs a nonlinear optimization model based on the above-mentioned road sign matching results, so that the optimal positioning correction information can be solved through the nonlinear optimization model.

[0065] The nonlinear optimization model described above can be implemented using a least squares algorithm. The principle of the least squares algorithm is to find the optimal function matching the data by minimizing the sum of squared errors. Of course, those skilled in the art can also flexibly select other types of nonlinear optimization models based on actual needs, and this is not specifically limited here.

[0066] Step S150 , determining positioning correction information through the nonlinear optimization model, and correcting the initial positioning information of the vehicle through the positioning correction information to obtain a final positioning result of the vehicle.

[0067] After completing the construction of the nonlinear optimization model, the nonlinear optimization model can be solved using a preset solution algorithm to obtain the optimal positioning correction information. In the embodiment of the present application, when solving the nonlinear optimization model, a preset gradient descent algorithm can be used, such as a GN (Gauss-Newton) algorithm or a LM (Levenberg-Marquardt) algorithm. Of course, which solution algorithm to use can be flexibly selected by those skilled in the art according to actual needs, and is not specifically limited here.

[0068] The above-mentioned positioning correction information can be understood as the relative transformation relationship between the pixel points in the road image and the road marking points in the high-precision vector map. After determining the relative transformation relationship, the relative transformation relationship can be used to correct the initial positioning information of the vehicle, that is, the initial positioning information can be converted into positioning information based on the high-precision vector map using the relative transformation relationship.

[0069] It should be noted that the vehicle positioning method of the embodiment of the present application can be used to correct the initial positioning information when the GPS or RTK positioning signal is poor. Of course, when the GPS or RTK positioning signal is good, the vehicle positioning method of the embodiment of the present application can also be used.

[0070] The embodiment of the present application uses data from a high-precision vector map to correct the initial positioning information of the vehicle, and solves the optimal positioning correction information by constructing a nonlinear optimization model, thereby further improving the positioning accuracy and positioning stability of the vehicle when the GPS or RTK positioning signal is poor. Compared with the positioning method using positioning equipment such as lidar, it greatly reduces the positioning cost and improves the positioning effect of structurally similar scenarios.

[0071] In one embodiment of the present application, the preprocessing of the semantic segmentation image according to a preset preprocessing strategy to obtain the road sign contour information corresponding to the semantic segmentation image includes: searching for road sign contours in the semantic segmentation image, and recording the contour points and corresponding road sign types in the road sign contours; merging the found road sign contours to obtain a merged road sign contour; filtering the merged road sign contour according to the number of contour points in the merged road sign contour to obtain a filtered road sign contour; and using the filtered road sign contour, the contour points and the corresponding road sign type in the filtered road sign contour as the road sign contour information corresponding to the semantic segmentation image.

[0072] When obtaining road sign contour information in a semantically segmented image, an embodiment of the present application can first find the road sign contour based on the road sign type of each pixel point in the semantically segmented image and the distance between the pixel points. For example, lane line points on the same lane line should have the same road sign type, and these lane line points are similar in geometric distance. In this way, all road sign contours contained in the semantically segmented image can be found, and the contour points contained in each road sign contour and the corresponding road sign type can be recorded.

[0073] Since the initially obtained road sign contours may have certain errors, in order to further reduce the errors, the embodiment of the present application may further perform merging and filtering on the road sign contours obtained in the aforementioned steps.

[0074] Specifically, in actual scenarios, since road signs may be obscured or no longer complete due to long-term wear and tear, the same road sign may be segmented into multiple different road signs. For such mis-segmentation, the road sign contours can be merged based on the geometric distance between two adjacent contour areas and the road sign type, that is, multiple road sign contours that should have been segmented into one road sign contour can be merged, thereby avoiding or reducing the impact of the above-mentioned mis-segmentation.

[0075] After completing the above-mentioned merging process of the road sign contours, the road sign contours can be further screened or eliminated according to the number of contour points contained in the merged road sign contours to avoid the merged road sign contours containing too few contour points, thereby affecting the subsequent matching process.

[0076] In one embodiment of the present application, matching the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain the road sign matching result includes: obtaining the initial positioning information of the vehicle; based on the initial positioning information, obtaining the road sign data in the corresponding high-precision vector map; matching the road sign contour information with the road sign data in the high-precision vector map to obtain the road sign matching result.

[0077] When matching road sign outline information with road sign data in a corresponding high-precision vector map, embodiments of the present application can first obtain the vehicle's initial positioning information. This initial positioning information can be obtained through a variety of positioning methods, such as positioning information output by a GNSS (Global Navigation Satellite System), positioning information output by an IMU (Inertial Measurement Unit) and a wheel odometer, either individually or after fusion, and positioning information output by a lidar. Of course, those skilled in the art can flexibly select the specific method for obtaining the initial positioning information based on actual needs, and this is not specifically limited here.

[0078] The initial positioning information includes the vehicle's initial position coordinates, etc. Based on the initial position coordinates, the road sign data in the high-precision vector map within a certain range from the position can be obtained. Finally, the road sign data in the obtained local high-precision vector map is matched with the road sign contour information to obtain the above-mentioned road sign matching result.

[0079] In one embodiment of the present application, the road sign contour information is matched with the road sign data in the corresponding high-precision vector map to obtain the road sign matching result, including: based on the camera extrinsic parameters of the vehicle, the road sign contour information is converted to the vehicle body coordinate system through inverse perspective transformation to obtain the 3D road sign contour in the vehicle body coordinate system; based on the initial positioning information of the vehicle, the road sign data in the high-precision vector map is converted to the vehicle body coordinate system to obtain the road sign data in the vehicle body coordinate system; the 3D road sign contour in the vehicle body coordinate system is matched with the road sign data in the vehicle body coordinate system to obtain the road sign matching result.

[0080] Since the road sign contour information obtained based on the road image in the embodiment of the present application and the road sign data in the corresponding high-precision vector map correspond to different coordinate systems, in order to ensure the accuracy of the matching results, a unified transformation of the coordinate system can be performed first.

[0081] Specifically, the calibrated camera extrinsics can be first obtained. Based on the camera extrinsics, the road sign outline information can be converted to the vehicle coordinate system using the IPM (Inverse Perspective Mapping) method to obtain the 3D road sign outline in the vehicle coordinate system. For road sign data in the high-precision vector map, the initial positioning information of the vehicle can be used to convert the road sign data in the high-precision vector map to the vehicle coordinate system to obtain the road sign data in the vehicle coordinate system.

[0082] The IPM method used above can be specifically expressed as:

[0083]

[0084] In the above formula (1), [X car ,Y car ,Z car ] T Represents the coordinates of the ground point in the vehicle coordinate system, [u,v,1] T represents the homogeneous pixel coordinates, R is the rotation matrix from the camera coordinate system to the vehicle coordinate system, t is the translation vector from the camera coordinate system to the vehicle coordinate system, d is the depth of the ground point in the camera coordinate system, and K is the camera intrinsic parameter matrix (K -1 represents the inverse of the matrix).

[0085] It should be noted that it is assumed here that the Z value of the ground point in the vehicle coordinate system is 0, that is, Z car = 0, and then we can use the following formula (2) to find d, and then find the whole [X car ,Y car ,Zcar ] T .

[0086]

[0087] In the above formula, r:3 means taking the value of the third row of the vector.

[0088] After the above coordinate system transformation process, the road sign contour information and the corresponding road sign data in the high-precision vector map can be unified into the vehicle body coordinate system, and finally the two can be matched with each other in the vehicle body coordinate system, thereby improving the matching accuracy.

[0089] In one embodiment of the present application, the 3D road sign contour includes a linear road sign contour and a non-linear road sign contour. The road sign contour information is converted into a vehicle body coordinate system through an inverse perspective transformation based on the camera extrinsic parameters of the vehicle, and the 3D road sign contour in the vehicle body coordinate system is obtained, including: for the linear road sign contour, retaining the 3D contour points in the linear road sign contour; for the non-linear road sign contour, determining the circumscribed rectangle of the non-linear road sign contour, and retaining each vertex of the circumscribed rectangle.

[0090] The 3D road sign contours obtained based on the above embodiments can be specifically divided into linear road sign contours and non-linear road sign contours. The linear road sign contours may include, for example, lane lines, stop lines, etc., and the non-linear road sign contours may include, for example, arrows, etc.

[0091] The road sign data in the high-precision vector map will also distinguish between the two types of road sign contours mentioned above. For linear road sign contours, all contour points in the linear road sign contour and the straight line equation obtained by fitting these contour points will be recorded. For non-linear road sign contours, the circumscribed rectangle of the non-linear road sign contour is usually determined, and the four vertices of the circumscribed rectangle are recorded.

[0092] Based on this, in order to improve the matching efficiency, the embodiment of the present application can directly retain all the contour points contained in the linear road marking contour in the 3D road marking contour, and for the non-linear road marking contour in the 3D road marking contour, the outer rectangle of the non-linear road marking contour can be calculated first, and the four vertices of the circumscribed rectangle can be recorded, so as to ensure the consistency with the form of the road marking data in the high-precision vector map and improve the matching efficiency.

[0093] Of course, it should be noted that for non-linear road sign contours, it is also possible not to calculate the circumscribed rectangle and directly retain all contour points, and the subsequent matching process can also be carried out.

[0094] In one embodiment of the present application, the road sign contour information includes a road sign contour, and the matching of the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result includes: matching the road sign contour with each road sign data corresponding to the same road sign type in the high-precision vector map according to the road sign type corresponding to the road sign contour to obtain multiple matching results corresponding to the road sign contour; and determining the road sign matching result based on the multiple matching results corresponding to the road sign contour.

[0095] The road sign outline information in the embodiment of the present application may include multiple road sign outlines. When matching each road sign outline, the road sign data in the high-precision vector map can be used to find road sign data of the same road sign type based on the road sign type corresponding to the road sign outline. For example, if the road sign type corresponding to a road sign outline is a lane line, the road sign data in the corresponding high-precision vector map can be filtered out before matching other road sign types such as stop lines and arrows, thereby improving matching efficiency.

[0096] Afterwards, when matching road sign contour information under the same road sign type, the geometric distance between each contour point in the road sign contour and the corresponding road sign point in the high-precision vector map can be calculated separately, and finally the matching result with the closest geometric distance is used as the final road sign matching result.

[0097] It should be further explained here that since the non-linear road sign contour only retains the four vertices of the circumscribed rectangle of the contour, subsequent matching may result in inaccurate matching due to the small number of feature points. Therefore, appropriate sampling can be performed on each side of the circumscribed rectangle to obtain denser feature point data and improve the accuracy of subsequent matching.

[0098] In one embodiment of the present application, the road sign matching result includes the distance between the contour points in the non-linear road sign contour and the matching road sign points in the road sign data, and the distance between the contour points in the linear road sign contour and the matching road sign lines in the road sign data; constructing a nonlinear optimization model based on the road sign matching result includes: constructing a matching point distance constraint based on the distance between the contour points in the non-linear road sign contour and the matching road sign points in the road sign data; constructing a point-to-straight line distance constraint based on the distance between the contour points in the linear road sign contour and the matching road sign lines in the road sign data; constructing the nonlinear optimization model based on the matching point distance constraint, the point-to-straight line distance constraint and the initial positioning information constraint.

[0099] Depending on the type of road sign contour, the road sign matching results of the embodiment of the present application may specifically include the distance between the contour point in the non-linear road sign contour and the matching road sign point in the road sign data, that is, the distance between the matching points, and the distance between the contour point in the linear road sign contour and the matching road sign line in the road sign data, that is, the distance from the point to the straight line.

[0100] Based on the road sign matching results from these two dimensions, we can further construct matching point distance constraints based on the distance between matching points corresponding to nonlinear road sign outlines, and point-to-line distance constraints based on the distance between points and lines corresponding to linear road sign outlines. By constraining the iterative process of the nonlinear optimization model with these two constraints, we can determine the optimal relative transformation relationship between pixels in the road image and road sign points in the high-precision vector map.

[0101] In addition, the embodiment of the present application also introduces an initial positioning information constraint condition, that is, the initial positioning information is used as a priori condition to constrain the iterative process of the model. Although the initial positioning information is not accurate enough, it can still provide a reference to a certain extent, and the introduction of the initial positioning information for further constraints can greatly improve the speed of model convergence.

[0102] To facilitate understanding of the various embodiments of the present application, the construction process of the nonlinear optimization model of the embodiment of the present application is further listed here in combination with the specific algorithm:

[0103] 1) State quantity to be optimized: the relative pose transformation T of the matching points unified in the current vehicle coordinate system delta Specifically, it refers to the relative pose transformation T between the 3D points of the road markings obtained after the camera image is processed by IPM and the 3D points of the road markings transferred from the high-precision vector map to the vehicle coordinate system. delta ;

[0104] 2) Matching point distance constraint:

[0105] The residual term of a pair of matching points:

[0106]

[0107] In the above formula (3), Norm(·) represents the modulus of the vector.

[0108] Overall optimization objective function:

[0109]

[0110] 3) Point-to-straight-line distance constraint:

[0111] Assuming that the analytical expression of the straight line is y=kx+b, the distance formula from point P(x0,y0) to the straight line is:

[0112]

[0113] Overall optimization objective function:

[0114]

[0115] 4) Initial positioning information constraints:

[0116] Residual term:

[0117]

[0118] Overall optimization objective function:

[0119]

[0120] In the above formulas (3)-(8), T delta is the state quantity to be optimized, that is, the relative position to be optimized, which includes rotation and translation, T ini is the initial relative pose, which can be solved by linear algebra such as SVD (Singular Value Decomposition) according to the matching points or initialized to a unit matrix, ∑ -1 is the information weight matrix, ∑ is the camera extrinsic parameter covariance matrix, different variables correspond to different information weights, log(·) ∨ Represents a logarithmic mapping. For example, the Lie group SE3 can be mapped to the Lie algebra se3, which is convenient for optimization and solution.

[0121] The embodiment of the present application also provides a vehicle positioning device 200, such as Figure 2 As shown, a schematic structural diagram of a vehicle positioning device in an embodiment of the present application is provided. The device 200 includes: an acquisition unit 210, a preprocessing unit 220, a matching unit 230, a construction unit 240, and a positioning correction unit 250, wherein:

[0122] An acquisition unit 210 is configured to acquire a road image collected by a vehicle and perform semantic segmentation on the road image using a preset semantic segmentation model to obtain a semantically segmented image;

[0123] A preprocessing unit 220 is configured to preprocess the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image;

[0124] A matching unit 230 is configured to match the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result;

[0125] A construction unit 240 is configured to construct a nonlinear optimization model according to the road sign matching result;

[0126] The positioning correction unit 250 is configured to determine positioning correction information using the nonlinear optimization model, and correct the initial positioning information of the vehicle using the positioning correction information to obtain a final positioning result of the vehicle.

[0127] In one embodiment of the present application, the preprocessing unit 220 is specifically used to: search for road sign contours in the semantic segmentation image, and record the contour points and corresponding road sign types in the road sign contours; merge the found road sign contours to obtain a merged road sign contour; filter the merged road sign contour according to the number of contour points in the merged road sign contour to obtain a filtered road sign contour; and use the filtered road sign contour, the contour points and corresponding road sign types in the filtered road sign contour as the road sign contour information corresponding to the semantic segmentation image.

[0128] In one embodiment of the present application, the matching unit 230 is specifically used to: obtain the initial positioning information of the vehicle; based on the initial positioning information, obtain the road sign data in the corresponding high-precision vector map; match the road sign contour information with the road sign data in the high-precision vector map to obtain the road sign matching result.

[0129] In one embodiment of the present application, the matching unit 230 is specifically used to: based on the camera extrinsic parameters of the vehicle, convert the road sign contour information to the vehicle body coordinate system through inverse perspective transformation to obtain the 3D road sign contour in the vehicle body coordinate system; based on the initial positioning information of the vehicle, convert the road sign data in the high-precision vector map to the vehicle body coordinate system to obtain the road sign data in the vehicle body coordinate system; match the 3D road sign contour in the vehicle body coordinate system with the road sign data in the vehicle body coordinate system to obtain the road sign matching result.

[0130] In one embodiment of the present application, the 3D road sign contour includes a linear road sign contour and a non-linear road sign contour, and the matching unit 230 is specifically used to: for the linear road sign contour, retain the 3D contour points in the linear road sign contour; for the non-linear road sign contour, determine the circumscribed rectangle of the non-linear road sign contour and retain each vertex of the circumscribed rectangle.

[0131] In one embodiment of the present application, the road sign contour information includes a road sign contour, and the matching unit 230 is specifically used to: match the road sign contour with each road sign data corresponding to the same road sign type in the high-precision vector map according to the road sign type corresponding to the road sign contour, to obtain multiple matching results corresponding to the road sign contour; and determine the road sign matching result based on the multiple matching results corresponding to the road sign contour.

[0132] In one embodiment of the present application, the road sign matching result includes the distance between the contour point in the non-linear road sign contour and the matching road sign point in the road sign data, and the distance between the contour point in the linear road sign contour and the matching road sign line in the road sign data; the construction unit 240 is specifically used to: construct a matching point distance constraint based on the distance between the contour point in the non-linear road sign contour and the matching road sign point in the road sign data; construct a point-to-straight line distance constraint based on the distance between the contour point in the linear road sign contour and the matching road sign line in the road sign data; and construct the nonlinear optimization model based on the matching point distance constraint, the point-to-straight line distance constraint and the initial positioning information constraint.

[0133] It can be understood that the above-mentioned vehicle positioning device can implement each step of the vehicle positioning method provided in the above-mentioned embodiment. The relevant explanations about the vehicle positioning method are applicable to the vehicle positioning device and will not be repeated here.

[0134] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0135] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0136] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0137] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a vehicle positioning device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0138] Acquire a road image collected by a vehicle, and perform semantic segmentation on the road image using a preset semantic segmentation model to obtain a semantic segmentation image;

[0139] Preprocessing the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image;

[0140] Matching the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result;

[0141] constructing a nonlinear optimization model based on the road sign matching results;

[0142] Positioning correction information is determined by the nonlinear optimization model, and the initial positioning information of the vehicle is corrected by the positioning correction information to obtain a final positioning result of the vehicle.

[0143] The above application Figure 1The methods performed by the vehicle positioning device disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be performed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0144] The electronic device may also perform Figure 1 The method executed by the vehicle positioning device in the vehicle positioning device is realized Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0145] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 1 The method performed by the vehicle positioning device in the embodiment shown is specifically used to perform:

[0146] Acquire a road image collected by a vehicle, and perform semantic segmentation on the road image using a preset semantic segmentation model to obtain a semantic segmentation image;

[0147] Preprocessing the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image;

[0148] Matching the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result;

[0149] constructing a nonlinear optimization model based on the road sign matching results;

[0150] Positioning correction information is determined by the nonlinear optimization model, and the initial positioning information of the vehicle is corrected by the positioning correction information to obtain a final positioning result of the vehicle.

[0151] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0155] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0156] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0157] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0158] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0159] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A vehicle positioning method, wherein: The method comprises: Acquire a road image collected by a vehicle, and perform semantic segmentation on the road image using a preset semantic segmentation model to obtain a semantic segmentation image; Preprocessing the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image; Matching the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result; constructing a nonlinear optimization model based on the road sign matching results; Determining positioning correction information through the nonlinear optimization model, and correcting the initial positioning information of the vehicle through the positioning correction information to obtain a final positioning result of the vehicle; Preprocessing the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image includes: Searching for road sign contours in the semantic segmentation image, and recording contour points in the road sign contours and corresponding road sign types; Merging the found road sign contours to obtain a merged road sign contour; filtering the merged road sign contour according to the number of contour points in the merged road sign contour to obtain a filtered road sign contour; Using the filtered road sign contours, contour points in the filtered road sign contours, and corresponding road sign types as road sign contour information corresponding to the semantic segmentation image; The road sign matching result includes the distance between the contour point in the non-linear road sign contour and the matching road sign point in the road sign data, and the distance between the contour point in the linear road sign contour and the matching road sign line in the road sign data; The constructing of a nonlinear optimization model according to the road sign matching result includes: Constructing a matching point distance constraint condition based on the distance between the contour point in the non-linear road sign contour and the matching road sign point in the road sign data; Constructing a point-to-straight-line distance constraint condition based on the distance between the contour point in the linear road sign contour and the road sign line matched in the road sign data; The nonlinear optimization model is constructed according to the matching point distance constraint, the point-to-straight line distance constraint, and the initial positioning information constraint.

2. The method according to claim 1, wherein: The step of matching the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result includes: Obtaining initial positioning information of the vehicle; Based on the initial positioning information, obtaining road sign data in the corresponding high-precision vector map; The road sign contour information is matched with the road sign data in the high-precision vector map to obtain the road sign matching result.

3. The method according to claim 1, wherein: The step of matching the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result includes: Based on the camera extrinsic parameters of the vehicle, the road sign contour information is converted into a vehicle body coordinate system through an inverse perspective transformation to obtain a 3D road sign contour in the vehicle body coordinate system; Based on the initial positioning information of the vehicle, converting the road marking data in the high-precision vector map into the vehicle body coordinate system to obtain the road marking data in the vehicle body coordinate system; The 3D road sign outline in the vehicle body coordinate system is matched with the road sign data in the vehicle body coordinate system to obtain the road sign matching result.

4. The method according to claim 3, wherein: The 3D road sign outline includes a linear road sign outline and a non-linear road sign outline. The converting of the road sign outline information into a vehicle body coordinate system through an inverse perspective transformation based on the vehicle's camera extrinsic parameters to obtain the 3D road sign outline in the vehicle body coordinate system includes: For the linear road sign outline, retaining 3D contour points in the linear road sign outline; For the non-linear road sign outline, a circumscribed rectangle of the non-linear road sign outline is determined, and each vertex of the circumscribed rectangle is retained.

5. The method of claim 1, wherein: The road sign outline information includes a road sign outline, and the road sign outline information is matched with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result, including: According to the road sign type corresponding to the road sign outline, the road sign outline is matched with each road sign data corresponding to the same road sign type in the high-precision vector map to obtain a plurality of matching results corresponding to the road sign outline; The road sign matching result is determined according to a plurality of matching results corresponding to the road sign contour.

6. A vehicle positioning device, wherein: The device comprises: an acquisition unit, configured to acquire a road image collected by a vehicle, and perform semantic segmentation on the road image using a preset semantic segmentation model to obtain a semantically segmented image; a preprocessing unit, configured to preprocess the semantic segmentation image according to a preset preprocessing strategy to obtain road sign contour information corresponding to the semantic segmentation image; a matching unit, configured to match the road sign contour information with the road sign data in the corresponding high-precision vector map to obtain a road sign matching result; A construction unit, configured to construct a nonlinear optimization model according to the road sign matching result; a positioning correction unit, configured to determine positioning correction information using the nonlinear optimization model, and correct the initial positioning information of the vehicle using the positioning correction information to obtain a final positioning result of the vehicle; The pre-processing unit is specifically used for: Searching for road sign contours in the semantic segmentation image, and recording contour points in the road sign contours and corresponding road sign types; Merging the found road sign contours to obtain a merged road sign contour; filtering the merged road sign contour according to the number of contour points in the merged road sign contour to obtain a filtered road sign contour; Using the filtered road sign contours, contour points in the filtered road sign contours, and corresponding road sign types as road sign contour information corresponding to the semantic segmentation image; The road sign matching result includes the distance between the contour point in the non-linear road sign contour and the matching road sign point in the road sign data, and the distance between the contour point in the linear road sign contour and the matching road sign line in the road sign data; The positioning correction unit is specifically used for: Constructing a matching point distance constraint condition based on the distance between the contour point in the non-linear road sign contour and the matching road sign point in the road sign data; Constructing a point-to-straight-line distance constraint condition based on the distance between the contour point in the linear road sign contour and the road sign line matched in the road sign data; The nonlinear optimization model is constructed according to the matching point distance constraint, the point-to-straight line distance constraint, and the initial positioning information constraint.

7. An electronic device comprising: processor; as well as A memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing one or more programs, wherein when the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device executes the method according to any one of claims 1 to 5.

Images