Vehicle positioning method and device, electronic equipment and computer program product

Abstract

The invention discloses a vehicle positioning method and device, electronic equipment and a computer program product, and the method comprises the steps: obtaining sensor data and high-precision map data of a vehicle, the sensor data comprising image data of a plurality of cameras and IMU data; performing semantic segmentation on the image data to obtain road sign segmentation results of the plurality of cameras; performing road mark matching according to the road mark segmentation result and the high-precision map data to obtain road mark matching results of the plurality of cameras; constructing a first vehicle pose optimization constraint according to the road sign matching results of the plurality of cameras and the IMU data; and solving the pose of the vehicle at the current moment according to the first vehicle pose optimization constraint and the optimization solution algorithm. According to the method, the vehicle pose optimization constraint is constructed by using the road identification information observed by the plurality of vehicle-mounted cameras, the problem of positioning failure caused by failure of a single camera is avoided, the problem of positioning continuity of scenes such as road identification information loss is solved by using the IMU data, and the positioning precision and stability are improved.

Description

Technical Field

[0001] This application relates to the technical field of vehicle positioning, and in particular, to a vehicle positioning method, device, electronic device, and computer program product. Background Art

[0002] In the field of vehicle positioning, especially for the positioning of autonomous vehicles, the Global Positioning System (GPS) is used as the core positioning means and can provide high-precision position information in open areas. However, when the vehicle travels to specific environments, such as tunnels, urban canyons (areas with dense high-rise buildings), and under viaducts and other complex scenarios, the GPS signal is often severely interfered, resulting in a weakened or even lost signal strength, and thus causing a significant decrease in positioning accuracy. This not only affects the independent positioning ability of the GPS system but also has an impact on the accuracy of the integrated navigation system that relies on GPS data, limiting its reliability and practicality in complex environments.

[0003] To make up for the deficiencies of GPS in these special scenarios, some solutions attempt to use other sensors for assisted positioning. Among them, the camera, as a low-cost and easily deployable visual perception device, shows great potential. By collecting image data of the road environment and applying deep learning models or traditional image processing algorithms, precise segmentation of road semantic information can be achieved, such as identifying key elements such as lane lines, traffic signs, and road surface markings. Matching these semantic information with the pre-stored data in the high-precision map can construct a vision-based positioning method, effectively improving the positioning accuracy and robustness of the vehicle in complex environments.

[0004] However, problems such as wear, pollution, or occlusion of road signs will interfere with the semantic segmentation in the image, resulting in damage to the integrity of the segmentation result, and thus affecting the vision-based positioning accuracy. In addition, the single-camera system also faces many challenges during the data collection process, such as adverse factors like light changes and field of view occlusion, which may cause the camera to malfunction, thereby limiting its stability and reliability in practical applications. Summary of the Invention

[0005] Embodiments of this application provide a vehicle positioning method, device, electronic device, and computer program product to improve vehicle positioning accuracy and positioning stability.

[0006] Embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a vehicle positioning method, where the vehicle positioning method includes:

[0008] Obtain the sensor data of the vehicle and the corresponding high-precision map data of the vehicle at present. The sensor data includes the image data collected by multiple cameras and the IMU data;

[0009] Perform semantic segmentation on the image data collected by multiple cameras respectively to obtain the road sign segmentation results of multiple cameras;

[0010] Perform road sign matching according to the road sign segmentation results of multiple cameras and the corresponding high-precision map data of the vehicle at present to obtain the road sign matching results of multiple cameras;

[0011] Construct the first vehicle pose optimization constraint according to the road sign matching results of multiple cameras and the IMU data;

[0012] According to the first vehicle pose optimization constraint, use the optimization algorithm to solve the pose of the vehicle at the current moment.

[0013] Optionally, the performing road sign matching according to the road sign segmentation results of multiple cameras and the corresponding high-precision map data of the vehicle at present to obtain the road sign matching results of multiple cameras includes:

[0014] Convert the corresponding high-precision map data of the vehicle at present into the camera images of each camera respectively;

[0015] Perform road sign matching on the road sign segmentation results of each camera and the high-precision map data in the camera images of each camera respectively to obtain the road sign matching results of multiple cameras.

[0016] Optionally, the converting the corresponding high-precision map data of the vehicle at present into the camera images of each camera respectively includes:

[0017] Calculate the reference pose of the vehicle at the current moment;

[0018] Based on the reference pose of the vehicle at the current moment, transform the corresponding high-precision map data of the vehicle at present into the vehicle body coordinate system to obtain the high-precision map data in the vehicle body coordinate system;

[0019] Calculate the relative pose of each camera to the current moment according to the IMU data;

[0020] According to the relative pose of each camera to the current moment and the camera parameters of each camera, convert the high-precision map data in the vehicle body coordinate system into the camera images of each camera respectively.

[0021] Optionally, the IMU data is the IMU data between the previous moment and the current moment, and the calculating the reference pose of the vehicle at the current moment includes:

[0022] Obtain the pose of the vehicle at the previous moment;

[0023] Calculate the reference pose of the vehicle at the current moment based on the previous moment pose of the vehicle and the IMU data between the previous moment and the current moment.

[0024] Optionally, the calculating the relative pose of each camera to the current moment according to the IMU data includes:

[0025] Determine the IMU data corresponding to the time stamps of the image data collected by each camera according to the IMU data and the time stamps of the image data collected by each camera;

[0026] Determine the IMU data corresponding to the time stamp of the current moment according to the IMU data and the time stamp of the current moment;

[0027] Calculate the relative pose of each camera to the current moment respectively according to the IMU data corresponding to the time stamps of the image data collected by each camera and the IMU data corresponding to the time stamp of the current moment.

[0028] Optionally, the road sign matching results of multiple cameras include road sign matching point pairs of multiple cameras. The constructing the first vehicle pose optimization constraint according to the road sign matching results of multiple cameras and the IMU data includes:

[0029] Construct a matching point pair residual constraint according to the coordinate point positions of the road sign matching point pairs of multiple cameras;

[0030] Construct a relative pose residual constraint of each camera to the current moment according to the reference pose of the vehicle at the current moment, the current pose of each camera, and the relative pose of each camera to the current moment;

[0031] Construct a relative pose residual constraint between multiple cameras according to the current pose of each camera and the relative pose between any two cameras;

[0032] Construct a total residual constraint according to the matching point pair residual constraint, the relative pose residual constraint of each camera to the current moment, and the relative pose residual constraint between multiple cameras as the first vehicle pose optimization constraint.

[0033] Optionally, after solving the pose of the vehicle at the current moment by using an optimization algorithm according to the first vehicle pose optimization constraint, the vehicle positioning method further includes:

[0034] In the case of failure to solve according to the first vehicle pose optimization constraint, construct a second vehicle pose optimization constraint according to the previous moment pose of the vehicle, the IMU data, and the RTK positioning data;

[0035] Solve the pose of the vehicle at the current moment by using an optimization algorithm according to the second vehicle pose optimization constraint;

[0036] Cache the current pose of the vehicle and update the reference pose of the vehicle at the current moment.

[0037] In a second aspect, an embodiment of the present application further provides a vehicle positioning device, where the vehicle positioning device includes:

[0038] An acquisition unit, configured to acquire sensor data of the vehicle and high-precision map data corresponding to the vehicle at the current time, where the sensor data includes image data collected by multiple cameras and IMU data;

[0039] A semantic segmentation unit, configured to perform semantic segmentation on the image data collected by multiple cameras respectively to obtain road sign segmentation results of multiple cameras;

[0040] A matching unit, configured to perform road sign matching according to the road sign segmentation results of multiple cameras and the high-precision map data corresponding to the vehicle at the current time to obtain road sign matching results of multiple cameras;

[0041] A first construction unit, configured to construct a first vehicle pose optimization constraint according to the road sign matching results of multiple cameras and the IMU data;

[0042] An optimization unit, configured to solve the current pose of the vehicle at the current moment by using an optimization algorithm according to the first vehicle pose optimization constraint.

[0043] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0044] A processor; and a memory arranged to store computer-executable instructions, where the executable instructions, when executed, cause the processor to execute any one of the foregoing vehicle positioning methods.

[0045] In a fourth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instruction, where the computer program or instruction, when executed by a processor, implements any one of the foregoing vehicle positioning methods.

[0046] The above at least one technical solution adopted in the embodiments of the present application can achieve the following beneficial effects: In the vehicle positioning method of the embodiments of the present application, first, sensor data of the vehicle and high-precision map data corresponding to the vehicle at present are obtained, and the sensor data includes image data collected by multiple cameras and IMU data; then, semantic segmentation is respectively performed on the image data collected by multiple cameras to obtain road sign segmentation results of multiple cameras; then, road sign matching is performed according to the road sign segmentation results of multiple cameras and the high-precision map data corresponding to the vehicle at present to obtain road sign matching results of multiple cameras; then, a first vehicle pose optimization constraint is constructed according to the road sign matching results of multiple cameras and IMU data; finally, according to the first vehicle pose optimization constraint, an optimization solving algorithm is used to solve the pose of the vehicle at the current moment. The vehicle positioning method of the embodiments of the present application constructs a vehicle pose optimization constraint by using road sign information observed by multiple cameras on the vehicle, so as to solve the optimal vehicle pose, avoiding the problem of positioning failure caused by the failure of a single camera; using IMU data solves the problem of positioning continuity in scenarios such as the lack of road sign information, improving the positioning accuracy and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The illustrative embodiments and descriptions thereof of the present application are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0048] Figure 1 is a schematic flow chart of a vehicle positioning method in an embodiment of the present application;

[0049] Figure 2 is a schematic structural diagram of a vehicle positioning device in an embodiment of the present application;

[0050] Figure 3 is a schematic structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] In order to make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0052] The following will describe in detail the technical solutions provided by each embodiment of the present application with reference to the drawings.

[0053] The embodiments of the present application provide a vehicle positioning method, as Figure 1As shown in the figure, a flowchart of a vehicle positioning method in an embodiment of the present application is provided. The vehicle positioning method at least includes the following steps S110 to S150:

[0054] Step S110, obtain the sensor data of the vehicle and the high-precision map data corresponding to the vehicle at present. The sensor data includes image data collected by multiple cameras and IMU data.

[0055] When performing vehicle positioning, it is necessary to first obtain the sensor data of the vehicle and the local high-precision map data corresponding to the current position of the vehicle. The sensor data mainly includes image data collected by multiple cameras deployed on the vehicle and IMU (Inertial Measurement Unit) data. The IMU data may specifically include accelerometer data and gyroscope data, etc.

[0056] Multiple cameras in the embodiments of the present application can be distributed around the vehicle, such as in the front, rear, left, and right directions, so as to effectively alleviate the influence of light changes and improve the accuracy and stability of vehicle positioning.

[0057] Step S120, perform semantic segmentation on the image data collected by multiple cameras respectively to obtain the road sign segmentation results of multiple cameras.

[0058] For the image data collected by multiple cameras, deep learning models or traditional image processing algorithms can be used to accurately segment the road semantic information in the images, segment key elements such as lane lines, stop lines, traffic signs, and road surface markings in the images collected by each camera, and then use the Leida criterion to fit the segmented lane lines, stop lines, etc., so as to improve the accuracy and robustness of the fitting. Of course, specifically which method is used for road sign segmentation and fitting can be flexibly determined by those skilled in the art in combination with the prior art, and will not be elaborated here.

[0059] Step S130, perform road sign matching according to the road sign segmentation results of multiple cameras and the high-precision map data corresponding to the vehicle at present to obtain the road sign matching results of multiple cameras.

[0060] The high-precision map contains detailed information of the road environment, such as information of various road signs such as lane lines, stop lines, traffic sign positions, and road surface markings. This information is collected by professional equipment and processed precisely, and has high precision and reliability.

[0061] Match the road sign semantic information of each camera obtained in the above steps with the high-precision map data around the current position of the vehicle respectively, and the corresponding relationship between the road signs in each camera image and the road signs pre-stored in the high-precision map data can be obtained respectively. These corresponding relationships can be used as one of the bases for solving the vehicle position subsequently.

[0062] Step S140, construct a first vehicle pose optimization constraint based on the road sign matching results of multiple cameras and the IMU data.

[0063] After obtaining the road sign matching results of multiple cameras, it is necessary to construct an optimization constraint for the vehicle pose by combining the road sign matching results of multiple cameras and the IMU data. Here, the vehicle pose optimization constraint is mainly a residual constraint. For the construction of the vehicle pose optimization constraint, on the one hand, the matching point pair constraint between the observation information of multiple cameras and the high-precision map data is considered to avoid the problem that a single camera fails (such as being blocked or malfunctioning) and thus unable to obtain valid data; on the other hand, scenarios such as missing road signs are considered, and the IMU data is used to predict the vehicle pose to improve the continuity of vehicle positioning; on the third hand, the rigid connection characteristics between multiple cameras are considered, which can reduce the influence caused by the jitter of a single camera during driving.

[0064] Step S150, according to the first vehicle pose optimization constraint, use an optimization algorithm to solve the vehicle pose at the current moment.

[0065] After constructing the above vehicle pose optimization constraint, a certain optimization algorithm such as the least squares method can be used to optimize and solve the vehicle pose, so as to obtain the final pose of the vehicle at the current moment.

[0066] The vehicle positioning method of the embodiment of the present application constructs a vehicle pose optimization constraint by using the road sign information observed by multiple cameras on the vehicle, so as to solve the optimal vehicle pose, avoiding the problem of positioning failure caused by the failure of a single camera, and using the IMU data to solve the positioning continuity problem in scenarios such as missing road sign information, improving the positioning accuracy and stability.

[0067] In some embodiments of the present application, the road sign matching of the road sign segmentation results of multiple cameras and the high-precision map data corresponding to the vehicle currently to obtain the road sign matching results of multiple cameras includes: converting the high-precision map data corresponding to the vehicle currently into the camera images of each camera respectively; performing road sign matching on the road sign segmentation results of each camera and the high-precision map data in the camera images of each camera respectively to obtain the road sign matching results of multiple cameras.

[0068] Since the road sign segmentation results of each camera are in the camera image coordinate system of each camera, while the original high-precision map data is in the world coordinate system, when performing road sign matching on the road sign segmentation results of multiple cameras and the high-precision map data corresponding to the vehicle currently, the coordinate systems can be aligned first, that is, unified into the same coordinate system before matching.

[0069] Specifically, in the embodiments of the present application, the local high-precision map data can be first transformed into the camera images of each camera based on the transformation relationship between different coordinate systems to obtain the high-precision map data in the camera images of each camera, and then the high-precision map data in the camera images of each camera is respectively matched with the road sign segmentation results in each camera image to obtain the road sign matching results of each camera.

[0070] In some embodiments of the present application, the transformation of the high-precision map data corresponding to the vehicle into the camera images of each camera respectively includes: calculating the reference pose of the vehicle at the current moment; based on the reference pose of the vehicle at the current moment, transforming the high-precision map data corresponding to the vehicle into the vehicle body coordinate system to obtain the high-precision map data in the vehicle body coordinate system; calculating the relative poses of each camera to the current moment according to the IMU data; and transforming the high-precision map data in the vehicle body coordinate system into the camera images of each camera respectively according to the relative poses of each camera to the current moment and the camera parameters of each camera.

[0071] On the one hand, when transforming the high-precision map data into the camera images of each camera respectively, the reference pose of the vehicle at the current moment can be first calculated. For example, it can be calculated according to the historical reference pose and the change of the vehicle driving state, etc. The reference pose at the current moment can be understood as a reference value or an initial value of the vehicle pose at the current moment. The setting and calculation of the reference pose can improve the efficiency and accuracy of subsequent pose optimization and solution. Based on the reference pose of the vehicle at the current moment, the corresponding road sign data such as lane lines and stop lines can be obtained from the high-precision map data, and then these road sign data in the high-precision map are transformed into the vehicle body coordinate system to obtain the high-precision map data in the vehicle body coordinate system.

[0072] On the other hand, since the IMU data can be cached in real time, the relative poses of each camera to the current moment when the vehicle is located can be calculated respectively according to the cached IMU data between the previous moment and the current moment. This process can be understood as a process of spatio-temporal synchronization processing of the data collected by multiple cameras. Because different cameras may have different acquisition frequencies, before vehicle positioning based on the observation information of multiple cameras, the time synchronization of different cameras can be performed through the IMU data, without the need for time synchronization of the cameras at the hardware level, effectively improving the engineering efficiency.

[0073] In some embodiments of the present application, the IMU data is the IMU data between the previous moment and the current moment, and the calculation of the reference pose of the vehicle at the current moment includes: obtaining the pose of the vehicle at the previous moment; and calculating the reference pose of the vehicle at the current moment according to the pose of the vehicle at the previous moment and the IMU data between the previous moment and the current moment.

[0074] When calculating the reference pose of the vehicle at the current moment, the final pose solved by the vehicle at the previous moment can be obtained first, that is, the optimized pose at the previous moment. Then, based on the optimized pose at the previous moment and the IMU data cached between two adjacent moments, the reference pose of the vehicle at the current moment can be calculated. By using the historical pose as the prior knowledge for calculating the current reference pose, the accuracy of the reference pose calculation is improved, and further the efficiency and accuracy of the subsequent pose optimization are improved.

[0075] In some embodiments of the present application, calculating the relative pose of each camera to the current moment according to the IMU data includes: determining the IMU data corresponding to the timestamp of the image data collected by each camera according to the IMU data and the timestamp of the image data collected by each camera; determining the IMU data corresponding to the timestamp of the current moment according to the IMU data and the timestamp of the current moment; calculating the relative pose of each camera to the current moment respectively according to the IMU data corresponding to the timestamp of the image data collected by each camera and the IMU data corresponding to the timestamp of the current moment.

[0076] Different cameras may have different acquisition frequencies. When performing vehicle positioning based on the observation information of multiple cameras, spatio-temporal synchronization processing of multiple cameras can be performed first. Specifically, the IMU data at the corresponding moment can be found in the cached IMU data according to the timestamp of the image data collected by each camera, and the IMU data corresponding to the current moment can also be found in the cached IMU data. Then, according to the IMU data at the corresponding moment of each camera and the IMU data corresponding to the current moment, the pose transformation data between the two moments can be calculated respectively.

[0077] For example, assume there are three cameras c1, c2, and c3, and the times for collecting image data are t1, t2, and t3 respectively, and the current moment is T0. Then the IMU data at t1, t2, t3, and T0 can be obtained. Then, according to the IMU data at t1 and the IMU data at T0, the relative pose of camera c1 to the current moment can be calculated. According to the IMU data at t2 and the IMU data at T0, the relative pose of camera c2 to the current moment can be calculated. According to the IMU data at t3 and the IMU data at T0, the relative pose of camera c3 to the current moment can be calculated.

[0078] In some embodiments of the present application, the road sign matching results of multiple cameras include road sign matching point pairs of multiple cameras. The construction of the first vehicle pose optimization constraint based on the road sign matching results of multiple cameras and the IMU data includes: constructing a matching point pair residual constraint according to the coordinate point positions of the road sign matching point pairs of multiple cameras; constructing a relative pose residual constraint of each camera to the current moment according to the reference pose of the vehicle at the current moment, the current poses of each camera, and the relative pose of each camera to the current moment; constructing a relative pose residual constraint between multiple cameras according to the current poses of each camera and the relative pose between any two cameras; constructing a total residual constraint according to the matching point pair residual constraint, the relative pose residual constraint of each camera to the current moment, and the relative pose residual constraint between multiple cameras as the first vehicle pose optimization constraint.

[0079] The vehicle position optimization constraints constructed in the embodiments of the present application mainly include the following aspects:

[0080] 1) Matching point pair residual constraint Err i line : It is constructed based on the residual between the road sign sampling points on the high-precision map and the semantic feature sampling points of the road signs in the image, and can be specifically expressed in the following form:

[0081]

[0082] Among them, i represents the i-th camera, k represents the k-th sampling point, p represents the coordinate point in the normalized coordinate system of the camera image, represents the coordinate point of the k-th sampling point in the normalized coordinate system of the camera image, P represents the high-precision map sampling point in the vehicle body coordinate system, P k hadmap represents the coordinate point of the k-th sampling point in the vehicle body coordinate system, π represents the transformation relationship from the point in the world coordinate system to the normalized coordinate system of the image obtained according to the internal and external parameters of the camera, represents the relative pose from the current moment to the i-th camera calculated according to the IMU data.

[0083] 2) Relative pose residual constraint Err of each camera to the current moment i ref : It can be specifically expressed in the following form:

[0084]

[0085] Among them, represents the reference pose of the vehicle at the current moment, T i ref represents the relative pose of the i-th camera to the current moment calculated according to the IMU data, T iRepresents the pose of the i-th camera, and T i are all parameters to be solved.

[0086] 3) Relative pose residual constraints between multiple cameras Specifically, it can be expressed in the following form:

[0087]

[0088] Among them, T i represents the pose of the i-th camera, and T j represents the pose of the j-th camera, represents the relative pose between the i-th camera and the j-th camera.

[0089] Finally, based on the above residual constraints 1)-3), the total residual constraint Err is calculated: The total residual constraint Err is the sum of all residuals corresponding to all the above cameras, and specifically can be expressed in the following form:

[0090] Err = ∑(Err i line + Err i ref ) + ∑Err cam , (4)

[0091] In some embodiments of the present application, after solving the current pose of the vehicle using the optimization algorithm according to the first vehicle pose optimization constraint, the vehicle positioning method further includes: in the case of failure to solve according to the first vehicle pose optimization constraint, constructing a second vehicle pose optimization constraint based on the previous pose of the vehicle, IMU data, and RTK positioning data; solving the current pose of the vehicle using the optimization algorithm according to the second vehicle pose optimization constraint; caching the current pose of the vehicle, and updating the current reference pose of the vehicle.

[0092] During the process of optimization and solution, the optimization algorithm may not be able to obtain the optimal solution due to various reasons such as missing road information, that is, the solution fails. For this situation, the embodiments of the present application can further use RTK positioning data for vehicle positioning prediction, so as to ensure the positioning stability and continuity of the vehicle in special scenarios.

[0093] Specifically, the current pose of the vehicle can be calculated first according to the finally solved vehicle pose at the previous moment and the IMU data from the previous moment to the current moment as the vehicle pose calculated based on the IMU data, and then a vehicle pose residual constraint is constructed based on the vehicle pose calculated from the IMU data and the vehicle pose in the RTK positioning data, and the final vehicle pose is solved using a solution algorithm such as the least squares method.

[0094] Of course, it should be noted that the vehicle pose residual constraint constructed based on the vehicle pose calculated from IMU data and the vehicle pose in RTK positioning data can also be applied as part of the first vehicle pose optimization constraint, i.e., the total residual constraint, in the foregoing embodiments.

[0095] Finally, the vehicle pose at the current moment obtained by the embodiments of the present application can be cached in the vehicle pose queue, and the reference pose at the current moment is updated as the positioning basis for subsequent moments.

[0096] In summary, the key points and technical effects achieved by the vehicle positioning method of the present application mainly include:

[0097] 1) Using semantic segmentation technology to detect road sign information from the image data collected by multiple on-vehicle cameras (such as front, rear, left, and right), constructing a local semantic map, and using the Chauvenet's criterion to fit the lane lines and stop lines in the semantic map, which improves the accuracy and robustness of the fitting;

[0098] 2) Obtaining the lane lines and stop lines in the high-precision map data based on the reference pose, and then positioning the vehicle according to the constraints between the current local semantic map and the high-precision map. In a short time, if road information is missing, the vehicle pose can be predicted based on IMU data and RTK measurement data, which improves the continuity of vehicle positioning;

[0099] 3) Constructing residuals in the camera normalization coordinate system and performing pose optimization, which avoids the registration error caused by the change of the road surface pitch angle during vehicle driving;

[0100] 4) Using multiple cameras to collect data simultaneously, which avoids the inability to obtain effective data due to the failure of a single camera (such as being blocked or malfunctioning), and since the multiple cameras are rigidly connected, the influence caused by the jitter of a single camera during driving is reduced;

[0101] 5) Using IMU data for time synchronization between multiple cameras, which reduces the difficulty of camera time synchronization at the hardware level.

[0102] In the embodiments of the present application, a vehicle positioning device 200 is also provided, as Figure 2 shown in the structural schematic diagram of a vehicle positioning device in the embodiments of the present application. The vehicle positioning device 200 includes: an acquisition unit 210, a semantic segmentation unit 220, a matching unit 230, a first construction unit 240, and an optimization unit 250, where:

[0103] The acquisition unit 210 is configured to acquire the sensor data of the vehicle and the high-precision map data corresponding to the vehicle at the current moment, and the sensor data includes the image data collected by multiple cameras and IMU data;

[0104] A semantic segmentation unit 220, configured to perform semantic segmentation on the image data collected by multiple cameras respectively, to obtain the road sign segmentation results of multiple cameras;

[0105] A matching unit 230, configured to perform road sign matching based on the road sign segmentation results of multiple cameras and the high-precision map data currently corresponding to the vehicle, to obtain the road sign matching results of multiple cameras;

[0106] A first construction unit 240, configured to construct a first vehicle pose optimization constraint according to the road sign matching results of multiple cameras and the IMU data;

[0107] An optimization unit 250, configured to solve the pose of the vehicle at the current moment by using an optimization algorithm according to the first vehicle pose optimization constraint.

[0108] In some embodiments of the present application, the matching unit 230 is specifically configured to: convert the high-precision map data currently corresponding to the vehicle into the camera images of each camera respectively; perform road sign matching on the road sign segmentation results of each camera and the high-precision map data in the camera images of each camera respectively, to obtain the road sign matching results of multiple cameras.

[0109] In some embodiments of the present application, the matching unit 230 is specifically configured to: calculate the reference pose of the vehicle at the current moment; based on the reference pose of the vehicle at the current moment, transform the high-precision map data currently corresponding to the vehicle into the vehicle body coordinate system, to obtain the high-precision map data in the vehicle body coordinate system; calculate the relative poses of each camera to the current moment according to the IMU data; according to the relative poses of each camera to the current moment and the camera parameters of each camera, convert the high-precision map data in the vehicle body coordinate system into the camera images of each camera respectively.

[0110] In some embodiments of the present application, the IMU data is the IMU data between the previous moment and the current moment, and the matching unit 230 is specifically configured to: obtain the pose of the vehicle at the previous moment; calculate the reference pose of the vehicle at the current moment according to the pose of the vehicle at the previous moment and the IMU data between the previous moment and the current moment.

[0111] In some embodiments of the present application, the matching unit 230 is specifically configured to: determine the IMU data corresponding to the time stamp of the image data collected by each camera according to the IMU data and the time stamps of the image data collected by each camera; determine the IMU data corresponding to the time stamp of the current moment according to the IMU data and the time stamp of the current moment; calculate the relative poses of each camera to the current moment respectively according to the IMU data corresponding to the time stamp of the image data collected by each camera and the IMU data corresponding to the time stamp of the current moment.

[0112] In some embodiments of the present application, the road sign matching results of multiple cameras include road sign matching point pairs of multiple cameras. The first construction unit 240 is specifically configured to: construct a matching point pair residual constraint according to the coordinate point positions of the road sign matching point pairs of multiple cameras; construct a relative pose residual constraint of each camera to the current moment according to the reference pose of the vehicle at the current moment, the current poses of each camera, and the relative pose of each camera to the current moment; construct a relative pose residual constraint between multiple cameras according to the current poses of each camera and the relative pose between any two cameras; construct a total residual constraint according to the matching point pair residual constraint, the relative pose residual constraint of each camera to the current moment, and the relative pose residual constraint between multiple cameras, and use it as the first vehicle pose optimization constraint.

[0113] In some embodiments of the present application, the vehicle positioning device 200 further includes: a first construction unit, configured to construct a second vehicle pose optimization constraint according to the pose of the vehicle at the previous moment, IMU data, and RTK positioning data when the solution of the first vehicle pose optimization constraint fails; the optimization unit is further configured to solve the current moment pose of the vehicle by using an optimization algorithm according to the second vehicle pose optimization constraint; the update unit is configured to cache the current moment pose of the vehicle and update the reference pose of the vehicle at the current moment.

[0114] It can be understood that the above vehicle positioning device can implement each step of the vehicle positioning method provided in the foregoing embodiments. The relevant explanations of the vehicle positioning method are applicable to the vehicle positioning device and will not be elaborated here.

[0115] Figure 3 It is a schematic structural diagram of an electronic device according to an embodiment of the present application. Please refer to Figure 3 , at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory, such as at least one disk memory, etc. Of course, the electronic device may also include other hardware required for other services.

[0116] The processor, network interface, and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 only a bidirectional arrow is used in

[0117] Memory, used to store programs. Specifically, the program can include program code, and the program code includes computer operation instructions. The memory can include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0118] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a vehicle positioning device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0119] Obtain the sensor data of the vehicle and the high-precision map data corresponding to the vehicle at present, and the sensor data includes the image data collected by multiple cameras and IMU data;

[0120] Perform semantic segmentation on the image data collected by multiple cameras respectively to obtain the road sign segmentation results of multiple cameras;

[0121] Perform road sign matching according to the road sign segmentation results of multiple cameras and the high-precision map data corresponding to the vehicle at present to obtain the road sign matching results of multiple cameras;

[0122] Construct a first vehicle pose optimization constraint according to the road sign matching results of multiple cameras and the IMU data;

[0123] According to the first vehicle pose optimization constraint, use an optimization solution algorithm to solve the pose of the vehicle at the current moment.

[0124] The above is as in this application Figure 1The method executed by the vehicle positioning device disclosed in the illustrated embodiment can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The above processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0125] The embodiments of the present application also propose a computer program product. The computer program product stores one or more programs. The one or more programs include instructions that, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 1 the method executed by the vehicle positioning device in the illustrated embodiment, and specifically used to execute:

[0126] Obtain the sensor data of the vehicle and the high-precision map data corresponding to the vehicle currently. The sensor data includes image data collected by multiple cameras and IMU data;

[0127] Perform semantic segmentation on the image data collected by multiple cameras respectively to obtain the road sign segmentation results of multiple cameras;

[0128] Perform road sign matching based on the road sign segmentation results of multiple cameras and the high-precision map data corresponding to the vehicle currently to obtain the road sign matching results of multiple cameras;

[0129] Construct a first vehicle pose optimization constraint based on the road sign matching results of multiple cameras and the IMU data;

[0130] According to the first vehicle pose optimization constraint, use an optimization algorithm to solve the vehicle pose at the current moment.

[0131] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0133] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0135] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0136] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0137] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0138] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0140] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, various modifications and variations can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A vehicle positioning method, wherein: The vehicle positioning method comprises: Acquire the vehicle's sensor data and the high-precision map data currently corresponding to the vehicle, wherein the sensor data includes image data collected by multiple cameras and IMU data; Semantic segmentation is performed on the image data collected by multiple cameras respectively to obtain the road sign segmentation results of multiple cameras; Road sign matching is performed based on the road sign segmentation results of multiple cameras and the high-precision map data corresponding to the vehicle, to obtain road sign matching results of multiple cameras; Constructing a first vehicle posture optimization constraint based on the road sign matching results of the multiple cameras and the IMU data; According to the first vehicle posture optimization constraint, the current posture of the vehicle is solved using an optimization solution algorithm.

2. The vehicle positioning method according to claim 1, wherein: The road sign matching is performed according to the road sign segmentation results of the multiple cameras and the high-precision map data currently corresponding to the vehicle, and the road sign matching results of the multiple cameras are obtained, including: Convert the high-precision map data currently corresponding to the vehicle into the camera images of each camera respectively; The road sign segmentation results of each camera are matched with the high-precision map data in the camera image of each camera to obtain the road sign matching results of multiple cameras.

3. The vehicle positioning method according to claim 2, wherein: The step of converting the high-precision map data currently corresponding to the vehicle into camera images of each camera includes: Calculate the current reference pose of the vehicle; Based on the current reference posture of the vehicle, the high-precision map data corresponding to the vehicle is transformed into the vehicle body coordinate system to obtain the high-precision map data in the vehicle body coordinate system; Calculate the relative position of each camera at the current moment according to the IMU data; According to the relative position of each camera to the current moment and the camera parameters of each camera, the high-precision map data in the vehicle coordinate system is converted into the camera image of each camera respectively.

4. The vehicle positioning method according to claim 3, wherein: The IMU data is the IMU data between the previous moment and the current moment, and the calculation of the current moment reference posture of the vehicle includes: Get the vehicle's last moment pose; The reference posture of the vehicle at a current moment is calculated based on the posture of the vehicle at a previous moment and the IMU data between the previous moment and the current moment.

5. The vehicle positioning method according to claim 3, wherein: Calculating the relative position of each camera to the current moment according to the IMU data includes: Determine the IMU data corresponding to the timestamp of the image data collected by each camera according to the IMU data and the timestamp of the image data collected by each camera; Determine the IMU data corresponding to the timestamp of the current moment according to the IMU data and the timestamp of the current moment; According to the IMU data corresponding to the timestamp of the image data collected by each camera and the IMU data corresponding to the timestamp of the current moment, the relative posture of each camera to the current moment is calculated respectively.

6. The vehicle positioning method according to claim 1, wherein: The road sign matching results of the multiple cameras include road sign matching point pairs of the multiple cameras, and constructing the first vehicle posture optimization constraint according to the road sign matching results of the multiple cameras and the IMU data includes: Construct matching point pair residual constraints based on the coordinate point positions of the road sign matching point pairs of multiple cameras; According to the current reference pose of the vehicle, the current pose of each camera, and the relative pose of each camera to the current moment, the relative pose residual constraint of each camera to the current moment is constructed; According to the current pose of each camera and the relative pose between any two cameras, the relative pose residual constraint between multiple cameras is constructed; A total residual constraint is constructed according to the residual constraints of the matching point pairs, the relative pose residual constraints of the cameras to the current moment, and the relative pose residual constraints between multiple cameras as the first vehicle pose optimization constraint.

7. The vehicle positioning method according to claim 1, wherein: After solving the vehicle's current posture at a certain moment by using an optimization solution algorithm according to the first vehicle posture optimization constraint, the vehicle positioning method further includes: In the case that the solution of the first vehicle posture optimization constraint fails, constructing a second vehicle posture optimization constraint according to the vehicle's last posture, IMU data and RTK positioning data; According to the second vehicle posture optimization constraint, solving the vehicle's current posture using an optimization solution algorithm; The vehicle's current position is cached and the vehicle's current reference position is updated.

8. A vehicle positioning device, wherein: The vehicle positioning device comprises: An acquisition unit, used to acquire sensor data of the vehicle and high-precision map data currently corresponding to the vehicle, wherein the sensor data includes image data collected by multiple cameras and IMU data; A semantic segmentation unit, used to perform semantic segmentation on the image data collected by multiple cameras respectively, to obtain road sign segmentation results of multiple cameras; A matching unit, used to match road signs according to the road sign segmentation results of multiple cameras and the high-precision map data currently corresponding to the vehicle, to obtain road sign matching results of multiple cameras; A first construction unit, configured to construct a first vehicle posture optimization constraint according to the road sign matching results of the multiple cameras and the IMU data; The optimization unit is used to solve the vehicle's current posture at a certain moment by using an optimization solution algorithm according to the first vehicle posture optimization constraint.

9. An electronic device, comprising: processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the vehicle positioning method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the vehicle positioning method according to any one of claims 1 to 7 is implemented.

Images