Vehicle positioning

Abstract

The present disclosure relates to vehicle positioning, in particular to a method for determining a vehicle pose. The method comprises: predicting a pose of a vehicle on a map based on sensor data acquired by a vehicle positioning system; transforming a set of map road references of a segment of the digital map from a global coordinate system to an image frame coordinate system of a vehicle-mounted camera based on map data and the predicted pose of the vehicle, the transformed set of map road references forming a set of multi-poly lines in the image frame coordinate system; identifying a set of corresponding image road reference features in an image acquired by the vehicle-mounted camera, each identified road reference feature defining a set of measured coordinates in the image frame; projecting each of the identified set of image road reference features onto the formed set of multi-poly lines in order to obtain a set of projection points, each projection point defining a set of projected coordinates; determining an error parameter based on a difference between the measured coordinates and the corresponding projected coordinates, and updating the predicted pose based on the determined error parameter.

Description

TECHNICAL FIELD

[0001] The present invention relates to the field of vehicle positioning, and more specifically to image plane measurement updates for high definition map (HD map) based positioning. BACKGROUND

[0002] In the past few years, research and development activities related to autonomous vehicles have grown exponentially and many different approaches are being explored. Increasingly, modern vehicles have advanced driver assistance systems (ADAS) to improve vehicle safety and more generally road safety. ADAS are electronic systems that can assist a vehicle driver while driving, which for example can be represented by adaptive cruise control (ACC), collision avoidance systems, forward collision warning, etc. Today, research and development is being conducted within many technical fields associated with both the ADAS and the autonomous driving (AD) field. ADAS and AD will be referred to herein as the generic term of autonomous driving systems (ADS) corresponding to all different levels of automation defined for example by the SAE J3016 levels of driving automation (0-5), and specifically for levels 4 and 5.

[0003] In the near future, ADS solutions are expected to be applied to a large part of the new cars put on the market. ADS can be understood as a complex combination of various components, which can be defined as a system in which the perception, decision making and operation of a vehicle are performed by electronics and machinery rather than a human driver, and as the introduction of automation into road traffic. This includes the handling of the vehicle, the destination, and the knowledge of the surrounding environment. While the automation system can control the vehicle, it allows the human operator to leave all or at least some of the responsibilities to the system. ADS typically combine various sensors to perceive the vehicle’s surrounding environment, such as for example radars, lidars, sonars, cameras, navigation systems (e.g. GPS), odometers and / or inertial measurement units (IMUs), advanced control systems can interpret the sensory information based on these sensors to identify appropriate navigation paths as well as obstacles, free space areas and / or related signs.

[0004] An important requirement for autonomous and semi-autonomous vehicles is that they are able to estimate the pose of the vehicle accurately and consistently, as this is an important safety aspect when the vehicle moves in traffic.

[0005] Traditionally, satellite-based positioning systems (global navigation satellite systems (GNSS)), like for example the global positioning system (GPS), global navigation satellite system (GLONASS), Galileo, Beidou, have been used for positioning purposes. However, these and other regional systems are typically not precise enough to rely on them alone for determining the position of a moving vehicle in autonomous applications. Furthermore, GNSS-based solutions have even lower precision in determining height information.

[0006] Other solutions involve the combination of GNSS data with vehicle IMU (Inertial Measurement Unit) signals, which can often be affected by large scale and bias errors leading to positioning errors and orientation estimation errors of several meters. Additionally, these methods and systems do not work in scenarios of poor or no satellite connection, such as in tunnels or close to high buildings.

[0007] Alternatively, there are systems and methods that exploit HD map information together with many different sensors, such as cameras, lidar, radar and other sensors for determining vehicle driving parameters such as speed and angular rate, etc., to improve the reliability of the map position. However, even given the current vehicle pose (position and orientation), it is still difficult to predict a robust vehicle pose estimate by dead reckoning only due to the measurement noise from different measuring sensors, e.g. motion sensors.

[0008] To this end, a solution is proposed that employs a landmark-based localization method. Here, external sensors are used to detect stationary objects, referred to as landmarks, whose geographic position is also available in the HD map data. Then, the position of the vehicle is estimated by sequentially comparing the sensor data with the positions of these landmarks according to the map. Examples of landmarks that are typically available in the HD map and can be detected by most car-level sensors are, for example, traffic signs and traffic lights.

[0009] However, there is still a need in the art for new and improved solutions for precise vehicle localization in autonomous applications. In general, improvements in performance should preferably be made without significant impact on the size, power consumption and cost of the on-board system or platform. SUMMARY

[0010] It is therefore an object of the present invention to provide a method for determining a vehicle pose, a corresponding computer readable storage medium, a device for determining a vehicle pose and a vehicle comprising such a device, which alleviate all or at least one of the above-described drawbacks of currently known solutions.

[0011] This object is achieved by a method for determining a vehicle pose, a corresponding computer readable storage medium, a device for determining a vehicle pose and a vehicle comprising such a device as defined in the appended claims. The term exemplary is to be understood in this context as used as an example, illustration, or instance.

[0012] According to a first aspect of the present invention, a method for determining a vehicle pose is provided. The method comprises predicting a pose (x k , y k , θ k). The method further comprises transforming a set of map road references of a segment of the digital map from a global coordinate system to an image frame coordinate system of the vehicle-mounted camera based on the map data and the predicted pose of the vehicle. The transformed set of map road references forms a set of multi-poly lines in the image frame coordinate system. Further, the method comprises identifying a set of corresponding image road reference features in the image acquired by the vehicle-mounted camera, wherein each identified road reference feature defines a set of measured coordinates (x i , y i ) in the image frame. The method further comprises projecting each of the identified set of image road reference features onto the formed set of multi-poly lines in order to obtain a set of projection points, wherein each projection point defines a set of projection coordinates (x p i , y p i ). Further, the method comprises determining an error parameter based on a difference between the measured coordinates and the corresponding projection coordinates, and updating the predicted pose based on the determined error parameter.

[0013] In other words, the above presented method can be understood as a vehicle localization method that generally defines a “prediction” and a “measurement update” process for estimating a current pose of a vehicle. In short, the “prediction” is performed based on information that the vehicle itself “possesses” (e.g. speed, acceleration, etc.). For example, if the vehicle keeps moving with the current speed and acceleration for 1 second, where will it be? The “measurement update” is performed based on information about the environment and the surrounding environment that is collected by vehicle-mounted sensors. Here, the measurement update can be understood as a process for “correcting” errors introduced in the prediction by comparing the observed surrounding environment with a “known” digital map.

[0014] In particular, in the vehicle localization method presented herein, the “measurement update” is performed in the image frame coordinate system, which mitigates drawbacks related to error propagation due to depth errors in single depth estimation. Further, by generating multi-poly lines to represent the transformed map road references in the image frame coordinate system, the measured / identified image road references can be validated in an efficient and robust manner, thereby improving the accuracy of the overall vehicle localization method.

[0015] According to a second aspect of the present invention, a (non-transitory) computer- readable storage medium storing one or more programs configured to be executed by one or more processors of a vehicle localization module is provided, the one or more programs comprising instructions for performing a method according to any of the embodiments disclosed herein. For this aspect of the present invention, similar advantages and preferred features exist as for the first aspect of the present invention discussed previously.

[0016] The term "non-transitory" as used herein is intended to convey that the computer-readable storage medium (or "memory") excludes computer-readable media or memory that propagate electromagnetic signals, but is otherwise not intended to limit the types of physical computer-readable storage devices that the phrase computer-readable medium or memory encompasses. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium can further be transmitted by transmission media or signals such as electrical, electromagnetic, or digital signals, which can be conveyed via a communication medium such as a network and / or a wireless link. Thus, the term "non-transitory" as used herein is a limitation of the medium itself (i.e., tangible, as opposed to a signal), and not a limitation of the durability of the data storage (e.g., RAM versus ROM).

[0017] Further, according to a third aspect of the application, there is provided a device for determining a vehicle pose, wherein the device comprises a control circuit. The control circuit is configured to predict a pose (x k , y k , Θ k ) of the vehicle on a map based on sensor data acquired by a vehicle positioning system. Further, the control circuit is configured to transform a set of map road references of a segment of the digital map from a global coordinate system to an image frame coordinate system of an on-board camera based on map data and the predicted pose of the vehicle. The transformed set of map road references forms a set of multi-poly lines in the image frame coordinate system. The control circuit is further configured to identify a set of corresponding image road reference features in an image acquired by the on-board camera, wherein each identified image road reference feature defines a set of measured coordinates (x i , y i ) in the image frame. Moreover, the control circuit is configured to project each of the identified set of road reference features onto the formed set of multi-poly lines in order to obtain a set of projection points, wherein each projection point defines a set of projection coordinates (x p i , y p i ), determine an error parameter based on a difference between the measured coordinates and the corresponding projection coordinates, and update the predicted pose based on the determined error parameter. For this aspect of the application, there are similar advantages and preferred features as for the first aspect of the application discussed previously.

[0018] Still further, according to a fourth aspect of the present application, there is provided a vehicle comprising a positioning system for monitoring the position of the vehicle, a vehicle mounted camera for capturing images of the surroundings of the vehicle, and a device for determining a vehicle pose according to any of the embodiments disclosed herein. For this aspect of the present application, there are similar advantages and preferred features as for the first aspect of the present application discussed previously.

[0019] Further embodiments of the present application are defined in the dependent claims. It is emphasized that the term "comprising" when used in this specification is used to specify the presence of stated features, integers, steps or components but does not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0020] These and other characteristics, features and advantages of the present application will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate several embodiments of the application. BRIEF DESCRIPTION OF DRAWINGS

[0021] Other objects, features and advantages of embodiments of the present application will appear from the following detailed description, from the attached claims as well as from the drawings to which reference is made. In the drawings:

[0022] Figure 1 is a schematic illustration of a measurement update and prediction process for one-dimensional (ID) vehicle positioning.

[0023] Figure 2 is a schematic flowchart representation of a method for determining a vehicle pose according to embodiments of the present application.

[0024] Figure 3a is a schematic perspective view of a vehicle on a road section with lane tracks.

[0025] Figure 3b is a schematic illustration of lane tracks from Figure 3a in an image frame coordinate system.

[0026] Figure 4 is a schematic illustration of a set of map road references after a transformation forming two polylines and corresponding image road references projected onto the polylines for a verification process according to embodiments of the present application.

[0027] Figure 5a is a schematic perspective view of a vehicle on a segment of a digital map according to embodiments of the present application.

[0028] Figure 5b is a schematic illustration of a set of map road references after a transformation forming two polylines and corresponding image road references projected onto the polylines for a verification process according to embodiments of the present application.

[0029] Figure 5cis a schematic diagram of a set of map road references forming two multi- fold lines after a transformation and corresponding image road references projected onto extensions of the multi-fold lines for a validation process according to an embodiment of the application.

[0030] Figure 5d is a schematic diagram of a set of map road references forming one multi- fold line after a transformation and corresponding image road references projected onto the multi-fold line for a validation process according to an embodiment of the application.

[0031] Figure 6 is a schematic side view of a vehicle comprising a device for determining a vehicle pose according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0032] Those skilled in the art will appreciate that the steps, services and functions explained herein can be implemented using a single hardware circuit, using software run using a programmed microprocessor or on a general purpose computer, using one or more application specific integrated circuits (ASICs) and / or using one or more digital signal processors (DSPs). It will further be appreciated that when the present application is described as a method, it can also be implemented as one or more programs running on one or more processors and stored on one or more storage media for execution by the one or more processors.

[0033] In the following description of example embodiments, like reference numbers represent the same or similar parts throughout the several figures.

[0034] Figure 1 is a schematic diagram for facilitating understanding of a simplified one-dimensional (ID) localization process of the present disclosure. In more detail, Figure 1 for illustrating the "prediction" and "measurement update" steps of a vehicle localization method.

[0035] In Figure 1 , the vehicle 1 is moving through a corridor having three doors 21a-21c. The vehicle is provided with a localization module configured to provide an estimate of the position of the vehicle 1 in the corridor. As mentioned, this is a simplified example where it is assumed that the vehicle 1 is only able to travel in one direction along the corridor and that it only has one sensor configured to determine whether there is a door 21a-21c to the left of the vehicle 1.

[0036] At a first point in time (to), the vehicle 1 has not yet started moving and the estimate 22a of the position of the vehicle within the corridor is a uniform distribution as indicated in the first graph. In other words, the localization module is not able to accurately estimate the position of the vehicle 1 in the corridor at time to.

[0037] At a later point in time (ti), the localization module reaches the first door 21a and performs a measurement update. The sensors of the vehicle detect the presence of a door on the left side. The localization module has map data indicating the number of doors 21a-21c and indicating the position of the doors 21a-21c in the corridor and can thus determine with a certain probability that the vehicle 1 is close to one of the doors 21a-21c, although the localization module cannot determine which of the three doors 21a-21c is close. Thus, the localization module determines that the probability 23a-23c that the vehicle 1 is close to one of the doors 21a-21c in the corridor increases and the probability that the vehicle 1 is in any other position in the corridor decreases, as indicated in the second graph 22b from the top.

[0038] The vehicle 1 continues to move along the corridor at a certain speed for a certain amount of time, e.g. 2 seconds at 1 m / s, and the position of the vehicle 1 is predicted at a later time instant (t2). If the speed estimate and the time estimate are accurate, the vehicle 1 has moved 2 meters in this case, and the peaks 23a-23c in the localization probabilities from the previous time point (ti) should have moved 2 meters forward. However, due to the uncertainty of the speed estimate and the timer, the height of the peaks 23a-23c will decrease, as indicated by the peaks 24a-24c of the probability distribution 22c. 1m ) should have moved 2 meters forward. However, due to the uncertainty of the speed estimate and the timer, the height of the peaks 23a-23c will decrease, as indicated by the peaks 24a-24c of the probability distribution 22c.

[0039] Further, the localization module of the vehicle 1 reaches the second door at time t2 and performs another measurement update. Here, the sensors detect the presence of a door on the left side of the vehicle 1 and the localization module updates the estimate 22d of the current position of the vehicle. This can be done, e.g., by multiplying the probability distribution 22c from the prediction step with the probability distribution of the second measurement step, which will be similar to the distribution 22b, resulting in a position estimate 22d as presented in the lowest graph in Figure 1 More in detail, the probability distribution 22d of the position of the vehicle 1 in the corridor has several "bumps" 25a, 25c, 25d, 25e and a single "peak" 25b at a position corresponding to the right side of the second door 21b.

[0040] Thus, the localization module can conclude that it is very likely (e.g. above a certain threshold) that the vehicle 1 is close to the second door 21b and that it is more likely than any other possible position in the corridor. Thus, the localization module can output a signal at time t2 indicating that the vehicle 1 is close to the second door 21b.

[0041] Thus, the above described process is performed repeatedly at a specified frequency or in a "cyclic" manner, such that the positioning module continuously predicts the position of the vehicle 1 based on sensor information of the motion of the vehicle 1 and a vehicle motion model, and then performs a measurement update based on the vehicle's perception data and map data to correct the predicted position. In other words, a "prediction" is performed based on "internal information" of the vehicle 1 (e.g. speed, acceleration, etc.). For example, the prediction is based on reasoning such as: if the vehicle 1 keeps moving at the current speed and acceleration for 1 second, where will it be? On the other hand, the "measurement update" is performed based on perception, i.e. based on the environment and surroundings (e.g. the door on the left side of the vehicle is detected in the illustrated example) collected by the vehicle's perception system. Then, by comparing the observed surroundings with map data (e.g. HD map data), the measurement update is used to "correct" the errors introduced in the prediction. Figure 1

[0042] Figure 2 is a schematic illustration of a method 200 for determining a vehicle pose according to some embodiments. The pose can be understood as the position and orientation of the ego vehicle and is in some embodiments represented by a 2D Cartesian position and the vehicle's yaw (x, y, 0). However, in some embodiments, the pose is a 6D pose, wherein the position is defined by a 3D Cartesian position and the orientation is defined by the vehicle's roll, pitch and yaw. The method 200 can comprise an optional step of initializing 201 the vehicle pose based on global navigation satellite system (GNSS) data and map data. The initialization 201 can for example be performed by utilizing a suitable satellite-based positioning system such as for example a GNSS or a corresponding regional system such as for example a Global Positioning System (GPS), a Global Navigation Satellite System (GLONASS), Galileo, Beidou, etc.

[0043] Further, the method 200 comprises predicting 202 the pose (x k , y k , 0 k ) of the vehicle based on sensor data acquired by a vehicle positioning system. The vehicle positioning system can for example comprise an inertial measurement unit (IMU). An IMU can be understood as a device configured to detect linear acceleration using one or more accelerometers and to detect rotational rates using one or more gyroscopes. Thus, in some embodiments, the sensor data can be in the form of sensor data obtained from an IMU. The output from the IMU is then used to estimate the change in the pose of the vehicle over time. In more detail, the prediction 202 of the pose of the vehicle can be estimated based on a vehicle motion model together with motion sensor data (e.g. data from accelerometers and gyroscopes, which will be referred to collectively as motion sensors herein). Furthermore, the prediction 202 of the pose of the vehicle can be performed using a Bayesian filter. ​

[0044] Still further, the method 200 comprises transforming a set of map road references of (at least) a segment of the digital map from a global coordinate system to an image frame coordinate system of the vehicle mounted camera based on the map data and the predicted pose of the vehicle. In the present context, a map road reference will be understood as a longitudinally repeating road reference (such as e.g. lane markings, road edges, road obstacles etc.) whose position in the map is indicated in the map data. The set of map road references after the transformation 204 forms a set of polylines in the image frame coordinate system. It should be noted that "a set of" (e.g. a set of map road references, a set of polylines etc.) as used herein will be interpreted as "one or more". A polyline can be understood as a set of points with line segments drawn between consecutive points. In other words, a polyline is a connected sequence of line segments created as a single object.

[0045] However, in some embodiments, the step of transforming 204 the set of map road references comprises converting 205 the set of map road references into a ego frame coordinate system and then transforming 206 the converted set into the image frame coordinate system. More specifically, based on the map data and the predicted pose, the set of map road references of the segment of the digital map is converted 205 from the global coordinate system into the ego frame coordinate system of the vehicle. Thus, once the predicted pose of the vehicle is known, the map road references provided by the digital map can be effectively transformed 205 to the local coordinate system of the vehicle (i.e. the ego frame coordinate system) using the predicted pose. Then, based on a set of calibration parameters of the vehicle mounted camera, the converted 205 set of map road references is transformed 206 from the ego frame coordinate system to the image frame coordinate system. The calibration parameters can comprise extrinsic parameters (which can also be referred to as "camera pose") and / or intrinsic parameters (e.g. focal length, lens distortion etc.). In more detail, the extrinsic parameters define the position and orientation of the camera relative to the ego frame and the intrinsic parameters allow for a mapping between camera coordinates and pixel coordinates in the image frame. Thus, the extrinsic parameters are external to the camera and can change with respect to the world frame and the intrinsic parameters are camera parameters that are internal and fixed to a particular camera / digitization setup. The terms conversion and transformation are used interchangeably when referring to coordinate transformations and are mainly used to facilitate the understanding of the present disclosure.

[0046] Turning temporarily to Figure 3a and Figure 3b which are used to illustrate the terms "ego frame coordinate system" 31 and "image frame coordinate system" 32, respectively. Thus, Figure 3a is a perspective view of the vehicle 1 in which the lane markings provided by the HD map are depicted as dashed lines 34 and the lane markings provided by the perception system of the vehicle are depicted as dotted lines 33. Here, in Figure 3aIn the case of HD map data 34, the HD map data 34 is transformed into ego frames 31 together with perception outputs 33. The perception outputs can for example be in the form of lane tracking by means of camera images. The ego frame coordinate system can for example be a Cartesian coordinate system with the origin at the center of the rear axis of the vehicle 1 (or any other suitable origin known in the art). Similarly, in the case of image frames 32, the corresponding data is shown in image frames 32, where the HD map data 34 is transformed into image frames 32. Then, the image frame coordinate system 32 can be interpreted as a 2D coordinate system with the origin at the upper left corner (or any other suitable origin known in the art). In other words, the image frames 32 are coordinate systems that measure the pixel positions in the image plane (i.e. the plane on which the image is formed). Figure 3b

[0047] Returning to Figure 2 , the method 200 further comprises identifying 207 corresponding image road references in the images acquired by the on-board camera, wherein each identified road reference defines a set of measured coordinates (x i , y i ) in the image frame. Thus, if right lane markings are used as map road references, right lane markings are identified 207 in the images acquired by the on-board camera. Similarly, if left road obstacles are used as map road references, left road obstacles are identified 207 in the images acquired by the on-board camera, and so on. Further, each of the set of identified image road references is projected 208 onto the set of multi-polygons formed in order to obtain a set of projection points. Each projection point defines a set of projection coordinates (x p i , y p i .

[0048] Turning temporarily to Figure 4 , which illustrates identified road references 42, a set of transformed map road references 43 forming multi-polygons 41, and projection points 44 of the identified road references onto the multi-polygons 41 in the image frame 32. In connection with the above example, the map road references 43 can be right lane markings obtained from a digital map (e.g. a HD map) that has been transformed to the image frame 32. Thus, in order to obtain the projection points 44, the identified road references (i.e. the right lane markings identified in the images acquired by the on-board camera) are orthogonally projected onto the multi-polygons 41. In other words, in some embodiments, the step of projecting 208 the set of identified road references comprises, for each identified image road reference, defining each multi-polygon 41 a nearest index of the image road reference 42 as a projection point 44 of the image road reference 42.

[0049] ​In other words, the white dots 42 are image road reference feature pixels (e.g. lane marking feature pixels) having a set of measured coordinates (x i , y i ) extracted by a perception module of the vehicle from raw video image frames of the on-board camera. Thus, assuming a predicted pose of the vehicle at the time instant when the image road reference feature pixels are extracted, map road reference features are extracted from the digital map based on the predicted vehicle pose, as indicated by the black dots 43 in Figure 4 . In more detail, the map road reference features (e.g. lane markings) are presented as indices of the multi-polyline 41.

[0050] Going back to Figure 2 , the method 200 further comprises determining 210 an error parameter based on a difference between the measured coordinates (x i , y i ) and the corresponding projected coordinates (x p i , y p i ), and updating 211 the predicted 202 pose based on the determined 210 error parameter. In more detail, the error parameter can be derived 210 from:

[0051]

[0052] and then used to perform the measurement update 211. Thus, the Euclidean distance between each set of measured coordinates and the corresponding projected coordinates is given by . More specifically, the update 211 of the predicted 202 pose can be done, for example, by a linear or non-linear filtering based on the predicted 202 pose and the determined 210 error parameter. The filter in the present context can be, for example, a Bayesian filter such as, for example, a Kalman filter, an Extended Kalman filter, a Cubature Kalman filter (CKF) or a Particle filter.

[0053] Going back again to Figure 4 , the transformed map road reference features 43 to the image frame can be understood as a hypothesis of what the on-board camera should have captured at the time instant when the vehicle is estimated to be in the predicted pose. However, the difference between the hypothesis (i.e. the black dots) and the actual measurements (i.e. the white dots) is the basis to perform the measurement update. More specifically, in order to determine the error parameter, an orthogonal projection from the image road reference features 42 to the multi-polyline 41 is performed to obtain a set of projected coordinates (x p i , y p i) of the projected point 44. Given the extracted / identified road reference features 42 and the predicted vehicle pose, the projected point 44 corresponds to the image frame road reference feature that the “expected” vehicle mounted camera “sees”. Thus, given the difference between the measured 42 and the expected 44, an error value can be derived and a “measurement update” can be performed.

[0054] In short, the positioning method presented herein generally comprises two main steps, namely, “prediction” and “measurement update”. To perform the latter step, a “measurement model” is defined, which models “what should a sensor mounted on the vehicle measure if the vehicle is hypothetically at a certain location?”. This is often referred to as “predicted measurement”.

[0055] As an optional step, the method 200 can further comprise selecting 203 a segment of the digital map based on the predicted pose of the vehicle and a set of characteristics of the vehicle mounted camera. More specifically, since the measurement update is performed by checking the local environment around the vehicle against the HD map, it is not necessary to transform the entire HD map into the image frame, but only an appropriately selected segment of the HD map is needed. Thus, the selection of the segment can be based on the set of characteristics of the camera indicative of the sensing range or field of view of the camera. In other words, in some embodiments, essentially only the map region that can be detected by the vehicle mounted camera region transformed at a given moment. However, in some embodiments, the selection 203 of the segment of the digital map can be based on other criteria (e.g. any criteria within a predetermined range of the ego vehicle frame).

[0056] Figure 5a is a schematic perspective view illustration of a vehicle 1 driving on a road portion of a digital map having a set of road references in the form of lane marks 51. Furthermore, Figure 5a indicates how the selection of the segment 50 of the digital map can be controlled. As mentioned, the selection of the segment 50 can be based on a predetermined range 52 in the ego vehicle frame, where the three axes indicate the origin of the ego vehicle frame at the center of the rear axis of the vehicle 1. In more detail, the selection of the segment 50 of the digital map, whose road references 51 are to be transformed to the image frame coordinate system, can be performed such that only a set of map road references within a predetermined range on the x-axis of the ego vehicle frame. In other words, map road references within a first predetermined distance 52a “in front” of the origin of the ego vehicle frame and a second predetermined distance 52b “behind” the origin of the ego vehicle frame.

[0057] Back to Figure 2And according to some embodiments, the method 200 can further comprise validating 209 the set of image road references identified 207. The validation 209 provides an advantage in terms of reducing errors caused by erroneous computation 210 of the error parameters in the measurement update 211 step. In more detail, since only segments of the digital map are used for the transformation 204 of the map road references, there is a risk of having image road reference features (i.e. road reference features captured by the on-board camera) falling outside the "range" of the formed polyline(s). Moreover, the use of a polyline is more robust and simple compared to a single transformed map road reference with a corresponding image road reference, and can provide further advantages in terms of more accurate error parameter computation.

[0058] Moreover, the method 200 can comprise a step of looping or repeating the above described steps in a loop fashion. In other words, the method 200 loops at a certain frequency (e.g. 10-100 Hz) such that the pose prediction and the subsequent measurement update steps are performed repeatedly as the vehicle moves. Moreover, the vehicle pose does not have to be a three-dimensional (3D) pose (lateral position, longitudinal position, yaw), but can be a six-dimensional (6D) pose (lateral position, longitudinal position, vertical position, roll, pitch, yaw) in some embodiments.

[0059] Moreover, the same procedure 200 can be extended to include other vehicle states than the pose (x k , y k , Θ k ). More specifically, in some embodiments, the prediction 202 further comprises a prediction of the longitudinal velocity of the vehicle, the lateral velocity of the vehicle, the vertical velocity of the vehicle, and the angular velocity (around the 3D axis) of the vehicle. Thus, the subsequent steps follow based on the predicted vehicle states, and the predicted states are updated in a similar manner as previously described.

[0060] With reference to Figure 5b Further illustrating the validation 209 procedure, Figure 5b two image road reference features 42a, 42b in the image frame coordinate system, and two polylines 41a, 41b formed by a set of transformed map road references 43a, 43b.

[0061] The vehicle localization method 200 disclosed herein is based on linear or nonlinear filtering techniques, and according to some embodiments, is based on Bayesian filtering. Furthermore, in some embodiments, the vehicle localization method 200 is based on a capacitive Kalman filter (CKF), a variant of the Kalman filter. Therefore, to handle the nonlinearity of the prediction and measurement update modeling, the CKF perturbs the predicted vehicle attitude, and the perturbed attitude is propagated using the prediction and measurement model, after which the average value of the propagated perturbation is derived. This average value is then used to determine the predicted attitude of the vehicle, and a transformed set of map road references 43a, 43b and associated polylines 41a, 41b are generated.

[0062] For the validation procedure 209, a validity check is performed on all volume points (i.e., based on the vehicle attitudes of all disturbances) because the transformed set of map road references 43a, 43b varies according to the volume points used (i.e., the vehicle attitudes of the disturbances). Therefore, in Figure 5b In the illustrated example, the transformed set of map road references for the digital map segment results in two polylines 41a and 41b, where the indices 43a and 43b of the two polylines are transformed using different volume points. For example... Figure 5b As described, for some image road reference features, some projections may be valid for some volume points, while they may be invalid for others. More specifically, the first image road reference feature 42a has two valid projection points 44a and 44b, while the second image road reference 42b has one invalid projection point 44c and one valid projection point 44d. Since the second image road reference feature 42b has at least one invalid projection point 44c, this image road reference feature and its associated projection point are discarded (or simply not considered) and not used for error parameter calculation.

[0063] In other words, for each image road reference feature, if one of the associated projection points is a non-orthogonal projection point, the verification of the identified set of image road reference features 42a, 42b can be performed by discarding the image road reference feature and the associated projection point. Therefore, the error parameter is determined solely based on the difference between the measured coordinates and the corresponding projected coordinates of the verified image road reference feature. Furthermore, since invalid image road references (i.e., extracted / measured road references from camera output) can be discarded in a simple and efficient manner, forming multi-segment lines based on the transformed map road reference provides a technical benefit to the verification procedure, resulting in a more accurate vehicle positioning estimate.

[0064] Furthermore, Figure 5cThe diagram illustrates an alternative verification process for the identified set of image road reference features 42'. As previously described, a validity check is performed on all volume points (i.e., based on all perturbations of vehicle pose) because the transformed set of map road references 43a', 43b' varies according to the volume points used (i.e., perturbations of vehicle pose). Therefore, in Figure 5c In the illustrated example, the transformation of a digital map segment into a set of map road references results in the formation of two polylines 41a' and 41b', where the indices 43a' and 43b' of the two polylines are transformed using different volume points.

[0065] Here, the image road reference feature 42' does not have projection points that can be orthogonally projected onto the polylines 41a', 41b'. Instead, the two projection points 44a', 44b' are generated by extending the polylines 41a', 41b' as indicated by the dotted lines. The extensions of the polylines 41a', 41b' can be considered as part of the polylines. To check the validity of the projection points 44a', 44b', the distance (d) between the projection points 44a', 44b' and the endpoints (i.e., end indices) 45a', 45b' of the polylines is compared with a threshold (T). If the distance (d) is less than the threshold (T), the image road reference feature 42' is considered valid; otherwise, it is considered invalid.

[0066] In addition, Figure 5c In the illustrated embodiment, the extension of the polyline is formed at the "bottom" portion of the polyline (i.e., towards the viewpoint (vehicle)). In some cases, the extension of the polylines 41a', 41b' may be made towards the horizontal line at the upper end. In this case, care must be taken to ensure that the polylines 41a', 41b' do not extend beyond the "horizontal line". Therefore, in some embodiments, the verification process includes checking whether any image road reference features 42' are arranged above the "horizontal line" in the image, and if so, discarding those image road reference features. Conventional computer vision techniques can be employed / utilized to identify the horizontal line. Furthermore, the threshold (T) can be a dynamic threshold. For example, a smaller threshold (T) can be used if the projection point is within a certain (predetermined) defined distance from the horizontal line compared to if the projection point is beyond the (predetermined) defined distance from the horizontal line. This is at least in part because measurements closer to the "horizontal line" are measurements of features farther from the camera, and are therefore associated with higher uncertainty. Therefore, in order to improve the accuracy of the method disclosed in this paper, a dynamic thresholding standard can be adopted by referring to the "distance to the horizontal line" in the image frame.

[0067] Furthermore, in Figure 5d The diagram illustrates an alternative verification process for a set of image road reference features 42” identified.

[0068] Even in Figure 5d Only one polyline 41" is illustrated in FIG. 22, but one of skill in the art will readily appreciate that the same verification process can be performed across multiple polylines as previously exemplified. Here, three image road reference 42a", 42b", 42c" are identified and projected onto the polyline 41". The first image road reference feature 42a" has an orthogonal projection point 44a" onto the polyline and is considered a valid image road reference. The second image road reference feature 42b" and the third image road reference feature 42c" do not have an orthogonal projection point onto the polyline 41". However, for such image road reference features, in some embodiments, the validity of the image road reference features is checked by comparing the distance (d) between the image road reference feature 42b", 42c" and the nearest end point (i.e., end index) 45" of the polyline 41" to a threshold distance (L). If the distance (d) is less than the threshold (L), the image road reference feature is considered valid, otherwise invalid.

[0069] Additionally, in some embodiments, there is another criteria for an image road reference to be considered valid. For example, even if the distance (d) between the third image road reference 42c" would be below the threshold (L), it would still be considered invalid because the nearest end point 45" of the polyline already has an associated "projection point" (from the second image road reference feature 42b"). In other words, if an image road reference feature 44a", 44b", 44c" does not have any orthogonal projection point onto the polyline, the image road reference feature can be projected onto the nearest end point of the polyline 41" (unless it is already "occupied" by another projection point) and then the image road reference feature is verified by comparing the distance (d) to the threshold (L). Similar to as described above, the threshold (L) can be dynamically set based on the distance between the relevant image road reference feature and the "horizontal line" in the image. Thus, if the image road reference feature is "close" to the horizontal line, a smaller threshold (L) is employed in order to validate / verify that the image road reference feature is further from the horizontal line than some other image road reference feature.

[0070] The inventors have realized that the current known vehicle localization module utilizing the “prediction and measurement update” approach suffers from technical drawbacks, since the measurement update is performed in the ego frame coordinate system. In more detail, in order to perform the measurement update in the ego frame coordinate system, the 2D road reference features acquired by the on-board camera from the image frame coordinate system need to be projected into the ego frame coordinate system by utilizing a single depth estimation, e.g. by utilizing a convolutional neural network (CNN). However, depth errors in the single depth estimation will then be propagated into the ego frame measurement update. Additionally, if multiple on-board cameras (e.g. both a front-facing camera and a rear-facing camera) are employed, multiple single depth errors are propagated into the ego frame measurement update.

[0071] Thus, by means of the vehicle localization method / module presented herein, the “camera measurements” are maintained in the image frame coordinate system and the need for a single depth estimator is alleviated, resulting in a more accurate vehicle localization. Furthermore, forming the multi-polyline from the transformed map road references provides further advantages in terms of robustness and accuracy.

[0072] Executable instructions for performing these functions are optionally comprised in a non-transitory computer readable storage medium or other computer program product configured to be executed by one or more processors.

[0073] Figure 6 is a schematic side view of a vehicle 1 comprising a device 10 (which can also be referred to as a vehicle localization module) for determining a vehicle pose. The vehicle 1 further comprises a perception system 6 and a localization system 5. The perception system 6 will in the present context be understood as the system responsible for acquiring raw sensor data from on-board sensors 6a, 6b, 6c, such as cameras, lidar and radar, ultrasonic sensors, and converting this raw data into scene understanding. In particular, the vehicle 1 has at least one on-board camera 6c for capturing images of at least a part of the vehicle’s surroundings. The localization system 5 is configured to monitor the geographical position and heading of the vehicle and can be in the form of a global navigation satellite system (GNSS), such as GPS. However, the localization system can alternatively be implemented as a real-time kinematic (RTK) GPS in order to improve accuracy. Furthermore, in the present context, it is assumed that the vehicle 1 has access to a digital map (e.g. an HD map), either in the form of a locally stored digital map or via a remote data repository (e.g. as a data stream) accessible via an external communication network 2. In some embodiments, access to the digital map can be provided, e.g. by the localization system 5.

[0074] Further, the vehicle 1 can be connected to one or more external networks 20 via e.g. a wireless link (e.g. for obtaining map data). The same or some other wireless link can be used for communication with other vehicles 2 in the vicinity of the vehicle or with local infrastructure elements. Cellular communication technology can be used for long distance communication such as to external networks, and if the cellular communication technology used has low latency, it can also be used for communication between vehicles, vehicle-to-vehicle (V2V) and / or vehicle-to-infrastructure (V2X). Examples of cellular radio technology are GSM, GPRS, EDGE, LTE, 5G and 5G NR, etc., also including future cellular solutions. In some solutions, however, medium to short distance communication technology is used, such as a wireless local area network (LAN), e.g. an IEEE 802.11 based solution. ETSI is working on a cellular standard for vehicle communication, and e.g. 5G is considered a suitable solution due to low latency as well as high bandwidth and efficient handling of communication channels.

[0075] The device or positioning module 10 comprises one or more processors 11, a memory 12, a sensor interface 13 and a communication interface 14. The processor(s) 11 can also be referred to as control loop 11, control unit 11, controller 11 or control circuit 11. The device 10 preferably comprises a plurality of software / hardware modules as described earlier, here summarized as “control circuit” 11. The control circuit 11 is configured to execute instructions stored in the memory 12 to perform a method for determining a vehicle pose according to any of the embodiments disclosed herein. In other words, the memory 12 of the control system 10 can comprise one or more (non-transitory) computer-readable storage media for storing computer-executable instructions which, when executed by the computer processor(s) 11, for example, can cause the computer processor(s) 11 to perform the techniques described herein. The memory 12 optionally includes high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices; and optionally includes non-volatile, computer-readable storage medium such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices.

[0076] In more detail, the control circuit 11 is configured to predict the pose (x k , y k , θ k). Further, the control circuit 11 is configured to transform the set of map road references of the segment of the digital map from the global coordinate system to the image frame coordinate system of the vehicle mounted camera based on the map data and the predicted pose of the vehicle. The transformed set of map road references forms a set of multi-poly lines in the image frame coordinate system. The control circuit is further configured to identify a set of corresponding image road reference features in the image acquired by the vehicle mounted camera, wherein each identified image road reference feature defines a set of measured coordinates (x i , y i ) in the image frame. Furthermore, the control circuit 11 is configured to project each of the identified set of road reference features onto the formed set of multi-poly lines in order to obtain a set of projection points, wherein each projection point defines a set of projection coordinates (x p i , y p i ), determine an error parameter based on a difference between the measured coordinates and the corresponding projection coordinates, and update the predicted pose based on the determined error parameter.

[0077] The application has been presented above with reference to specific embodiments. However, other embodiments than the above described are possible and within the scope of the application. Within the scope of the application there can be, for example, different steps in the methods presented above, and the method steps presented above can be executed in a different order. The application thus encompasses also any such modifications and alterations of the application. According to an example embodiment, there is thus provided a non-transitory computer readable storage medium storing one or more programs configured to be executed by one or more processors of a vehicle control system, the one or more programs comprising instructions for performing a method according to any of the embodiments discussed above. Alternatively, according to another example embodiment, a cloud computing system can be configured to perform any of the methods presented herein. The cloud computing system can comprise distributed cloud computing resources jointly performing the methods presented herein under control of one or more computer program products.

[0078] Generally, a computer-accessible medium can include any tangible or non-transitory storage media or memory media such as electronic, magnetic, or optical media (e.g., a disk or CD / DVD-ROM that is coupled to a computer system via a bus). As used herein, the terms "tangible" and "non-transitory" are intended to describe a computer-readable storage medium (or memory) that does not include a propagated signal or carrier wave that is ephemeral in nature. For example, the terms "non-transitory computer-readable medium" or "tangible memory" are intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored on tangible computer-accessible storage media can further be transmitted by transmission media or signals such as electrical, electromagnetic, or digital signals, which can be conveyed electronically, optically, acoustically, or by any other form of traversing a communication signal.

[0079] The processor(s) 11 (associated with the control device 10) can be or include any number of hardware components for conducting data or signal processing or for executing computer code stored in the memory 12. The device 10 has an associated memory 12, and the memory 12 can be one or more devices for storing data and / or computer code for completing or facilitating the various methods described in the specification. The memory can include volatile memory or non-volatile memory. The memory 12 can include database components, object code components, script components, or any other type of information structure for supporting the various activities described in the specification. According to an exemplary embodiment, any distributed or local memory device can be utilized with the systems and methods of the specification. According to an exemplary embodiment, the memory 12 is communicably connected to the processor 11 (e.g., via a circuit or any other wired, wireless, or network connection) and includes computer code for executing one or more processes described herein.

[0080] It should be understood that the sensor interface 14 can also provide the possibility to acquire sensor data directly or via a dedicated sensor control circuit 4 in the vehicle. The communication / antenna interface 14 can further provide the possibility to send output to a remote location, e.g. a remote operator or control centre, by means of an antenna 5. Furthermore, some sensors in the vehicle can communicate with the control device 10 using local network setups such as CAN bus, I2C, Ethernet and optical fiber etc. The communication interface 14 can be arranged to communicate with other control functions of the vehicle and can thus also be seen as a control interface; however, a separate control interface (not shown) can be provided. The local communication within the vehicle can also be of a wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth or similar medium / short range technology.

[0081] It should thus be understood that parts of the described solution can be implemented in the vehicle, in a system located outside the vehicle or in a combination of inside and outside the vehicle; e.g. in a server in communication with the vehicle, a so-called cloud solution. For example, sensor data can be sent to an external system and this system performs the step of comparing the sensor data (motion of another vehicle) with a predetermined behaviour model. Different features and steps of the embodiments can be combined in other combinations than described.

[0082] It should be noted that the word 'comprising' does not exclude the presence of elements or steps other than those listed and the word 'a' or 'an' preceding the

[0083] Although the drawings can show a specific order of method steps, the order of the steps can differ from what is depicted. Also, two or more steps can be performed concurrently or with partial concurrence. Such variation will depend on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the application. Likewise, software implementations could be accomplished with standard programming techniques with rule based logic and other logic to accomplish the various connection steps, processing steps, comparison steps and decision steps. The above-described embodiments are merely possible examples of the application, obtained from the full scope of the application described. Other solutions, uses, objects and functions within the scope of the application as described in the application as claimed below will be apparent to those skilled in the art upon consideration of the description.

Claims

1. A method for determining vehicle attitude, the method comprising: Based on sensor data obtained from the vehicle positioning system, the attitude of the vehicle on the map is predicted (202); Based on map data and the predicted attitude of the vehicle, a set of map road references for a segment of the digital map is transformed (204) from the global coordinate system to the image frame coordinate system of the vehicle camera, wherein the transformed set of map road references forms a set of polylines in the image frame coordinate system. Identify (207) a set of corresponding image road reference features in the image acquired by the vehicle-mounted camera, and each identified road reference feature defines a set of measurement coordinates in the image frame; Project (208) each of the identified road reference features in the set of images onto the formed set of polylines to obtain a set of projection points, where each projection point defines a set of projection coordinates; Based on the difference between the measured coordinates and the corresponding projected coordinates, the error parameter (210) is determined; and Based on the determined error parameters, update the attitude predicted by (211).

2. The method according to claim 1, wherein, The steps to change the road reference of this set of maps include: Based on map data and the predicted attitude, the road references for that segment of the digital map are transformed (205) from the global coordinate system to the vehicle's own frame coordinate system; and Based on a set of calibration parameters of the vehicle-mounted camera, the transformed map road reference of the section is transformed (206) from the self-frame coordinate system to the image frame coordinate system.

3. The method according to claim 2, wherein, The calibration parameters include a set of external camera parameters and a set of internal camera parameters.

4. The method according to any one of claims 1 to 3, wherein, The steps for predicting the attitude of the vehicle (202) include: Based on sensor data acquired by the vehicle positioning system and a predefined vehicle motion model, the vehicle's attitude on the map is predicted.

5. The method according to claim 4, wherein, The steps for identifying the road reference features of this set of images by projection (208) include: For each identified image road reference feature, the nearest index of each polyline relative to the image road reference feature is defined as the projection point for the image road reference feature.

6. The method of claim 5, further comprising: The road reference features identified in (209) of the image set were verified in the following manner: For each image road reference feature, if one of the projection points is a non-orthogonal projection point, then the image road reference feature and the corresponding projection point are discarded. The determination of the error parameter is based solely on the difference between the measured coordinates and the corresponding projected coordinates of the verified road reference features.

7. The method according to claim 1, wherein, The step of predicting (202) the attitude of the vehicle includes: The attitude of the vehicle is predicted using a Bayesian filter.

8. The method according to claim 1, wherein, The step of predicting (202) the attitude of the vehicle includes: The current vehicle pose is estimated by the disturbance and the disturbed vehicle pose is propagated; and The transformation of the road reference in this set of maps is based on the perturbed vehicle attitude.

9. The method according to claim 8, wherein, The disturbance estimates the current vehicle attitude and propagates the disturbed vehicle attitude based on a prediction and measurement model using a capacitive Kalman filter, wherein the disturbed vehicle attitude corresponds to a capacitive point.

10. The method of claim 1, further comprising: Based on the predicted attitude of the vehicle and a set of characteristics of the onboard camera, the segment of the digital map is selected (203).

11. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors (11) of a vehicle positioning module, the one or more programs including instructions for performing the method according to any one of the preceding claims.

12. A device (10) for determining the attitude of a vehicle (1), the device (10) comprising a control circuit (11) configured to: Based on sensor data acquired by the vehicle positioning system (5), the attitude of the vehicle (1) on the map is predicted; Based on map data and the predicted attitude of the vehicle (1), a set of map road references for a segment of the digital map is transformed from the global coordinate system to the image frame coordinate system of the vehicle-mounted camera, wherein, The transformed map road references form a set of polylines in the image frame coordinate system; Identify a set of corresponding image road reference features in the images acquired by the vehicle-mounted camera, and define a set of measurement coordinates in the image frame for each identified image road reference feature; Each of the identified road reference features is projected onto the resulting set of polylines to obtain a set of projection points, where each projection point defines a set of projection coordinates. The error parameters are determined based on the difference between the measured coordinates and the corresponding projected coordinates; and The predicted attitude is updated based on the determined error parameters.

13. A vehicle (1), comprising: A positioning system (5) is used to monitor the location of the vehicle (1); An onboard camera is used to capture images of the surrounding environment of the vehicle (1); as well as The device (10) for determining vehicle attitude according to claim 12.

Images