Method and system for vehicle positioning from camera images

Abstract

A method for determining a vehicle position is provided, the method comprising obtaining a 2D image with a vehicle-based camera, the image showing a vehicle's surroundings; classifying pixels in the image such that each classified pixel belongs to one of a predetermined set of classes, thereby forming a classified 2D image. The classification is performed by a learning algorithm. The method further comprises determining an initial estimated vehicle position and defining possible vehicle positions based on the initial position. Then, the method comprises matching the classified 2D image to a three-dimensional map comprising a plurality of geometric objects, each geometric object belonging to one of said predetermined set of classes, by comparing the classification of geometric objects for the possible vehicle positions in the 3D map with the classification of at least one corresponding pixel in the classified 2D image; determining a matching score for at least one possible vehicle position based on the matching pixels of the 2D image; and determining the vehicle position based on the scores of the possible vehicle positions.

Description

Technical Field

[0001] This invention relates to a method and system for determining vehicle location from camera images based on semantic segmentation. Background Technology

[0002] With the introduction of driver assistance systems and fully or partially self-driving cars, accurately determining the location of vehicles has become increasingly important.

[0003] For example, it is proposed to use multiple redundant technologies to solve the localization of autonomous vehicles, among which the combination of local sensing (such as radar, laser detection and measurement system, and camera) and identifiable landmark map is a major component.

[0004] However, one problem with using camera images for localization is that it creates a map that remains unchanged despite visual changes in the environment surrounding the vehicle. Changes in lighting during the day, changing weather, seasonal changes, etc., can cause significant variations in appearance under the same conditions, making it difficult to extract features from images that remain constant.

[0005] Current methods can extract image features (e.g., SIFT, SURF, ORB), build maps based on these features, and then quickly locate vehicles within those maps. To ensure a consistently working map, some experiments have been conducted to continuously add image features as their appearance changes over time, thus keeping the map updated. However, due to the increasing number of maps, this method suffers from the following drawback: each region must be accessed relatively quickly to avoid too many changes in the final feature sets that could lead to a failure to locate the vehicle.

[0006] Another approach involves using triangulation of road signs to identify locations that are not currently in use with the same precision, but the goal is to roughly determine where the image was taken. This method has proven to be more robust to temporary changes, but its accuracy is too low for use with self-driving cars.

[0007] Therefore, there is still room for improvement in using camera images to locate vehicles. Summary of the Invention

[0008] In view of the above and other deficiencies of the prior art, one object of the present invention is to provide an improved method and system for determining the position of a vehicle in a vehicle.

[0009] According to a first aspect of the invention, a method for determining a vehicle location is provided, the method comprising capturing a two-dimensional image—a 2D image—using a vehicle-based camera, the image showing a portion of the environment surrounding the vehicle; classifying a plurality of pixels in the image such that each classified pixel belongs to a category of a predetermined classification group, thereby forming a classified two-dimensional image, wherein classification is performed by a learning algorithm. The method further comprises determining an initial estimated vehicle location and defining a plurality of possible vehicle locations based on the initial estimated vehicle location. The method then matches the classified two-dimensional image with a three-dimensional map—a 3D map—including a plurality of geometric objects, each geometric object belonging to a category of a predetermined classification group, by comparing the classification of geometric objects for possible vehicle locations in a 3D map with the classification of at least one corresponding pixel in the classified two-dimensional image; determining a matching score for at least one possible vehicle location based on the plurality of matching pixels in the two-dimensional image; and determining the vehicle location based on the score of the at least one possible vehicle location.

[0010] In the current description, it is assumed that a 3D map is formed prior to the use of the method, and that the 3D map is thus stored in or available to the vehicle. Furthermore, the predetermined classification groups used to classify pixels in the 2D image are based on existing predetermined classifications in the 3D map.

[0011] The main principle of the method is that, for a given captured 2D image, the image is matched against a 3D map representing possible vehicle locations. Each possible vehicle location corresponds to an observation point in the 3D map. Therefore, the captured 2D image is matched against different "observation points" in the 3D map corresponding to possible vehicle locations. In this simplified description, for the possible vehicle location closest to the actual location, it can be assumed that the categorical pixels of the 2D image have the best correspondence with the categorical geometry of the 3D map, thereby determining the correct vehicle location.

[0012] This invention is based on the understanding that a more efficient method can be achieved by classifying targets in a 3D map and pixels in a 2D image, and then matching the classified targets. This eliminates the need for specific identifiable targets in the surrounding environment, which consume significant storage space because a separate identifier for each individual target must be stored. Furthermore, feature-based localization for target recognition in images using image feature descriptors is sensitive to changes in the appearance of image features, meaning that a given descriptor cannot recognize image features if the appearance has changed. Such changes may occur naturally over time, including seasonal variations. Target classification, on the other hand, is more likely to remain unchanged by such changes.

[0013] Furthermore, by utilizing learning algorithms, the classification of two-dimensional images can be improved over time to increase classification accuracy and thus increase the accuracy of location determination.

[0014] Therefore, by utilizing target and pixel classification, the method provides a way to locate vehicles with improved efficiency and accuracy.

[0015] According to one embodiment of the invention, matching may include projecting a plurality of geometric targets in a 3D map onto a two-dimensional plane, and comparing the classification of the plurality of geometric targets projected onto the two-dimensional plane with the classification of at least one corresponding pixel in a classified two-dimensional image. Thereby, the projected targets form a corresponding "two-dimensional map," simplifying the matching between the 3D map and the two-dimensional image.

[0016] According to one embodiment of the invention, the two-dimensional plane corresponds to the image plane of the two-dimensional image. Therefore, only targets in the three-dimensional map that are assumed to be visible in the two-dimensional image are projected onto the two-dimensional plane. Consequently, geometric targets not within the line of sight of the assumed vehicle position are not projected. This reduces the number of projected targets and simplifies the matching process.

[0017] According to one embodiment of the invention, matching may include projecting at least one pixel of a two-dimensional image onto the 3D map to form a cone-shaped volume in the 3D map, the cross-section of which corresponds to the cross-section of the pixel. A pixel typically has a square cross-section, meaning that the four corners of the pixel are projected onto the 3D map to form a cone (i.e., a pyramid) in the 3D map. Thus, if a target is within the volume, matching is performed by comparing the pixel's classification with the target classification of the projected volume in the 3D map.

[0018] According to one embodiment of the invention, determining a matching score may include assigning a weight to each geometric target within a cone-shaped volume, wherein the weight is based on the distance between the geometric target in the 3D map and the image plane of the 2D image. Thus, when the volume includes multiple geometric targets, targets closer to the image plane of the 2D image can be given a higher weight when matching pixel classification and target classification compared to targets farther from the image plane of the 2D image. Therefore, even if the cone-shaped volume includes targets invisible from the vehicle's perspective, such targets can be given a lower weight. The weight may, for example, be 0 or 1.

[0019] According to one embodiment of the invention, matching may include projecting at least one pixel of a two-dimensional image onto a geometric target in a 3D map, and comparing the classification of the geometric target with the classification of at least one pixel projected in the two-dimensional image.

[0020] According to one embodiment of the present invention, the position of a point in a 3D map corresponds to a two-dimensional point in an image plane. An image plane is a plane in three-dimensional space where the position of a pixel can be determined by two coordinates. Therefore, projecting pixels from a two-dimensional image onto a 3D map can be considered as forming an image plane in three-dimensional space.

[0021] According to one embodiment of the present invention, classifying pixels in a two-dimensional image may include classifying only pixels in the two-dimensional image that correspond to the locations of geometric targets in the 3D map. Therefore, pixels in the two-dimensional image can be mapped to corresponding geometric targets in the 3D map before pixel classification, thereby making the method more computationally efficient due to fewer pixels needing to be classified.

[0022] According to one embodiment of the invention, the method may further include selecting a subset of categories and matching only pixels and geometric targets belonging to that subset. Here, pixel classification is performed, but only pixels belonging to the subset of categories are then matched with geometric targets in the 3D map, which allows for faster location determination. If a known target does not belong to some categories near the vehicle, a subset of categories can be selected, for example, based on estimated location. Furthermore, the known location of the camera in the vehicle can also be used to select a subset of categories.

[0023] According to one embodiment of the present invention, geometric targets in a 3D map can be represented by points in the 3D map, wherein the points correspond to a pixel in a classified two-dimensional image.

[0024] Furthermore, geometric targets in a 3D map can be represented by surfaces or volumes. In principle, the size of a surface or volume might correspond to only one pixel. However, in practice, each surface or volume will correspond to multiple pixels in the categorized 2D image.

[0025] According to one embodiment of the invention, the initial vehicle position may be based on a previously determined vehicle position. Therefore, if a method for determining vehicle position has recently been implemented, the approximate position of the vehicle is known. The estimate of the initial vehicle position may, for example, be based on a previously determined combination of position and vehicle speed.

[0026] According to one embodiment of the present invention, the initial vehicle position may be based on the vehicle position obtained from GPS measurements. Alternatively, the initial vehicle position may be determined based on a combination of the methods described above.

[0027] According to one embodiment of the invention, the predetermined classification group may include predetermined semantic classifications, meaning that the classifications can be associated with actual targets such as houses, buildings, or trees. An advantage of using semantic classification is that it enables the use of guided machine learning algorithms, where human input is used to define the targets to train the algorithms.

[0028] According to one embodiment of the invention, a predetermined classification group is determined by a learning algorithm. The classification group is then determined during the learning phase of a learning algorithm, which can be executed, for example, when forming a 3D map. By enabling the learning algorithm to define the classification, it is possible to obtain classification definitions that are more suitable for localization than human-defined classifications. Furthermore, unsupervised machine learning algorithms can be more time-efficient if the need for human interaction during the learning process is reduced or eliminated.

[0029] According to one embodiment of the present invention, the learning algorithm may be a convolutional neural network, which has been proven to be well-suited for image classification. Alternatively, a support vector machine may be used.

[0030] According to one embodiment of the present invention, defining multiple possible vehicle positions based on an initial estimated vehicle position includes utilizing a Bayesian filter.

[0031] According to a second aspect of the invention, a system for determining the position of a vehicle in a vehicle is provided, the system comprising: a camera configured to capture a two-dimensional image showing a portion of the environment surrounding the vehicle; a storage unit storing thereon a three-dimensional map (3D map) including a plurality of geometric objects, each geometric object belonging to a category of a predetermined classification group; and an image processing unit. The image processing unit is configured to: form a categorized two-dimensional image by categorizing a plurality of pixels in the categorized image such that each categorized pixel belongs to a category of the predetermined classification group, wherein classification is performed by a learning algorithm; determine an initial estimated vehicle position; define a plurality of possible vehicle positions based on the initial estimated vehicle position; match the categorized two-dimensional image with the 3D map by comparing the classification of the geometric objects for the possible vehicle positions in the 3D map with the classification of at least one corresponding pixel in the categorized two-dimensional image; determine a matching score for at least one possible vehicle position based on the plurality of matching pixels in the two-dimensional image; and determine the vehicle position based on the score of the at least one possible vehicle position.

[0032] The additional effects and features of the second aspect of the invention are largely similar to those described above in conjunction with the first aspect of the invention.

[0033] Those skilled in the art will recognize that different features of the present invention can be combined to produce embodiments other than those described below without departing from the scope of the invention. Attached Figure Description

[0034] These and other aspects of the invention will now be described in more detail with reference to the accompanying drawings, which illustrate exemplary embodiments of the invention, wherein:

[0035] Figure 1 This is a flowchart depicting the general steps of a method according to an embodiment of the present invention;

[0036] Figure 2 A system according to an embodiment of the present invention is illustrated schematically;

[0037] Figure 3A and 3B The features of a method according to an embodiment of the present invention are illustrated schematically; and

[0038] Figure 4 The features of a method according to an embodiment of the present invention are illustrated schematically. Detailed Implementation

[0039] The description primarily refers to various embodiments of the method and system according to the invention for determining the position of a car. However, the method is equally applicable to other types of vehicles.

[0040] Figure 1 This is a flowchart depicting the general steps of a method for determining vehicle location according to an embodiment of the present invention. (Refer to...) Figure 2 The system 200 shown and the selected steps of the method are illustrated schematically. Figure 3A -B Discussion Figure 1 The method.

[0041] Figure 2 The vehicle 204 is shown, which includes an onboard camera 206 configured to capture images of the vehicle's surroundings and an image processing unit 208 coupled to the camera 206. The camera 206 may be a camera 206 that captures real-time image input of the surroundings, or it may be a still camera 206 that captures images at selected time intervals or points.

[0042] In the first step of the method, camera 206 captures 102 two-dimensional images (i.e., 2D images) that depict a portion of the environment surrounding the vehicle. 2D image 302, as shown... Figure 3A It is shown schematically.

[0043] Next, the method involves classifying multiple pixels in image 302 such that each pixel belongs to a category within a predetermined classification group, thereby forming a classified 2D image. Classification 104 is performed using a learning algorithm. Typically, the classifier determines the probability that each pixel belongs to each category, i.e., there will be N numbers for each pixel, where N is the number of categories, and the sum of the N numbers for each pixel will be 1 (since a pixel must have one category). Example output for a pixel: tree 0.6, building 0.05, ground 0.3, vehicle 0.05 (if we only have those four categories). This can be reduced to one category by selecting the most probable category. One can also match the probability function of the points in the image to the corresponding probability function in the detected pixels and compute a probability discretization of numbers from 0 (same probability function) to 1 (completely different probability functions).

[0044] One straightforward approach is to classify all pixels in a 2D image. However, it might be possible to classify only a subset of pixels to obtain a sufficiently accurate 2D image. The current simplified example is shown using three distinct semantic categories: “buildings,” “trees,” and “roads,” thus defining predetermined classification groups. Here, it is assumed that only pixels considered to belong to one of the three categories will be classified.

[0045] In the following steps, 106 initial vehicle positions are determined. The initial vehicle position can be based, for example, on a vehicle position previously determined according to the method described. It can also be determined based on GPS position and / or inertial measurement system, etc. Based on the initially estimated vehicle position, 108 possible vehicle positions are defined. For example, a Bayesian filter can be used to define the possible vehicle positions. In a Bayesian filter used for localization, possible positions are represented by probability densities of states that are typically continuous in the 3D position, and there may be additional elements coded in the states other than position, such as velocity. The Bayesian filter is further based on the assumption that there is a relationship between the state distributions over time that can be described by a motion model. It is often assumed that the motion model is a Markov process, meaning that if the distribution is known from the previous time step, modeling subsequent time steps does not require additional information prior to the previous time step. The Bayesian filter can be constructed to describe the distribution of possible positions using a number of particles in a particle filter, a point mass filter, or various types of Kalman filters.

[0046] Therefore, once more than 108 possible vehicle locations have been defined, the following matching step 110 is performed for at least one location and preferably for more than one possible location.

[0047] The possible vehicle locations define the vehicle coordinates in the 3D map. The 3D map is described as being stored on memory 207 in vehicle 204, thereby being usable by image processing unit 208. However, it is also possible that the 3D map, or at least a portion thereof, is stored at a distance from the vehicle and can be accessed wirelessly.

[0048] For a given set of coordinates, a classified 2D image is matched with a 3D map. Pixels in the 2D image are matched with geometric targets in the 3D map, where the geometric targets can be represented by points, surfaces, or volumes in the 3D map—that is, 1D, 2D, or 3D targets—each with a classification and coordinates. For simplicity, Figure 3B The targets in the 3D map 304 shown can be considered as represented by surfaces, i.e., two-dimensional targets. However, the matching principle remains the same regardless of the dimensionality of the targets.

[0049] Pixel 306 in the 2D image is projected onto image plane 308 in the 3D map 304. Pixel 306 has been classified as belonging to the "building" category. Pixel 306 is thus projected onto the target in the 3D map, which is here a surface with the "building" classification. Therefore, there is a match between the classification of pixel 306 and the classification of target 310. The 2D target can be represented, for example, by surface triangles, Gaussian processes, or other known 2D target representations.

[0050] Here, the projection is referred to as the projection of pixels onto the two-dimensional image plane of the 3D map. However, this is equivalent to the projection of the geometric objects of the 3D map onto the classification two-dimensional image, and it can be assumed that the two projections provide the same results.

[0051] If it is known or can be assumed that a vehicle is in a location with a high probability that the target belongs to a specific category that does not exist, i.e., if the vehicle is driving on a city street without trees, then the classification "tree" can be omitted when classifying two-dimensional images and when performing matching, thus providing a faster classification and matching process.

[0052] Once a selected number of pixels, or all pixels, have been matched in the appropriate manner, a total matching score of 112 is determined for possible vehicle locations. New possible vehicle locations can then be selected, resulting in a slightly different representation of the 3D map, which will likely subsequently provide different matching scores.

[0053] Finally, after matching has been performed for all selected possible vehicle locations, the locations of 114 vehicles can be determined based on one or more matching scores. If the scores are carefully selected to correspond to the probability of generating an observed classification image for that location, a standard Bayesian filter can be used to combine that score with the density from previous time steps to form a posterior density. From this posterior density, the estimated location can be extracted, for example, by finding the maximum value of the probability density (MAP estimation) or by minimizing the expected error of the estimation (MMSE estimation).

[0054] Figure 4 An example embodiment of the method is illustrated schematically, wherein pixels 306 of a two-dimensional image 302 are projected onto a 3D map 304 to form a cone-shaped volume 402 in the 3D map, having a cross-section (here, a rectangular cross-section) corresponding to the pixel cross-section, thereby providing a pyramid volume. Matching then involves comparing the pixel classification with the classification of targets within the cone-shaped volume 402. Figure 4 It can be seen that volume 402 intersects with the "house" target 310.

[0055] In the above description, it is assumed that a 3D map with object classification exists in advance and is available inside the vehicle. The 3D map can be formed by a convolutional neural network that performs semantic or non-semantic object classification. Therefore, a convolutional neural network can be used to classify pixels in the obtained two-dimensional image.

[0056] Although the invention has been described with reference to specific exemplary embodiments therein, many different variations, modifications, etc., will be apparent to those skilled in the art. It should also be noted that components of the methods and systems may be omitted, substituted, or arranged in various ways, yet the methods and systems will still be able to perform the functions of the invention.

[0057] Additionally, variations of the disclosed embodiments will readily be understood and implemented by those skilled in the art who practice the claimed invention, based on a study of the drawings, specification, and appended claims. In the claims, the word "comprising" does not exclude the presence of other elements or steps, and the indefinite article "a" does not exclude a plurality. The mere fact that some measurements are recorded in different dependent claims does not imply that combinations of these measurements cannot be utilized.

Claims

1. A method for determining the position of a vehicle, the method comprising: Two-dimensional images—2D images—are captured using vehicle-based cameras, showing a portion of the environment surrounding the vehicle; The classification of multiple pixels in the image is performed such that each classified pixel belongs to a category in a predetermined classification group, thereby forming a classified 2D image, wherein the classification is performed by a learning algorithm; Determine the initial estimated vehicle position; Based on the initial estimated vehicle position, multiple possible vehicle positions are defined; The 2D image is matched with a 3D map that includes multiple geometric objects, each belonging to one of the predetermined classification groups, by comparing the classification of geometric objects in the 3D map for possible vehicle locations with the classification of at least one corresponding pixel in the classified 2D image. A matching score for at least one possible vehicle location is determined based on multiple matching pixels of the 2D image. and The vehicle position is determined based on the score of the at least one possible vehicle position.

2. The method according to claim 1, wherein, The matching process involves projecting multiple geometric targets from the 3D map onto a two-dimensional plane and comparing the classification of the multiple geometric targets projected onto the two-dimensional plane with the classification of at least one corresponding pixel in the classified 2D image.

3. The method according to claim 2, wherein, The two-dimensional plane corresponds to the image plane of the 2D image.

4. The method according to claim 1, wherein, Matching includes projecting at least one pixel of the 2D image onto the 3D map to form a cone-shaped volume in the 3D map, the cross section of which corresponds to the cross section of the pixel.

5. The method according to claim 1, wherein, The matching process includes projecting at least one pixel of the 2D image onto a geometric target in the 3D map, and comparing the classification of the geometric target with the classification of at least one pixel projected in the classified 2D image.

6. The method according to claim 5, wherein, The location of a point in a 3D map corresponds to a two-dimensional point in the image plane.

7. A system (200) for determining the position of a vehicle (204), the system comprising: A camera (206) is configured to capture 2D images showing a portion of the environment surrounding the vehicle; It stores a 3D map storage unit (207) containing multiple geometric objects, each geometric object belonging to a category within a predetermined classification group; and The image processing unit (208) is configured as follows: The classification of multiple pixels in the image such that each classified pixel belongs to a category in the predetermined classification group forms a classified 2D image, wherein the classification is performed by a learning algorithm; Determine the initial estimated vehicle position; Based on the initial estimated vehicle position, multiple possible vehicle positions are defined; The 2D image is matched with the 3D map by comparing the classification of geometric targets for possible vehicle locations in the 3D map with the classification of at least one corresponding pixel in the classified 2D image. A matching score for at least one possible vehicle location is determined based on multiple matching pixels of the 2D image; and The vehicle position is determined based on the score of at least one possible vehicle position.

Images