Image processing device and method for advanced driver assistance system

Abstract

The present invention provides an image processing device (100) for generating a scene map based on multiple images of a scene. The image processing device (100) includes a processing circuit (101) for iteratively generating the map by processing the multiple images in the following manner: (a) dividing a first image of the multiple images into multiple image parts; (b) extracting multiple feature points from each image part, and classifying at least one feature point of the multiple feature points as at least one successful feature point of the corresponding image part when at least one of the feature points is associated with a static background of the scene; (c) determining a confidence value for each image part of the first image based on the at least one successful feature point; and (d) repeating (a) to (c) for another image of the multiple images, wherein in (b), the number of the multiple feature points extracted from the corresponding image part of the another image depends on the confidence value of the corresponding image part of the first image. The present invention also discloses an image processing method.

Description

Technical Field

[0001] The present invention relates to the field of image processing or computer vision, and more particularly to an image processing device and method for an advanced driver assistance system. Background Art

[0002] Advanced driver assistance systems (ADAS) can alert the driver to dangerous situations and / or actively participate in driving.

[0003] A major challenge for advanced driver assistance systems (ADAS) is the mapping of the vehicle environment. Typically, mapping involves the estimation of camera trajectories and environmental structures (e.g., 3D point clouds), which are used for localization tasks. Mapping relies on visual input, usually in the form of video input from one or more cameras, and requires detecting a sufficient number of feature points from the static background in the scene.

[0004] Simultaneous localization and mapping (SLAM) is a computational problem of building or updating a map of an unknown environment while tracking the position of a vehicle in the environment. For example, CN105989586 A, US 9574883B2, and US 9758305B2 disclose techniques for combining SLAM with semantic information (also known as semantic mapping and localization).

[0005] In traditional composition techniques, moving objects distort the composition results and cause them to fail. In some cases, traffic scenes contain many moving objects (e.g., cars, pedestrians, etc.). In some other cases, not enough feature points are found due to lack of unique scene points, image blur, poor lighting conditions, etc. Traditional techniques mainly rely on points or lines or edges to detect unique feature points in the scene.

[0006] Therefore, traditional mapping or localization techniques may fail if there are too many moving objects or not enough good feature points. Sometimes, even if mapping techniques are feasible, not enough good feature points (outliers or points with no corresponding points) are collected in the map to achieve accurate and robust localization. Extracting many feature points is usually a very computationally intensive task.

[0007] In view of the foregoing, there exists a need for improved image processing apparatus and methods that allow for robust and efficient mapping and localization. Summary of the invention

[0008] Embodiments of the invention are defined by the features of the independent claims, and further advantageous embodiments of the embodiments are defined by the features of the dependent claims.

[0009] To describe the embodiments of the present invention in detail, the following terms, abbreviations and symbols will be used:

[0010] Scene: The surrounding environment relative to a reference object. For example, a camera's scene is the portion of the environment visible to the camera.

[0011] ADAS: Advanced Driver Assistance System.

[0012] 2D Image: A normal two-dimensional image or picture (RGB or Chroma-Luminance) acquired using a camera.

[0013] Texture: Areas within an image that show content with significant variations in (color) intensity.

[0014] 3D point cloud: A collection of points in 3D space.

[0015] 2D feature point: A location in image coordinates that represents a feature point in the scene.

[0016] 3D feature points: feature points in a 3D scene.

[0017] Composition: Create a 3D structure / 3D point cloud in some environment's global coordinate system, including position support (e.g. coordinates, ...).

[0018] Localization: Estimate the current position of an entity (e.g., a camera) relative to the global coordinate system of the provided map.

[0019] Semantic Segmentation: A method that divides an image into different regions based on semantic context. For example, pixels depicting cars are all red, pixels depicting roads are all blue, and so on.

[0020] Object instance: A single object within a group of objects of the same class.

[0021] Instance-level semantic segmentation: A method that segments an image into different regions and object instances based on semantic attribution. Individual objects can be identified and separated from each other.

[0022] Label: An identifier (e.g., an integer) used to determine the class type of an item / entity.

[0023] Dynamic objects: Objects in a scene that typically move or change their position.

[0024] Static background: All parts of the scene remain stationary, such as buildings, trees, roads, etc.

[0025] Global Coordinate System: A coordinate system relative to a common global reference.

[0026] Local Coordinate System: The coordinate system relative to the reference object selected in the global reference object.

[0027] Mapping Cycle: Typically, a specific vehicle path is chosen for the environment to be mapped. This path may be traversed multiple times (multiple cycles) to improve the final map accuracy and consistency.

[0028] Inliers: A corresponding pair of image feature points (from two image frames) where each point points to the same static background 3D point in the scene.

[0029] Outliers: A corresponding pair of image feature points (from two image frames) that point to two different 3D points in the scene.

[0030] In general, embodiments of the present invention are based on the idea of ​​providing robust and efficient mapping and localization by increasing the number of successfully extracted feature points (eg, inlier feature points or "inliers" for short) that correspond to a static background in a scene.

[0031] More specifically, according to a first aspect, the invention relates to an image processing device for generating a scene map based on a plurality of images of the scene, each image comprising a plurality of pixels. The image processing device comprises a processing circuit for iteratively generating the map by processing the plurality of images one by one in the following manner:

[0032] (a) dividing a first image of the plurality of images into a plurality of image portions;

[0033] (b) extracting a plurality of feature points from each image portion and, if at least one of the feature points is associated with a static background of the scene, classifying at least one feature point of the plurality of feature points as at least one target feature point (e.g., an inlier) of the corresponding image portion;

[0034] (c) determining a confidence value for each image portion of the first image based on the at least one target feature point; and

[0035] (d) repeating (a) to (c) for another image of the multiple images, wherein, in (b), the number of the multiple feature points extracted from the corresponding image portion of the other image depends on the confidence value of the corresponding image portion of the first image.

[0036] The image processing apparatus according to the first aspect of the invention allows increasing the chances of extracting useful target feature points (i.e., feature points associated with the static background of the scene) and using them in the composition and positioning process. Thus, a robust and efficient apparatus for generating a scene map is provided.

[0037] In another possible implementation manner of the first aspect, the processing circuit is used to divide the first image and another image of the multiple images into multiple rectangles, especially square image parts.

[0038] In another possible implementation manner of the first aspect, the rectangular image parts have the same size.

[0039] In another possible implementation manner of the first aspect, the processing circuit is used to determine the confidence value of each image part as a ratio of the number of target feature points to the total number of feature points of the corresponding image part.

[0040] In another possible implementation of the first aspect, the processing circuit is used to determine the confidence value of each image portion as the product of the ratio of the number of target feature points to the total number of feature points of the corresponding image portion, and to determine the confidence value of the corresponding image portion of a previously processed image.

[0041] In another possible implementation manner of the first aspect, the map is a semantic map of the scene, including semantic information of at least some of the multiple feature points.

[0042] In another possible implementation manner of the first aspect, the processing circuit is further configured to assign a semantic class C to each of the plurality of feature points, and determine, for each image portion, a corresponding main semantic class C having most feature points.

[0043] In another possible implementation of the first aspect, the processing circuit is used to determine the confidence value of each image part as the ratio of the number of target feature points to the total number of feature points of the corresponding image part, and the ratio is obtained by weighted calculation according to a first weighting factor, provided that the main semantic class C of the corresponding image part of the image is the same as the main semantic class of the corresponding image part of the previously processed image; or the ratio is obtained by weighted calculation according to a second weighting factor, provided that the main semantic class C of the corresponding image part of the image is different from the main semantic class of the corresponding image part of the previously processed image, wherein the first weighting factor is greater than the second weighting factor.

[0044] In another possible implementation manner of the first aspect, the processing circuit is used to iteratively generate the map based on a simultaneous localization and mapping (SLAM) algorithm.

[0045] In another possible implementation manner of the first aspect, the number of the plurality of feature points extracted from the corresponding image portion of the other image is proportional to the confidence value of the corresponding image portion of the first image.

[0046] In another possible implementation manner of the first aspect, the image processing apparatus further includes an image acquisition device, in particular a camera, for acquiring a plurality of images of the scene.

[0047] According to a second aspect, the present invention relates to an advanced driver assistance system for a vehicle, wherein the advanced driver assistance system comprises an image processing device according to the first aspect of the present invention or any one of its implementations.

[0048] According to a third aspect, the present invention relates to an image processing method for generating a scene map based on a plurality of images of the scene, wherein the image processing method comprises the following steps:

[0049] (a) dividing a first image of the plurality of images into a plurality of image portions;

[0050] (b) extracting a plurality of feature points from each image portion, and classifying at least one feature point of the plurality of feature points as at least one target feature point (e.g., an inlier feature point) of the corresponding image portion if at least one of the feature points is associated with a static background of the scene;

[0051] (c) determining a confidence value for each image portion of the first image based on the at least one target feature point; and

[0052] (d) repeating steps (a) to (c) for another image of the multiple images, wherein, in step (b), the number of the multiple feature points extracted from the corresponding image portion of the other image depends on the confidence value of the corresponding image portion of the first image.

[0053] Therefore, a robust and efficient method for generating scene maps is provided.

[0054] The image processing method according to the third aspect of the present invention can be performed by the image processing device according to the first aspect of the present invention. Other features of the image processing method according to the third aspect of the present invention are directly derived from the functions of the image processing device according to the first aspect and its different implementation modes described in the context.

[0055] According to a fourth aspect, the present invention relates to a computer program product, comprising: a program code for executing the method according to the third aspect of the present invention when running on a computer.

[0056] The following drawings and description will set forth one or more embodiments in detail. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Hereinafter, various embodiments of the present invention will be described in more detail with reference to the accompanying drawings, in which:

[0058] Figure 1 is a block diagram showing an example of an image processing device provided by an embodiment of the present invention;

[0059] Figure 2 It is shown by Figure 1 A schematic diagram of an example of an image having multiple image parts that is processed by the image processing apparatus;

[0060] Figure 3 It is shown in Figure 1 A flowchart of an example of processing steps implemented in the image processing device; and

[0061] Figure 4 It is shown in Figure 1 A flowchart of another example of the processing steps implemented in the image processing device.

[0062] In the following, identical reference signs refer to identical or at least functionally equivalent features. DETAILED DESCRIPTION

[0063] The following description is made in conjunction with the accompanying drawings, which illustrate specific aspects of embodiments of the present invention or specific aspects in which embodiments of the present invention may be used by way of diagrammatic illustration. It is understood that embodiments of the present invention may be used in other aspects and include structural or logical changes not shown in the accompanying drawings. Therefore, the following detailed description is not intended to be limiting, and the scope of the present invention is defined by the appended claims.

[0064] For example, it is understood that disclosures relating to the described methods are also applicable to corresponding devices or systems for performing the methods, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) for performing the described one or more method steps (e.g., a unit that performs one or more steps, or multiple units that perform one or more steps), even if such one or more units are not explicitly described or illustrated in the accompanying drawings. On the other hand, for example, if a specific device is described based on one or more units (e.g., functional units), the corresponding method may include steps for performing the functions of the one or more units (e.g., a step that performs the functions of the one or more units, or multiple steps that perform the functions of the one or more units), even if such one or more steps are not explicitly described or illustrated in the accompanying drawings. In addition, it is understood that the features of the various exemplary embodiments and / or aspects described herein may be combined with each other, unless otherwise stated.

[0065] Figure 1is a block diagram showing an example of an image processing apparatus 100 provided by an embodiment of the present invention. In one embodiment, the image processing apparatus 100 further comprises an image acquisition device 103 for acquiring multiple images of a scene, in particular a camera. In one embodiment, the image processing apparatus 100 is implemented as a part of a vehicle advanced driver assistance system (ADAS) or interacts with it.

[0066] As described in more detail below, the image processing device 100 is used to generate a scene map based on a plurality of images of the scene. To this end, the image processing device 100 comprises a processing circuit 101, which is used to iteratively generate the map by processing the plurality of images one by one in the following manner:

[0067] (a) dividing a first image of the plurality of images into a plurality of image portions;

[0068] (b) extracting a plurality of feature points from each image portion and, if at least one of the feature points is associated with a static background of the scene, classifying at least one feature point of the plurality of feature points as at least one target feature point (e.g., an inlier) of the corresponding image portion;

[0069] (c) determining a confidence value P for each image part of the first image based on the at least one target feature point; and

[0070] (d) repeating (a) to (c) for another image of the multiple images, wherein, in (b), the number of the multiple feature points extracted from the corresponding image portion of the other image depends on the confidence value of the corresponding image portion of the first image.

[0071] Figure 2 It is shown by Figure 1 Schematic diagram of an example of an image 200 having a plurality of image parts (identified by a pair of indices (m, n)) to be processed by the image processing apparatus 100 described in FIG. Figure 2 As can be seen from the exemplary image 200 shown, in one embodiment, the processing circuit 101 is used to divide multiple images (such as the image 200) into multiple rectangular, especially square image parts. In one embodiment, the rectangular image parts have the same size.

[0072] Figure 3 It is shown in Figure 1 Flow chart of an example of multiple processing steps 300 implemented in the image processing device 100. The multiple processing steps 300 include the following steps:

[0073] 301: Acquire an image from a plurality of images for further processing.

[0074] 303: Divide the image of the plurality of images into an NxM rectangular grid of image portions.

[0075] 305: Assume K is the total number of feature points to be detected in the entire image. Set the processing circuit 101 to detect or extract P(m, n)*K feature points in the image region (m, n). First, set all P(m, n) to 1. Assume S(m, n) is the number of target feature points (inliers) in the image region (m, n), and T(m, n) is the total number of feature points detected in the image region (m, n). Therefore, in one embodiment, the number of multiple feature points extracted from the corresponding image portion of the image is proportional to the confidence value P(m, n) of the corresponding image portion of the previously processed image.

[0076] 307: Semantic Segmentation assigns a class or label C describing its semantic class to each pixel in the image. The assigned class indicates to which semantic class the pixel belongs (e.g., car, road, building, etc.). In the case where a pixel cannot be classified, it can be assumed to be associated with a dynamic feature and thus it can be defined as an outlier.

[0077] 309: Each feature point has a position in pixel coordinates (ultimately sub-pixel precision). Therefore, each feature point can be associated with its nearest pixel. If the semantic class of the nearest pixel is a dynamic object (car, pedestrian, truck, bicycle, etc.), then the feature point is removed from the set of detected feature points, i.e., it is not a target feature point.

[0078] 311: The confidence value P of each pixel part (m, n) is updated as:

[0079]

[0080]

[0081] Therefore, in one embodiment, the processing circuit 101 of the image processing device 100 is used to determine the confidence value P(m, n) of each image portion as the ratio of the number of target feature points S(m, n) to the total number of feature points T(m, n) of the corresponding image portion. In addition, in one embodiment, the processing circuit 101 is used to determine the confidence value P(m, n) of each image portion as the ratio product of the number of target feature points S(m, n) to the total number of feature points T(m, n) of the corresponding image portion, and to determine the confidence value of the corresponding image portion of the previously processed image.

[0082] against Figure 2Some exemplary confidence values ​​P(m,n) are shown for different image portions (m,n) of the illustrated image 200. It is understood that the processing circuit will extract most feature points in image portions (1,2), (1,3) and (2,3) because they have the highest confidence values ​​P. The sum of the confidence values ​​P for all image portions of the image should be 1, i.e.:

[0083]

[0084] 313: Update the map using the marked feature points. In one embodiment, the semantic map is a map in the simultaneous localization and mapping (SLAM) process, which is used to locate the vehicle. Updating the semantic map means updating the map according to the SLAM algorithm, but the additional information comes from the semantic segmentation (step 307). In this case, it also contains the corresponding semantic class C for each feature point. Therefore, in one embodiment, the processing circuit 101 of the image processing device 100 is used to iteratively generate the map based on the simultaneous localization and mapping (SLAM) algorithm.

[0085] During the composition process, the map contains the 3D point locations of the image feature points and the camera position and orientation. As mentioned above, the map can be updated at each new image, that is, the composition process is iterative. For example, new points can be added, or the current camera position and orientation can be added (e.g., like nodes in the graph). Sometimes, some larger updates can be made, such as going back a few nodes in time (this is called a bundle adjustment process), where the camera position and / or orientation and / or 3D points are fine-tuned to further improve the estimation accuracy. This is an overall optimization process.

[0086] 315: Output the (updated) semantic map.

[0087] Figure 4 It is shown in Figure 1 Flow chart of another example of a plurality of processing steps 300 implemented in the image processing apparatus 100. Figure 3 Compared to the illustrated processing steps 300 , the plurality of processing steps 300 further incorporates semantic information about each image part into the calculation and update of the confidence value P by including an additional step 310 and a modified step 311 .

[0088] More specifically, Figure 4 The plurality of processing steps 300 shown in FIG. 1 take into account the main semantic class C of the corresponding image portion of the processed image. The main semantic class of the corresponding image portion is defined as the semantic class with the largest number of pixels. Figure 3 In the case of the processing step 300 shown, the confidence values ​​of all image regions of the currently processed image should first be normalized, that is, P(m, n)=1. Figure 3 In the case of the processing step 300 shown, S(m,n) represents the number of target feature points (ie, inliers) of the image region (m,n), and T(m,n) represents the total number of feature points detected in the image region (m,n).

[0089] 310: Determine the main semantic class of each image region. C(m,n) represents the main semantic class of the image region (m,n). As mentioned above, this means that most pixels belong to class C(m,n).

[0090] 311: The confidence value P of each image region (m, n) is updated by the processing circuit 101 based on the following equation:

[0091]

[0092]

[0093] in,

[0094] The weight D here is a measure of the frequency of changes in the main semantic class of each image region over time. More frequent changes reduce the reliability of the image region containing useful target feature points. This is reflected by introducing the weight D in the above equation (1). Therefore, the higher the frequency of change of the main semantic class, the smaller the average weight D over time.

[0095] Therefore, in one embodiment, the processing circuit 101 is used to assign a semantic class C to each of the plurality of feature points, and to determine, for each image portion, a corresponding primary semantic class C having the majority of feature points. In addition, in one embodiment, the processing circuit 101 is used to determine the confidence value P(m, n) of each image portion as a ratio of the number of target feature points S(m, n) to the total number of feature points T(m, n) of the corresponding image portion, the ratio being weighted and calculated according to a first weighting factor (e.g., D=1), provided that the primary semantic class C of the corresponding image portion of the image is the same as the primary semantic class of the corresponding image portion of the previously processed image; or the ratio being weighted and calculated according to a second weighting factor (e.g., D=0.75), provided that the primary semantic class C of the corresponding image portion of the image is different from the primary semantic class of the corresponding image portion of the previously processed image.

[0096] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the described device embodiments are merely exemplary. For example, the unit division is only a logical function division and can be other divisions in actual implementation. For example, multiple units or components can be merged or integrated into another system, or some features can be ignored or not performed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be realized by using some interfaces. The direct coupling or communication connection between devices or units can be realized by electronic, mechanical or other forms.

[0097] The units described as separate parts may or may not be physically separated, and the parts described as units may or may not be physical units, may be located in one location, or may be distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0099] Embodiments of the present invention also include an apparatus comprising processing circuitry for performing any of the methods and / or processes described herein.

Claims

1. An image processing device (100) for generating a scene map based on a plurality of images of a scene, characterized in that: The image processing device (100) comprises a processing circuit (101) for iteratively generating the map by processing the plurality of images one by one in the following manner: (a) dividing a first image of the plurality of images into a plurality of image portions; (b) extracting a plurality of feature points from each image portion, and classifying at least one feature point of the plurality of feature points as at least one target feature point of the corresponding image portion if at least one of the feature points is associated with a static background of the scene; (c) determining a confidence value for each image portion of the first image based on the at least one target feature point; as well as (d) repeating (a) to (c) for another image of the multiple images, wherein, in (b), the number of the multiple feature points extracted from the corresponding image portion of the other image depends on the confidence value of the corresponding image portion of the first image; the confidence value of the corresponding image portion of the first image is determined as the product of the ratio of the number of target feature points in the corresponding image portion of the first image to the total number of feature points and the confidence value of the corresponding image portion of the previously processed image.

2. The image processing device (100) according to claim 1, characterized in that: The processing circuit (101) is used to divide the first image and another image of the plurality of images into a plurality of rectangular image portions.

3. The image processing device (100) according to claim 2, characterized in that: The rectangular image portions have the same size.

4. The image processing device (100) according to any one of claims 1 to 3, characterized in that: The map is a semantic map of the scene, and the map includes semantic information of at least some feature points.

5. The image processing device (100) according to claim 4, characterized in that: The processing circuit (101) is also used to assign a semantic class C to each of the plurality of feature points, and determine a main semantic class C for each image part; the main semantic class C is the semantic class with the largest number of pixels.

6. The image processing device (100) according to claim 5, characterized in that: The processing circuit (101) is used to determine the confidence value of each image part as the ratio of the number of target feature points to the number of feature points of the corresponding image part, wherein the ratio is obtained by weighted calculation according to a first weighting factor, provided that the main semantic class C of the corresponding image part of the image is the same as the main semantic class of the corresponding image part of the previously processed image; or the ratio is obtained by weighted calculation according to a second weighting factor, provided that the main semantic class C of the corresponding image part of the image is different from the main semantic class of the corresponding image part of the previously processed image, wherein the first weighting factor is greater than the second weighting factor.

7. The image processing device (100) according to claim 4, characterized in that: The processing circuit (101) is used to generate the map based on a simultaneous positioning and map building algorithm.

8. The image processing device (100) according to claim 1, characterized in that: The number of the plurality of feature points extracted from the corresponding image portion of the other image is proportional to the confidence value of the corresponding image portion of the first image.

9. The image processing device (100) according to claim 1, characterized in that: The image processing apparatus (100) further comprises an image acquisition device (103) for acquiring a plurality of images of the scene.

10. An advanced driver assistance system for a vehicle, characterized in that: The advanced driver assistance system comprises an image processing device (100) according to any one of claims 1 to 9.

11. An image processing method for generating a scene map based on multiple images of a scene, characterized in that: The image processing method (200) comprises the following steps: (a) dividing a first image of the plurality of images into a plurality of image portions; (b) extracting a plurality of feature points from each image portion, and classifying at least one feature point of the plurality of feature points as at least one target feature point of the corresponding image portion if at least one of the feature points is associated with a static background of the scene; (c) determining a confidence value for each image portion of the first image based on the at least one target feature point; and (d) repeating steps (a) to (c) for another image of the multiple images, wherein, in step (b), the number of the multiple feature points extracted from the corresponding image portion of the other image depends on the confidence value of the corresponding image portion of the first image; the confidence value of the corresponding image portion of the first image is determined as the product of the ratio of the number of target feature points in the corresponding image portion of the first image to the total number of feature points and the confidence value of the corresponding image portion of the previously processed image.

12. A computer program product, characterized in that include: The program code is used to execute the method according to claim 11 when running on a computer or a processor.

Images