A method of performing a geometric and / or semantic map learning via a framework, computer program and computing device

Abstract

The invention relates to a method of performing a map learning via a unified framework comprising the steps on a computing device (10): - Retrieving data comprising features that are geospatially referenced and / or localized with respect to geospatially referenced; (S1) - Arranging the features in relation to each other on the basis of their localization and / or their geospatially referenced features, whereby a lane is determined and / or aligned; (S2) - Aggregate and / or cluster geometric features; (S3) - Determine a lane divider and / or lane segments based on the geometric features and / or based on lanes traveled; (S4) - Deriving a lane topology based on a connectivity of the lane segments; (S5) and - Semantic learning based on the lane topology, whereby features are assigned a function and the map is thus created or expanded. (S6) Furthermore, the invention relates to computer program and a computing device (10).

Description

Applicant’s Ref.: 2023P04546WOAttorney’s Ref.: 938111Mercedes-Benz Group AG NegiA METHOD OF PERFORMING A GEOMETRIC AND / OR SEMANTIC MAP LEARNING VIA A FRAMEWORK, COMPUTER PROGRAM AND COMPUTING DEVICEFIELD OF THE INVENTION

[0001] The present invention relates to the field of vehicles, in particular cars, vans, or trucks. More specifically, the present invention relates to a method to provide a map for a vehicle via a framework for geometric and / or semantic map learning. Furthermore, the present invention relates to a corresponding computer program, as well as to a computing device.BACKGROUND INFORMATION

[0002] Accurate and up-to-date maps with lane information are crucial for a variety of assistance systems and autonomous driving. However, the map creation process is time consuming and inefficient due to the lack of a flexible and efficient map learning development. Many developments are implemented with different frameworks, which leads to continuous re-implementation and inefficiencies due to a lack of comparability.SUMMARY OF THE INVENTION

[0003] Therefore, it is the object of the present invention to provide a method, a computer program as well as a computing device by which a geometric and / or semantic map learning can be performed very efficiently.

[0004] This object is solved by a method, a computer program as well as a computing device according to the independent claims. Advantageous embodiments are presented in the dependent claims, the description and the drawings.Applicant’s Ref.: 2023P04546WOAttorney’s Ref.: 938112

[0005] A first aspect of the present invention relates to a method of performing a geometric and / or semantic map learning via a unified framework, comprising the steps on a computing device:

[0006] A first step is retrieving data comprising features, in particular features comprising useful information for a map, that are geospatially referenced and / or localized with respect to geospatially referenced features, whereby the data is retrieved in particular from different sources, via a unified interface.

[0007] A second step is, in particular, geometrically, arranging the features in relation to each other on the basis of their localization and / or their geospatially referenced features, whereby a lane, in particular for a vehicle, is determined and / or aligned.

[0008] A third step is aggregating and / or clustering geometric features or features that contain geometrically information. In particular the aggregation and / or clustering takes place after lane alignment.

[0009] A fourth step is determining a lane divider and / or lane segments based on the geometric features and / or based on lanes traveled, in particular via at least one vehicle, which provided or generated the date retrieved in the first step.

[0010] A fifth step is deriving a lane topology, in particular as a map, based on a connectivity of the lane segments, which is, in particular, derived via determining the lane dividers.

[0011] The sixth step is a semantic learning based on the lane topology, whereby features are assigned a function, and the map is thus created or expanded.

[0012] In other words, the unified framework provides generic modules: A first module for data ingestion, wherein the unified interface is used to ingest data from, in particular, various sources, like vehicle sensors of different vehicles. Prerequirement for the data acquisition is that the features in the data are either geospatially referenced or referenced relative to features that are geospatially referenced (transitive geospatial reference).

[0013] Another module is for lane or trace alignment and / or localization, which is used to align individual traces with each other by performing relative localization on the geospatially referenced features of its trace (pools, signs, lane boundaries, etc.). TheApplicant’s Ref.: 2023P04546WOAttorney’s Ref.: 938113 trace may give strong constraints between geometric features along the trace and the localization may give strong constraints between multiple traces.

[0014] The next module of the unified framework is in particular a geometric aggregation module, which is used after alignment to cluster and / or aggregate geometric features. So, multiple observations of a point object may be combined into one.

[0015] Another module is for lane topology: the aggregated geometric representation contains or comprises visible lane dividers, so in a next step of the method those are used in combination with driven traces to derive the lane segments and their connectivity which gives the overall lane topology.

[0016] In a last module, the semantic module, learning is performed, where based on the geometric representation, the lane topology and the driven traces, semantic lane features are derived. Those include but are not limited to traffic lights, to lane assignments, ride of way, side lane assignments, etc. So, after the semantic learning, the map is created and / or expanded with features and their function.

[0017] The key advantage of this map learning framework is a unified interface for different sources of inputs. More specifically, it could be vehicles on the road from different carlines (e.g., different types of vehicles with different sensors, SW versions etc.).

[0018] The map could be generated precisely because a multitude of different inputs can be used through unified interface, improving precision and also completeness (e.g., mapping areas where sufficient data basis is only available after consolidating multiple types of inputs).

[0019] The method is preferably performed in a backend.

[0020] In an embodiment of the present invention the data are sensor data from a sensor device of at least one vehicle and, in particular, the data are sensor data from various sensors of more than one vehicle. The data may be camera data, which contains information about poles, signs, lane boundaries, etc. The data may also be an orientation and / or position, as well as the velocity of the at least one vehicle. In other words, sensor data, in particular from different sources, are used via the method. Thereby, the data may contain any information, which could be used to generate a map on geometric orApplicant’s Ref.: 2023P04546WOAttorney’s Ref.: 938114 semantic learning methods. An advantage is that the map could be generated very precisely in dependence of the data.

[0021] In another embodiment of the present invention, the recorded and generated data and / or data from other sources are referenced, stored, indexed and / or versioned in a data warehouse. The recorded data is in particular the raw data from the sensors retrieved in the first step. The then generated data is data derived from the method, for example the created or expanded map. Reference data from other sources may be used for validation of the generated data. In other words, the data provided for or generated by the method is processed via central data base optimized for analysis purpose, which combines data from different, usually heterogeneous sources. An advantage is that the method could be performed very efficiently.

[0022] In still another embodiment of the present invention, a map learning process or the steps of the method are displayed three-dimensionally by an output unit. In other words, a generic visualization tool allows to render all steps of the map learning process in 3D. The output unit may comprise a screen. An advantage is that a user may easily understand and / or reproduce the learning done via the method.

[0023] The visualization can give introspection into the individual steps that the map learning framework performs, allowing validation of the method, help with debugging, and data science.

[0024] In another embodiment of the present invention, at least one test object is created and / or executed on the map or at least on one of its levels. In other words, a map validation may be a module of the unified framework, which may generate a suit of test cases, or test objects, and / or define a suit of test cases, and execute them on the generated map layers. An advantage is that via map validation the map may be created or expanded very precisely.

[0025] In still another embodiment of the present invention, a test report is created in dependence on the test object. In other words, for the suit of test cases the test report may be generated, which may also be stored in the data warehouse, in particular as metadata. An advantage is the automatization of the test report generation and thereby completeness of the test documentation.Applicant’s Ref.: 2023P04546WOAttorney’s Ref.: 938115

[0026] In yet another embodiment of the present invention, the steps of the method are performed on a backend and / or server, which, in particular, may have communication means for communication with the at least one vehicle comprising the sensors for data collection. Alternatively or additionally, the map is delivered to the at least one vehicle, in particular from the backend via the communication means. In other words, there is a central computing unit or the computing device performing the method is a central computing unit built as a backend and / or server. Vie the central computing unit an actual map could be delivered very efficiently to all vehicles, which may communicate with the backend. An advantage is that the map used are always in an accurate and / or actual state.

[0027] In another embodiment of the present invention, the framework is designed as a domain framework. In other words, the main framework builds a programming framework for specific problems. The map learning and its modules, or, in particular, its functions and / or structures are fitted to solve the map learning. An advantage is that compared to any application or class framework it could be used very efficiently.

[0028] A second aspect of the present invention relates to a computer program comprising code means for providing a framework for performing a method according to any one of the preceding claims.

[0029] The computer program may be stored on a non-transitory computer-readable storage medium. That is, the non-transitory computer-readable storage medium may include at least the computer program according to the preceding aspect. Embodiments and advantages of the first aspect of the present invention are to be regarded as advantages and embodiments of the second aspect of the present invention and vice versa.

[0030] A third aspect of the present invention relates to a computing device, wherein the computing device is configured to perform a method according to the first aspect of the present invention.

[0031] The advantages and embodiments already described for the method and the computer program also apply to the computing device and vice versa.

[0032] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawings. TheApplicant’s Ref.: 2023P04546WOAttorney’s Ref.: 938116 features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figures and / or shown in the figures alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention.BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0034] The drawings show in:

[0035] Fig. 1 a schematic flow chart of a method for performing a map learning; and

[0036] Fig. 2 a schematic view of a computing device for performing the method and a corresponding vehicle.

[0037] In the figures the same elements or elements having the same function are indicated by the same reference signs.DETAILED DESCRIPTION

[0038] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0039] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure isApplicant’s Ref.: 2023P04546WOAttorney’s Ref.: 938117 to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0040] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0041] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0042] Fig. 1 shows a schematic diagram in a style of a flow chart for a method of performing a geometric and / or semantic map learning via a unified framework on a computing device 10 shown in Fig. 2 for generating or expanding the map, which may be used in at least one vehicle 12.

[0043] The method comprises the following steps:

[0044] In a first step S1 retrieving data comprising features that are geospatially referenced and / or localized with respect to geospatially referenced features via a unified interface.

[0045] A second step S2 is arranging the features in relation to each other on the basis of their localization and / or their geospatially referenced features, whereby a lane is determined and / or aligned.

[0046] A third step S3 is aggregate and / or cluster geometric features.Applicant’s Ref.: 2023P04546WOAttorney’s Ref.: 938118

[0047] A fourth step S4 is determine a lane divider and / or lane segments based on the geometric features and / or based on lanes traveled by at least the vehicle 12.

[0048] A fifth step S5 is deriving a lane topology, in particular as a map, based on a connectivity of the lane segments derived in step S4.

[0049] The sixth step S6 is a semantic learning based on the lane topology, whereby features are assigned a function, and the map is thus created or expanded.

[0050] The data are, in particular, sensor data from a sensor device 14 of the at least one vehicle 12. The exchange between the unified interface of the computing device 10 and the vehicle 12 may be provided via a communication unit 16.

[0051] The data may comprise, in particular, camera data with images, and a, for example, computer vision may extract information from the images, where the information are the features comprised in the data. The features may be pools, lanes, signs, lane boundaries, the velocity of the vehicle 12, its position, its orientation, landmarks, etc.

[0052] A data warehouse 18, in particular, is part of the computing device 10, which is in the embodiment shown here built as a backend or server. All ingested as well as generated data may be indexed and versioned in the data warehouse 18. Additionally, referenced data from other sources may also be stored in the data warehouse 18.

[0053] After the semantic and / or geometric map learning provided by the steps shown in Fig. 1, a map visualization as a module of the framework could provide a generic visualization tool to render all steps of the map learning process in three dimensions. Furthermore, a map validation may be performed where a suit of test cases may be defined and executed on a generated map layer. A test report may be generated that is also stored in the data warehouse 18 as metadata.

[0054] Fig. 2 shows the computing device 10 and the vehicle 12. The computing device 10 may comprise generic modules as part of the unified framework. Thereby the unified framework has map learning as a special purpose and is therefore built as, in particular, as a domain framework for map learning.

[0055] In other words, via the method a unified interface is used to ingest data from various sources. Key requirements may be that features are geospatially referenced orApplicant’s Ref.: 2023P04546WOAttorney’s Ref.: 938119 referenced relative to the features that are geospatially referenced. Individual traces or lanes are aligned with each other by performing relative localization on the geospatially referenced features of each trace.

[0056] Traces give strong constraints between geometric features along the trace, and the localization gives strong constraints between multiple traces. Therefore, geometric aggregations could be performed. In particular, after alignment, the geometric features are then clustered and aggregated so that multiple observations in the data, for example of different vehicles 12, are combined into one.

[0057] The aggregated geometric representation may contain visible lane dividers, or visible lane dividers could be aggregated from the geometric representation. Those are used in combination with the driven traces of the vehicle 12 which may be comprised of the data to derive the lane segments and their connectivity, which gives the overall lane topology and therefore a geometrical representation based on which the lane topology and the driven traces, semantic map features are derived via map learning. Those semantic map features may include but are not limited to traffic lights, lane assignments, ride of way, lane assignments, etc.

[0058] Advantages of the shown method, the corresponding computer program and the computing device 10 are that they compatible with different kinds of data inputs via a unified and, in particular, well-defined interface implementation. Another advantage is the modularity of code to support implementation of different model architectures. Another advantage is the abstraction of each layer allows to use the most suitable module per task. Therefore, another advantage is the efficiency, in particular, because the framework is reusable or uses reusable modules. A comparability through a harmonized validation framework for comparing the results is also possible. An option could also be to open source or collaborate with different organizations on individual modules.

[0059] So, a framework for geometric and semantic module learning is provided.Applicant’s Ref.: 2023P04546WOAttorney’s Ref.: 9381110Reference signs10 computing device12 vehicle14 sensor device16 computing unit18 data warehouse

Claims

Applicant’s Ref.: 2023P04546WOAttorney’s Ref.: 9381111Mercedes-Benz Group AG NegiCLAIMS1. A method of performing a geometric and / or semantic map learning via a unified framework comprising the steps on a computing device (10):Retrieving data comprising features that are geospatially referenced and / or localized with respect to geospatially referenced features via a unified interface; (S1)Arranging the features in relation to each other on the basis of their localization and / or their geospatially referenced features, whereby a lane is determined and / or aligned; (S2)Aggregate and / or cluster geometric features; (S3)Determine a lane divider and / or lane segments based on the geometric features and / or based on lanes traveled; (S4)Deriving a lane topology based on a connectivity of the lane segments; (S5) and- Semantic learning based on the lane topology, whereby features are assigned a function and the map is thus created or expanded. (S6)2. The method according to claim 1 , characterized in that data are sensor data from a sensor device of at least one vehicle.

3. The method according to claim 1 or 2, characterized in that the recorded and generated data and / or data from other sources are referenced, stored, indexed and / or versioned in a data warehouse (18).Applicant’s Ref.: 2023P04546WOAttorney’s Ref.: 93811124. The method according to any one of claims 1 to 3, characterized in that the map learning process is displayed three-dimensionally by an output unit.

5. The method according to any one of the preceding claims, characterized in that at least one test object is created and / or executed on the map or at least one of its levels.

6. The method according to claim 5, characterized in that a test report is created in dependence on the test object.

7. The method according to any one of the preceding claims, characterized in that the steps are performed on a backend and / or the map is delivered to a vehicle.

8. The method according to any one of the preceding claims, characterized in that the framework is designed as a domain framework.

9. A computer program comprising code means for providing a Framework for performing a method according to any one of the preceding claims.

10. Computing device (10), wherein the computing device (10) is configured to perform a method according to any one of claims 1 to 8.

Images