Method for creating a road model and method for operating an autonomous vehicle
By using a base model and locally applicable sub-models in autonomous vehicles to dynamically generate road models, the problems of outdated maps and insufficient environmental detection are solved, and efficient and stable autonomous driving path planning is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2026-04-07
AI Technical Summary
Existing technologies for creating road models for autonomous vehicles suffer from problems such as outdated maps and insufficient identification of areas that environmental detection sensor systems cannot recognize, leading to inaccurate positioning and models that are not adapted to environmental changes.
By employing an AI-based base model and locally applicable supplementary sub-models, the sub-models are trained using environmental detection sensor data, dynamically updated, and fused to generate a road model. The total output vector is formed by superimposing the base model and sub-models, enabling end-to-end road model perception.
It improves the recognition performance and stability of road models, reduces storage and transmission complexity, ensures the real-time adaptability of models to the environment, reduces the risk of map data becoming outdated, and achieves high-precision autonomous driving path planning.
Smart Images
Figure CN121816490A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The invention relates to a method for creating a road model in a predetermined environmental area of a vehicle.
[0002] The invention also relates to a method for operating an autonomous vehicle. BACKGROUND
[0003] For operating automated, in particular highly automated or autonomous, vehicles, high-resolution maps (HD maps) are used, which provide extensive and precise environmental information for the driving scenario and are used for planning the driving operation of the automated vehicle.
[0004] When creating a road model using a high-resolution map, this map has to be registered in the vehicle, which has to be converted from the bird's eye view in the global coordinate system to the view of the environmental detection sensor system in the vehicle coordinate system. In this case, a high-precision local positioning of the vehicle is required in order to be able to fuse the map data with the environmental data detected by the environmental detection sensor system. There is a risk in this case that the map is outdated and the map data contained therein is no longer suitable for the detected environmental data. Through the great progress in sensor technology and artificial intelligence, the quality of the road model creation can be achieved by environmental detection alone, with the efficiency of the information obtained by the environmental detection sensor system being improved in the so-called end-to-end processing. In this case, so-called generative artificial intelligence methods are used for end-to-end road model perception. Such methods include, for example, the so-called Detektions-Transformer (DETR) architecture.
[0005] From "Liao, B. et al.: MapTR - Structured Modeling and Learning for Online Vectorized HD Map Construction; see: arXiv: 2208.14437 v2 [cs.CV], 30 January 2023", a structured end-to-end Transformer for online construction of vectorized HD maps is known. Here, a uniform arrangement equivalent modeling method is used, in which map elements are modeled as a set of equivalent arrangements of points with a shape describing the map element and stabilizing the learning process. A hierarchical query embedding scheme is used to flexibly encode the structured map information and hierarchical two-part matching is performed for the learning of map elements. SUMMARY
[0006] It is an object of the present invention to provide a new method for creating a road model and a new method for operating an autonomous vehicle.
[0007] According to the application, this object is achieved by a method for creating a road model having the features of claim 1 and by a method for operating an autonomous vehicle having the features of claim 8.
[0008] Advantageous design forms of the application are the subject matter of the dependent claims.
[0009] The method for creating a road model in a predetermined environmental region of a vehicle according to the application is characterized in that
[0010] - a base model of an artificial intelligence-based road for a specific geographical region is determined, trained with data of the geographical region and stored in the vehicle,
[0011] - a plurality of additional / additive artificial intelligence-based submodels for the geographical region are created, which each have local applicability for a limited local region of the geographical region and are trained using only data from the respective local region,
[0012] - the additional submodels are stored in a computing unit external to the vehicle, for example in a backend server,
[0013] - during a driving operation of the vehicle, before reaching an environmental region of the vehicle, the additional submodel applicable to the environmental region is called from the computing unit and transferred to the vehicle,
[0014] - environmental data of the environmental region are detected by an environmental detection sensor system of the vehicle, and
[0015] - a road model of the environmental region is created using the base model and the submodels only on the basis of the detected environmental data,
[0016] wherein, when creating the road model, for a plurality of road sections of the road respectively:
[0017] - an input vector determined from the environmental data and describing a course of the respective road section is provided as input data to the base model and the additional submodel respectively,
[0018] - a base output vector describing the road section in the road model is generated with the base model,
[0019] - a sub output vector describing the road section in the road model is generated with the additional submodel, and
[0020] - the base output vector and the sub output vector are added to form a total output vector of the road model.
[0021] The road model is understood here, for example, as a model of the static world, which does not contain data of moving objects. In this case, the road model contains at least a representation of the course of the so-called ego lane, i.e. the lane in which the vehicle is located. The topology of other lanes, for example the topology of all lanes in the vehicle's environment region, and / or the course of stop lines, boundaries and / or curbs can also be contained in the road model. In particular, the road model is created in the vehicle coordinate system from the perspective of the environment detection sensor system.
[0022] By simply superimposing the base output vector and the sub-output vector to form the total output vector, the application allows the road model to be created particularly simply and reliably.
[0023] By means of the present method, the road model can be created in a particularly advantageous manner without the use of a high-resolution map. By means of the present method, by using a base model trained on the basis of extensive data, for example according to "Liao, B. et al.: MapTR - Structured Modeling and Learning for Online Vectorized HD Map Construction; see: arXiv:2208.14437 v2 [cs.CV], 30 January 2023", and by means of additional sub-models, the road model can be created in the driving operation of the vehicle without the need for a high-resolution map, using only the environment data detected by the environment detection sensor system. However, the base model and the sub-models can be trained using a high-resolution map.
[0024] In this case, the method enables the algorithm for generating the road model to be adapted in a particularly simple manner to local peculiarities in the respective environment region of the vehicle. For example, in this case, the additional sub-models are replaced only by additional sub-models that are applicable to the new location or new environment region. This in particular means that the respective additional sub-models are used temporarily while the vehicle is located in the respective environment region. Since the training of the sub-models is carried out using only data from the respective locally assigned region, the training complexity of the sub-models is lower compared to the training of the base model, and the sub-models contain less data than the base model. This makes it possible to carry out a location-specific update of the algorithm in the form of the sub-models with only a small amount of data being transmitted from the computing unit to the vehicle. For this transmission, it is possible in a particularly advantageous manner to use existing and known infrastructure, for example already used for transmitting high-resolution maps to vehicles, without any modifications being necessary. Training the sub-models with a small amount of data also allows the sub-models to be created quickly and updated quickly if necessary.
[0025] Thus, by means of the present method, end-to-end road model perception can be seamlessly integrated in a vehicle in a particularly advantageous manner. In comparison to map-based approaches, the present method essentially covers the functionality of vehicle localization and road model fusion, thereby reducing the complexity of the architecture required for performing the method. In contrast to conventional end-to-end road model perception approaches, in which the potential of high-resolution maps to provide content that cannot be recognized by the environment detection sensor system and the stability of the externally aggregated map data is lost, the present method enables such recognition while at the same time achieving high stability. By using additional sub-models for local specialization / spezialisierung, the recognition performance of the method is thus significantly improved.
[0026] According to one possible design of the method, raw data of the environment detection sensor system are used as environment data. This makes it possible for the data detected by the environment detection sensor system to be used directly as environment data without further processing.
[0027] According to another possible design of the method, the base model and / or the sub-models are trained by an end-to-end learning process. By such a learning process, all intermediate steps required for achieving the desired result can be integrated into one unified base model. The base model can be machine-trained using artificial neural networks.
[0028] According to another possible design of the method, an additional sub-model applicable to the environment region in which the vehicle is to be driven next is called up from the computing unit and transmitted to the vehicle after leaving the environment region in which the vehicle is currently driven. By means of the base model and the sub-model, a road model of the environment region in which the vehicle is to be driven next is created from the detected environment data. This makes it possible to minimize the number of sub-models stored in the vehicle and the required storage space resulting therefrom.
[0029] According to another possible design of the method, an additional sub-model applicable to the environment region in which the vehicle is currently driven is deleted in the vehicle after leaving the environment region. This also makes it possible to minimize the number of sub-models stored in the vehicle and the required storage space resulting therefrom.
[0030] According to another possible design of the method, an additional sub-model applicable to the environment region in which the vehicle is currently driven is transmitted to the central computing unit after leaving the environment region. This makes it possible to further train the sub-model while driving the environment region, and then, when the vehicle or another vehicle subsequently drives the environment region, the trained sub-model can be transmitted from the computing unit to the vehicle or the other vehicle.
[0031] According to a further possible design of the method, when creating the road model from the detected environmental data and by means of the respective additional sub-model, the presence and location of fixed / static environmental features in the respective environmental region are learned. In a subsequent drive of the environmental region, the learned fixed environmental features are implicitly present if they are not present in the environmental data detected during this subsequent drive of the environmental region. If the algorithm implementing the method learns, for example, by means of an additional model trained specifically for a certain urban region of a city, that there is always an environmental feature designed as a parking strip behind the zebra crossing at a roundabout in this urban region, the algorithm can detect this parking strip even if the environmental detection sensor system fails to detect it, for example, due to an occlusion. That is, in this case, the parking strip is present, is not detected by the environmental detection sensor system, but is recognized by the algorithm. By taking into account this so-called hallucination, the reliability of the road model created by the present method can be increased and the road model can be generated completely and precisely even in the case of occlusions of the field of view of the environmental detection sensor system. That is, the superior recognition performance of an end-to-end architecture based on artificial intelligence is achieved, but the road model estimation exceeds the range visible to the environmental detection sensor system.
[0032] According to a further possible design of the method, the geographical region is used as the geographical whole of a national territory, a continent or the earth. Other finite regions can also be used as the geographical region.
[0033] According to a further possible design of the method, the current position of the vehicle in the geographical region is determined by means of a satellite-supported positioning system. This makes it possible to reliably and precisely determine the position of the vehicle, for which purpose the positioning system already present in the vehicle can be used.
[0034] In a method for operating an autonomous vehicle, according to the present application, the road model generated by the above-described method and the object and / or obstacle data determined on the basis of the environmental data are used in a planning and control module for planning a target trajectory along which the vehicle is to be autonomously guided.
[0035] By using a road model that does not need to be created with high resolution maps in the driving operation, on the one hand, the vehicle can be located more easily and, on the other hand, there is no risk of the maps data becoming outdated in relation to the detected environmental data. Furthermore, the algorithm for generating the road model can be adapted to local peculiarities in the respective environmental region of the vehicle in a particularly simple manner. By invoking additional sub-models that are thus always up to date, a target trajectory can be planned for an autonomous driving operation of the vehicle in a particularly reliable and precise manner. The driving operation can thus be carried out particularly precisely and reliably.
[0036] According to one possible design of the method, adjustment commands are generated for actuators of the vehicle in order to guide the vehicle along the planned nominal trajectory in a longitudinal and lateral adjustment manner, on the basis of the road model and on the basis of the object and / or obstacle data. This makes reliable automatic execution of the driving operation possible.
[0037] According to another possible design of the method, adjustment commands are generated for acceleration actuators and / or deceleration actuators and / or steering actuators of the vehicle. BRIEF DESCRIPTION OF DRAWINGS
[0038] Embodiments of the application are explained in more detail below with reference to the drawings.
[0039] wherein:
[0040] Figure 1 a block diagram of an apparatus for operating an automated driving vehicle according to the prior art is schematically shown;
[0041] Figure 2 a diagram of an image of an environment region of a vehicle with superimposed road model is schematically shown;
[0042] Figure 3 a block diagram of an apparatus for operating an automated driving vehicle is schematically shown;
[0043] Figure 4 a block diagram of a base model and additional sub-models in the case of processing of an input vector into a total output vector is schematically shown, and
[0044] Figure 5 a training of additional sub-models is schematically shown. DETAILED DESCRIPTION
[0045] Corresponding parts are denoted by the same reference numerals in all the figures.
[0046] Figure 1 a block diagram of an apparatus 1 for operating an automated driving vehicle 2 according to the prior art is shown. Figure 2 The block diagram of the apparatus 1 of the automated driving vehicle 2 is shown in more detail in Fig. 1.
[0047] The automated, in particular highly automated or self-driving vehicle 2 comprises an environment detection sensor system 3, by means of which, inter alia, optical lane recognition for providing a road model SM2 is carried out. For this purpose, fusion of environment data UD detected by various sensors of the environment detection sensor system 3 in an environment region of the vehicle 2 is carried out by a fusion unit 7. Furthermore, object and / or obstacle data OD of objects and obstacles located in the environment region are determined from the fused environment data UD.
[0048] Since this optical lane recognition is limited in its performance and range of action and is easily obscured, for the execution of an automatic driving operation, the known concept additionally uses high-resolution road models SM2 provided by a computing unit 4 external to the vehicle, in particular a so-called back end, which are based on map data KD. Figure 5 in order to improve the perception-based road model SM2 by means of the map data KD.
[0049] For this purpose, the position POS of the vehicle 2 is determined by means of a satellite-supported positioning system 6. For an environmental region located within a predetermined circumference around this position POS, a map section representing this environmental region is requested from the computing unit 4 by means of a communication unit 8 and transmitted from the computing unit 4 to the vehicle 2 in the form of map data KD. Figure 5
[0050] The map data KD and the locally determined localization features LM determined by means of sensor fusion are supplied to a processing unit 9, by means of which an exact and precise vehicle-local localization is carried out in the map section. In this case, the map data KD are registered from the own perspective of the vehicle 2, i.e. from the perspective of the environmental detection sensor system 3, and a map-based road model SM1 is created.
[0051] Subsequently, the perception-based road model SM2 and the map-based road model SM1 are fused in a further fusion unit 10 to form a road model SM.
[0052] The road model SM and the object and / or obstacle data OD generated in this way are then supplied to a planning and control module 11, which plans a target trajectory T along which the vehicle 2 is to be automatically guided. In particular, in accordance with the target trajectory T, adjustment commands are generated for actuators of the vehicle 2 in order to guide the vehicle 2 along the planned target trajectory T in a longitudinal and lateral control manner.
[0053] The above-described use of map data KD for the creation of a road model SM has the disadvantage, on the one hand, that in order to be able to fuse the map data KD with the environmental data UD detected by means of the environmental detection sensor system 3, a great deal of effort is required to achieve the required high-precision localization of the vehicle 2, on the other hand, the map data KD are only slowly adapted to changes in the real world and are therefore often outdated due to the long update release process. Figure 5
[0054] Figure 2 An image B of an environmental region of the vehicle 2 with superimposed road model SM is shown.
[0055] The road model SM is a static world model and does not contain data on moving objects. The road model SM must at least contain a representation of the lane orientation SV of the so-called Ego lane (i.e., the lane where vehicle 2 is located). The road model SM may also contain the topology of other lanes, such as the topology of all lanes in vehicle 2's environment area, and / or the orientation of stop lines, boundary RBs, and / or curbs.
[0056] In the road model SM shown, lane directions SV are represented by dotted lines. Boundaries RB are represented by solid lines, and prohibited and / or uncrossable areas BE are represented by dashed lines. Furthermore, the planned trajectory T of vehicle 2 is represented by dashed lines. However, this planned trajectory T is not a component of the road model SM.
[0057] Figure 3 A block diagram is shown of a possible embodiment of a device 1 for operating an autonomous vehicle 2, by means of which a possible embodiment of a method according to the invention for operating an autonomous vehicle 2 is performed.
[0058] To eliminate the use of high resolution Figure 5 To address the aforementioned drawbacks in creating a road model SM, and to determine the appropriate approach for... Figure 5 The model GM, a road based on artificial intelligence, is shown in detail in the diagram for a specific geographic region REG. This model is trained using data from the geographic region REG and stored in vehicle 2. The geographic region REG can be a national territory, a continent, or the entire Earth. Furthermore, other limited areas can also be used as geographic region REGs.
[0059] The base model GM is trained on the basis of extensive data such that the base model GM is able to create a road model SM with as little distortion or illusion as possible that covers various scenarios. For example, this training takes place using the so-called SafeAD method according to https: / / www.safead.de / (as of September 7, 2023, 11:03 a.m.) and https: / / www.autonomousvehicleinternational.com / news / expo-news-safead-presents-offline-hd-maps-and-3d-perception-system.html (as of September 7, 2023, 11:08 a.m.). In this case, artificial intelligence-based algorithms are used for continuous environmental perception during autonomous driving of the vehicle 2, in which object recognition and road model estimation are carried out. Alternatively or additionally, the training of the base model GM can also take place in accordance with "Liao, B. et al.: MapTR - Structured Modeling and Learning for Online Vectorized HD Map Construction; see: arXiv: 2208.14437 v2 [cs.CV], January 30, 2023".
[0060] In addition, for the geographical region REG, a plurality of additional, artificial intelligence-based sub-models TM1 to TMn of the road are created, which are each locally applicable for a limited local region OB of the geographical region REG and are trained using only data from the respective local region OB. These trained additional sub-models TM1 to TMn are stored in the computing unit 4 external to the vehicle. The training of the additional sub-models TM1 to TMn takes place, for example, analogously to the training of the base model GM, but specifically for the limited local region OB. Figure 5
[0061] The additional sub-models TM1 to TMn are designed, for example, according to the so-called Low Rank Adaptation (short: LoRA). Due to the limited range of the local region OB, the sub-models TM1 to TMn can be trained very easily and quickly. With the aid of the sub-models TM1 to TMn, the base model GM can be expanded without changing the base model GM. Thus, by downloading the dedicated additional sub-models TM1 to TMn from the computing unit 4 according to the geographical reference position POS of the vehicle 2 and inserting them into the base model GM, the sub-models can be dynamically activated and deactivated.
[0062] This means that, during the driving operation of the vehicle 2, before reaching an environmental region of the vehicle 2, the applicable additional sub-models T1 to Tm for this environmental region are called from the computing unit 4 by the communication unit 8 and transmitted to the vehicle 2. Using the base model GM and the transmitted sub-models TM1 to TMn, a road model SM of this environmental region is created only from the environmental data UD detected by the environmental detection sensor system 3, for which, during the driving operation of the vehicle 2, the environment of the vehicle 2 is permanently detected in the form of environmental data UD by means of the environmental detection sensor system 3 and the position POS of the vehicle 2 is detected by means of the positioning system 6. For example, raw data of the environmental detection sensor system 3 are used as environmental data UD.
[0063] By means of the local specialization achieved by means of the sub-models TM1 to TMn, the recognition performance is significantly improved. Furthermore, the use of the sub-models TM1 to TMn makes it possible to at least supplement and extend the part of the road model SM in regions which cannot be detected by the environmental detection sensor system 3. This is achieved by "hallucinating" the environmental features of these regions from the learned context using the respective sub-models TM1 to TMn. In most applications, this hallucination is a negative effect due to the insufficient breadth of the training data TD, there is a risk of producing plausible but incorrect results. In the present application, however, the hallucination is exploited, here the road structure known to exist at a particular location is deliberately paid attention to. It is also possible to assign lower weights to the additional sub-models TM1 to TMn; to obtain a better perception of reality.
[0064] After leaving the environmental region in which the vehicle 2 is currently driving, the additional sub-models TM1 to TMn applicable to the environmental region left are deleted in the vehicle 2. It is also possible to further train the sub-models TM1 to TMn during their use in the vehicle 2, the further trained sub-models TM1 to TMn being transferred to the central computing unit 4 after the end of use.
[0065] The generated road model SM and the object and / or obstacle data OD determined by means of the model 12 trained end-to-end are then provided to the planning and control module 11, which plans a target trajectory T along which the vehicle 2 is to be automatically guided. In particular, from the target trajectory T, adjustment commands are generated for actuators of the vehicle 2 in order to guide the vehicle 2 along the planned target trajectory T in a longitudinal and lateral control manner.
[0066] Figure 4 A block diagram of the base model GM and the additional sub-models TM1 to TMn in the case of processing of the input vector EV into the total output vector GAV is shown. This processing is carried out, for example, in accordance with the low-rank adaptation described above.
[0067] In this case, when creating the road model SM, the input vectors EV determined from the environment data UD and describing the course of the respective road section are provided as input data to the base model GM and the additional sub-models TM1 to TMn used in each case, respectively, for the respective road section of the road.
[0068] The base output vectors GRAV describing the road sections in the road model SM are generated using the base model GM.
[0069] The sub-output vectors TAV describing the road sections in the road model SM are generated using the additional sub-models TM1 to TMn.
[0070] The base output vectors GRAV and the sub-output vectors TAV are added to form the total output vectors GAV of the road model SM.
[0071] This procedure is carried out for all road sections of the components of the road model SM.
[0072] Figure 5 The training of the additional sub-models TM1 to TMn is shown schematically.
[0073] In this case, a specific local region OB is selected from the region REG, and the additional sub-models TM1 to TMn are trained from the data basis 13 using the training data TD limited for this local region OB. The base model GM remains unchanged here.
Claims
1. A method for creating a road model (SM) in a predetermined environmental area of a vehicle (2), Its features are, - Determine the base model (GM) of the road based on artificial intelligence for a specific geographic region (REG), train it with data from that geographic region (REG) and store it in the vehicle (2), - Create multiple additional, AI-based road sub-models (TM1 to TMn) for the geographic region (REG). These sub-models are locally applicable to a limited local area (OB) of the geographic region (REG) and are trained using only data from the respective local area (OB). - The additional sub-models (TM1 to TMn) are stored in the computing unit (4) outside the vehicle. - During the operation of the vehicle (2), before reaching an environmental area of the vehicle (2), the additional sub-models (TM1 to TMn) applicable to that environmental area are called from the computing unit (4) and transmitted to the vehicle (2). - The environmental data (UD) of the environmental area is detected by the environmental detection sensor system (3) of the vehicle (2), and - Using the base model (GM) and sub-models (TM1 to TMn), a road model (SM) is created for this environmental region based solely on the detected environmental data (UD), where, When creating a road model (SM), the multiple road segments are handled separately: - The input vector (EV) determined from the environmental data (UD) and describing the orientation of the corresponding road segment is provided as input data to the base model (GM) and the additional sub-models (TM1 to TMn). - Generate basis output vectors (GRAVs) describing road segments in the road model (SM) using the base model (GM). - Generate sub-output vectors (TAVs) describing road segments in the road model (SM) using additional sub-models (TM1 to TMn), and - The base output vector (GRAV) and the sub-output vector (TAV) are added together to form the total output vector (GAV) of the road model (SM).
2. The method according to claim 1, Its features are, The raw data from the environmental monitoring sensor system (3) is used as environmental data (UD).
3. The method according to claim 1 or 2, Its features are, The base model (GM) and / or sub-models (TM1 to TMn) are trained through an end-to-end learning process.
4. The method according to any one of the preceding claims, Its features are, Before leaving the current driving environment area of the vehicle (2), the additional sub-models (TM1 to TMn) applicable to the next driving environment area of the vehicle (2) are called from the computing unit (4) and transmitted to the vehicle (2). The road model (SM) of the environment area is created based on the detected environmental data (UD) using the base model (GM) and the sub-models (TM1 to TMn).
5. The method according to any one of the preceding claims, Its features are, After leaving the current driving environment area of the vehicle (2), the additional sub-models (TM1 to TMn) applicable to the left environment area are deleted from the vehicle (2) and / or transferred to the central computing unit (4).
6. The method according to any one of the preceding claims, Its features are, - When creating a road model (SM) based on the detected environmental data (UD) and with the aid of corresponding additional sub-models (TM1 to TMn), the existence and location of fixed environmental features within the corresponding environmental region are learned, and - When driving through the environment area subsequently, the learned fixed environmental features are implicitly present if they are not present in the environmental data (UD) detected during the subsequent driving of the environment area.
7. The method according to any one of the preceding claims, Its features are, Use the geographical whole of a country’s territory, continent, or Earth as a geographic region (REG).
8. A method for operating an autonomous vehicle (2), wherein, The road model (SM) generated by the method according to any one of the preceding claims and the object and / or obstacle data (OD) determined based on environmental data (UD) are used in the planning and control module (11) to plan the rated trajectory (T) of the automatically guided vehicle (2) along it.
9. The method according to claim 9, Its features are, Based on the road model (SM) and object and / or obstacle data (OD), adjustment commands are generated for the actuators of the vehicle (2) to guide the vehicle (2) along the planned nominal trajectory (T) in a longitudinal and lateral adjustment manner.