Method for creating road models and operating autonomous vehicles
Patent Information
- Application Number
- JP2026517272
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-09-21
- Filing Date
- 2024-09-04
- Publication Date
- 2026-10-14
Smart Images
Figure 2026534153000001_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present invention relates to a method for creating a road model in a defined area around a vehicle.
[0002] The present invention further relates to a method for operating an autonomous driving vehicle.
BACKGROUND ART
[0003] High-definition maps (HD maps) are used for the operation of autonomous driving vehicles, particularly highly automated driving vehicles or autonomous driving vehicles. These maps provide extensive and precise surrounding information related to driving scenes, and are used to generate driving plans for autonomous driving vehicles.
[0004] When creating a road model using a high-definition map, it is necessary to register a map converted from a bird's-eye view in the global coordinate system to the vehicle coordinate system, which is the viewpoint of the surrounding environment recognition sensor, in the vehicle. In this case, high-precision vehicle positioning is required so that the map data can be fused with the surrounding data recognized by the surrounding environment recognition sensor. The problem in this case is that the map becomes outdated, and the map data contained therein no longer matches the recognized surrounding data. With the improved availability of sensor technology and the rapid advances in artificial intelligence, the efficiency of information acquisition by surrounding environment recognition sensors in so-called end-to-end processing has been improved, and a certain quality level in road map creation should be achievable even with surrounding environment recognition alone. Here, so-called generative artificial intelligence schemes are used for end-to-end road model perception. Such schemes include architectures such as so-called object detection transformers.
[0005] A structured end-to-end transformer for building vectorized HD maps online is disclosed in "Liao, B. et al.: MapTR - Structured Modeling and Learning for Online Vectorized HD Map Construction; In: arXiv: 2208.14437v2 [cs.CV], 30. January 2023". This employs a unified permutation-equal-transformation modeling technique, where map elements are modeled as point sets by equivalent permutation groups that describe the shape of the map elements and stabilize the learning process. A hierarchical query embedding scheme is used to flexibly encode the structured map information, thereby performing hierarchical bipartite matching for learning the map elements. [Overview of the Initiative] [Problems that the invention aims to solve]
[0006] The present invention aims to provide a novel method for creating a road model and a novel method for operating an autonomous vehicle.
[0007] This objective is achieved by a method for creating a road model having the features described in claim 1, and a method for operating an autonomous vehicle having the features described in claim 8, based on the present invention.
[0008] Advantageous embodiments of the present invention are subject to the dependent claims.
[0009] A method for creating a road model in a predetermined surrounding area around a vehicle is based on the present invention. - For a specific geographical area, an AI-based basic model for roads is detected, trained using geographical area data, and stored in the vehicle. -For that geographical region, multiple AI-based additional submodels related to roads are created, each locally effective for a limited local area within that geographical region, and each is trained using only the data for that local area. -This additional submodel is stored in an external computing unit, such as a backend server. -During vehicle operation, before the vehicle reaches a particular peripheral region, an additional submodel valid for that peripheral region is invoked from the computing unit and transmitted to the vehicle. -The vehicle's surrounding environment recognition sensor is used to recognize environmental data about the surrounding area. - Using the basic model and sub-models, a road model of the surrounding area is created based solely on recognized environmental data. When creating a road model for each of the multiple sections of a road, - Input vectors, detected from environmental data and describing the trajectory of each section, are supplied as input data to the basic model and additional submodels, respectively. -Using the basic model, a basic output vector describing the section within the road model is formed. - An additional sub-model is used to form a partial output vector that describes the section within the road model. -To form the total output vector of the road model, the basic output vector and the partial output vector are summed together. It is characterized by the following:
[0010] In this case, the road model is understood to be a model of a static world that does not include data on moving objects, for example. This road model includes, for example, at least one representation of the view of the so-called ego lane, i.e., the lane in which the vehicle is traveling. This road model may also include, for example, the topology of other lanes, such as all lanes in the area surrounding the vehicle, and / or the trajectories of stop signs, boundary markers, and / or curbs. In particular, the road model is created in the vehicle's coordinate system from the perspective of the surrounding environment perception sensor.
[0011] According to the present invention, a road model can be created particularly easily and reliably by simply adding the basic output vector and the partial output vector together to form a total output vector.
[0012] This method offers a particularly advantageous approach to creating road models without using high-resolution maps. For example, by using a basic model trained on a wide range of data, such as "Liao, B. et al.: MapTR - Structured Modeling and Learning for Online Vectorized HD Map Construction; In: arXiv: 2208.14437v2 [cs.CV], 30. Januar 2023," and by using an additional sub-model based on environmental data recognized solely using surrounding environment recognition sensors, it becomes possible to create road models while a vehicle is in motion without high-resolution maps. However, high-resolution maps can be used to train the basic and sub-models.
[0013] In this case, this method makes it possible to adapt the algorithm for generating road models to the local characteristics of each surrounding area of a vehicle, particularly in a simple manner. For example, these additional submodels are replaced only by additional submodels applied to new locations or new surrounding areas. This means that each additional submodel is used temporarily, specifically only while the vehicle is in that surrounding area. Since the training of submodels is performed using data from their respective local ranges, this training is less complex than training the base model, and the amount of data included in the submodels is less than that of the base model. This means that when updating the algorithm locally in the form of submodels, only less data needs to be transmitted from the computing unit to the vehicle. This transmission can utilize existing, well-known infrastructure, such as that already used to transmit high-resolution maps to vehicles, in a particularly advantageous manner without requiring modification. Furthermore, training submodels with less data allows these submodels to be created quickly and updated quickly as needed.
[0014] Therefore, this method makes it possible to seamlessly integrate end-to-end road model perception into vehicles in a particularly advantageous manner. Unlike map-based approaches, the complexity of the architecture required to implement this method is reduced because the functions of vehicle localization and road model fusion are basically covered. Conventional approaches to end-to-end road model perception lose the potential of high-resolution maps that provide what surrounding environment perception sensors cannot perceive, as well as the stability of externally aggregated map data. Conversely, this method enables such perception while simultaneously maintaining high stability. Therefore, the recognition performance of this method is greatly improved by local specificity using additional submodels.
[0015] According to a possible embodiment of this method, raw data from a surrounding environment recognition sensor is used as environmental data. As a result, the data recognized using the surrounding environment recognition sensor can be used directly as environmental data in its raw form without any additional processing.
[0016] According to further possible embodiments of this method, the base model and / or submodels are trained using an end-to-end learning process. Such a learning process makes it possible to integrate all the intermediate steps necessary to achieve the desired result into a unified base model. In this case, the base model can be trained mechanically using an artificial neural network.
[0017] According to a further possible embodiment of this method, before the vehicle leaves the surrounding area it is currently traveling through, an additional submodel effective for the next surrounding area the vehicle will pass through is called from the computing unit and transmitted to the vehicle. Using the basic model and submodels, a road model of the surrounding area is created from the recognized environmental data. This makes it possible to minimize the number of submodels stored in the vehicle, and consequently, minimize the memory space required.
[0018] According to a further possible embodiment of this method, after the vehicle leaves the surrounding area in which it is currently traveling, any additional submodels that are valid for the left surrounding area are removed from the vehicle. This also minimizes the number of submodels stored in the vehicle, and consequently, minimizes the required memory space.
[0019] According to a further possible embodiment of this method, after a vehicle leaves the surrounding area it is currently traveling through, an additional submodel effective for the left surrounding area is transmitted to a central computing unit. This allows the submodel to be further learned while traveling through the surrounding area, and this learned submodel can then be transmitted from the computing unit to the vehicle or another vehicle the next time they travel through that area.
[0020] According to further possible embodiments of this method, when creating a road model from recognized environmental data and their respective additional submodels, the presence and location of immovable environmental features in the relevant surrounding area are learned. When driving through this surrounding area again, if the previously learned immovable environmental features are not found in the surrounding data acquired during the re-driving of the area, the presence of these features is estimated. If the algorithm performing this method learns, for example, through an additional submodel specifically trained for a particular area of a city, that there is always an environmental feature formed as a hard shoulder behind a roundabout behind a pedestrian crossing in that area, the algorithm can detect this hard shoulder even if the surrounding environment recognition sensor cannot recognize it for reasons such as an obstruction. In other words, in this case, the hard shoulder exists and is recognized by the algorithm even if it is not recognized by the surrounding environment recognition sensor. By considering this type of situation, also known as hallucination, the reliability of road model creation performed by this method is increased, and a complete and accurate road model can be generated even when the field of view of the surrounding environment recognition sensor is obstructed. In other words, it becomes possible to predict road models that go beyond the range visible to the surrounding environment recognition sensor, while achieving the superior recognition performance of an AI-based end-to-end architecture.
[0021] According to further possible embodiments of this method, a national territory, a continent, or the entire globe may be used as the geographical area. Other limited areas may also be used as the geographical area.
[0022] According to a further possible embodiment of this method, the current location of a vehicle in a geographical area is detected using a satellite-assisted positioning system. This enables reliable and accurate vehicle location detection, and for this purpose, a positioning system already installed on the vehicle can be used.
[0023] In one operating method for an autonomous driving vehicle, according to the present invention, a road model generated using the aforementioned method, and object data and / or obstacle data detected on the basis of environmental data are used in a planning and control module, whereby a target trajectory is planned, and the vehicle is automatically controlled along the target trajectory.
[0024] By using the road model created without a high-definition map during driving operations, on the one hand, vehicle positioning is facilitated, and on the other hand, there is no risk of using outdated map data that no longer matches the perceived environmental data. Furthermore, it becomes possible to adapt the algorithm for generating the road model to local characteristics in the respective surrounding area of the vehicle in a particularly simple manner. As a result, by invoking an additional submodel that is always maintained in an up-to-date state, reliable and accurate target trajectory planning for autonomous driving of the vehicle is enabled. As a result, this driving operation can be performed in a particularly accurate and reliable manner.
[0025] According to a further possible embodiment of the present method, operation commands for actuators of the vehicle are generated in accordance with the road model and in accordance with the object data and / or obstacle data, whereby the vehicle is controlled in the longitudinal direction and the lateral direction along the planned target trajectory. This enables highly reliable automation of driving operations.
[0026] According to a further possible embodiment of the present method, operation commands for an acceleration actuator and / or a deceleration actuator and / or a steering actuator of the vehicle are generated.
[0027] In the following, embodiments of the present invention will be described in detail based on the drawings. [Brief Description of the Drawings]
[0028] [Figure 1] 1 is a schematic block diagram of an apparatus for operating an autonomous driving vehicle according to the prior art. [Figure 2] FIG. 2 is a schematic diagram of a surrounding area of a vehicle on which a road model is superimposed. [Figure 3] This is a schematic block diagram of the equipment used to operate an autonomous vehicle. [Figure 4] This is a schematic block diagram of the basic model and additional submodels for processing input vectors to create a summed output vector. [Figure 5] This is a schematic diagram of the learning process for additional submodels. [Modes for carrying out the invention]
[0029] In all of the figures, corresponding parts are denoted by the same reference numeral.
[0030] Figure 1 shows a schematic block diagram of the device 1 for operating the conventional autonomous vehicle 2, which is shown in detail in Figure 2.
[0031] Autonomous vehicles, particularly highly automated vehicles, or autonomous vehicles 2, include a surrounding environment recognition sensor 3, which is used, in particular, to perform optical lane recognition and generate a road model SM2. To this end, a fusion unit 7 is used to fuse environmental data UD recognized by various sensors of the surrounding environment recognition sensor 3 in the area surrounding the vehicle 2. Furthermore, object data and / or obstacle data OD regarding objects and obstacles present in the surrounding area are detected from the fused environmental data UD.
[0032] Because such optical lane recognition has limited performance and range and is vulnerable to obstacles, a well-known concept for implementing an autonomous driving mode is to additionally use a high-resolution map 5 provided from an external computing unit 4, specifically a so-called backend, thereby improving the perception-based road model SM2 using map data KD.
[0033] For this purpose, the vehicle's position POS is determined using, for example, a positioning system 6 formed with satellite assistance. For the surrounding area within a predetermined range centered on this position POS, the computing unit 4 requests a map fragment of the map 5 that depicts this surrounding area using the communication unit 8, and transmits it from the computing unit 4 to the vehicle 2 in the form of map data KD.
[0034] The map data KD and positioning features LM identified by sensor fusion are supplied to the processing unit 9, which performs accurate and precise vehicle positioning within the map segment. In this case, the map data KD is registered from the vehicle's own viewpoint, i.e., from the viewpoint of the surrounding environment recognition sensor 3, and a map-based road model SM1 is created.
[0035] Next, the perception-based road model SM2 and the map-based road model SM1 are fused in another fusion unit 10 to form a single road model SM.
[0036] Next, the road model SM generated in this manner, along with object data and / or obstacle data OD, is supplied to the planning and control module 11, which plans a target trajectory T and controls the vehicle 2 along it. Specifically, according to the target trajectory T, operation commands are generated for the actuators of the vehicle 2, thereby controlling the vehicle 2 in the longitudinal and lateral directions along the planned target trajectory T.
[0037] The drawbacks of creating a road model SM using the aforementioned map data KD are, on the one hand, that high-precision positioning of vehicle 2 is required to fuse the map data KD with the environmental data UD recognized using the surrounding environment recognition sensor 3, which requires a lot of effort to achieve; on the other hand, the map 5 has a long update and publication process, which can lead to delays in responding to changes in the real world and often results in it becoming outdated.
[0038] Figure 2 is image B of the area surrounding vehicle 2 with the road model SM superimposed.
[0039] In this case, the road model SM is a static world model that does not include data on moving objects, for example. The road model SM includes at least one representation of the so-called ego lane SV, i.e., the lane in which vehicle 2 is traveling. The road model SM may also include the topology of other lanes, such as all lanes in the area surrounding vehicle 2, and / or the trajectories of stop signs, boundary signs RB, and / or curbs.
[0040] In the illustrated road model SM, lanes SV are shown as dotted lines. Boundary markers RB are shown as solid lines, and no-entry and / or impassable areas BE are shown as dashed lines. Furthermore, the planned trajectory of vehicle 2 is shown as a dashed line. However, the planned trajectory T is not a component of the road model SM.
[0041] Figure 3 shows a schematic block diagram of a possible embodiment of the device 1 for operating the autonomous vehicle 2, and a possible embodiment of the method according to the present invention for operating the autonomous vehicle 2 is carried out using this device 1.
[0042] To overcome the aforementioned drawbacks of using high-resolution map 5 for creating road model SM, an artificial intelligence-based basic road model GM is detected for specific geographical regions REG, as detailed in Figure 5, learned from the data of the geographical regions REG, and stored in vehicle 2. In this case, the geographical regions REG may be national territories, continents, or the entire globe. Other limited areas can also be used as geographical regions REG.
[0043] The base model GM is trained on a wide range of data, enabling the creation of a road model SM that minimizes distortion and hallucination across diverse scenarios. For example, this training is performed using the so-called SafeAD method, as described at https: / / www.safead.de / (accessed around 11:03 AM on September 7, 2023) and https: / / www.autonomousvehicleinternational.com / news / expo-news-safead-presents-offline-hd-maps-and-3d-perception-system.html (accessed around 11:08 AM on September 7, 2023). Here, an artificial intelligence-based algorithm is used for consistent perception of the surrounding environment during autonomous driving of vehicle 2, performing object recognition and road model prediction. Alternatively or additionally, training the basic GM model can also be performed according to "Liao, B. et al.: MapTR - Structured Modeling and Learning for Online Vectorized HD Map Construction; In: arXiv: 2208.14437v2 [cs.CV], 30. January 2023".
[0044] Furthermore, for each geographical region REG, multiple additional, AI-based road submodels TM1 to TMn are created, each locally effective for a limited local area OB (shown in detail in Figure 5) of the geographical region REG, and are trained using only the data for their respective local area OBs. These trained additional submodels TM1 to TMn are stored in the external computing unit 4. The training of the additional submodels TM1 to TMn is carried out in a similar manner to, for example, the training of the basic model GM, but is specialized for the limited local area OB.
[0045] The additional submodels TM1 to TMn are designed according to, for example, so-called low-rank adaptation (abbreviation: LoRA). In this case, the submodels TM1 to TMn can be learned very easily and quickly based on a limited local range OB. By using the submodels TM1 to TMn, the base model GM can be extended without modification. That is, the submodels can be dynamically enabled and disabled by downloading special additional submodels TM1 to TMn from the computing unit 4 and adding them to the base model GM according to the georeferenced location POS of vehicle 2.
[0046] In other words, during the operation of vehicle 2, before vehicle 2 reaches a particular surrounding area, the communication unit 8 is used to call additional submodels T1 to Tn valid for that surrounding area from the calculation unit 4 and transmit them to vehicle 2. Using the basic model GM and the transmitted submodels TM1 to TMn, a road model SM for the surrounding area is created based solely on the environmental data UD recognized by the surrounding environment recognition sensor 3. For this reason, during the operation of vehicle 2, the area around vehicle 2 is continuously recognized as environmental data UD by the surrounding environment recognition sensor 3, and the position POS of vehicle 2 is detected using the position identification system 6. In this case, raw data from the surrounding environment recognition sensor 3, for example, is used as environmental data UD.
[0047] Recognition capabilities are significantly improved through localized specialization achieved using submodels TM1-TMn. Furthermore, the use of submodels TM1-TMn allows for the supplementation and extension of at least a portion of the road model SM in areas that cannot be recognized by the surrounding environment recognition sensor 3. This is made possible by "hallucination" of the surrounding features of this region from the learned context using the corresponding submodels TM1-TMn. In most application cases, this hallucination is an undesirable effect caused by insufficient training data TD, and carries the risk of producing plausible but erroneous results. Here, conversely, hallucination is utilized, intentionally focusing on road structures known to exist in specific locations. It is also possible to weight the additional submodels TM1-TMn more heavily, thereby improving live perception.
[0048] After vehicle 2 leaves the surrounding area it is currently traveling through, any additional submodels TM1 to TMn that are effective in the left surrounding area are removed from vehicle 2. It is also possible that submodels TM1 to TMn can be further learned during their use in vehicle 2 and then transmitted to the central computing unit 4 in their further learned state after their use.
[0049] Next, the generated road model SM and object data and / or obstacle data OD detected using the end-to-end learned model 12 are sent to the planning and control module 11, which plans the target trajectory T and controls the vehicle 2 along it. Specifically, according to the target trajectory T, operation commands are generated for the actuators of the vehicle 2, thereby controlling the vehicle 2 in the longitudinal and lateral directions along the planned target trajectory T.
[0050] Figure 4 shows a schematic block diagram of the basic model GM and additional submodels TM1 to TMn when the input vector EV is processed to produce the total output vector GAV. This process is performed, for example, according to the low-rank adaptation described above.
[0051] In this case, when creating a road model for multiple sections of a road, input vectors EV, which are detected from environmental data UD and describe the trajectory of each section, are sent as input data to the basic model GM and the additional submodels TM1 to TMn used therein.
[0052] Using the basic model GM, a basic output vector GRAV is formed that describes the sections within the road model SM.
[0053] Using the additional submodels TM1 to TMn, a partial output vector TAV is formed that describes the sections within the road model SM.
[0054] The basic output vector GRAV and the partial output vector TAV are summed to form the total output vector GAV of the road model SM.
[0055] This is performed for all segments of the road model SM.
[0056] Figure 5 shows a schematic diagram illustrating the learning process of the additional submodels TM1 to TMn.
[0057] In this case, a specific local range OB is selected from the regional REG, and training data TD, limited to this local range OB, is used from database 13 to train the additional submodels TM1 to TMn. In this case, the base model GM remains unchanged. [Prior art documents] [Non-patent literature]
[0058] [Non-Patent Document 1] Liao,B.et al.:MapTR - Structured Modeling and Learning for Online Vectorized HD Map Construction;In:arXiv:2208.14437v2 [cs.CV],30.Januar 2023
Claims
1. In a method for creating a road model (SM) in a predetermined surrounding area around a vehicle (2), - For a specific geographical region (REG), an artificial intelligence-based basic model (GM) for roads is detected, trained using data from the said geographical region (REG), and stored in the vehicle (2). - For the aforementioned geographical region (REG), multiple artificial intelligence-based additional submodels (TM1 to TMn) relating to roads are created, each locally effective for a limited local area (OB) within the geographical region (REG), and are trained using only the data of each of the aforementioned local areas (OB). - The additional submodels (TM1 to TMn) are stored in the external computing unit (4) of the vehicle. - During the operation of the vehicle (2), before the vehicle (2) reaches a peripheral region, additional submodels (TM1 to TMn) effective for this peripheral region are called from the calculation unit (4) and transmitted to the vehicle (2). - Using the surrounding environment recognition sensor (3) of the vehicle (2), environmental data (UD) relating to the surrounding area is recognized, - Using the basic model (GM) and the sub-models (TM1 to TMn), the road model (SM) of the surrounding area is created based solely on the recognized surrounding data (UD). When creating the aforementioned road model (SM) for each of the multiple sections of a road, - Input vectors (EVs) detected from the environmental data (UD) and describing the trajectory of each section are supplied as input data to the basic model (GM) and the additional submodels (TM1 to TMn), respectively. - Using the basic model (GM), a basic output vector (GRAV) describing the section within the road model (SM) is formed. - Using the additional submodels (TM1 to TMn), a partial output vector (TAV) describing the section within the road model (SM) is formed. - In order to form the total output vector (GAV) of the road model (SM), the basic output vector (GRAV) and the partial output vector (TAV) are summed together. A method characterized by the following features.
2. As environmental data (UD), the raw data from the surrounding environment recognition sensor (3) is used. The method according to claim 1, characterized in that
3. The basic model (GM) and / or the submodels (TM1 to TMn) are trained using an end-to-end learning process. The method according to claim 1 or 2, characterized in that
4. Before the vehicle (2) leaves the surrounding area it is currently traveling through, additional submodels (TM1 to TMn) effective for the next surrounding area that the vehicle (2) will pass through are called from the calculation unit (4) and transmitted to the vehicle (2). Using the basic model (GM) and the submodels (TM1 to TMn), the road model (SM) of the surrounding area is created from the recognized surrounding data (UD). The method according to any one of claims 1 to 3, characterized in that
5. After the vehicle (2) leaves the surrounding area in which it is currently traveling, the additional submodels (TM1 to TMn) that are valid for the detached surrounding area are removed from the vehicle (2) and / or transmitted to the central computing unit (4). The method according to any one of claims 1 to 4, characterized in that
6. - When creating the road model (SM) from the recognized environmental data (UD) and the respective additional submodels (TM1 to TMn), the existence and location of immovable environmental features in the relevant surrounding area are learned. - When driving through this surrounding area again, if the previously learned fixed environmental features are not found in the environmental data (UD) acquired during the re-driving of the surrounding area, the existence of these environmental features is inferred. The method according to any one of claims 1 to 5, characterized in that
7. The aforementioned geographical region (REG) may be a national territory, a continent, or the entire globe. The method according to any one of claims 1 to 6, characterized in that
8. A method for operating an autonomous vehicle (2), wherein a road model (SM) generated using the method according to any one of claims 1 to 7 and object data and / or obstacle data (OD) detected based on the environmental data (UD) are used in a planning and control module (11) so that a target trajectory (T) is planned and the vehicle (2) is automatically controlled along it.
9. The system is characterized in that, according to the road model (SM) and according to the object data and / or obstacle data (OD), operation commands are generated for the actuators of the vehicle (2), thereby controlling the vehicle (2) in the longitudinal and lateral directions along the planned target trajectory (T). The method according to claim 9.