Method and system for creating an environment model

Abstract

The invention relates to a method for operating an artificial neural network (3) of a vehicle (1) having a sensor system (2) for detecting a region surrounding the vehicle (1) and an artificial neural network (3) for processing the information provided by the sensor system (2), the method enabling the weighting factors of the artificial neural network to be adapted to the geographical position at which the vehicle is currently located.

Description

[0001] 202401245

[0002] - 1 -

[0003] Method and system for creating an environmental model

[0004] The invention relates to a method for operating an artificial neural network of a vehicle and a driver assistance system that processes sensor information regarding the vehicle's environment by means of an artificial neural network.

[0005] It is known to use sensors on a vehicle to detect the surrounding area and to use an artificial neural network to identify and classify objects. The sensors can include, in particular, one or more radar sensors, LiDAR sensors (LiDAR: light detection and ranging), cameras, stereo cameras, and / or ultrasonic sensors.

[0006] The objects that need to be detected, automatically recognized, and classified can vary greatly depending on the region or country. For example, traffic signs, traffic lights, road markings, and even moving objects like buses or animals can look very different from region to region.

[0007] Artificial neural networks must be trained using suitable training data to detect and classify objects.

[0008] Artificial neural networks are typically trained using a training dataset tailored to a specific country or region. This means that such a network cannot be used in a different country or region where the objects to be recognized look different.

[0009] Alternatively, a very large artificial neural network can be used, trained using training data from different countries and regions. This requires a significant training effort, as well as high computing power and a large [202401245]

[0010] - 2 -

[0011] Storage requirements mean that this is not feasible with limited computing resources in a vehicle.

[0012] Based on this, the object of the invention is to provide a method for operating an artificial neural network of a vehicle which, despite limited computing resources of a computing unit on which the artificial neural network is operated, enables location-adaptive detection and recognition of environmental objects.

[0013] The problem is solved by a method having the features of independent claim 1. A driver assistance system is the subject of dependent claim 12. Preferred embodiments are the subject of the dependent claims.

[0014] According to a first aspect, a method for operating an artificial neural network of a vehicle is disclosed. The vehicle has sensors for detecting its surroundings and an artificial neural network for processing the information provided by the sensors. The method comprises the following steps:

[0015] First, several sets of weighting factors are provided. Each set of weighting factors is intended for use in the artificial neural network. The weighting factors of the sets of weighting factors were determined during training phases of the artificial neural network such that the network is trained to capture environmental characteristics that depend on geographical location and can differ, at least partially. In other words, the sets of weighting factors were determined using training data that depict location-dependent environmental scenarios. 202401245

[0016] - 3 -

[0017] The vehicle receives location information that specifies the geographical position where the vehicle is located.

[0018] Depending on the received location information, at least one set of weighting factors is loaded.

[0019] After loading the set of weighting factors, the vehicle's surroundings are captured using the artificial neural network, which uses the weighting factors of a loaded set of weighting factors.

[0020] The technical advantage of the proposed method lies in the use of a very compact artificial neural network that utilizes weight factors tailored to the vehicle's current geographic position. These weight factors are loaded and applied based on the vehicle's actual location. By adapting the artificial neural network through the selection of appropriate weight factors, it is configured to detect and classify the objects present at that location.

[0021] According to one embodiment, the weighting factors of the sets of weighting factors differ only partially. Alternatively, a first part of the weighting factors of the neural network remains unchanged, and only a second part of the weighting factors is adapted depending on location.

[0022] According to one embodiment, the weighting factors of at least two sets of weighting factors are the same in at least one input layer, and the weighting factors differ in at least one output layer. 202401245

[0023] - 4 -

[0024] In particular, input layers of a feature extractor can always use the same weighting factors, while output layers of the feature extractor and a classifier use location-adapted weighting factors.

[0025] According to one embodiment, the artificial neural network is trained to recognize and classify traffic signs, traffic lights, and / or road markings. These objects, in particular, can vary significantly depending on location and can be better recognized through adaptation of the artificial neural network.

[0026] According to one embodiment, the artificial neural network is designed to recognize different objects, in particular different types of traffic signs, depending on the loaded set of weighting factors. The objects can be stationary (e.g., traffic signs, traffic lights, and / or road markings) or moving objects, such as vehicles that vary in appearance depending on their location, like buses, trucks, etc., or even animals whose presence and appearance are location-dependent.

[0027] According to one embodiment, the structure of the artificial neural network is adapted depending on the loaded set of weighting factors. Thus, depending on the vehicle's geographical position, not only can the weighting factors be adapted, but structural changes (i.e., changes to the topology) of the artificial neural network can also be made.

[0028] According to one embodiment, the structure of the artificial neural network is adapted in one or more output-side layers, whereas one or more input-side layers are adapted in 202401245

[0029] - 5 - their structure remains unchanged. In particular, the output layers of the feature extractor and the classifier can be adapted to perform different classifications depending on the recognized features of an object, especially so-called high-level features. It should be noted that the layers progressively extract more complex and abstract features as the data flows through the artificial neural network. The deeper the artificial neural network, the more complex the learned features, enabling the model to capture complex patterns and relationships in the input data. The so-called low-level features are the most basic and universal features, such as local edges and vertices.These low-level features, provided by the feature extractor, can be reused and combined in various ways by subsequent layers to create more complex and meaningful representations (e.g., the shape of stop signs is hexagonal). Since the low-level features learned in the input layers are often more general and task-independent, it is not necessary to modify the weighting factors of the input layers. High-level features, on the other hand, are extracted by output layers of the feature extractor and are object-dependent or location-dependent. Therefore, it is advantageous to adapt the weighting factors of the output layers. Furthermore, the classifier can also be extended to be location-dependent in order to recognize other types of objects.

[0030] According to one embodiment, at least one set of weighting factors is downloaded from an external data provisioning unit via a wireless communication interface of the vehicle. This allows the artificial neural networks of a large number of vehicles to be adapted to their respective local conditions. 202401245

[0031] - 6 -

[0032] According to one embodiment, several sets of weighting factors are stored in a memory unit of the vehicle, and at least one set of weighting factors is loaded from this memory unit. This allows the artificial neural network to adapt even without a data connection between the vehicle and the external data provision unit.

[0033] In one embodiment, several sets of weighting factors are loaded. These sets of weighting factors relate to geographical regions that are close to the vehicle's current geographical position. For example, if the vehicle is currently located very close to several national borders, multiple sets of weighting factors can be loaded, enabling the artificial neural network to adapt as quickly as possible.

[0034] According to another aspect, a computer program is disclosed. The computer program comprises instructions which, when executed by a computing unit, cause the computing unit to execute the method according to one of the preceding embodiments.

[0035] According to a further aspect, a driver assistance system is disclosed. The driver assistance system comprises an artificial neural network trained to receive information from the vehicle's sensors. The artificial neural network is trained to process the information provided by the sensors. Furthermore, the driver assistance system is trained to load several different sets of weighting factors. Each set of weighting factors is used in the artificial neural network 202401245.

[0036] - 7 - provided and the weighting factors of the sets of weighting factors were determined in training phases of the artificial neural network such that the artificial neural network is trained, by means of the sets of weighting factors, to detect different environmental characteristics that depend on the geographical position. The driver assistance system is configured to perform the following steps:

[0037] - Receiving location information that specifies the geographical position where the vehicle is located;

[0038] - Loading at least one set of weighting factors depending on the received location information;

[0039] - Capturing the vehicle's surroundings using an artificial neural network that utilizes the weighting factors of a loaded set of weighting factors.

[0040] According to one embodiment of the driver assistance system, at least some of the sets of weighting factors are assigned adaptation information based on which the structure of the artificial neural network can be adapted to the respective loaded set of weighting factors. This also allows the network topology to be adapted to the set of weighting factors used.

[0041] According to one embodiment of the driver assistance system, a second set of weighting factors is derived from a first set of weighting factors through transfer learning. Thus, based on a first set of weighting factors adapted to a first region and a reduced training dataset containing training data from a second region that focuses on the regional differences to the first region, a second set of 202401245 can be generated.

[0042] - 8 -

[0043] Weighting factors are obtained that are adapted for a second region.

[0044] The terms “approximately”, “essentially” or “about” mean, within the meaning of the invention, deviations from the respective exact value by + / - 10%, preferably by + / - 5% and / or deviations in the form of changes that are insignificant for the function.

[0045] Further developments, advantages, and possible applications of the invention will also become apparent from the following description of exemplary embodiments and from the figures. All features described and / or illustrated are, individually or in any combination, fundamentally the subject matter of the invention, irrespective of their compilation in the claims or their cross-reference. The content of the claims is also incorporated into the description.

[0046] The invention will be explained in more detail below with reference to exemplary embodiments shown in the figures. The figures show:

[0047] Fig. 1 shows an exemplary schematic representation of a vehicle with sensors for detecting the vehicle's surroundings;

[0048] Fig. 2 shows, by way of example and schematically, a geographical map with boundaries that delineate several map areas, each of which is assigned a set of weighting factors;

[0049] Fig. 3 shows an exemplary and schematic representation of an artificial neural network with a feature extractor and a classifier, whose weighting factors are regionally adjustable; and 202401245

[0050] - 9 -

[0051] Fig. 4 shows an example of a block diagram illustrating the process steps for

[0052] Illustrated operation of an artificial neural network.

[0053] Figure 1 schematically shows a vehicle 1 moving in the direction of travel FR on a roadway. The vehicle includes a sensor system 2 by means of which the area surrounding the vehicle 1 can be detected. The sensor system 2 comprises one or more sensors configured to detect the area surrounding the vehicle 2. The sensor system 2 can, for example, include one or more of the following sensors: radar sensor, LiDAR sensor, camera, stereo camera, ultrasonic sensor, etc.

[0054] Vehicle 1 has a computer unit in which an artificial neural network 3 is implemented. The artificial neural network 3 is configured to process the information provided by the sensors and to recognize and classify objects.

[0055] The trained artificial neural network 3 has several weighting factors that were determined during training. This training uses data in which objects are assigned labels (so-called ground-truth data). The neural network's weighting factors are adjusted so that the artificial neural network recognizes the objects contained in the training data according to their labels. An example of an object could be a traffic sign labeled, for example, as a "stop sign" or a "no entry sign." Another object type could be an object that can be located on the road, such as "person," "car," "truck," or even animals like "moose," "kangaroo," "cow," "sheep," etc. 202401245

[0056] - 10 -

[0057] The objects that need to be detected in the sensor information for the specific region where vehicle 1 is currently moving can vary. In Germany, for example, "No Entry" signs have a different shape than in the USA. Furthermore, kangaroos do not need to be classified as objects in Germany, whereas in Australia, for example, they do.

[0058] To avoid the artificial neural network 3 being able to detect objects in all regions, which leads to very large network structures and thus requires high computing power from the computing unit that implements the artificial neural network 3, the artificial neural network 3 of vehicle 1 uses several sets of weighting factors W1, W2, W3, which were determined using training data that only includes training data from a specific region, thus limiting the applicability of each set of weighting factors to a specific geographical region.

[0059] Fig. 2 shows a rough schematic of a map 10 in which several regions are delimited by boundary lines G. These boundary lines G result in several regional areas B1, B2, B3. Each of these regional areas B1, B2, B3 is assigned a set of weighting factors W1, W2, W3. In the illustrated embodiment, the regional areas B1, B2, B3 do not overlap. Alternatively, regional overlap can be provided to improve the functionality of the driver assistance system even in the area of ​​the boundary lines G.

[0060] Depending on the regional position and thus the deployment location of vehicle 1, the artificial neural network 3 can be operated with the respective set of weighting factors W1, W2, W3, which were determined by means of a training phase for this regional area B1, B2, B3. For example, if vehicle 1 is located in regional area 202401245

[0061] - 11 -

[0062] If vehicle 1 is located in region B1, neural network 3 is operated with the set of weighting factors W1. If, however, vehicle 1 is located in regional region B2, neural network 3 is operated with the set of weighting factors W2.

[0063] The sets of weighting factors W1, W2, W3 can be stored in a storage unit 4 of the vehicle 1 or in an external data provision unit 5. The external data provision unit 5 can, for example, be a server of a data provision service. The vehicle 1 can have a wireless communication interface 6 through which the vehicle can download the respective set of weighting factors W1, W2, W3 from the external data provision unit 5.

[0064] The vehicle is equipped with sensors that can determine its geographical position. These sensors may include, for example, a GPS sensor. Based on the position information provided by the sensors, the regional area B1, B2, B3 in which the vehicle is located is determined. Based on this determined regional area B1, B2, B3, a set of weighting factors W1, W2, W3 is selected that contains weighting factors appropriate for this regional area B1, B2, B3.

[0065] This allows the artificial neural network 3 to recognize and classify the respective objects that occur or are used in this regional area B1, B2, B3, despite the limited computing resources of the computing unit, by means of regionally adapted weighting factors.

[0066] Fig. 3 shows a schematic block diagram of an artificial neural network 3 for object recognition and object classification. The artificial neural network 3 includes a feature extractor (202401245).

[0067] - 12 - feature extractor) 3.1, which groups several variables from large datasets into groups (so-called features). The information provided by the sensors or data derived from them (e.g., through suitable further processing) is transmitted to the feature extractor 3.1 and processed by it. The output information provided by the feature extractor 3.1 is transmitted to a classifier 3.2, which determines output data based on information provided by the feature extractor 3.1. The output data of the classifier 3.2 indicates, for example, what type of object it is and / or into which object class it should be classified.

[0068] In Feature Extractor 3.1, several weighting factors are used to perform feature extraction. A first part of Feature Extractor 3.1 can use fixed weighting factors that are not task- or location-dependent and therefore do not need to be adapted to regional conditions. This first part of Feature Extractor 3.1 can have several input layers in which the fixed weighting factors are used. A second part of Feature Extractor 3.1, on the other hand, uses task- or location-dependent weighting factors. This second part of Feature Extractor 3.1 can comprise several output layers, i.e., layers that connect to the output of Feature Extractor 3.1.

[0069] In addition, the classifier 3.2 includes location-dependent weighting factors, which are loaded depending on the geographical position of vehicle 1 and are applied after loading.

[0070] Fig. 4 shows a flowchart illustrating the process steps for operating an artificial neural network 3 of a vehicle. 202401245

[0071] - 13 -

[0072] First, several sets of weighting factors are provided (S10). Each set of weighting factors is intended for use in the artificial neural network. The weighting factors of the sets of weighting factors were determined in training phases of the artificial neural network such that the artificial neural network is trained, using the sets of weighting factors, to capture environmental characteristics that depend on the geographical location and can differ, at least partially.

[0073] Subsequently, location information is received by the vehicle, specifying the location where the vehicle is situated (S11).

[0074] Depending on the received location information, at least one set of weighting factors is loaded (S12).

[0075] Finally, the area surrounding the vehicle is captured using the artificial neural network, which uses the weighting factors of a loaded set of weighting factors (S13).

[0076] The invention has been described above using exemplary embodiments. It is understood that numerous modifications and adaptations are possible without thereby departing from the scope of protection defined by the patent claims.

[0077] 202401245

[0078] - 14 -

[0079] Reference symbol list

[0080] 1 vehicle

[0081] 2 Sensors

[0082] 3 artificial neural network

[0083] 3.1 Feature Extractor

[0084] 3.2 Classifiers

[0085] 4 storage units

[0086] 5 External Data Provisioning Unit

[0087] 6 Wireless communication interface

[0088] 10 cards

[0089] B1, B2, B3 regional areas

[0090] G boundary line

[0091] W1, W2, W3 sets of weighting factors

Claims

202401245 - 15 - Patent claims 1) Method for operating an artificial neural network of a vehicle (1) which has a sensor system (2) for detecting an area surrounding the vehicle (1) and an artificial neural network (3) for processing the information provided by the sensor system (2), wherein the method comprises the following steps: - Providing several sets of weighting factors (W1 , W2, W3), wherein each set of weighting factors (W1, W2, W3) is intended for use in the artificial neural network (3) and the weighting factors of the sets of weighting factors (W1 , W2, W3) have each been determined in training phases of the artificial neural network (3) such that the artificial neural network (3) is trained by means of the sets of weighting factors (W1 , W2, W3) to detect environmental characteristics that depend on the geographical location and may differ at least partially (S10); - Receiving location information by the vehicle (1) specifying the geographical position where the vehicle (1) is located (S11); - Loading at least one set of weighting factors (W1 , W2, W3) depending on the received location information (S12); - Capturing the surrounding area of ​​the vehicle (1 ) using the artificial neural network (3) where the weighting factors of a loaded set of weighting factors (W1 , W2, W3) are used (S13). 202401245 - 16 - 2) Method according to claim 1 , characterized in that the weighting factors of the sets of weighting factors (W1 , W2, W3) differ only partially. 3) Method according to claim 2, characterized in that the weighting factors of at least two sets of weighting factors (W1 , W2, W3) are the same in at least one input layer and the weighting factors differ in at least one output layer. 4) Method according to one of the preceding claims, characterized in that the artificial neural network (3) is trained to recognize and classify traffic signs, traffic lights and / or road markings. 5) Method according to one of the preceding claims, characterized in that the artificial neural network (3) is configured to recognize different objects, in particular different types of traffic signs, depending on the loaded set of weighting factors (W1 , W2, W3). 6) Method according to one of the preceding claims, characterized in that the structure of the artificial neural network (3) is adapted depending on the loaded set of weighting factors (W1, W2, W3). 7) Method according to claim 6, characterized in that the adaptation of the structure of the artificial neural network (3) takes place in one or more output-side layers, whereas 202401245 - 17 - one or more input-side layers are unchanged in their structure. 8) Method according to one of the preceding claims, characterized in that the at least one set of weighting factors (W1 , W2, W3) is downloaded via a wireless communication interface of the vehicle (1 ) from an external data provision unit (5). 9) Method according to one of claims 1 to 7, characterized in that several sets of weighting factors (W1 , W2, W3) are stored in a storage unit (4) of the vehicle (1 ) and the loading of at least one set of weighting factors (W1 , W2, W3) is carried out from the storage unit. 10) Method according to one of the preceding claims, characterized in that when loading several sets of weighting factors (W1 , W2, W3) these sets of weighting factors refer to geographical regions that are close to the geographical position where the vehicle (1 ) is currently located. 11) Computer program comprising instructions which, when executed by a computer unit, cause the computer unit to execute the method according to one of the preceding claims. 12) Driver assistance system comprising an artificial neural network (3) configured to receive information from a sensor system (2) of the vehicle (1), wherein the artificial neural network (3) is configured to process the information provided by the sensor system (2), wherein the driver assistance system is configured to 202401245 - 18 - is trained to load several different sets of weighting factors (W1 , W2, W3), each set of weighting factors (W1 , W2, W3) being intended for use in the artificial neural network (3), and the weighting factors of the sets of weighting factors (W1 , W2, W3) being determined in training phases of the artificial neural network (3) such that the artificial neural network (3) is trained by means of the sets of weighting factors (W1 , W2, W3) to detect different environmental characteristics that depend on the geographical position, the driver assistance system being configured to perform the following steps: - Receiving location information that specifies the geographical position where the vehicle (1) is located; - Loading at least one set of weighting factors (W1, W2, W3) depending on the received location information; - Capturing the area surrounding the vehicle (1) using the artificial neural network (3) which uses the weighting factors of a loaded set of weighting factors (W1, W2, W3). 13) Driver assistance system according to claim 12, characterized in that at least partially the sets of weighting factors (W1, W2, W3) are assigned adaptation information based on which the structure of the artificial neural network (3) can be adapted to the respective loaded set of weighting factors (W1, W2, W3). 14) Driver assistance system according to claim 12 or 13, characterized in that a second set of weighting factors (W2) 202401245 - 19 - is derived from a first set of weighting factors (W1) through transfer learning.

Images