Method for detecting an environment of a first sensor system

Abstract

A method for detecting an environment of a first sensor system is proposed, the method having the following steps: providing a time series of data of the first sensor system for detecting the environment; generating an input tensor for a trained neural network using the time series of data of the first sensor system, wherein the neural network has been set up and trained for recognizing at least one partial region of the environment on the basis of the input tensor in order to improve the detection of the environment by means of a second sensor system; generating a control signal for the second sensor system by means of an output signal of the trained neural network in order to improve the detection of the environment in the at least one partial region.

Description

Background Technology

[0001] The automation of driving is accompanied by the equipping of vehicles with increasingly extensive and high-performance sensor systems for environmental detection and to support driving functions and / or to control and guide the vehicle at least partially automatically.

[0002] For this purpose, multiple different sensors of different modalities or based on different technologies are typically used, such as radar sensors and video sensors. The data generated by each sensor of different modalities is typically processed individually and independently first; that is, the received radar waves are processed independently of the optical sensors. Data correlation or computation is then performed in later processing steps, such as correlation between radar reflections and video pixels, or fusion of calculated radar objects and video objects. Even in more highly integrated systems, the sensors perform measurements independently of each other.

[0003] Video cameras, for example, send their images to the detection device, while radar sensors similarly send their signals to the detection device. Summary of the Invention

[0004] If a video camera is considered an uncontrolled sensor system used for assisted or automated driving, which constantly tries to detect the entire scene as much as possible, it can be compared to the visual perception of a driver on a mobile platform. However, the driver additionally uses interior and exterior rearview mirrors to detect the entire vehicle environment and uses additional aids, such as low beam or high beam headlights, for better visual perception in twilight or at night.

[0005] Here, the driver's heightened attention tends to be localized, meaning that clear vision and corresponding brain processing are primarily forward-oriented while driving. Conversely, in the periphery, the driver primarily processes motion. Therefore, the driver actively and situationally focuses their gaze and attention on the relevant areas of the environment—it is impossible for a single driver to visually detect the entire 360° environment at all times.

[0006] Similar to human perception (scan-and-watch), the targeted use and active correlation of multiple sensors can improve systems for driver assistance or automated driving, especially when using sensors of different modalities.

[0007] Corresponding to various aspects of the invention, a method for detecting the environment of a first sensor system, a method for training a neural network, a neural network, and a detection device are provided according to the features of the independent claims. Advantageous configurations are the subject matter of the dependent claims and the descriptions below.

[0008] Throughout the description of this invention, the order of the method steps is shown in a manner that makes the method easy to understand. However, those skilled in the art will recognize that many of these method steps can be performed in other orders and yield the same or corresponding results. In this sense, the order of these method steps can be changed accordingly. Some features are set with numerals to improve readability or to make the association more unique and explicit, but this does not imply the existence of a particular feature.

[0009] According to one aspect, a method for detecting the environment of a first sensor system is proposed, the method comprising the following steps:

[0010] In one step, a time series of data from the first sensor system is provided for detecting the environment. In another step, the time series of data from the first sensor system is used to generate an input tensor for a trained neural network, wherein the neural network has been configured and trained to identify at least one partial region of the environment based on the input tensor to improve the detection of the environment using a second sensor system. In yet another step, a control signal is generated for the second sensor system using the output signal of the trained neural network to improve the detection of the environment in the at least one partial region.

[0011] In the case of neural networks, the signals at the connections of artificial neurons can be real numbers, and the output of an artificial neuron is calculated as a nonlinear function of the sum of its inputs. Typically, the connections of artificial neurons have weights that are adjusted as learning continues. These weights increase or decrease the strength of the signals at the connections. Artificial neurons may have thresholds, so that they output a signal only if the total signal is greater than the threshold.

[0012] Typically, multiple artificial neurons are grouped into layers. Different layers can perform different types of transformations on their inputs. The signal propagates from the first layer—the input layer—to the last layer—the output layer; possibly after passing through these layers multiple times.

[0013] A neural network essentially consists of at least three layers of neurons: an input layer, hidden layers, and an output layer. This means that all neurons in the network are divided into layers, where a neuron in one layer is always connected to all neurons in the layer below it. Apart from the input layer, different layers consist of neurons determined by a non-linear activation function and are connected to neurons in the layer below it. Deep neural networks can have multiple such hidden layers.

[0014] Such neural networks must be trained for their specific tasks. Here, each neuron in the corresponding architecture of the neural network receives, for example, random initial weights. Input data is then fed into the network, and each neuron weights the input signal according to its weights, further feeding the result to neurons in the next layer. The final result is then provided at the output layer. The magnitude of the error, and the share of that error held by each neuron, can be calculated so that the weights of each neuron are then changed in the direction that minimizes the error. This process of recursively traversing, remeasuring the error, and matching the weights continues until the error is less than a pre-given bound.

[0015] For example, this method can be implemented using a passive sensor as a first sensor system and an active sensor as a second sensor, which are connected either directly or through a processing unit. Here, the first sensor system measures and processes the environment of the first sensor system; using the data from the first sensor system, a control signal can be generated according to the method, which is provided to the second sensor system. Therefore, the second sensor system, in response to the control signal, identifies a portion of the environment of the first sensor system, and the identification of this portion of the environment by the second sensor system improves the detection of the environment. For example, the control signal for the active sensor, i.e., for the second sensor system, can control the second sensor system to more accurately determine a certain area of ​​the environment of the first sensor system, such as angle and / or distance and / or elevation angle.

[0016] Here, the control signal is not limited to accurately determining only the area where an object has already been identified, but rather the control signal can control the second sensor system to more accurately determine suitable portions or regions of the environment of the first sensor system, so as to achieve earlier and / or more accurate identification and / or recognition and / or determination of objects in the environment of the first sensor system. Furthermore, the method described for detecting the environment of the first sensor system does not require the introduction of expert knowledge into the method, as the relevant knowledge can be learned data-driven using neural networks.

[0017] By using this method, it is advantageous to derive improved robustness and performance in detecting relevant traffic participants, such as vehicles and pedestrians, in the environment without relying on expert knowledge. This is because traditional model-based control of active sensors requires expert knowledge to identify improvement potential and control the second sensor system in a suitable manner.

[0018] Furthermore, it is advantageous to derive a better-performing and more cost-effective overall system for the selected task, as the sensing devices and algorithms used to identify relevant traffic participants are jointly optimized.

[0019] Furthermore, it advantageously yields the identification of relevant objects in relation to the application and the method's simple scalability.

[0020] In other words, this method enables, for example, the environment of the first sensor system to be determined in a directional manner after the object or part of the environment has been determined using data from the first sensor system, for example, by supplementing the data from the second sensor system, so as to more accurately determine the object and / or the part of the environment.

[0021] Here, using a second sensor system can, for example, more accurately determine, for example, the extended scale and / or location and / or region and / or angle and / or distance and / or elevation angle and / or speed and / or possible future location and / or region with unreliably determined objects and / or object size with unreliably determined objects.

[0022] Control signals for the second sensor system can relate to spatial detection of the environment, and / or can be complexly constructed and relate to, for example, the exposure of the second sensor system, such as a video system. Therefore, control signals for the second sensor system can relate to all characteristics of the second sensor system, and by controlling these characteristics, the detection of the environment can be altered by using the second sensor system.

[0023] The method for detecting the environment using a first sensor system does not presuppose the identification of objects. Instead, it extracts features from the environment within a neural network, which are used to deduce specific regions to improve the detection of that environment. These features primarily include structural occlusions, such as buildings or vegetation, and / or road topology or road orientation, and / or objects, such as vehicles and pedestrians.

[0024] In particular, it can more accurately detect areas that do not contain identified objects but have a high probability that "the object may appear there in the future".

[0025] Using the method for detecting the environment of the first sensor system, objects can be identified faster and earlier because it also detects areas in which objects have not been previously identified.

[0026] In addition, this method can be applied to applications that do not require object recognition, such as semantic segmentation and / or recognition of lane boundaries and / or compensation for sensor degradation in multi-sensor systems.

[0027] This method improves robustness and recognition rates for relevant traffic participants—such as vehicles and pedestrians—and regulatory elements—such as boundary lines, lanes, and pedestrian platforms. Furthermore, it enables highly targeted and accurate detection and processing of small environmental areas using a controllable second sensor system. This is particularly useful for detecting small objects at considerable distances, such as lost cargo, partially obscured objects (e.g., pedestrians behind vehicles), locating potential obstacles, and accurately detecting markers such as posts, manhole covers, and traffic lights.

[0028] Furthermore, this method enables simpler assurance of autonomous systems through increased redundancy at the sensing device level, as a controllable second sensor system can, for example, confirm or refute measurements from the first sensor system. Therefore, this method can provide a more economically advantageous overall sensor system with improved overall performance, consisting of at least one first sensor system and a controllable second sensor system. Additionally, this method can also combine a sensor that slowly measures the overall environment as the first sensor system with a sensor that measures more quickly within a smaller detection area of ​​that environment as the second sensor system.

[0029] According to one aspect, the neural network is configured to process time series of data and / or characterize the time-related states of the neural network. Such a neural network configuration is particularly well-suited for the described method.

[0030] According to one perspective, the neural network is a recurrent neural network.

[0031] The following neural network is called a recurrent neural network (RNN) or a back-coupled neural network: Unlike other networks, its characteristic is that neurons in one layer are connected to neurons in the same layer or to neurons in a previous layer. Through back-coupling, a state vector can be represented, which is propagated (weigereichted) time-step by time and changed as needed. By using a RNN, this method is particularly suitable for training to generate corresponding control signals because such a RNN can access sensor data from previous time steps for the current time step. This can improve the environmental detection of the first sensor system.

[0032] According to one approach, the input tensor of the neural network incorporates data from a second sensor system to improve the detection of the environment by the first sensor system. Advantageously, additional data from the second sensor system can be used to improve the representation of the environment by the first sensor system. Here, the input tensor may contain data from the second sensor system from the current time step and / or previous time steps. Training the neural network can be matched to the corresponding time step in which the data from the second sensor system is generated.

[0033] According to one approach, the input tensor of the neural network has data from a second sensor system from a time step prior to the current time step, in order to improve the detection of the environment by the first sensor system.

[0034] Advantageously, the generated control signal is primarily based on sensor data from the second sensor system at previous time steps, and thus it is possible to use data from the second sensor system used in neural networks to improve the representation of the environment of the first sensor system.

[0035] According to one approach, the input tensor of the neural network has data from a second sensor system at the current time step, and the second sensor system is manipulated using control signals generated from previous time steps in order to improve the detection of the environment by the first sensor system.

[0036] Advantageously, data from the second sensor system at the current time step and control signals from previous time steps can be used to improve the representation of the environment of the first sensor system, because the data from the corresponding sensor systems are generated at the same time step and therefore have higher synchronization. The training of the neural network can be matched to the corresponding time step in which the data from the second sensor system is generated.

[0037] According to one approach, the neural network is set up and trained to create a list of objects from the environment of a first sensor system. Advantageously, such a list of objects can be used to train the neural network, or the identified objects in the list can be provided to other active systems used for detecting the environment.

[0038] According to one perspective, the first sensor system is a passive sensor system, and the second sensor system is an active sensor system.

[0039] Here, an active sensor system can be one whose signal, the data it generates, or its detection characteristics can be selectively adapted and / or controlled according to control signals for specific detection requirements in the detection of the environment. Such detection characteristics of a controllable sensor system can particularly relate to the position of an object within the detection area of ​​the controllable sensor system, for example, in a selected partial area and / or a limited distance range and / or elevation angle. The scope can be defined, but it can also involve measurement characteristics such as location resolution. Accordingly, a controllable sensor system can be an active sensor system. Examples of controllable sensor systems are radar sensors, lidar sensors, infrared sensors, time-of-flight sensors, and optical sensors.

[0040] Generally, sensors that can be controlled by a controllable signal source that acts on an object or environment, such as thermal imaging sensors, can be used. These thermal imaging sensors determine temperature changes that are controlled at a targeted location on the object to be determined by introducing eddies or other thermal inputs.

[0041] The wavelength of the signal source can usually be changed for the introduced controllable signal source, for example, in the range of 0.7μm-1000μm for infrared sources, or in the range of 300-1600nm for lidar sensors.

[0042] One approach proposes using control signals to reduce the data from the second sensor system.

[0043] Alternatively or additionally, the determination of an object using the first sensor system can also be performed by a controllable second sensor system by correspondingly changing the detection characteristics of the controllable second sensor system, wherein the change in detection characteristics particularly affects the spatial size of the determination range of the environment of the mobile platform.

[0044] By manipulating the second sensor system in this way, it is possible to reduce the amount of sensor data, thereby saving computation time and bandwidth. An example of this is a video sensor that provides only selected image segments rather than the entire image based on control signals.

[0045] According to one aspect, the data from the first sensor system includes the turning angle of the mobile platform and / or geographic map data and / or planned route orientation and / or road classification and / or weather conditions.

[0046] Alternatively or additionally, the first sensor system may also be an active sensor system, which is either used in accordance with a passive sensor system or controlled by other control parameters, such as the steering angle signal and / or positioning signal of the mobile platform.

[0047] To leverage map data to manipulate the second sensor system, the first sensor system can be a positioning sensor that determines a location on a map or in the environment. The map data can also be used to derive features and specific areas for detecting improvements to the environment.

[0048] Meaningful, inferable examples of specific regions include road directions and, in particular, road vanishing points. Objects first become visible near these vanishing points. This is in... Figure 3 Several examples are shown in sections a through 3d. Areas that should be detected more accurately are highlighted with shadows.

[0049] Alternatively or additionally, a single active sensor can be used as a second sensor system, without using the data from that second sensor system for generating control signals, and without providing data from the first sensor system as input tensors to the neural network. The neural network's input tensors are then used for generating control signals, such as signals from an existing mobile platform, either individually or collectively, and the trained neural network generates control signals for the second sensor system. Examples of such input data are:

[0050] The turning angle of the mobile platform and / or map data and / or the direction of the planned route for the mobile platform and / or road classification (city, highway, ...) and / or weather conditions.

[0051] Alternatively or additionally, the input tensor may have multiple time series of data from multiple first sensor systems. Alternatively or additionally, the first sensor systems may generate multiple control signals, which are provided to multiple active sensor systems.

[0052] According to one aspect, the first sensor system is an optical camera system, and the second sensor system is a lidar sensor and / or a radar sensor and / or an ultrasonic sensor.

[0053] An optical camera system can be either a strictly optical camera or a video system.

[0054] According to one aspect, the first sensor system is the same sensor system as the second sensor system; and in subsequent time steps, the control signals of the sensor system are used to detect improvements in the environment.

[0055] Such subsequent time steps can be separate time steps.

[0056] Correspondingly, the input data for the neural network may not include data from the first sensor system. Therefore, the manipulation of the second sensor system can be based, in particular, on data generated by the second sensor system in time steps prior to the current time step.

[0057] For example, an active lidar sensor can operate in alternating "active" and "passive" modes over time. In "passive" mode, an overview of the scene can be generated by detecting the entire field of view of a second sensor system at low location resolution. Using the information obtained, a small area of ​​that field of view can be measured very accurately in "active" mode at subsequent time steps.

[0058] According to one approach, a control signal controls a second sensor system to detect a portion of the environment in order to improve the environment detection capabilities of the first sensor system.

[0059] According to one perspective, the input tensor of the neural network has the turning angle of the mobile platform and / or geographic map data and / or planned route orientation and / or road classification and / or weather conditions and / or the current task of the autonomous system and / or a list of objects and / or areas that should be measured more accurately.

[0060] Here, an example of the current task of the autonomous system could be a driving task, such as parking and / or avoiding obstacles.

[0061] A method for training a neural network to generate control signals for a second sensor system is proposed, the method comprising the following steps: In one step, a time series of data from the first sensor system used to detect the environment of the first sensor system is used to provide an input tensor to the neural network, wherein the input tensor of the neural network has data from the second sensor system from time steps prior to the current time step. In another step, at least one object of the environment of the first sensor system is generated using the neural network and the input tensor. In another step, a control signal is generated using the neural network and the input tensor. In another step, the generated at least one object is compared with at least one correspondingly associated reference object. In another step, data of the second sensor system is generated for the next time step based on the control signal. In yet another step, the neural network is adapted to minimize the deviation from the corresponding reference object when determining the object of the environment.

[0062] For this method of training a neural network, the neural network can be configured with a time series for processing data and / or to characterize the time-related states of the neural network. Alternatively or additionally, the neural network used in this method of training a neural network can be a recurrent neural network.

[0063] The reference object is an object that is specifically generated for training a neural network in conjunction with corresponding input data for that neural network, and the object is accordingly labeled.

[0064] According to one aspect, for training the neural network, the at least one object and the corresponding associated reference object are each an object list of at least one object in an environment having a first sensor system, and / or the at least one object and the corresponding associated reference object are each a high-resolution representation of the environment of the first sensor system. The high-resolution representation of the environment may, for example, be an optically generated image of the environment, and / or a representation generated using a LiDAR system.

[0065] According to one approach, in order to train a neural network, at least one object from the object list is generated using an object detector.

[0066] In other words, a separate object detector can be used to train a neural network, particularly a recurrent neural network, by training the neural network to generate control signals, and by generating at least one object, such as at least one object from a list of objects, for example, based on data from a first sensor system. Generating the at least one object enables the object detector to provide the necessary feedback for learning control signals through the neural network, wherein the object detector itself can either remain unchanged or be incorporated into the training of the neural network.

[0067] According to one perspective, in order to train the neural network, the time series of data from the first sensor system is either real data from the first sensor system or simulated data used for the first sensor system.

[0068] To train neural networks, especially recurrent neural networks, using annotated data, data from a second sensor system is particularly important, as this data is generated based on control signals from the second sensor system. Therefore, the second sensor system provides data decisively related to the manipulation of that system. Consequently, it is not possible to collect data from the second sensor system that can subsequently be used to train the neural network in an unchanged form.

[0069] In addition, the sensor data from the first and second sensor systems must be annotated. As is common in object recognition, each relevant object is annotated with a bounding box and additional attributes, such as object type and / or velocity.

[0070] During training, the neural network is adapted using a loss function that describes the optimization objective. Such a loss function can have at least two parts. The first part can be a loss function commonly used in the field of object recognition (multi-task loss), which has regression and classification components.

[0071] When adapting neural networks, for example, objects can be compared to annotated data (ground-truth). The second part of the loss function can express a further optimization objective based on control signals. An example of this is minimizing the number of data points in a LiDAR system, in order to achieve cost reduction for the corresponding sensor system, for example.

[0072] Neural networks can be adapted and trained using backpropagation. By using the described loss function, the neural network learns, for example, to recognize relevant objects during training. Since the control signals to the second sensor system can have a significant impact on how well the object can be recognized, the neural network will modify the control signals to optimize object recognition.

[0073] In addition to the methods described for training neural networks, other known methods from the field of machine learning can be applied for training.

[0074] Some of these methods are:

[0075] Deep learning

[0076] Reinforcement learning

[0077] Active learning

[0078] Unsupervised learning / partially supervised learning

[0079] According to one aspect, in order to train a neural network, data of a second sensor system is generated using a high-resolution sensor system, and / or using a simulation program to simulate the second sensor system, and / or using the second sensor system in the environment of the first sensor system to generate data of the second sensor system.

[0080] In other words, to train a neural network, data from not only the first sensor system but also the second sensor system can be generated using a simulation program. Alternatively or additionally, to train the neural network, the data from the second sensor system can be generated using data from a high-resolution sensor system, in such a way that data corresponding to the second sensor system is generated from the high-resolution sensor system data according to a control signal. For example, data corresponding to at least one partial region of the environment can be selected from the high-resolution sensor system data based on the control signal. Alternatively or additionally, to train the neural network, data with a lower resolution can be selected from the high-resolution sensor system data according to the control signal. Accordingly, a high-resolution sensor will be used to generate and store high-resolution data, which is selected according to the control signal for training to simulate the data from the second sensor system.

[0081] Alternatively or additionally, neural network training can be performed using data from real-world sensor measurements, specifically by directly using data from a second sensor system. With the aid of a reference system, using annotated object and / or high-resolution sensor data, it is possible to simultaneously generate the reference data needed for training.

[0082] Neural networks can also be trained without annotated data by minimizing a reconstruction loss as the objective function. This allows for learning control signals for a second sensor system, as such a reconstruction loss contains the most valuable information from an information theory perspective. For example, it can be learned that a building that has already been measured does not need to be measured again in subsequent time steps because the building does not move or change. However, pedestrians can be measured more accurately at each time step because pedestrians can move and their appearance changes, which cannot be easily predicted.

[0083] To this end, the structure of the neural network is matched so that high-resolution sensor data is output at the output, replacing the object list. A loss function compares the neural network's output signal with high-resolution sensor data, for example, from simulated second sensor data, and uses a metric to determine the degree of consistency of the data. The sum of absolute differences can primarily be used as the metric. In this way, the neural network learns to design control signals such that high-resolution sensor data can be generated from low-resolution sensor data derived from past data and the first and / or second sensor data.

[0084] According to one aspect, the neural network is configured to process a time series of data and / or characterize the time-related state of the neural network, and / or the neural network is a recurrent neural network.

[0085] A neural network is proposed, which is set up and trained according to any of the methods described above for training neural networks.

[0086] Corresponding to any of the methods described above for detecting the environment of the first sensor system, a method is proposed that has a neural network, which is set up and trained in accordance with any of the methods described above for training a neural network.

[0087] A detection device is proposed, which is configured to perform any of the methods described above for detecting the environment of a first sensor system.

[0088] A mobile platform is proposed that is at least partially automated, and that the mobile platform has any of the detection devices described above for detecting the environment of the mobile platform, and / or wherein the mobile platform has a first sensor system and a second sensor system as described above.

[0089] This allows for high-quality detection of mobile platform environments with relatively low economic expenditure.

[0090] A mobile platform can be understood as a mobile, at least partially automated system, and / or a driver assistance system. An example could be a vehicle that is at least partially automated or has a driver assistance system. That is, in this respect, a system that is at least partially automated includes a mobile platform in terms of its at least partially automated functionality, but a mobile platform also includes vehicles and other mobile machines, including driver assistance systems. Other examples of mobile platforms could be driver assistance systems with multiple sensors, or mobile multi-sensor robots—such as robotic vacuum cleaners or lawnmowers.

[0091] The described method for detecting the environment of the first sensor system can be used on mobile platforms and / or multi-sensor monitoring systems and / or production machines and / or personal assistants and / or access control systems.

[0092] Each of these systems can be fully or partially automated.

[0093] A computer program with instructions, when executed by a computer, causes the computer to perform any of the methods described above. With such a computer program, the methods described above can be made available, for example, to a mobile platform in a simple manner.

[0094] A machine-readable storage medium is proposed on which the computer program described above is stored. With the aid of this machine-readable storage medium, the aforementioned computer program product is portable.

[0095] A control signal as described above is proposed for the manipulation of an external sensor system. Therefore, alternatively or additionally, this control signal can be used to control a second sensor system and / or another external sensor system.

[0096] A use of a mobile platform for controlling at least one part of the automation of any of the methods described above for detecting the environment of the first sensor system is proposed.

[0097] A method is proposed in which control signals for operating a vehicle that is at least partially automated are generated based on the environment of a first sensor system detected according to any of the methods described above; and / or warning signals for alerting vehicle occupants are generated based on the detected environment of the first sensor system. Attached Figure Description

[0098] Reference to embodiments of the present invention Figures 1 to 8 This is shown and will be explained in more detail below. The accompanying diagram shows:

[0099] Figure 1 A mobile platform with at least one controllable sensor is shown, which has sensors with different modes.

[0100] Figure 2a , 2b This illustrates a recurrent neural network;

[0101] Figure 3 The highlighted areas show different traffic conditions;

[0102] Figure 4 The method steps for detecting the environment are shown;

[0103] Figure 5 A detection device with a data stream is shown;

[0104] Figure 6 This shows the data stream during the training of the detection device;

[0105] Figure 7This illustrates a modified setup for training the testing equipment;

[0106] Figure 8 This illustrates possible applications of the method. Detailed Implementation

[0107] Figure 1 The diagram schematically illustrates a vehicle 170 having a system 100 for detecting the environment of a mobile platform. The vehicle 170 has a video camera 110, which acts as a first sensor system and is signal-coupled to a detection device 130 to provide its generated images to the detection device 130.

[0108] The controllable radar sensor 120 of vehicle 170 is bidirectionally coupled to detection device 130 in a signal manner and provides its signal to detection device 130. The controllable radar sensor is equivalent to a second sensor system. Here, the controllable radar sensor 120 can be controlled in its perception of the environment using the generated control signals.

[0109] With such a system, targeted manipulation of an active second sensor system 120, such as a radar sensor or a lidar sensor, via control signals enables directional detection of the environment. Here, the control signal can be transmitted to the second sensor system 120 via a connection signal 125, which transmits signals from the controller 130 to the second sensor system 120.

[0110] The detection device 130 is configured to implement the method described above for detecting the environment using data from the video camera 110 and radar sensors.

[0111] Furthermore, the detection device can be coupled to a controller 140, which may be coupled to, for example, a brake 160 or a steering system 150. Thus, the controller 140 can, based on signals from the detection device 130, control automated emergency braking or adjust automated evasive maneuvers via the steering system.

[0112] Figure 2a The structure of a recurrent neural network (RNN) 200 is shown, which has an input connection 210 and an output terminal 220. The input connection can provide an input tensor, and the output terminal can provide, for example, control signals and / or object lists, as described above. Here, the recurrent structure of the neural network is indicated by a state variable V 230 and arrows.

[0113] Figure 2bThe structure of the recurrent neural network 200 is shown, in which the loop structure is illustrated using an "ausgerollte" recurrent neural network. At the input connections 210a to 220c, for example, time series of data from the first sensor system can be provided for time steps t-1, t, t+1, and the corresponding output signals 220a to 220c are generated from these time series via the neural network. t-2 V t-1 V t V t+1 It is used to characterize the current state of the recurrent neural network 200. Here, the neural network itself remains unchanged in the method used to detect the environment of the first sensor system.

[0114] In other words, at time point t, the input data x t The state vector V at the previous time step t-1 This is processed by neural network 200. Here, outputs 220a to 220c and a new state V are generated. t State V t+1 It is used again as an input parameter of the neural network in the next time step t+1.

[0115] Figure 3 Four different road scenes or traffic conditions are drawn from 3D to 4D. Shaded areas are used to highlight certain regions, such as those that can be detected by a second sensor system with the aid of control signals to improve the environmental detection of the first sensor system, as described above.

[0116] Figure 4 The method steps for detecting the environment of the first sensor system 510 are as follows:

[0117] In step S1, which is located before the current time step, the second sensor system 520 generates data that is provided for generating the input tensor.

[0118] In another step S2, a time series of data from the first sensor system 510 for detecting the environment is provided at the current time step, the data including, for example, a large area of ​​the environment.

[0119] In another step S3, the time series of data from the first sensor system 510 and the data provided by the second sensor system 520 are used to generate an input tensor for a trained neural network, wherein the neural network has been set up and trained to identify at least one partial region of the environment based on the input tensor in order to improve the detection of the environment by means of the second sensor system.

[0120] In another step S4, control signals 540 for the second sensor system 520 are generated using the output signal of the trained neural network 200 to improve the detection of the environment in the at least one partial region. For this purpose, the neural network 200 can internally extract features, such as occlusion and / or road direction and / or objects, and output these features as an object list 530 if necessary. The extracted features can be used internally within the neural network 200 to determine areas that should be measured more accurately, such as angles and / or distances. There is a high probability that "new objects become visible or already identified objects can be detected more accurately" in these areas. Therefore, the neural network 200 can generate corresponding control signals for the second sensor system 520 based on these areas.

[0121] In another step S5, the control signal controls the second sensor system 520 to detect the aforementioned portion of the environment, thereby improving the environment detection by the first sensor system 510. With the aid of the control signal 540, the second sensor system can, for example, limit its detection area. The second sensor system 520 can then generate data for such areas with improved accuracy, and this data can then be used to improve the detection of the environment by the first sensor system.

[0122] Figure 5 The data flow during operation of a detection device used to detect the environment of the first sensor system 510 is depicted, with three side-by-side representations of the detection device. The previous time step is arranged on the far left (501), the current time step in the middle (502), and the future time step on the right. Only the middle representation of the detection device shows its corresponding details. Here, the detection device has a trained recurrent neural network 200. For example, the first sensor system 510 may be a passive video sensor and / or lidar sensor system, while the second sensor system 520 may be an actively controllable lidar sensor system. The sensor systems periodically provide sensor data.

[0123] Here, data from the first sensor system 510 can be used to control the second sensor system 520 to more accurately detect suitable areas in the environment of the first sensor system 510. This, for example, enables earlier and more reliable identification of relevant traffic participants and / or reduces the number of ghost objects.

[0124] At time point t, an input signal 210b with data from the first sensor system 510 is provided to the neural network 200 trained using deep learning. This applies accordingly to either the preceding time point t-1 with input data 210a or the following time point t+1 with input data 210c.

[0125] Additionally, the trained neural network 200 is provided with sensor data c from the second sensor system 520 from a previous time point t-1. t-1 540a and the state vector d of the neural network from the previous time point t-1 t-1 230b, and thus the input tensor is generated.

[0126] State vector d t-1 230b represents the state vector of the neural network from time point t-1, which enables information to be stored across time steps.

[0127] In addition, the input tensor can also have other data 210b, such as vehicle speed and / or vehicle steering angle.

[0128] The neural network 200 generates control signals b for the second sensor system 520 based on the described input data and the logic learned by the neural network. t 540 and a list of detected objects a t 530, for example, a list of relevant traffic participants identified, which are characterized by their location and / or size and / or orientation and / or object type.

[0129] To this end, neural network 200 summarizes information from sensor systems 510 and 520 across time steps. This information can then be used with the help of a list of relevant traffic participants, a. t The particularly accurate estimates are used to control mobile platforms, such as vehicles.

[0130] signal b t It may have specifications, such as angle range and distance range, which should be measured by an actively controllable second sensor system.

[0131] With the help of control signal b t 540, using the second sensor system 520, the detection of the environment by the first sensor system 510 is controlled in a subsequent future time step t+1. To this end, data c from the second sensor system 520 at time point t can be provided for the next time step t+1. t The state vector d of 540b and neural network 200 t 230c.

[0132] Figure 6 The data flow during the training of the detection device used to detect the environment of the first sensor system is plotted.

[0133] In addition to corresponding to Figure 5In addition to the data stream described in the description of the detection device for detecting the environment of the first sensor system 510, there is also... Figure 6 The image shows a list of objects. t 530 and control signal b t The data stream from 540 to the loss function 610 enables a comparison to be performed with at least one object generated by the neural network 200 and a correspondingly associated reference object, as described above, using a reference object not shown here.

[0134] In order to according to control signal b t 540 generated Figure 5 Data from the second sensor system 520, in Figure 6 The second sensor system 520 is replaced by a simulation program 620. The method and steps for training the neural network 200 have been described above.

[0135] Figure 7 The data stream with the modified settings of the neural network 200 is plotted when the neural network 200 is trained to detect the environment of the first sensor system.

[0136] In this training of neural network 200, neural network 200 is trained with the help of a separate object detector 710 to generate control signals for the second sensor system.

[0137] Here, not only the neural network 201 but also the object detector 710 obtains sensor data c from the second sensor system 520 from a previous time point t-1. t-1 540a. Therefore, neural network 200 learns to generate control signal b. t 540, and detector 710 generates object list a based on data from second sensor system 520. t At least one object of 530. Therefore, detector 710 provides an object list a t The at least one object is used to provide feedback for training the neural network to generate control signals b. t 540, and can remain unchanged or be optimized in the same way. Object detector 710, for example, uses object list a... t The state represented by 530 can be used as the corresponding state vector d in the next time step t+1. t 230c is used as an input parameter for neural network 201. Alternatively or additionally, the input signal 210b for neural network 201 and object detector 710 can be provided as an input parameter. In other words, neural network 200 is configured to generate control signal b. t 540 neural network 201 and object detector 710.

[0138] Figure 8 Draw diagrams from a to 8d showing other possible application areas for any of the methods described.

[0139] Figure 8 a. Draw a diagram illustrating the application of any of the described methods in the context of an automated inspection system, for example, for inspecting components using thermal imaging, eddy current, and conventional optics in order to reliably sort out defective components.

[0140] Figure 8 b. Draw any of the described methods for use in automated lawnmower applications, such as for reliably identifying or classifying objects, particularly distinguishing between obstacles and non-obstacles.

[0141] Figure 8 c. Draw a diagram of any of the described methods for an application of automated access control, such as optical and acoustic personnel identification and automated door opening.

[0142] Figure 8 d. Draw an application of any of the methods described for monitoring a site or building, such as for inspecting hazardous goods, using cameras and lidar sensors.

Claims

1. A method for detecting the environment of a first sensor system (510), the method comprising the steps of: Provides a time series of data from the first sensor system (510) for detecting the environment; The time series data from the first sensor system (510) is used to generate an input tensor for a trained neural network (200), wherein the neural network (200) has been set up and trained to identify at least one partial region of the environment based on the input tensor in order to improve the detection of the environment by means of the second sensor system (520). A control signal (540) is generated using the output signal of the trained neural network, and the control signal is provided to the second sensor system (520) to improve the detection of the environment in the at least one partial region. The second sensor system (520) limits its detection area using the control signal (540). The input tensor of the neural network (200) has data from the second sensor system (520) from the time step prior to the current time step, in order to improve the detection of the environment by the first sensor system (510).

2. The method according to claim 1, wherein, The neural network (200) is configured with a time series for processing data and / or a time-related state characterizing the neural network (200).

3. The method according to claim 1 or 2, wherein, The first sensor system (510) is a passive sensor system, and the second sensor system (520) is an active sensor system.

4. The method according to claim 1 or 2, wherein, The first sensor system (510) is the same sensor system as the second sensor system (520); and in subsequent time steps, the control signals of the sensor system are used to detect improvements in the environment.

5. The method according to claim 1 or 2, wherein, The control signal controls the second sensor system (520) to detect a portion of the environment in order to improve the environment detection of the first sensor system (510).

6. A testing apparatus, the testing apparatus being configured to perform the method according to any one of claims 1 to 5.

Images