METHOD FOR DETECTING AN ENVIRONMENT OF A FIRST SENSOR SYSTEM

Description

State of the art

[0001] The automation of driving goes hand in hand with the equipping of vehicles with increasingly comprehensive and powerful sensor systems for environmental detection and for supporting driving functions and / or for at least partially automated control and guidance of the vehicle.

[0002] For this purpose, several different sensors of different modalities or based on different technologies are often used, such as radar sensors and video sensors. The data generated by the respective sensors of different modalities is typically initially processed individually and independently of one another, i.e., the received radar waves are processed independently of the optical sensors. Only in later processing steps is the data linked or calculated, for example, by associating radar reflections and video pixels or fusing calculated radar and video objects. Even in more highly integrated systems, the sensors measure independently of one another.

[0003] For example, a video camera sends its images to a detection device, and a radar sensor also sends its signals to this detection device. Relevant prior art includes: R. Cheng et al., Geometry-Aware Recurrent Neural Networks for Active Visual Recognition, 03-11-2018, arXiv:1811.01292; S. Chen et al., VERAM: View-Enhanced Recurrent Attention Model for 3D Shape Classification in IEEE Transactions on Visualization & Computer Graphics, vol. 25, no. 12, pp. 3244-3257, Dec. 2019, doi: 10.1109 / TVCG.2018.2866793; D. Jayaramam and K. Grauman, Look-ahead before you leap: end-to-end active recognition by forecasting the effect of motion, 05-08-2016, arXiv:1605.00164v2. Disclosure of the invention

[0004] If a video camera is viewed as a non-controllable sensor system for assisted or automated driving that always detects the entire scene, it is comparable to the visual perception of a driver of a mobile platform. However, a driver also uses interior and exterior mirrors to capture the entire vehicle surroundings, and at dusk or at night, additional aids, such as low or high beams, are used for improved visual perception.

[0005] The driver's increased attention is more local, meaning that sharp vision and the corresponding brain processing occur primarily in the forward direction while driving. In the periphery, however, humans primarily process movements. Thus, the driver actively and situationally directs their vision and attention to the relevant areas of the environment – a single driver cannot achieve a visual perception of the entire 360° environment at all times.

[0006] A targeted use and active linking of multiple sensors similar to human perception (scan-and-watch) can improve driver assistance or automated driving systems, especially when using sensors of different modalities.

[0007] According to aspects of the invention, a method for detecting the environment of a first sensor system, a method for training a neural network, a neural network, and a detection device are proposed according to the features of the independent claims. Advantageous embodiments are the subject of the dependent claims and the following description.

[0008] Throughout this description of the invention, the sequence of method steps is presented in such a way that the method is easily understandable. However, those skilled in the art will recognize that many of the method steps can also be performed in a different order and lead to the same or a similar result. In this sense, the order of the method steps can be changed accordingly. Some features are provided with counter words to improve readability or to make the assignment clearer; however, this does not imply the presence of certain features.

[0009] According to one aspect, a method for detecting an environment of a first sensor system is proposed, comprising the following steps: In one step, a temporal sequence of data from the first sensor system is provided for detecting the environment. In a further step, an input tensor with the temporal sequence of data from the first sensor system is generated for a trained neural network, wherein the neural network has been set up and trained to identify at least one sub-area of the environment based on the input tensor in order to improve the detection of the environment by means of a second sensor system. In a further step, a control signal for the second sensor system is generated using an output signal of the trained neural network in order to improve the detection of the environment in at least one sub-area.

[0010] In neural networks, the signal at a connection of artificial neurons can be a real number, and the output of an artificial neuron is calculated as a nonlinear function of the sum of its inputs. The connections of the artificial neurons typically have a weight that adjusts as learning progresses. The weight increases or decreases the strength of the signal at a connection. Artificial neurons can have a threshold, so that a signal is only output when the total signal exceeds this threshold.

[0011] Typically, a large number of artificial neurons are grouped into layers. Different layers may perform different types of transformations on their inputs. Signals travel from the first layer, the input layer, to the last layer, the output layer, possibly after passing through the layers several times.

[0012] Neural networks generally consist of at least three layers of neurons: an input layer, a hidden layer, and an output layer. This means that all neurons in the network are divided into layers, with a neuron in one layer always connected to all neurons in the next layer. Except for the input layer, the different layers consist of neurons that are subject to a nonlinear activation function and are connected to the neurons in the next layer. A deep neural network can have many such intermediate layers.

[0013] Such neural networks must be trained for their specific task. For example, each neuron of the corresponding neural network architecture receives a random initial weight. The input data is then fed into the network, and each neuron weights the input signals with its weight and passes the result on to the neurons of the next layer. The overall result is then provided at the output layer. The magnitude of the error can be calculated, as well as the contribution each neuron made to this error, in order to then change the weight of each neuron in a direction that minimizes the error. Recursive runs are then performed, remeasuring the error and adjusting the weights until the error is below a specified limit.

[0014] For example, the method can be carried out with a passive sensor as the first sensor system and an active sensor as the second sensor system, which are connected either directly or via a processing unit. The first sensor system measures and processes the environment of the first sensor system; the data from this system can then be used according to the method to generate a control signal which is provided to the second sensor system so that the second sensor system can identify sub-regions in the environment of the first sensor system according to the control signal, the detection of which with the second sensor system can improve the detection of the environment. For example, the control signal for the active sensor, i.e. for the second sensor system, can control the second sensor system to determine a certain region of the environment of the first sensor system, such as an angle and / or a distance and / or an elevation, more precisely.

[0015] The control signal is not limited to precisely determining only those areas in which objects have been detected; rather, the control signal can control the second sensor system to more precisely determine suitable subareas or regions of the environment of the first sensor system in order to achieve earlier and / or more precise identification and / or detection and / or determination of objects in the environment of the first sensor system. Furthermore, with the described method for detecting the environment of the first sensor system, it is not necessary to incorporate expert knowledge into the method, since the corresponding knowledge can be learned data-driven using the neural network.

[0016] The use of this method advantageously results in increased robustness and performance when detecting the environment for relevant road users, such as vehicles, pedestrians, etc., without requiring expert knowledge. A traditional model-based control of the active sensor would require expert knowledge to identify potential for improvement and appropriately control the second sensor system.

[0017] Furthermore, the result is a more efficient and cost-effective overall system for the selected task, as the sensors and the algorithms for detecting the relevant road users are optimized together.

[0018] Furthermore, this advantageously results in application-related recognition of relevant objects and easy extensibility of the method.

[0019] In other words, this method can be used to carry out a directed determination of the environment of the first sensor system, for example supplemented with data from the second sensor system, after, for example, an object or a sub-area of the environment has been determined with data from the first sensor system in order to determine the object and / or the sub-area more precisely.

[0020] For example, an extent and / or a position and / or a region and / or an angle and / or a distance and / or an elevation and / or a speed and / or a future probable location and / or a region with objects that were determined with uncertainty and / or an object dimension that was determined with uncertainty can be determined more precisely with the second sensor system.

[0021] The control signal for the second sensor system can relate to a spatial detection of the surroundings and / or the control signal can also be complex and, for example, relate to the exposure of the second sensor system, such as a video system. The control signal for the second sensor system can thus relate to all properties of the second sensor system, via whose control the detection of the surroundings can be changed using the second sensor system.

[0022] The method for detecting the environment of a first sensor system does not require any detected objects. Rather, features from the environment are extracted within the neural network, which serve to derive a sub-area for improved detection of the environment. This can include, among other things, structural occlusions, such as buildings or vegetation, and / or road topology or a road layout, and / or objects such as vehicles and pedestrians.

[0023] In particular, areas that do not have a detected object but have a high probability that an object could appear there in the future can be recorded more precisely.

[0024] With the method for detecting an environment of the first sensor system, objects can be detected faster and earlier, since even partial areas in which no object has been detected before are detected.

[0025] Furthermore, the method is applicable to applications that do not require object detection, such as semantic segmentation and / or detection of lane boundaries and / or compensation of sensor degradations in a multi-sensor system.

[0026] This method can increase the robustness and detection rate for relevant road users, such as vehicles, pedestrians, etc., and regulatory elements such as boundary lines, lanes, sidewalks, etc. Furthermore, a controllable second sensor system can be used to capture and process a small area of the environment very precisely and with high accuracy. This is particularly useful for small objects at large distances, such as lost cargo, in cases of partial obscuration, such as a pedestrian behind a car, or for locating potential obstacles, for the precise detection of markers such as posts, manhole covers, traffic lights, etc.

[0027] Furthermore, this method allows for easier validation of autonomous systems through increased redundancy at the sensor level, since, for example, the controllable second sensor system confirms or falsifies the measurement of the first sensor system. Thus, this method can be used to provide a more cost-effective overall sensor system consisting of at least a first sensor system and a controllable second sensor system with improved overall performance. Furthermore, this method can also be used to combine a sensor that measures the entire environment slowly, as the first sensor system, with a sensor that measures faster within a smaller detection range of the environment, as the second sensor system.

[0028] According to one aspect, it is proposed that the neural network be configured to process time series of data and / or to characterize a time-dependent state of the neural network. A neural network configured in this way is particularly suitable for the described method.

[0029] According to one aspect, it is proposed that the neural network is a recurrent neural network.

[0030] Recurrent or feedback neural networks are neural networks that, in contrast to other networks, are characterized by connections between neurons in one layer and neurons in the same or a previous layer. The feedback can be used to represent a state vector that is passed on from time step to time step and modified if necessary. Using a recurrent neural network makes it particularly suitable for training the method for generating a corresponding control signal, since such a recurrent neural network can access sensor data from previous time steps for a current time step. This can improve the detection of the environment of the first sensor system.

[0031] According to one aspect, it is proposed that the input tensor of the neural network comprise data from the second sensor system in order to improve the detection of the environment of the first sensor system. Advantageously, the representation of the environment of the first sensor system can be improved using the additional data from the second sensor system. The input tensor can comprise the data from the second sensor system from a current time step and / or a previous time step. The training of the neural network can be adapted according to the respective time step in which the data from the second sensor system is generated.

[0032] According to one aspect, it is proposed that the input tensor of the neural network comprises data of the second sensor system from a time step preceding a current time step in order to improve the detection of the environment of the first sensor system.

[0033] Advantageously, the generated control signal is based, among other things, on the sensor data of the second sensor system from the previous time step and can thus improve the representation of the environment of the first sensor system with the data of the second sensor system for the neural network.

[0034] According to one aspect, it is proposed that the input tensor of the neural network comprises data from the second sensor system from a current time step, and that the second sensor system is controlled with the generated control signal from a previous time step in order to improve the detection of the environment of the first sensor system. Advantageously, the representation of the environment of the first sensor system can be improved with the data from the second sensor system from the current time step and a control signal from a previous time step, since the data from the respective sensor systems are generated in the same time step and thus have a higher level of synchronicity. The training of the neural network can be adapted according to the respective time step in which the data from the second sensor system is generated.

[0035] According to one aspect, it is proposed that the neural network be configured and trained to create an object list from the environment of the first sensor system. Advantageously, such an object list can be used to train the neural network or to provide the objects identified in this way from the object list to other active systems for sensing the environment.

[0036] According to one aspect, it is proposed that the first sensor system is a passive sensor system and the second sensor system is an active sensor system.

[0037] An active sensor system can be a sensor system whose signal or generated data or detection properties can be selectively adapted and / or controlled to specific detection requirements depending on a control signal with respect to the detection of an environment. Such detection properties of the controllable sensor system can, in particular, relate to the position of an object within the detection range of the controllable sensor system, such as determination in selected sub-areas and / or limited distance ranges and / or elevation angle ranges. However, it can also relate to measurement properties, such as spatial resolution, etc. Accordingly, a controllable sensor system can be an active sensor system. Examples of controllable sensor systems are radar sensors, LIDAR sensors, infrared sensors, time-of-flight sensors, and optical sensors.

[0038] In general, sensors can be used that can be controlled by a controllable signal source acting on the object or the environment, such as thermography sensors that determine temperature changes that are controlled at a specific point on the object to be determined by introducing eddy currents or other heat input.

[0039] In general, the wavelength of the introduced controllable signal source can also be changed, e.g. in the range of 0.7µm - 1000µm for an infrared source or in the range of 300 - 1600nm for a LIDAR sensor.

[0040] According to one aspect, it is proposed that the data of the second sensor system be reduced by means of the control signal.

[0041] Alternatively or additionally, the determination of the object with the first sensor system can also be carried out by the controllable second sensor system by a corresponding change in the detection properties of the controllable second sensor system, wherein this change in the detection properties relates in particular to a spatial size of a determination area of the environment of the mobile platform.

[0042] By using this control of the second sensor system, the amount of sensor data can be reduced, for example, to save computing time and bandwidth. One example of this is a video sensor that, in response to the control signal, only delivers a selected image section instead of the entire image.

[0043] According to one aspect, it is proposed that the data of the first sensor system comprise a steering angle of a mobile platform and / or geographical map data and / or a planned route and / or a category of a road and / or weather conditions.

[0044] Alternatively or additionally, the first sensor system may also be an active sensor system that is either used in accordance with a passive sensor system or is controlled by another control variable, such as a steering angle signal of a mobile platform and / or a localization signal.

[0045] To control the second sensor system using map data, the first sensor system can be a localization sensor that determines a position on a map or in an environment. The map data can also be used to derive features and sub-areas for improved detection of the environment.

[0046] An example of useful sub-areas that can be derived from this are the course of the road and, in particular, the vanishing point of the road. Near the vanishing point, objects become visible for the first time. This is shown in the Figures 3a to d A few examples are shown. The areas that require more detailed coverage are highlighted by hatching.

[0047] Alternatively or additionally, a single active sensor can be used as the second sensor system without the data from the second sensor system being used to generate a control signal, and no data from a first sensor system is provided to the neural network as an input tensor. The input tensor for the neural network then comprises, for example, existing signals from a mobile platform for generating the control signal, with which the trained neural network either alone or together generates a control signal for the second sensor system. Examples of such input data are: a steering angle of a mobile platform and / or map data and / or a course of a planned route for a mobile platform and / or a road category (city, motorway, ...) and / or weather conditions.

[0048] Alternatively or additionally, the input tensor may comprise a plurality of temporal sequences of data from a plurality of first sensor systems. Alternatively or additionally, the first sensor system may generate a plurality of control signals that are provided to a plurality of active sensor systems.

[0049] According to one aspect, it is proposed that the first sensor system is an optical camera system and the second sensor system is a LIDAR sensor and / or a RADAR sensor and / or an ultrasonic sensor.

[0050] An optical camera system can be an optical camera in the narrow sense or a video system.

[0051] According to one aspect, it is proposed that the first sensor system is the same sensor system as the second sensor system; and the control signal from the sensor system is used in a subsequent time step for improved detection of the environment.

[0052] Such a subsequent time step can be the respective following time step.

[0053] According to this aspect, the input data for the neural network may not include any data from a first sensor system. Thus, the control of the second sensor system can be based, in particular, on the data from the second sensor system that was generated in a time step preceding the current time step.

[0054] For example, an active LIDAR sensor can be operated alternately in "active" and "passive" modes. In "passive" mode, an overview of the surrounding scene can be generated by capturing the entire field of view of the second sensor system with low spatial resolution. Using the information obtained, small areas of the field of view can be measured very precisely in the next time step in "active" mode.

[0055] According to one aspect, it is proposed that the control signal controls the second sensor system to detect the sub-area of the environment in order to improve the detection of the environment of the first sensor system.

[0056] According to one aspect, it is proposed that the input tensor of the neural network comprises a steering angle of a mobile platform and / or geographical map data and / or a planned route and / or a category of a road and / or weather conditions and / or a current task of an autonomous system and / or a list of objects and / or areas to be measured more precisely.

[0057] Examples of a current task of an autonomous system could be: a driving task, such as parking and / or avoiding an obstacle, etc.

[0058] A method for training a neural network for generating a control signal for a second sensor system is proposed, comprising the following steps: In one step, an input tensor for the neural network is provided with a temporal sequence of data from a first sensor system for detecting an environment of the first sensor system, wherein the input tensor of the neural network comprises data from a second sensor system from a time step preceding a current time step. In a further step, at least one object in the environment of the first sensor system is generated using the neural network and the input tensor. In a further step, a control signal is generated using the neural network and the input tensor. In a further step, the generated at least one object is compared with at least one correspondingly assigned reference object.In a subsequent step, data from a second sensor system is generated based on the control signal for the next time step. In a further step, the neural network is adapted to minimize deviations from the respective reference object when determining the object in the environment.

[0059] For the method of training a neural network, the neural network can be configured to process time series of data and / or to characterize a time-dependent state of the neural network. Alternatively or additionally, the neural network used for the training method can be a recurrent neural network.

[0060] Reference objects are objects that were generated specifically for the training of a neural network together with the corresponding input data for the neural network and that are labeled accordingly.

[0061] According to one aspect, it is proposed that for training the neural network, the at least one object and the correspondingly assigned reference object each be an object list with at least one object of the environment of the first sensor system and / or the at least one object and the correspondingly assigned reference object each be a high-resolution representation of the environment of the first sensor system. Such a high-resolution representation of the environment can, for example, be an optically generated image of the environment and / or a representation generated using a LIDAR system.

[0062] According to one aspect, it is proposed that, in order to train the neural network, at least one object of the object list is generated by means of an object detector.

[0063] In other words, the training of the neural network, and in particular the recurrent neural network, can be carried out with a separate object detector by training the neural network to generate the control signal, and generating at least one object, for example from an object list, by the object detector, for example based on the data from the first sensor system. By generating the at least one object, the object detector provides the necessary feedback for the neural network to learn the control signal. The object detector itself can either remain unchanged or be incorporated into the training of the neural network.

[0064] According to one aspect, it is proposed that for training the neural network, the temporal sequence of data of the first sensor system is data of a real first sensor system or simulated data for the first sensor system.

[0065] For training the neural network, and especially for training recurrent neural networks with annotated data, data for the second sensor system must be provided, as this data is generated depending on the control signal from the second sensor system. The second sensor system thus provides data that depends significantly on its control. Therefore, it is not possible to collect data for the second sensor system that can subsequently be used in its unaltered form for training a neural network.

[0066] Furthermore, the sensor data for the first and second sensor systems must be annotated. As is common in object detection, each relevant object is annotated with a bounding box and additional attributes, such as an object type and / or speed, etc.

[0067] During training, the neural network is adapted using a loss function that describes the optimization objective. Such a loss function can have at least two parts. The first part can be a common loss function (multi-task loss) from the field of object recognition, which has a regression and classification part.

[0068] When adapting the neural network, for example, objects can be compared with annotated data (ground truth). A second part of the

[0069] The loss function can formulate further optimization objectives based on the control signals. One example of this is minimizing the number of data points in a LIDAR system, for example, to achieve cost reduction for the respective sensor system.

[0070] The neural network can be adapted and trained using backpropagation. By applying the loss function described above, the neural network learns to recognize relevant objects during training, for example. Since the control signal for the second sensor system can have a significant influence on how well objects are detected, the neural network will modify the control signal to ensure the best possible detection of the objects.

[0071] In addition to the described methods for training the neural network, other well-known methods from the field of machine learning can be used for training.

[0072] Some of these methods are: · Deep learning · Reinforcement learning · Active learning · Unsupervised / semi-supervised learning

[0073] According to one aspect, it is proposed that for training the neural network, the data of the second sensor system are generated by means of a high-resolution sensor system and / or the data of the second sensor system are generated by means of a simulation program for simulating the second sensor system and / or the data of the second sensor system are generated with the second sensor system in the environment of the first sensor system.

[0074] In other words, for training the neural network, both the data from the first sensor system and the data from the second sensor system can be generated using a simulation program. Alternatively or additionally, the data from the second sensor system for training the neural network can be generated using data from a high-resolution sensor system. Depending on the control signal, data corresponding to a second sensor system can be generated from the data of a high-resolution sensor system. For example, based on the control signal, the data corresponding to at least a partial area of the environment could be selected from the data of a high-resolution sensor system. Alternatively or additionally, depending on the control signal, data with a lower resolution could be selected from the data of the high-resolution sensor system for training the neural network.According to this aspect, the high-resolution sensor would be used to generate and store high-resolution data, which would be selected for training according to the control signal to simulate data from a second sensor system.

[0075] Alternatively or additionally, the neural network can be trained with data from real-world sensor measurements, using data directly from the second sensor system. Using a reference system, the reference data required for training, including annotated objects and / or high-resolution sensor data, could be generated simultaneously.

[0076] The neural network can also be trained without annotated data by using a reconstruction loss as the minimizing objective function. This makes it possible to learn the control signal for the second sensor system, since such a reconstruction loss contains the most valuable information from an information-theoretic perspective. For example, it can be learned that a building that has already been measured does not need to be measured again in the next time step, as it will not move or change. However, a pedestrian is measured more accurately in each time step, as they can move and change their appearance, which is not easy to predict.

[0077] To do this, a neural network structure is adapted to output high-resolution sensor data instead of an object list. The loss function can compare the neural network's output signal with high-resolution sensor data for the second sensor system, for example, from a simulation, and determine how well the data match using a metric. Metrics can include, among others, the sum of the absolute differences. In this way, the neural network learns to design the control signal in such a way that high-resolution sensor data can be generated from past data and the low-resolution sensor data of the first sensor system and / or the second sensor system.

[0078] According to one aspect, it is proposed that for training the neural network, the neural network is configured to process time series of data and / or to characterize a time-dependent state of the neural network and / or the neural network is a recurrent neural network.

[0079] A neural network is proposed which is set up and trained according to one of the methods for training the neural network described above.

[0080] A method is proposed, corresponding to one of the methods described above, for detecting an environment of a first sensor system, which has a neural network that is set up and trained according to one of the methods described above for training the neural network.

[0081] A detection device is proposed which is configured to carry out one of the methods described above for detecting an environment of a first sensor system.

[0082] A mobile platform is proposed which is at least partially automated and has one of the above-described detection devices for detecting the environment of the mobile platform and / or wherein the mobile platform has a first sensor system and a second sensor system as described above.

[0083] This allows the environment of the mobile platform to be recorded with less economic effort and high quality of recording.

[0084] A mobile platform can be understood as an at least partially automated system that is mobile and / or a driver assistance system. An example can be an at least partially automated vehicle or a vehicle with a driver assistance system. This means that in this context, an at least partially automated system includes a mobile platform with respect to at least partially automated functionality, but a mobile platform also includes vehicles and other mobile machines, including driver assistance systems. Other examples of mobile platforms can be driver assistance systems with multiple sensors, mobile multi-sensor robots such as robot vacuum cleaners or lawnmowers.

[0085] The described method for detecting an environment of a first sensor system can be used for mobile platforms and / or also for multi-sensor monitoring systems and / or a manufacturing machine and / or a personal assistant and / or an access control system.

[0086] Each of these systems can be a fully or partially automated system.

[0087] A computer program is proposed that has instructions that, when executed by a computer, cause the computer to perform one of the methods described above. Using such a computer program, the methods described above can be made available in a simple manner, for example, on a mobile platform.

[0088] A machine-readable storage medium is proposed on which the computer program described above is stored. The computer program product described above is transportable by means of such a machine-readable storage medium.

[0089] The use of the control signal as described above for controlling an external sensor system is proposed. Thus, the control signal can be used alternatively or additionally for controlling the second sensor system and / or another external sensor system.

[0090] The use of one of the methods described above for detecting an environment of a first sensor system for controlling an at least partially automated mobile platform is proposed.

[0091] A method is proposed in which, based on an environment of a first sensor system detected according to one of the methods described above, a control signal for controlling an at least partially automated vehicle is generated; and / or based on the detected environment of the first sensor system, a warning signal for warning a vehicle occupant is generated. Example

[0092] An embodiment of the invention is described with reference to the Figures 1 to 8 presented and explained in more detail below. They show: Figure 1 shows a mobile platform with sensors of different modalities with at least one controllable sensor; Figure 2a shows a recurrent neural network; Figure 3 shows highlighted areas of different traffic situations; Figure 4 shows method steps for detecting an environment; Figure 5 shows a detection device with data flows; Figure 6 shows data flows during training of a detection device; Figure 7 shows a modified setup for training a detection device; and Figure 8 shows possible applications of the method.

[0093] The Figure 1 schematically shows a vehicle 170 having a system 100 for detecting the surroundings of a mobile platform. The vehicle 170 has a video camera 110, which corresponds to a first sensor system and is signal-coupled to a detection device 130 in order to provide images generated by the camera to the detection device 130.

[0094] A controllable radar sensor 120 of the vehicle 170, which corresponds to a second sensor system, is bidirectionally coupled to the detection device 130 and provides its signals to the detection device 130. The controllable radar sensor 120 can be controlled with respect to its perception of the surroundings using a generated control signal.

[0095] With such a system, a control signal can be used to specifically control an active second sensor system 120, such as a radar or lidar sensor, so that a targeted detection of the surroundings can be achieved. The control signal can be transmitted to the second sensor system 120 via the connection signal 125, which transmits signals from the control unit 130 to the second sensor system 120.

[0096] The detection device 130 is configured to carry out one of the methods described above for detecting the surroundings using the data from the video camera 110 and the radar sensor.

[0097] Furthermore, the detection device can be coupled to a control unit 140, which is coupled, for example, to a brake 160 or a steering system 150. As a result, the control unit 140 can, for example, control an automatic emergency braking depending on signals from the detection device 130 or regulate an automatic evasive maneuver using the steering system.

[0098] The Figure 2aoutlines a structure of a recurrent neural network (RNN) 200 with an input terminal 210 to which the input tensor can be provided and an output 220 to which, for example, a control signal and / or an object list as described above can be provided. The state variable V 230 and the arrow indicate the recurrent structure of the neural network.

[0099] The Figure 2b2 outlines the structure of the recurrent neural network 200, illustrating the recurrent structure with a "rolled-out" recurrent neural network. For example, temporal sequences of data from the first sensor system can be provided at the input terminals 210a to c, for time steps t-1, t, t+1, from which corresponding output signals 220a to 220c are generated by the neural network. The respective previous state of the recurrent neural network 200 can be used via a state vector V t-2 , V t-1 , V t , V t+1 to characterize the current state of the recurrent neural network 200. The neural network itself remains unchanged during the process for detecting the environment of a first sensor system.

[0100] In other words, at time t, input data xt and the state vector of a previous time step Vt-1 are processed by the neural network 200. This generates the output 220a to 220c and a new state Vt. The state Vt+1 again serves as an input for the neural network in the next time step t+1.

[0101] The Figure 3 a to d outlines four different street scenes or traffic situations in which the hatched fields highlight sub-areas that can be detected, for example, by the second sensor system using the control signal in order to improve the detection of the surroundings of the first sensor system, as already explained above.

[0102] The Figure 4outlines method steps for detecting the environment of the first sensor system 510: In a step S1, which lies before a current time step, data is generated by a second sensor system 520, which is provided for generating the input tensor.

[0103] In a further step S2, a temporal sequence of data from the first sensor system 510 for detecting the environment is provided in a current time step, which data, for example, covers a large area of the environment.

[0104] In a further step S3, an input tensor is generated with the temporal sequence of data from the first sensor system 510 and the data provided by the second sensor system 520 for a trained neural network, wherein the neural network has been set up and trained to identify at least a partial area of the environment based on the input tensor in order to improve the detection of the environment by means of a second sensor system.

[0105] In a further step S4, a control signal 540 is generated for the second sensor system 520 using an output signal from the trained neural network 200 in order to improve the detection of the surroundings in at least one partial area. To this end, the neural network 200 can internally extract features, such as occlusions and / or a road layout and / or objects, and optionally also output them as an object list 530. These extracted features can be used internally in the neural network 200 to determine areas, such as angles and / or distances, that are to be measured more precisely. In these areas, there is a high probability that new objects will become visible or that already detected objects can be detected more accurately. Thus, the neural network 200 can generate the appropriate control signal for the second sensor system 520 depending on these areas.

[0106] In a further step S5, the control signal controls the second sensor system 520 to detect the partial area of the surroundings in order to improve the detection of the surroundings of the first sensor system 510. Using this control signal 540, the second sensor system can, for example, limit its detection range. The second sensor system 520 can then generate data for such areas that have increased accuracy and can then be used to improve the detection of the surroundings of the first sensor system.

[0107] The Figure 5outlines data flows during operation of a detection device for detecting an environment of a first sensor system 510, with three representations of the detection device arranged side by side, wherein a previous time step is arranged on the far left 501, a current time step is arranged in the middle 502, and a future time step is arranged on the right. Only in the middle representation of the detection device are the respective details of the detection device shown. The detection device has a trained recurrent neural network 200. The first sensor system 510 can, for example, be a passive video sensor and / or a LIDAR sensor system, and the second sensor system 520 can be an actively controllable LIDAR sensor system. These sensor systems periodically supply sensor data.

[0108] Data from the first sensor system 510 should be able to be used to control the second sensor system 520 so that suitable areas in the vicinity of the first sensor system 510 are detected more accurately. This allows, for example, relevant road users to be detected earlier and more reliably and / or the number of ghost objects to be reduced.

[0109] At time t, input signals 210b containing data from the first sensor system 510 are provided to the deep learning-trained neural network 200. The same applies to a previous time t-1 with the input data 210a and to the subsequent time t+1 with the input data 210c.

[0110] In addition, sensor data c t-1 540a from the second sensor system 520 from a previous time t-1 and a state vector of the neural network d t-1 230b from a previous time t-1 are provided to the trained neural network 200, thereby generating an input tensor.

[0111] The state vector d t-1 230b represents a state vector of the neural network at time t-1, with the help of which the neural network 200 can store information over time steps.

[0112] In addition, the input tensor may include further data 210b, such as a vehicle speed and / or a vehicle link angle, etc.

[0113] Based on the described input data and its learned logic, the neural network 200 generates a control signal bt 540 for the second sensor system 520 and a list of detected objects at 530, such as a list of recognized relevant road users characterized by their position and / or dimensions and / or orientation and / or object type.

[0114] For this purpose, the neural network 200 aggregated information from the sensor systems 510, 520 over previous time steps. The particularly accurate estimate, based on the object list at of the relevant road users, can then be used to control a mobile platform, such as a vehicle.

[0115] The signal bt can contain information such as angle and distance ranges that the active controllable second sensor system is to measure.

[0116] By means of the control signal bt 540, a detection of the surroundings of the first sensor system 510 by the second sensor system 520 is controlled in a subsequent future time step t+1. For this purpose, data ct 540b of the second sensor system 520 and the state vector dt 230c of the neural network 200 at time t can be provided for the next time step t+1.

[0117] The Figure 6 outlines data flows during training of a detection device for detecting an environment of a first sensor system.

[0118] In addition to the data flows defined in Figure 5 described detection device for detecting an environment of a first sensor system 510 have already been described, are described in the Figure 6the data flows from the object list at 530 and the control signal bt 540 to the loss function 610 are shown in order to be able to carry out a comparison of the at least one generated object by the neural network 200 with a correspondingly assigned reference object, as described above, by means of reference objects that are not shown here.

[0119] To collect data from the second sensor system 520 of the Figure 5 depending on the control signal bt 540 is to be generated in the Figure 6 A simulation program 620 is provided instead of a second sensor system 520. The process steps of training the neural network 200 have already been described above.

[0120] The Figure 7 outlines the data flows with a modified setup of the neural network when training a neural network 200 to detect the environment of the first sensor system.

[0121] During this training of the neural network 200, the neural network 200 is trained with the aid of a separate object detector 710 to generate a control signal for the second sensor system.

[0122] Both the neural network 201 and the object detector 710 receive sensor data c t-1 540a from the second sensor system 520 from a previous time t-1. The neural network 201 thus learns to generate the control signal bt 540, and the detector 710 generates the at least one object from the object list at 530 based on the data from the second sensor system 520. The detector 710 thus provides the feedback for training the neural network to generate the control signal bt 540 by providing the at least one object from the object list at and can itself remain unchanged or be optimized. The respective state of the object detector 710, which is characterized, for example, by an object list at 530, can be used in the next time step t+1 as an input variable for the neural network 201 according to the state vector dt 230c.Alternatively or additionally, the input signal 210b can be provided as an input variable for the neural network 201 and the object detector 710. In other words, the neural network 200 is divided into a neural network 201 for generating the control signal bt 540 and the object detector 710.

[0123] The Figure 8a to d outlines further possible areas of application of one of the described methods.

[0124] The Figure 8a outlines the application of one of the described methods in automated testing systems, e.g. for the inspection of components using thermography, eddy current and conventional optics in order to reliably sort out defective components.

[0125] The Figure 8b outlines the application of one of the described methods for an automated lawnmower, e.g. for the reliable identification or classification of objects, in particular the differentiation of an obstacle from a non-obstacle.

[0126] The Figure 8c outlines the application of one of the described procedures for automatic access control, e.g. for optical and acoustic personal identification and automatic door opening.

[0127] The Figure 8d outlines the application of one of the described methods for monitoring places or buildings, e.g. for checking dangerous goods, for example with a camera and a LIDAR sensor.

Claims

1. Method for detecting an environment of a first sensor system (510), having the steps of: providing a temporal sequence of data from the first sensor system (510) for detecting the environment; generating an input tensor with the temporal sequence of data from the first sensor system (510) for a trained neural network (200), wherein the neural network (200) was set up and trained to identify, based on the input tensor, at least one section of the environment for improving the detection of the environment by means of a second sensor system (520); generating a control signal (540), which is provided for the second sensor system (520), by means of an output signal from the trained neural network, in order to improve the detection of the environment by means of the second sensor system (520) in the at least one section.

2. Method according to Claim 1, wherein the neural network (200) is configured to process time series of data and / or to characterize a time-dependent state of the neural network (200).

3. Method according to one of the preceding claims, wherein the input tensor of the neural network (200) has data from the second sensor system (520) from a time step preceding a current time step in order to improve the detection of the environment of the first sensor system (510).

4. Method according to one of the preceding claims, wherein the first sensor system (510) is a passive sensor system and the second sensor system (520) is an active sensor system.

5. Method according to one of the preceding claims, wherein the first sensor system (510) is the same sensor system as the second sensor system (520); and the control signal from the sensor system is used in a subsequent time step for improved detection of the environment.

6. Method according to one of the preceding claims, wherein the control signal controls the second sensor system (520) to detect the section of the surroundings in order to improve the detection of the environment of the first sensor system (510).

7. Method for training a neural network (200) for generating a control signal for a second sensor system (520), having the steps of: providing an input tensor for the neural network (200) with a temporal sequence of data from a first sensor system (510) for detecting an environment of the first sensor system (510), wherein the input tensor of the neural network (200) has data from a second sensor system (520) from a time step preceding a current time step; generating at least one object in the surroundings of the first sensor system (510) by means of the neural network (200) and the input tensor; generating a control signal (540) by means of the neural network (200) and the input tensor; comparing the generated at least one object with at least one accordingly assigned reference object; generating data from a second sensor system (520) based on the control signal (540) for a next time step; and adapting the neural network (200) in order to minimize a deviation from the respective reference object when determining the object in the surroundings.

8. Method according to Claim 7, wherein the at least one object (530) and the accordingly assigned reference object are each an object list with at least one object in the environment of the first sensor system (510), and / or the at least one object and the accordingly assigned reference object are each a high-resolution representation of the surroundings of the first sensor system (510).

9. Method according to Claim 7, wherein the at least one object in the object list (530) is generated by means of an object detector (710).

10. Method according to one of Claims 7 to 9, wherein the temporal sequence of data from the first sensor system (510) are data from a real first sensor system (510) or simulated data for the first sensor system (510).

11. Method according to one of Claims 7 to 10, wherein the data from the second sensor system (520) are generated by means of a high-resolution sensor system, and / or the data from the second sensor system (520) are generated by means of a simulation program for simulating the second sensor system, and / or the data from the second sensor system (520) are generated with the second sensor system (520) in the environment of the first sensor system.

12. Method according to Claims 7 to 11, wherein the neural network (200) is configured to process time series of data and / or to characterize a time-dependent state of the neural network (200), and / or the neural network (200) is a recurrent neural network.

13. Neural network (200) which is set up and trained according to one of Claims 7 to 12.

14. Method according to Claims 1 to 6, with a neural network (200) according to Claim 13.

15. Detection device configured to carry out a method according to one of Claims 1 to 6 or 14.