Dynamic spatial scene analysis

Abstract

The invention relates to a method and a system (100) for preparing data of a dynamic spatial scene (10), a computer-assisted method and a system (200) for training an artificial neural network (1), and a computer-assisted method and a system (300) for analyzing sensor data. A representation (20) of the time profile of an angular sector obscured by other objects (12, 12a, 12b, 12c) from the perspective of an autonomous object (11a) is generated. For this purpose, the time profile is determined from the sensor data, wherein the sensor data is suitable for characterizing a dynamic spatial scene (10) involving the autonomous object (11a) and at least one other object (12, 12a, 12b, 12c).

Description

Technical Field

[0001] The invention relates to a method and a system for preparing data of a dynamic spatial scene for processing by an artificial neural network, a computer-supported method and a system for training an artificial neural network, and a computer-supported method and a system for analyzing sensor data. Background Art

[0002] Modern vehicles are increasingly equipped with driver assistance systems (ADAS, Advanced Driver Assistance Systems) that support the driver in certain driving situations. This support ranges from the mere display of potentially important information (e.g. warnings via a lane change assistant) to semi-automatic intervention (e.g. regulation of the torque applied to the axles via an anti-lock braking system) to fully automatic intervention in the control of the vehicle (e.g. adaptive speed regulation via a cruise control, ACC, Adaptive Cruise Control).

[0003] Sensor data, such as signals provided by ultrasonic sensors, radar sensors or cameras, usually form the basis of such driver assistance systems, with the aid of which the current driving situation can be determined and the functions of the corresponding driver assistance system can be implemented in response to the current driving situation. In particular, in driver assistance systems that (automatically) intervene in the control of the vehicle, it must be possible to classify the current driving situation with a high degree of reliability using the sensor data.

[0004] Specific rules or criteria are usually defined that are assigned to the driving situation, and when they are met, it can be inferred that a known driving situation exists. Meeting the rules or criteria serves, for example, as a triggering condition for an action of the driver assistance system. For example, a traffic situation in which an adjacent vehicle in front of the ego vehicle, i.e., in front of the vehicle equipped with the driver assistance system, merges into the same lane can be detected, i.e., the lateral distance to the adjacent vehicle detected by the sensor at right angles to the driving direction decreases and finally assumes at least substantially the value 0 when the adjacent vehicle is directly in front of the ego vehicle.

[0005] The programming of algorithms suitable for (reliable) recognition of such driving situations or scenarios is usually very complex and may even be impossible, especially in the case of a large number of driving situations that need to be taken into account. Machine learning is therefore increasingly being used to automatically find features in sensor data that indicate driving situations. The basic sensor data usually involve, for example, sensor data that have been manually classified for different driving situations (labeled sensor data in English), from which an artificial neural network can, for example, extract important criteria for the corresponding driving situation. In this case, the classified sensor data usually have to be in a predetermined form, that is, they have to be prepared accordingly in order to be able to be processed reasonably by the artificial neural network.

[0006] So-called grid maps are a practical approach for such preparation, which show the corresponding driving situation from a bird's eye view. For example, by plotting all the positions occupied over time by the corresponding road users in the corresponding grid map, the dynamics of the basic driving situation are reproduced.

[0007] Such grid maps are discussed, for example, in Gruner et al., "Spatial-temporal Representation of Driving Scenarios and Classification Using Neural Networks" (2017, IEEE Intelligent Vehicle Conference, pp. 1782-1788).

[0008] In Cara and de Gelder, "Classification of safety-critical vehicle-cyclist scenarios using machine learning" (2015, IEEE, 18th International Conference on Intelligent Transportation Systems, pp. 1955-2000), algorithms for classifying vehicle-cyclist scenarios are compared. In this case, a training set of labeled objects is used in supervised learning to train a so-called classifier. After training, the classifier is able to label new, unknown scenarios in such a way that it assigns them to an estimated class. Summary of the invention

[0009] The object of the present invention is to further improve the processing of sensor data suitable for characterizing a dynamic spatial scene involving an autonomous object and at least one other object. In particular, the object of the present invention is to improve the preparation of these sensor data for processing by an artificial neural network, in particular to prevent or at least reduce information losses during the preparation.

[0010] This object is achieved by a method and a system for data preparation, a method and a system for training an artificial neural network, and a method and a system for analyzing sensor data according to the independent claims.

[0011] A first aspect of the invention relates to a method for preparing data of a dynamic spatial scene, in particular a traffic scene, for processing by an artificial neural network. In this case, a graphical representation of the time profile of an angular sector that is obscured by other objects, in particular other vehicles, from the perspective of an autonomous object, in particular an autonomous vehicle, is generated. For this purpose, the time profile is determined from sensor data, wherein the sensor data is suitable for characterizing a dynamic spatial scene involving the autonomous object and at least one other object.

[0012] According to the invention, a spatial scene is formed in particular by a temporal sequence of spatial, in particular static, scenes. In this context, a spatial scene describes, for example, the spatial arrangement of at least one other object relative to the autonomous object, such as the situation of a traffic participant. A spatial scene can in particular include a driving situation in which the driver assistance system at least partially controls the autonomous vehicle, i.e. the vehicle equipped with the driver assistance system, for example automatically performs at least one vehicle function of the autonomous vehicle.

[0013] According to the invention, dynamic means in particular a time series or a time profile. Traffic scenes are dynamic, for example, because individual road users move relative to one another over time, ie their positions change relative to one another.

[0014] According to the invention, a graphic representation is particularly a graphical display, such as a picture or a diagram. A graphic representation preferably includes a two-dimensional or three-dimensional diagram. Alternatively or additionally, a graphic representation may also include a mathematical representation, such as a distributive law or a function. A graphic representation may, for example, show the time profile of an angular sector that is obscured by other objects from the perspective of the main object in a map-like diagram.

[0015] According to the invention, sensor data are data generated by real or simulated sensors, in particular environmental sensors, in particular in the form of signals, that is to say real or simulated sensor data. The sensors are preferably provided to detect the surroundings of the autonomous vehicle and to generate corresponding sensor data, so that the sensor data characterize the surroundings of the autonomous vehicle. The sensor data are preferably sensor fusion data, which are obtained from the combination of the signals provided by the sensors and have been processed, at least to a certain extent, if necessary.

[0016] According to the invention, an autonomous object is in particular an object from whose point of view a dynamic spatial scene is viewed. An autonomous object can be, for example, an (autonomous) vehicle from whose point of view a traffic scene is detected and, if necessary, analyzed and / or evaluated, for example in order to be able to appropriately control a driver assistance system or to react to the detected traffic scene. Thus, for example, the sensor data prepared for further processing are preferably generated by environmental sensors of the autonomous vehicle, wherein environmental sensors, for example ultrasonic sensors, radar sensors, cameras and / or the like, are provided for detecting the surroundings of the autonomous vehicle.

[0017] According to the invention, an angular sector refers in particular to the area occupied by other objects from the perspective of the autonomous object. From the perspective of the autonomous vehicle, an angular sector corresponds, for example, to an area covered by other vehicles. The angular sector is preferably defined by a cross section of the object or a contour of the object. The angular sector can be defined, for example, by the maximum extent of the cross section or contour along a predetermined direction, in particular along the horizontal direction.

[0018] The invention is based in particular on a practical concept that information about a dynamic spatial scene, such as a traffic situation, determined from sensor data is encoded in a diagram. The diagram can therefore also be understood as an abstraction of the scene and is used in particular to summarize different types of information, such as the position, translation, quantity and / or similar information of objects.

[0019] The representation is preferably generated based on the time development of angular sectors in which the field of view of the autonomous object is obscured by other objects. The angular sectors can be used to illustrate in particular the position of the objects relative to one another in a spatial situation. The representation is preferably a graphic, in particular a two-dimensional or three-dimensional, diagram that shows the position and movement of the objects relative to one another in an abstract manner. The movement of the objects can be represented in the diagram, for example, by a shape, in particular a geometric shape.

[0020] Preferably, a representative shape can be found based on the analysis of such a diagram. In this case, the shape is used in particular to characterize a dynamic spatial scene involving the autonomous object and at least one other object. In other words, the representative shape is a pattern that can be found, for example, by means of an artificial neural network by analyzing one such diagram, or optionally a plurality of such diagrams generated based on sensor data.

[0021] If, on the other hand, characteristic shapes or patterns for different dynamic spatial scenes are known and exist, for example, as templates, the corresponding dynamic spatial scene is determined by evaluating the representation.

[0022] The advantage of the angular sector-based representation of the temporal evolution of a dynamic spatial scene is that its information content is independent of the changes in the scene or at least only slightly dependent. Different variations of the scene, such as particularly fast or slow driving maneuvers, produce different images in the representation, but have essentially the same or at least similar shapes.

[0023] The plotting of angular sectors that are obscured by other vehicles with respect to the field of view of the autonomous vehicle and therefore show the position of the other vehicles relative to the autonomous vehicle, for example, generates a pattern over time, the shape of which characterizes the driving maneuvers performed by the other vehicles. If the other vehicle, for example, moves away from the autonomous vehicle, the obscured angular sectors, for example, become smaller, and when approaching, the angular sectors become correspondingly larger. If the other vehicle, for example, moves laterally with respect to the direction of travel of the autonomous vehicle, the obscured angular sectors move. The pattern generated here, for example, with a characteristic curvature, can then be reliably assigned to a driving maneuver, even if the diagram usually changes, such as being compressed or stretched, depending on the execution of the driving maneuver, for example an offensive or defensive driving maneuver.

[0024] The graphical representation of the temporal evolution of the angular sectors thus allows for the summary of different types of information in a compact manner. Since the time delay is shown here, for example, overlapping of image elements and the associated information losses are avoided or at least reduced. At the same time, for example, storage space can be saved and the evaluation of the sensor data can be accelerated. The graphical representation simultaneously enables the reliable identification of dynamic spatial scenes.

[0025] For example, the graphical representation of the temporal evolution of an angular sector has the advantage that the same information can be represented with a much smaller data set for a grid map. Individual pixels in a grid map that are not occupied by other vehicles (that is to say "empty" pixels) do not contribute to the information content that can be used, for example, by an artificial neural network. The data set required for training an artificial neural network can be reduced by a factor of 27, in particular, by using the graphical representation of the temporal evolution of an angular sector.

[0026] Overall, the invention allows further improvement of the processing of sensor data. In particular, the preparation of sensor data for processing by an artificial neural network can be improved, for example, in such a way that information losses during the preparation are prevented or at least reduced.

[0027] Preferred embodiments of the invention and their developments are described below, which can be combined as desired with one another and with aspects of the invention still to be described below, unless this is explicitly excluded.

[0028] In a preferred embodiment, the representation is output to an artificial neural network or provided for processing by an artificial neural network. The representation can be stored, for example, as a data packet, in particular as a digital image, on a storage medium or directly transmitted to the artificial neural network via an interface. Thus, a large amount of sensor data can also be used for training the artificial neural network, in particular for determining patterns in the representation that characterize a predetermined scene, such as the shape of a figure in the representation.

[0029] A second aspect of the invention relates to a computer-supported method for training an artificial neural network based on sensor data, which are suitable for characterizing a known dynamic spatial scene involving an autonomous object and at least one other object. In this case, a graphical representation of the time profile of an angular sector, which is obscured by other objects, in particular other vehicles, from the perspective of the autonomous object, in particular the autonomous vehicle, is generated based on the sensor data. The generated graphical representation is fed to the artificial neural network together with information about the spatial scene. This makes it possible to train the artificial neural network particularly quickly and reliably to recognize patterns that characterize different scenes.

[0030] The information about the spatial scene preferably includes a description of the classification (label) of the scene, by means of which the scene can preferably be clearly and unambiguously identified. The information can already be contained in the sensor data, for example when it is simulated sensor data generated by a simulator when simulating a specific scene. However, the information can alternatively also be supplied to the artificial neural network as a separate data stream, for example when the sensor data have already been analyzed and classified with respect to at least one scene. This classification can be carried out manually, for example by analyzing an image stream corresponding to the sensor data.

[0031] A third aspect of the invention relates to a computer-supported method for analyzing sensor data suitable for characterizing a dynamic spatial scene involving an autonomous object and at least one other object. Based on the sensor data, a graphical representation of the time profile of an angular sector is generated, which is obscured by the other object, in particular the other vehicle, from the perspective of the autonomous object, in particular the autonomous vehicle. The generated graphical representation is compared with at least one predefined template of a known dynamic spatial scene. This makes it possible to detect the presence of a known scene particularly quickly and reliably, for example, using a sensor data stream provided by an environmental sensor of a vehicle.

[0032] The predefined template is preferably a generic representation of the time development of an angular sector, which, for example, contains all important characteristic patterns of a scene. The predefined template can be, for example, a representation that contains at least one graphic whose shape characterizes a driving maneuver of the vehicle that is performed within the scope of a traffic scene.

[0033] The predefined template is preferably determined based on a plurality of illustrations representing known dynamic spatial scenes. A plurality of illustrations representing known scenes can in particular be averaged. These illustrations can be generated in advance, for example, from sensor data after classification of the scene. This allows a high degree of reliability to be achieved when comparing the generated illustrations with the predefined template.

[0034] In a preferred specific embodiment, the dynamic spatial scene is classified based on the comparison, for example by assigning the determined representation to at least one predefined template. This allows the scene to be reliably identified and, if necessary, a reaction of a driver assistance system to be triggered.

[0035] Preferably, a degree of similarity is determined when comparing the generated representation with a predefined template, based on which, for example, scenes can be classified and the generated representation can be assigned to a scene. For example, it is conceivable to assign the generated representation to a scene on which the generated representation can be optimally drawn with the aid of a method for elastic matching (elastic matching, or nonlinear template matching), i.e. for which the degree of similarity achieved by the method is the greatest. This allows a particularly reliable identification of scenes.

[0036] In another preferred embodiment, a template for a new dynamic spatial scene is defined when the generated representation cannot be assigned to at least one predefined template of a known dynamic spatial scene. This allows new dynamic scenes to be identified, in particular to be classified. In particular, new dynamic scenes can thus be identified substantially in real time.

[0037] For example, a template for a new dynamic spatial scene can be defined if the degree of similarity determined when comparing the generated representations does not reach a predetermined threshold value. This makes it possible to generate a series of predefined templates for known dynamic spatial scenes in a simple manner, in particular in real time.

[0038] In another preferred embodiment, the generated representation also reproduces the distance, in particular the lateral distance, of the autonomous object from other objects and / or the speed, in particular the lateral speed, of the autonomous object, wherein the distance and / or the speed are likewise determined from the sensor data. The corresponding distance information or speed information is preferably encoded in the representation in such a way that it can be read at least independently of the spatial position of the other objects relative to the autonomous object. By taking into account the distance and / or the speed, the current dynamic spatial scene can be particularly reliably identified. Taking into account the distance and / or the speed also makes it possible to distinguish between different dynamic spatial scenes in particular in a particularly fine manner.

[0039] The speed of the autonomous object can have in particular a transverse speed, in particular a speed component substantially perpendicular to the main (longitudinal) direction of movement of the autonomous object, for example the direction of travel. The speed of the autonomous object reproduced in the generated representation is preferably formed only by the transverse speed.

[0040] In the case of a straight-ahead exit, the lateral speed of the ego vehicle is, for example, zero, whereas in the case of a lane change it rises to a maximum value and then falls back to zero again.

[0041] By means of the speeds reproduced in the diagram, it is possible, for example, to distinguish between a lane change performed by the ego vehicle and a lane change performed by another vehicle. Since the two driving maneuvers involving the relative movement of the ego vehicle and the other vehicle do not differ from one another, that is, the diagram shows, for example, the same graphic in both cases, taking into account the lateral speed of the ego vehicle makes it possible to reliably determine which vehicle is changing lanes.

[0042] The distance between the autonomous object and the other objects can in particular have a transverse distance, that is to say a distance component substantially perpendicular to the main (longitudinal) direction of movement of the autonomous object, for example the direction of travel. The distance between the autonomous object and the other objects reproduced in the generated representation is preferably formed only by the transverse distance.

[0043] If the ego vehicle is overtaken, for example, by another vehicle in an adjacent lane, the absolute distance between the ego vehicle and the other vehicle changes. However, the lateral distance remains constant as long as neither the ego vehicle nor the other vehicle leaves its respective lane. The change in the lateral distance thus allows (further) inferences to be drawn, for example, about the maneuvers performed within the context of a scenario.

[0044] In another preferred embodiment, the time curve of the angular sector is shown by a line, the width of which indicates the value of the current angular sector. The distance of the autonomous object from other objects and / or the speed of the autonomous object are alternatively or additionally shown by stored values ​​or by coloring at the corresponding locations of the line at the time points at which the corresponding distance and / or speed existed.

[0045] Alternatively or additionally, the stored values ​​or the coloring of the lines can be determined by means of a particularly universal function, into which the distance between the autonomous object and the other object, in particular the lateral distance, and / or the speed of the autonomous object, in particular the lateral speed, are entered as input variables. With the aid of the function, for example, the distance and the speed can be added or multiplied, if necessary taking into account weighting. In particular, the speed can be weighted with the distance or the distance with the speed. The information obtained in this way, which is reproduced in the diagram or by means of the stored values, allows a particularly reliable and detailed differentiation of driving maneuvers or corresponding dynamic spatial scenes.

[0046] The function is preferably used here to control the perception of the artificial neural network. In other words, the function can be selected in such a way that the resulting representation influences the pattern recognized by the artificial neural network. This makes it possible to distinguish between different dynamic spatial scenes particularly reliably.

[0047] Alternatively or additionally, other parameters other than the distance between the autonomous object and the other object and / or the speed of the autonomous object can also be selected as input variables of the function. The parameters can be selected, for example, based on a dynamic spatial scene, in particular a known dynamic spatial scene. Preferably, parameters are selected as input variables that at least partially characterize the dynamic spatial scene.

[0048] A fourth aspect of the invention relates to a system for preparing data of a dynamic spatial scene, in particular a traffic scene, for processing by an artificial neural network. The system has a determination module, which is configured to determine the time profile of an angular sector from the sensor data, which angular sector is covered by other objects, in particular other vehicles, from the perspective of an autonomous object, in particular an autonomous vehicle. The sensor data is particularly suitable for characterizing a dynamic spatial scene involving the autonomous object and at least one other object. The system also has a generation module, which is configured to generate a graphical representation of the determined time profile.

[0049] A fifth aspect of the invention relates to a system for training an artificial neural network based on sensor data, the sensor data being suitable for characterizing a known dynamic spatial scene involving an autonomous object and at least one other object. The system comprises a generation module, which is configured to generate a graphical representation of the time profile of an angular sector, which is obscured by other objects, in particular other vehicles, from the perspective of the autonomous object, in particular the autonomous vehicle, based on the sensor data. The system further comprises an interface, which is configured to feed the generated graphical representation together with information about the spatial scene to the artificial neural network.

[0050] A sixth aspect of the invention relates to a system for analyzing sensor data, the sensor data being suitable for characterizing a dynamic spatial scene involving an autonomous object and at least one other object. The system comprises a generation module, which is configured to generate, based on the sensor data, a graphical representation of a time profile of an angular sector, which is obscured by other objects, in particular other vehicles, from the perspective of the autonomous object, in particular the autonomous vehicle. The system further comprises a comparison module, which is configured to compare the generated graphical representation with a predefined template of a known dynamic spatial scene.

[0051] The features and advantages described with reference to the first aspect of the invention and its advantageous designs also apply, at least where technically reasonable, to the second, third, fourth, fifth and sixth aspects of the invention and their advantageous designs, and vice versa.

[0052] Further features, advantages and possible applications of the invention are apparent from the following description in conjunction with the accompanying drawings, in which identical or corresponding elements of the invention are always provided with the same reference numerals. The accompanying drawings at least partially schematically illustrate: BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Preferred exemplary embodiments of a system according to the invention for preparing data and a system according to the invention for training an artificial neural network are shown respectively;

[0054] Figure 2 A preferred exemplary embodiment of the system according to the invention for analyzing sensor data is shown;

[0055] Figure 3 An example for finding an angle sector in a traffic scene is shown;

[0056] Figure 4 A first example is shown for explaining the relationship between the representation according to the invention and the corresponding dynamic spatial scene;

[0057] Figure 5 A second example is shown for explaining the relationship between the representation according to the invention and the corresponding dynamic spatial scene;

[0058] Figure 6 Preferred exemplary embodiments of a method according to the invention for preparing data and a method according to the invention for training an artificial neural network are shown, respectively; and

[0059] Figure 7 Preferred exemplary embodiments of the method according to the invention for evaluating data are shown. DETAILED DESCRIPTION

[0060] Figure 1Preferred embodiments of a system 100 according to the invention for preparing data of a dynamic spatial scene and a system 200 according to the invention for training an artificial neural network 1 are shown. The system 100 for preparing data has a determination module 2 and a generation module 3, wherein the determination module 2 is configured to determine the time profile of an angular sector from the sensor data S, which angular sector is obscured by other objects from the perspective of the autonomous object, and wherein the generation module 3 is configured to generate a representation of the determined time profile. The system 200 for training an artificial neural network 1 has a generation module 3 and an interface 4, wherein the interface 4 is configured to supply the generated representation to the artificial neural network 1 together with information about the dynamic spatial scene.

[0061] The sensor data S are generated, for example, by environmental sensors of the autonomous vehicle when detecting a traffic scene and characterize, for example, the number of adjacent vehicles, the relative arrangement of other vehicles to the autonomous vehicle, in particular the relative distance, the speed of the autonomous vehicle and / or similar information. The ascertainment module 2 can preferably ascertain from the sensor data S how wide at least one angular sector covered by other vehicles is in the field of view of the autonomous vehicle and at what position this angular sector is arranged, for example, relative to the driving direction of the autonomous vehicle.

[0062] Since the environment sensor preferably detects the surroundings of the autonomous vehicle continuously, but at least at a high frequency, the time profile of the angular sector, in particular the change in the width of the angular sector and / or the change in the position of the angular sector as viewed from the perspective of the autonomous vehicle, is also determined from the generated sensor data S. The time profile can be used by the generation module 3 to generate a graphic representation that abstractly depicts the traffic scene. In other words, the generation module 3 is configured to encode the information about the traffic scene contained in the sensor data S in the graphic representation, in particular in compressed form.

[0063] The sensor data S about the dynamic spatial scene are preferably classified sensor data, that is to say that the sensor data S are assigned, for example, to one of a plurality of traffic scene categories, such as an overtaking maneuver or a lane change maneuver. This classification can be done, for example, manually, for example by sorting and filtering the image data stream. The representation generated based on the determined time profile of the angular sector can thus be transmitted from the interface 4 to the artificial neural network 1 together with the information about the dynamic spatial scene.

[0064] The artificial neural network 1 is preferably configured to recognize at least one pattern in each of all images assigned to the same traffic scene category. With the aid of the patterns thus recognized, templates that characterize known traffic scenes can be defined. The templates thus defined can be stored in a database 5 for further use, for example for evaluating sensor data generated during normal operation of the vehicle.

[0065] Figure 2 A preferred embodiment of a system 300 for analyzing sensor data S is shown, which are suitable for characterizing a dynamic spatial scene involving an autonomous object and at least one other object. The system 300 has a generation module 3, which is configured to generate a graphical representation of the time curve of an angular sector that is covered by other objects from the perspective of the autonomous object based on the sensor data S. The system 300 also has a comparison module 6, which is configured to compare the generated graphical representation with a predefined template of a known dynamic spatial scene. For this purpose, the comparison module 6 preferably calls a database 5, in which at least one predefined template is stored.

[0066] The result of the comparison is preferably output by the comparison module 6 and can be used, for example, to control a driver assistance system with which the autonomous vehicle is equipped. If, for example, at least a predetermined degree of agreement is determined between the generated representation and at least one predefined template, for example by analyzing the degree of similarity produced during the comparison, it can be inferred that a known scenario is present and the comparison module 6 can issue, for example, a scenario class as an output signal. Alternatively or additionally, the start and / or end of the identified driving maneuver can also be output, in particular signaled.

[0067] Figure 3 An example is shown for determining an angular sector Φ in a dynamic spatial scene 10 in which an autonomous object 11 moves along a direction of movement, here along the x-axis of a coordinate system, and is surrounded by other vehicles 12. In the present example, the spatial scene 10 is a traffic scene with an autonomous vehicle 11a, which moves along the driving direction in the central lane and is surrounded by other vehicles 12a, 12b in adjacent lanes.

[0068] Figure 3 A shows a traffic scene from a bird's eye view, wherein the time sequence of the spatial scene is summarized in the illustrated diagram. At different points in time, other vehicles 12a, 12b are in different positions relative to the autonomous vehicle 11a, which are indicated by the different fillings of the rectangles representing the vehicles 12a, 12b. Here, a tighter filling corresponds to a position that is later in time. As can be clearly seen in this illustration, the first vehicle 12a changes lanes from the outer lane to the lane used by the autonomous vehicle 11a and merges in front of the autonomous vehicle 11a. First, the second vehicle 12b, which is approximately at the same height as the autonomous vehicle 11a with respect to the direction of movement, lags behind in the course of time. Since the position change of the second vehicle 12b is small, the positions occupied by the second vehicle 12b at different points in time overlap in this illustration.

[0069] In order to generate an alternative representation of the traffic scene, an angular sector Φ can be determined for each of the two vehicles 12a, 12b, which describes the area in the field of vision of the ego vehicle 11a that is covered by the respective vehicle 12a, 12b. Figure 3 B is shown by way of example for the first vehicle 12a. The contour 13 of the first vehicle 12a, which is obtained with reference to the angle of the ego vehicle 11a, is shown as a solid line and spans an angular sector Φ. From the perspective of the ego vehicle 11a, the position of the angular sector Φ and thus of the first vehicle 12a can be indicated relative to a predetermined direction, for example the direction of movement. If the vehicle 12a moves relative to the position of the ego vehicle 11a, the width of the angular sector Φ and its position All are subject to change.

[0070] This is Figure 3 C. The bar graph drawn there forms Figure 3 Abstract illustration of the traffic scene shown in A, with the position of the angular sector Φ in the bar graph Draw out the Figure 3 The reverse distance d between the ego vehicle 11a and the other vehicles 12a, 12b is shown as a black arrow in A. The three bars on the right correspond to the angular sector Φ covered by the first vehicle 12a, and the three bars on the left correspond to the angular sector Φ covered by the second vehicle 12b. Figure 3 As in A, the different time points are indicated by the corresponding filling of the bars.

[0071] The angular sector Φ covered by the first vehicle 12a is thus moved in the direction of the 0° position before merging into the autonomous vehicle 11a, wherein the 0° position of the angular sector Φ corresponds to a position before the autonomous vehicle 11a. The width of the angular range Φ covered by the first vehicle 12a is calculated as the distance d between the first vehicle 12a and the autonomous vehicle 11a, as in Figure 3 The degree of decrease is shown in A, and the increase is. In addition to the width of the angular range Φ, the distance d between the first vehicle 12a and the ego vehicle 11a is also encoded in the height of the bar, which increases over time.

[0072] The angular range Φ covered by the second vehicle 12b is then as far away from the 0° position as the second vehicle 12b is behind the ego vehicle 11a. Since the distance between the second vehicle 12b and the ego vehicle 11a is increased here, the height and width of the strips are also reduced.

[0073] Figure 4 A first example is shown for illustrating the relationship between a preferred embodiment of the diagram 20 according to the invention and a corresponding dynamic room scene 10. For this purpose, Figure 4A shows a diagram 20 of the time profile of the angular sectors Φa, Φb, Φc which are obscured by other objects from the perspective of an autonomous object, for example the autonomous vehicle 11a. In the diagram 20, the positions of the angular sectors Φa, Φb, Φc are shown. The time t is plotted, where the 0° position corresponds to a position in front of the autonomous object.

[0074] The time profiles of the angular sectors Φa, Φb, Φc are shown as lines, the width of which corresponds to the distance of the respective object from the autonomous object. A first object, such as the first vehicle 12a, is therefore initially located at a certain distance in front of the autonomous object at t=0. A second object, such as the second vehicle 12b, is initially closer to the autonomous object, but in a position laterally to the autonomous object. At t=40, for example, the second object changes its position and moves between the autonomous object and the first object. The time profile of the angular sector Φb covered by the second object overlaps with the profile of the angular sector Φa covered by the first object from this point in time.

[0075] At t=70, for example, another angular sector Φc is covered by a third object, such as a third vehicle 12c. The third object is located to the side of the autonomous object, more precisely on the side of the autonomous object opposite to the side on which the second object was located at t=0. The position of the angular sector Φc covered by the third object Then there is a movement in the direction of the 0° position. This time profile can be caused, for example, by a movement of the third object parallel to the direction of movement of the autonomous object, wherein the distance of the third object from the autonomous object increases.

[0076] Such a time profile of the angular sectors Φa, Φb, Φc can, for example, characterize Figure 4 The spatial dynamic scene 10 shown in B is a traffic scene. Figure 4 The trajectory of the vehicle on the lane is shown in B by the dots.

[0077] Initially, autonomous vehicle 11a is at position (x=0, y=0) and first vehicle 12a is, for example, at position (x=60, y=0). First vehicle 12a thus drives ahead of autonomous vehicle 11a in the middle lane, for example, at a distance Δx=60. Second vehicle 12b is initially at position (x=45, y=-4), for example, wherein the second vehicle is thus in the adjacent lane, that is to say, driving between autonomous vehicle 11a and first vehicle 12a, displaced laterally with respect to autonomous vehicle 11a.

[0078] In another time profile, the second vehicle 12b merges between the autonomous vehicle 11a and the first vehicle 12a in the middle lane. The merging motion of the second vehicle 12b begins, for example, at a position (x=100, y=-4) and ends, for example, at a position (x=150, y=0). After the other vehicle 12b merges, the autonomous vehicle 11a, the first vehicle 12a, and the second vehicle 12b continue to move together in the middle lane. Since the second vehicle 12b now blocks the view of the first vehicle 12a from the perspective of the autonomous vehicle 11a, Figure 4 In the corresponding illustration 20 in A, only the line in the 0° position can still be seen.

[0079] The third vehicle 12c is initially behind the autonomous vehicle 11a, so that the third vehicle is not detected by the surrounding sensors of the autonomous vehicle 11a for a while. However, the third vehicle 12c is moving at a higher speed than the autonomous vehicle 11a, so that in the further course of the process the third vehicle overtakes the autonomous vehicle 11a in another adjacent lane at y=4. The third vehicle 12c is only seen by the autonomous vehicle 11a at the time of overtaking, so that the time curve of the angle range Φc covered by the third vehicle 12c is shown in FIG. Figure 4 In the corresponding diagram 20 of A, the process starts only from this point in time.

[0080] Figure 4 The diagram 20 generated in A by the time profile of the angular sector has a pattern that characterizes the described spatial dynamic scene 10. If a large number of such diagrams are generated based on the sensor data collected repeatedly when detecting such scenes, the artificial neural network can learn this pattern or be trained to recognize this pattern. In normal operation of the vehicle, the time profile of the angular sector can be displayed essentially in real time from the sensor data generated here and analyzed by the trained artificial neural network, in particular compared with the learned pattern. It is thus preferably determined whether or at which point in time a known traffic scene is present.

[0081] exist Figure 4 C, for this purpose it is shown that Figure 4 The result of this comparison between the diagram 20 and the corresponding template is shown in A, where the output signal a is plotted against the time t. For example, at the time t=30, the output signal a jumps to the value 6, thus signaling the presence of a known traffic scenario, in this case a merge of the second vehicle 12b. For example, at the time t=85, the output signal a falls back to the value zero, thus signaling the end of the merge maneuver.

[0082] In addition to the start and duration of the driving maneuver, the classification of the traffic scene can also be output via the value of the output signal a. In another example, not shown, a comparison of the generated diagram with different templates can, for example, indicate the greatest degree of agreement with the template assigned to the maneuver of leaving the original lane, and the output signal a can have other values ​​accordingly.

[0083] Figure 5 A second example is shown for explaining the relationship between a preferred embodiment of the diagram 20 according to the invention and a corresponding dynamic room scene 10. For this purpose, Figure 5 A shows a scene from a traffic situation corresponding to the dynamic spatial scene 10, in this case the merging of the autonomous vehicle 11a into a lane indicated by an arrow, in which another first vehicle 12a is already traveling. In addition to the first vehicle 12a, another second vehicle 12b is traveling in another lane.

[0084] The temporal development of the dynamic space scene 10 is Figure 5 B shows the trajectories of vehicles 11a, 12a, 12b marked with dots. Initially, ego vehicle 11a is, for example, at position (x=0, y=2.5), and first vehicle 12a is, for example, at position (x=35, y=0) in the middle lane ahead of ego vehicle 11a, that is, for example, at a distance Δx=35. Second vehicle 12b is, for example, at position (x=25, y=-2.5) at the beginning.

[0085] In the further time profile, the autonomous vehicle 11a begins merging into the middle lane at position (x=60, y=2), for example, and ends the driving maneuver at position (x=100, y=0), for example. The other two vehicles 12a, 12b continue to drive straight ahead at a slightly higher speed, so that the autonomous vehicle 11a slowly falls further behind.

[0086] exist Figure 5 c, the time variation of the angular sectors Φa, Φb respectively covered by the first vehicle 12a or the second vehicle 12b from the perspective of the ego vehicle 11a is shown in the diagram 20. Here, the position of the angular sectors Φa, Φb is Plot time t. Figure 5 The 0° position shown in c corresponds to a position in front of the ego vehicle 11 a .

[0087] In the time profile, the angular sectors Φa, Φb move in the direction of the 0° position, since the first and second vehicles 12a, 12b, as explained above, are moving away from the ego vehicle 11a due to their higher speed. At the time when the ego vehicle 11a merges, for example at time t=30, the time profiles of the angular sectors Φa, Φb additionally bend in the direction of the 0° position. The angular sector Φa covered by the first vehicle 12a then extends along the 0° position, since, as in Figure 5 As shown in FIGS. 5A and 5B , first vehicle 12 a is traveling ahead of ego vehicle 11 a in the same lane.

[0088] Based on the generated pattern, only statements can be made about the relative movement of autonomous vehicle 11a relative to first and second vehicles 12a, 12b, that is, it is not yet clear from the shape of the time profile of angular sectors Φa, Φb whether autonomous vehicle 11a changes lanes to the middle lane or whether first and second vehicles 12a, 12b respectively perform a lane change, in particular essentially simultaneously, with first vehicle 12a turning into the lane of autonomous vehicle 11a and second vehicle 12b turning into the lane previously used by first vehicle 12a.

[0089] In order to be able to distinguish these two situations from one another, the diagram 20 shows, in addition to the angular sectors Φa, Φb, a value that characterizes the distance, in particular the transverse distance, between the ego vehicle 11a and the respective other vehicle 12a, 12b at time t and / or the speed, in particular the transverse speed, of the ego vehicle 11a. The transverse distance or transverse speed here refers to the transverse component of the distance or speed, that is to say the transverse component of the distance or speed. Figure 5 B shows the y component of the diagram.

[0090] The values ​​characterizing the distance and / or speed are shown in the diagram 20 as the time curve of the angular sector colored by color. In this case, dark colors correspond, for example, to a high lateral speed of the ego vehicle 11a. Figure 5 As can be seen from diagram 20 in C, the autonomous vehicle begins a lateral movement, ie a movement perpendicular to the lane course, at time t=20, for example, wherein autonomous vehicle 11a reaches a maximum lateral speed at t=30, for example, and ends the lateral movement at t=40, for example.

[0091] Since ego vehicle 11a does not change its (lateral) speed when executing the lane change by one of other vehicles 12a , 12b , it is concluded in the present case that ego vehicle 11a has turned into the lane used by first vehicle 12a .

[0092] Figure 6Preferred exemplary embodiments of a method V1 according to the invention for preparing data and a method V2 according to the invention for training an artificial neural network are respectively shown.

[0093] In method step S1, sensor data are generated, for example by detecting the surroundings of the autonomous vehicle with sensors, and the sensor data are classified, that is to say assigned to different dynamic spatial scenarios. The classification can be performed manually, for example by evaluating an image data stream. Alternatively, the sensor data can also be classified automatically, in particular when it comes to sensor data generated by a simulator when simulating different dynamic spatial scenarios.

[0094] In a further method step S2, the temporal profile of the angular sector that is obscured by the other object from the perspective of the autonomous object can be determined using the sensor data. For example, the contour of the other object, in particular a cross section, can be determined and its width in the field of view of the autonomous object or in a partial field of view of the autonomous object can be determined. Furthermore, the geometric center of gravity of the contour or cross section is determined and its position in the field of view of the autonomous object, in particular relative to the direction of movement of the autonomous object, can be determined.

[0095] In a further method step S3, a diagram, in particular a graphic representation, is preferably generated from the time profile of the angular sector, which plots the time profile. In such a diagram, the time profile of the angular sector can form, for example, a pattern, such as a graph. In this case, for each time point at which sensor data are generated or at least one angular sector covered by other objects is determined, the width of this angular sector, in particular its share of the field of vision of the autonomous object, and its position in the field of vision of the autonomous object, in particular relative to the direction of movement of the autonomous object, are plotted.

[0096] In method step S3, the speed of the autonomous object, in particular the lateral speed and / or the distance of the autonomous object from other objects, in particular the lateral distance, is also preferably taken into account when generating the representation. In particular, a value can be determined and stored, or the representation can be colored accordingly, for example, by means of a function, wherein parameters characterizing the dynamic spatial scene, such as the speed and / or the distance, are preferably entered as input variables into the function. The generated representation thus preferably provides information about the width and position of the angular sector in the field of view of the autonomous object that is obscured by other objects, as well as the speed of the autonomous object and / or the distance of the autonomous object from other objects.

[0097] In a further method step S4, the generated representation is fed to an artificial neural network, for example by means of an interface, which is thus trained in particular to recognize patterns in the generated representation. For this purpose, information about the dynamic spatial scene, according to which the sensor data were classified in method step S1, is also preferably fed to the artificial neural network, so that the artificial neural network can associate the generated representation or the recognized pattern with one of the dynamic spatial scenes.

[0098] Figure 7 A preferred embodiment of the method V3 according to the invention for analyzing sensor data suitable for characterizing a dynamic spatial scene involving an autonomous object and at least one further object is shown. The sensor data are preferably sensor data generated by environmental sensors of the autonomous vehicle.

[0099] In method step S3, a graphical representation of the time profile of an angular sector is generated from the sensor data. An angular sector corresponds to an area in the field of view of the autonomous object that is obscured by other objects. Such a graphical representation is, for example, a diagram in which the time profile of the angular sector forms a pattern, such as a graph.

[0100] The time profile of the angular sector can here also be determined, if necessary, in a separate, preceding method step (not shown) based on the sensor data.

[0101] In a further method step S5 , the generated representation is compared with at least one predefined template of a known dynamic spatial scene. It is thereby possible to determine in which dynamic spatial scene the autonomous object is currently located and, if necessary, to control the driver assistance system accordingly.

[0102] If it is determined that the generated representation does not correspond or does not correspond sufficiently to at least one predefined template assigned to a known dynamic spatial scene, the generated representation can optionally be stored as a further predefined template, for example in a database, in a further method step S6. In this way, a series of predefined templates suitable for identifying dynamic spatial scenes can be generated, in particular substantially in real time.

[0103] Reference numerals list

[0104] 1 Artificial Neural Networks

[0105] 2. Generate Module

[0106] 3 Obtaining modules

[0107] 4 Interfaces

[0108] 5 Database

[0109] 6 Comparison Module

[0110] 10 Spatial Dynamic Scene

[0111] 11 Autonomous Objects

[0112] 11a Autonomous Vehicles

[0113] 12 Other vehicles

[0114] 12a First vehicle

[0115] 12b Second vehicle

[0116] 12c Third vehicle

[0117] 13. Outline

[0118] 20 Graphics

[0119] 100 Systems for preparing data

[0120] 200 Systems for training artificial neural networks

[0121] 300 Systems for analyzing sensor data

[0122] Φ Angular sector

[0123] Position of the angular sector

[0124] S Sensor data

[0125] y direction of movement

[0126] d Spacing

[0127] a Output signal

[0128] V1 Methods for preparing data

[0129] V2 Methods for training artificial neural networks

[0130] V3 Methods for analyzing sensor data

[0131] S1-S6 Method Steps

Claims

1. A method for preparing data of a dynamic spatial scene for processing by an artificial neural network, wherein: The dynamic space scene represents a traffic scene, and the method comprises: Acquiring sensor data, wherein the sensor data represents a dynamic spatial scene involving the autonomous vehicle and at least one other object; and A representation in the form of a pattern is generated based on a temporal development of an angular sector determined from the sensor data, wherein the temporal development of the angular sector is obscured by the at least one other object from the perspective of the autonomous vehicle.

2. The method according to claim 1, wherein: The representation is output to an artificial neural network or is processed by the artificial neural network.

3. A computer-supported method for training an artificial neural network based on sensor data suitable for characterizing a known dynamic spatial scene involving an autonomous vehicle and at least one other object, the method comprising: A representation is generated based on the temporal development of an angular sector from the sensor data, wherein the temporal development of the angular sector is obscured by the at least one other object from the perspective of the autonomous vehicle, and wherein the generated representation is fed to the artificial neural network together with information about the spatial scene, which serves as a hint for classification.

4. A computer-supported method for analyzing sensor data suitable for characterizing a dynamic spatial scene involving an autonomous vehicle and at least one other object, the method comprising: A representation is generated based on a temporal development of an angular sector from the sensor data, wherein the temporal development of the angular sector is obscured by the at least one other object from the perspective of the autonomous vehicle, and wherein the generated representation is compared to at least one predefined template of a known dynamic spatial scene.

5. The method according to claim 4, wherein: The dynamic spatial scene is classified based on the comparison.

6. The method according to claim 4, wherein: When the generated representation cannot be assigned to at least one predefined template of a known dynamic spatial scene, a template is defined for a new dynamic spatial scene.

7. The method according to claim 1, wherein: The generated representation also reproduces the distance of the ego vehicle to the at least one further object and / or the speed of the ego vehicle, wherein the distance and / or the speed are likewise determined from the sensor data.

8. The method according to claim 1, wherein: The progression of the angular sectors over time is shown by a line, the width of the line indicating the value of the respective current angular sector, and / or wherein the distance of the autonomous vehicle relative to the at least one other object and / or the speed of the autonomous vehicle are shown by means of stored values ​​or coloring at corresponding locations of the line at the time points at which the distance and / or speed existed.

9. A system for preparing data of a dynamic spatial scene for processing by an artificial neural network, wherein: The dynamic space scene represents a traffic scene, and the system has: a first computer module which determines the temporal development of an angular sector from sensor data, wherein the temporal development of the angular sector is obscured by at least one other object from the perspective of the autonomous vehicle, wherein the sensor data are suitable for characterizing a known dynamic spatial scene involving the autonomous vehicle and the at least one other object; and A second computer module which generates a representation of the time development of the determined angular sector.

10. A system for training an artificial neural network based on sensor data suitable for characterizing a known dynamic spatial scene involving an autonomous vehicle and at least one other object, wherein: The known dynamic spatial scene describes a known traffic scene, and the system has: - a computer module that generates a representation based on the temporal development of an angular sector from said sensor data, wherein the temporal development of said angular sector is obscured by said at least one other object from the perspective of the autonomous vehicle; An interface which feeds the generated representation together with information about the spatial scene to an artificial neural network.

11. A system for analyzing sensor data suitable for characterizing a dynamic spatial scene involving an autonomous vehicle and at least one other object, the system comprising: - a first computer module which generates a representation based on the temporal development of an angular sector from said sensor data, wherein the temporal development of the angular sector being obscured by the at least one other object from the perspective of the autonomous vehicle; and - A second computer module which compares the generated representation with at least one predefined template of a known dynamic spatial scene.

12. The method according to claim 1, wherein: The at least one other object includes another vehicle.

13. The computer-supported method of claim 3, wherein: The at least one other object includes another vehicle.

14. The computer-supported method of claim 4, wherein: The at least one other object includes another vehicle.

15. The system according to claim 9, wherein: The dynamic spatial scene is a traffic scene, and the at least one other object includes another vehicle.

16. The system of claim 10, wherein: The at least one other object includes another vehicle.

17. The system of claim 11, wherein: The at least one other object includes another vehicle.

Images