Method and apparatus for analyzing a sensor data stream and method for guiding a vehicle - Patents.com

Abstract

The present invention relates to a method and apparatus for analyzing a sensor data stream characterizing a vehicle's surroundings for the presence of a traffic scenario, and a method for guiding a vehicle. A similarity index indicating the degree of match between a section of the sensor data stream and at least one pattern stored in a database is determined by mapping the section of the sensor data stream onto the at least one pattern, preferably using a dynamic time warping method, where the pattern characterizes a known traffic scenario. If the similarity index satisfies a predetermined similarity criterion, the known traffic scenario is assigned to the section of the sensor data stream.

Description

[Technical field]

[0001] The present invention relates to a method and device for analyzing a sensor data stream characterizing a vehicle's surroundings for the presence of a traffic scenario, and to a method for guiding the vehicle. [Background technology]

[0002] Modern vehicles are increasingly equipped with driver assistance systems (Advanced Driver Assistance Systems, ADAS) that assist the driver in certain driving situations. Such assistance ranges from the mere display of potentially relevant information (e.g. a warning output by the lane change assistant) through semi-autonomous intervention (e.g. control of the torque applied to the axles by the anti-lock braking system) to fully autonomous or at least semi-autonomous intervention in the control of the vehicle (e.g. adaptive speed control through distance control, Adaptive Cruise Control, ACC).

[0003] The basis for such driver assistance systems is generally sensor data, e.g. signals provided by ultrasonic sensors, radar sensors or cameras, with which the current driving situation is determined and in response to which the respective functions of the driver assistance system can be executed. In particular, driver assistance systems which (autonomously) intervene in the control of the vehicle must be able to identify the current driving situation with a very high degree of reliability on the basis of the sensor data.

[0004] In this case, a specific rule or criterion is generally created that is assigned to a driving situation, and if the rule or criterion is fulfilled, the existence of a known driving situation can be inferred. In this case, fulfillment of the rule or criterion acts as an actuator for the operation of a driver assistance system, for example. For example, a traffic scenario in which an adjacent vehicle enters the same lane in front of a vehicle equipped with a driver assistance system, called an ego-Fahrzeug, can be recognized by the fact that the lateral distance perpendicular to the direction of travel with the adjacent vehicle detected by the sensor decreases and finally assumes a value of at least approximately zero when the adjacent vehicle is in front of the ego-Fahrzeug.

[0005] In order to test such a driver assistance system, in particular the reaction of the driver assistance system in a known traffic scenario, the driver assistance system to be tested can be input with sensor data that characterizes the known traffic scenario. In order to reliably test the driver assistance system, a large number of sensor data are generally required, which may also characterize simple variations of the traffic scenario.

[0006] From DE 10 2005 11 2005 A1 an automatic generation of simulation scenarios for validating a driver assistance system is known, whereby a number of such simulation scenarios can in particular be generated by variants of an accepted scenario, which variants are based on a data stream that is generated by separation of differences between similar accepted scenarios. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] International Publication No. 2017 / 210222 [Non-patent literature]

[0008] [Non-Patent Document 1] Rakthanmanon et al., “Searching and mining trillions of time series subsequences under dynamic time warping”, Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 12, 2018, p.262-270. [Non-Patent Document 2] Petitjean et al., “A global averaging method for dynamic time warping, with applications to clustering”, Pattern recognition, 44(3), 2011, p.678-693. Summary of the Invention [Problem to be solved by the invention]

[0009] The object of the invention is to further improve the analysis of a sensor data stream with regard to the presence of a traffic scenario, in particular to make an existing traffic scenario more reliably and / or easier to recognize by analysis of the sensor data stream. [Means for solving the problem]

[0010] This problem is solved by a method and a device for analyzing a sensor data stream characterizing the vehicle's surroundings for the presence of a traffic scenario and by a method for operating a driver assistance system according to the independent claims.

[0011] A first aspect of the invention relates to a method for analysing a sensor data stream characterising a vehicle's surroundings for the presence of a traffic scenario, the method comprising the following work steps: (i) determining a similarity measure indicative of the degree of match between a section of the sensor data stream and at least one pattern stored in a databank by mapping the section of the sensor data stream onto at least one pattern characterising a known traffic scenario, preferably using a dynamic time warping method, and (ii) assigning the known traffic scenario to the section of the sensor data stream if the similarity measure satisfies a predefined similarity criterion.

[0012] In particular, the method is computer-assisted.

[0013] A sensor data stream in the sense of the present invention is in particular a time series of sensor data, in particular corresponding signals characterizing the vehicle surroundings at each point in time. In other words, the sensor data stream can provide continuous information in particular about the vehicle surroundings. The sensor data stream can be provided or created, for example, by a sensor device, preferably having one or more, if necessary different sensors for detecting the vehicle surroundings. Alternatively, the sensor data stream can also be created artificially, for example by simulation. Preferably, the sensor data stream is formed from preprocessed, particularly prepared, for example fused sensor data, and contains information on the relative spacing between traffic participants or other objects, in particular taking into account the curvature of the road.

[0014] A section of a sensor data stream in the sense of the present invention is in particular a temporal section of the sensor data stream. A section of a sensor data stream can be, for example, a section from a sensor data stream. In other words, a section of a sensor data stream can include sensor data that has been provided or will be provided within a time frame, possibly within a predefined time frame. A section can in particular include a series of values, in particular a time series of values.

[0015] A pattern in the sense of the present invention may in particular be a sequence of values, in particular a time series of values. The pattern may in particular describe the driving behavior of at least one vehicle. Preferably, the pattern is a section, in particular a generic section, of the sensor data stream that is characteristic for a particular traffic scenario, in particular a known traffic scenario.

[0016] The assignment of sections of a sensor data stream to known traffic scenarios in the sense of the present invention is in particular a classification of the sensor data stream, in particular a classification (in English: label) of a section of the sensor data stream. Preferably, the sections of the sensor data stream are correspondingly labeled during the assignment, for example by setting markers or values ​​that are respectively characteristic for the known traffic scenario.

[0017] The rendering of a section of a sensor data stream onto a pattern in the sense of the present invention is in particular an adaptation, in particular a transformation, of the section and / or the pattern, so that the section, in particular the change over time of the section, corresponds at least approximately, in particular as well as possible, to the pattern, in particular the change over time of the pattern, e.g., the section and / or the pattern can be compressed and / or expanded during the rendering, so that for example the shape of the time series of values ​​contained in the section corresponds at least approximately to the shape of the time series of values ​​contained in the pattern.

[0018] Dynamic time warping (DTW) in the sense of the present invention is in particular a method, in particular an algorithm, for the overlapping representation of value series, for example time series, such as sections of a sensor data stream, of different lengths as required. In this case, a matrix is ​​preferably created, which contains as matrix elements distances between the elements of the value series, for example Euclidean distances, Manhattan distances or Mahalanobis distances. Based on a cost function into which these distances are input, a minimum cost for various classifications of the elements of the value series, which are represented through distances, can be calculated in order to find the most accurate representation of the value series.

[0019] A similarity in the sense of the present invention is in particular a value that characterizes the similarity between two value series, for example between time series, such as sections of a sensor data stream. Preferably, the similarity is based on a distance, for example the Euclidean distance, the Manhattan distance or the Mahalanobis distance. In this case, the similarity can be proportional to the distance, in particular the same as the distance. Alternatively, the similarity can be inversely proportional to the distance. In this case, the similarity preferably corresponds to a distance calculated via an optimization function. In other words, the similarity can characterize the distance, possibly an abstract distance, between sections of the sensor data stream and the pattern.

[0020] A traffic scenario in the sense of the present invention is a time-dependent change of scene elements within a scene sequence, preferably starting with a start scene. Unlike a scene, a scenario covers a certain time interval. A scene preferably describes a snapshot of the surroundings, including in particular all spatially stationary and dynamic elements.

[0021] The invention is based in particular on the approach of comparing sections extracted from a sensor data stream, provided for example by a sensor device detecting the vehicle surroundings or by a simulator, with patterns characterizing known traffic scenarios, for example the spatial positioning of traffic participants and / or their dynamic development, in particular at least one driving maneuver. Depending on the comparison result, a classification of the sections of the sensor data stream can be made. For example, the sections can be classified as belonging to a known traffic scenario characterized by a pattern.

[0022] Based on the classification thus performed, for example, information regarding known traffic scenarios can be output or provided and communicated, for example via an interface, to a driver assistance system.

[0023] In this case, the comparison of the section of the sensor data stream with the pattern is preferably performed through dynamic time warping of the section and the pattern, i.e., through drawing the section on the pattern. The similarity determined in this case, in particular the distance between the section and the pattern, can provide a clue as to how well the section and the pattern match, for example, how much the vehicle surroundings characterized by the section of the sensor data stream differ from the known traffic scenario. If the similarity satisfies the similarity criterion, i.e. if the difference between the section and the pattern is not too large, the known traffic scenario characterized by the pattern is assigned to the section of the sensor data stream. In other words, the unknown traffic scenario characterized by the section of the sensor data stream is in this case identified with the known traffic scenario.

[0024] In this case, the dynamic time warping method allows a particularly fast and reliable identification of unknown traffic scenarios characterized by sensor data streams, in particular by sections. In particular, it is not necessary to extract multiple parameters from the sensor data streams, possibly in a complex manner, and to check whether said parameters fulfill various criteria. Instead, the invention allows only one parameter, in particular the similarity, to be determined by the dynamic time warping method, preferably implemented with an algorithm that has a particularly short computation time, for example the algorithm described in non-patent document 1, and to use said parameter for a reliable identification of the traffic scenario.

[0025] By using the dynamic time warping method, it is possible to reliably assign known traffic scenarios to sections of the sensor data stream that characterize variants of the traffic scenario, in particular by mapping them onto a comprehensive pattern with respect to the known traffic scenarios. This makes it possible to prevent sections of the sensor data stream from being classified differently in these cases. This allows for an effective processing of the sensor data stream.

[0026] Overall, the present invention makes it possible to further improve the analysis of a sensor data stream regarding the presence of a traffic scenario and in particular makes it possible to recognize an existing traffic scenario more reliably and / or in an easier manner by the analysis of the sensor data stream.

[0027] In the following, preferred embodiments of the present invention and developments thereof are described, which can be combined with each other in any way and with the further described aspects of the invention, unless expressly excluded.

[0028] In a preferred embodiment, the sections of the sensor data stream are stored in a data bank when assigned to a known traffic scenario. In this case, the sections are preferably classified accordingly, i.e. labeled as belonging to the known traffic scenario. The sections can in particular be assigned to a cluster of sections already stored in the data bank and assigned to the known traffic scenario or form part of such a cluster. This allows the sections of the sensor data stream stored in this way to be taken into account during future (further) processing of the data stored in the data bank, for example during future adaptation of at least one pattern.

[0029] In a further preferred embodiment, the method further comprises the work step of storing the section of the sensor data stream as a further pattern in the data bank if the similarity does not meet the predefined similarity criterion. The further pattern can be preferably used in the following to assign further, in particular future, sections of the sensor data stream to a traffic scenario, in particular hitherto unknown, which is characterized by the further pattern. In this way, the data bank can be expanded. In particular, new traffic scenarios, for example new driving maneuvers, can be added to the data bank. In other words, in this way, the learning of new traffic scenarios and a corresponding dynamic expansion of the data bank can be realized.

[0030] This may occur when a rare traffic scenario occurs, for example when a vehicle performs an abnormal steering maneuver that has not yet been detected by the patterns in the database. In this case, a section of the sensor data stream can characterize this rare and previously unknown traffic scenario, which does not match any of the patterns stored in the database, at least within the framework set by the similarity criterion, i.e., the section cannot be depicted sufficiently accurately, in particular by dynamic time warping, into one of the patterns and cannot be stored in the database as a new pattern and therefore cannot form, for example, a new traffic scenario group.

[0031] In a further preferred embodiment, the method further comprises the following work steps: (i) checking whether at least one pattern is stored in the data bank, and (ii) depending on the result of the check, adding a section of the sensor data stream as a pattern to the data bank. In particular, if no pattern has been stored in the data bank so far, the section of the sensor data stream is stored in the data bank. In this way, the data bank can be advantageously initialized or built. In particular, this makes it possible to omit a possibly complex preparation or addition of a data bank. In a further preferred embodiment, the method further comprises the work step of performing an adaptation of the pattern if the similarity simultaneously satisfies a predefined similarity criterion and a predefined adaptation criterion. For example, if a section of the sensor data stream is assigned to a known traffic scenario characterized by a pattern, but there exists a certain difference between the section and the pattern, an adaptation of the pattern can be performed. This allows further, in particular future, sections of the sensor data stream to be assigned to the correct traffic scenario in a particularly reliable manner.

[0032] For example, based on the distance between a section of the sensor data stream identified by the dynamic time warping method and a pattern, the section may be assigned to a cluster of sections corresponding to a known traffic scenario, but if it is not located near the center of gravity of the cluster, the pattern may be adapted, specifically by recalculation.

[0033] In a further preferred embodiment, during the adaptation of the pattern, all sections of the sensor data streams, which are assigned to known traffic scenarios, in particular those stored so far in a data bank, are averaged by dynamic time warping. The averaging is preferably carried out using the method of barycenter averaging of dynamically time warped time series (in English: dynamic time warping barycenter averaging), known, for example, from [2]. This makes it possible to avoid repeated overlapping averaging, which advantageously reduces the computational burden. In addition, it is possible to use the average for any sequence of sensor data contained in the stored sections of the sensor data stream, so that the length, in particular the time length, of the thus adapted pattern is advantageously not enlarged.

[0034] Thus, preferably, the method forms a real-time version of the so-called K-means algorithm (in English k-means clustering), in which the patterns stored in the data bank form the centroids of various clusters, each corresponding to a known traffic scenario. In this case, if the similarity criterion is met, new elements, i.e. further sections of the sensor data stream, can be added to the cluster. However, if at the same time the adaptation criterion is also met, for example if the representation of the section of the sensor data stream on the pattern does not reach a certain quality, the centroid of the corresponding cluster, i.e. the pattern, is adapted, in particular recalculated, in particular through the average of all elements of the cluster. If the similarity criterion is no longer met, i.e. if a section of the sensor data stream cannot be meaningfully represented on one of the patterns, in particular by dynamic time warping, the section forms a new cluster. By configuring the method as a K-means algorithm, it is possible to assign sections of the sensor data stream to the correct traffic scenario with particular certainty and with a particularly high probability, or to dynamically adapt or expand the data bank.

[0035] In a further preferred embodiment, if the similarity falls below a predefined similarity threshold, the predefined similarity criterion is met or is met. In this case, it is preferably investigated whether the distance between the section of the sensor data stream and the pattern, which is determined by the dynamic time warping method, is smaller than a predefined similarity threshold. This makes it possible to determine reliably and unambiguously, especially when saving computational power, whether the section of the sensor data stream should be assigned to the known traffic scenario characterized by the pattern.

[0036] In a further preferred embodiment, a predefined adaptation criterion is met or is met if the similarity exceeds an adaptation threshold that depends on a similarity criterion, in particular a similarity threshold, whereby it can be determined reliably and unambiguously, in particular when saving computational power, whether the pattern characterizing the known traffic scenario should be adapted, for example recalculated.

[0037] In this case, it is preferably checked whether the distance between the section of the sensor data stream and the pattern, which is determined by the dynamic time warping method, is greater than a predefined adaptation threshold. This allows the pattern to be adapted, especially if the degree of match between the section of the sensor data stream and the pattern is not too high. In other words, the adaptation of the pattern is preferably only performed if the section of the sensor data stream currently depicted on the pattern deviates from the pattern at least to a certain extent, and therefore the adaptation of the pattern is also appropriate. In particular, unnecessary adaptation of the pattern can be avoided.

[0038] Preferably, the dependency of the adaptation criterion, in particular the adaptation threshold, is characterized through a function, in particular a mathematical function. For example, the value of said function can express the adaptation criterion, in particular the adaptation threshold, if at least a similarity criterion, in particular a similarity threshold, is selected as an input variable of the function. In this case, said function can be set to weight the similarity criterion, in particular the similarity threshold, for example by multiplying it by a coefficient smaller than 1. In other words, the adaptation criterion can correspond to a weighting of the similarity criterion.

[0039] In a further preferred embodiment, the method further comprises the work step of setting a similarity criterion, in particular a similarity threshold, on the basis of a comparison of a number of similarities, in particular all similarities, which are determined during the representation, in particular by a dynamic time warping method, from at least a section of the sensor data stream, which is assigned to a known scenario, in particular stored in a data bank. Preferably, during the comparison, a similarity criterion characterizing a minimum degree of match between the section of the sensor data stream and the pattern, for example a maximum distance between the section of the sensor data stream and the pattern, determined by the dynamic time warping method, is determined, from which the similarity criterion, in particular the similarity threshold, is formed. This allows the similarity criterion, in particular the similarity threshold, to be dynamically adapted, for example to the quality of the sensor data contained in the sensor data stream.

[0040] For example, all elements of a cluster corresponding to a known traffic scenario, whose centroid is formed by the corresponding pattern, can each be depicted on the pattern by dynamic time warping, and from the distances between the elements, i.e. between the sections of the sensor data stream and the patterns, the maximum distance is selected, for example filtered, and used as a similarity criterion, in particular as a similarity threshold.

[0041] In a further preferred embodiment, a number of similarities are determined by mapping a section of the sensor data stream onto a respective one of a number of patterns stored in a databank, each characterizing another known traffic scenario, and the section of the sensor data stream is assigned to a known traffic scenario based on a comparison between the determined similarities. In other words, an unknown traffic scenario characterized by a section of the sensor data stream is compared with the plurality of known traffic scenarios and preferably identified based on said comparison.

[0042] Preferably, during the comparison, a similarity measure characterizing the highest degree of match between the section of the sensor data stream and the pattern is determined, for example the smallest distance between the section of the sensor data stream and the pattern, determined during dynamic time warping of the section of the sensor data stream, and the section is assigned to a known traffic scenario characterized by the corresponding pattern, whereby a traffic scenario characterized by a section of the current sensor data stream can be particularly depicted and identified reliably and quickly.

[0043] In a further preferred embodiment, the sensor data stream characterizes at least one, in particular transverse, distance between two traffic participants of a traffic scenario, which extends perpendicularly to the direction of travel. For example, if two vehicles are traveling side by side on two adjacent lanes, the distance between the vehicles does not change. If one vehicle performs a maneuver, for example by accelerating or overtaking or by changing into the respective other lane, the distance between the vehicles changes. The change can be characteristic for the maneuver or the traffic scenario. The change in the distance during the maneuver, in particular the change over time, can thus form a pattern that is stored in a data bank for the traffic scenario. This allows a section of the sensor data stream to be assigned with a high probability to the correct traffic scenario.

[0044] In a further preferred embodiment, the sensor data stream is provided by a sensor device of the vehicle during operation of the vehicle. The sensor device preferably comprises a number of sensors, for example at least one camera, at least one radar sensor, at least one lidar sensor, at least one ultrasonic sensor and / or others, which generate corresponding sensor data, preferably at least approximately continuously, and the sensor data stream is preferably formed by fusion of the sensor data. Based on the assignment of sections of the sensor data stream to known traffic scenarios, information on the traffic scenario thus identified is output in situ, i.e. approximately in real time, and can be provided, for example, to a driver assistance system. Furthermore, for example within the framework of a test drive, the data bank can be filled with patterns, so that the data bank or the data contained in the data bank can be taken into account at a later time for identifying traffic scenarios. By providing the sensor data stream during operation of the vehicle, the data bank can be expanded both in situ, i.e. in real time.

[0045] In a further preferred embodiment, the method further comprises the work step of selecting a section from the sensor data stream, in which the start of the section of the sensor data stream and / or the end of the section of the sensor data stream are selected such that the start of the section of the sensor data stream and the end of the section of the sensor data stream are spaced apart from one another over a predefined period of time. Preferably, the section of the sensor data stream selected in this way forms a time frame within which the sensor data contained in the sensor data stream, for example provided by a sensor device of the vehicle or a simulation, can be taken into account, for example, when identifying an existing traffic scenario. By setting the period, i.e. by selecting the length of the time frame, a particularly reliable assignment of the section of the sensor data stream to a known traffic scenario is obtained.

[0046] Preferably, the end of a section is formed by current sensor data generated by a sensor device or by a simulation, which allows, for example, the identification of the current existing traffic situation in a reliable manner.

[0047] In a further preferred embodiment, at least the selection of the section of the sensor data stream, the determination of the similarity and the assignment of the section of the sensor data stream are performed repeatedly. Preferably, in the repeated selection of the section of the sensor data stream, the start and / or the end of the section of the sensor data stream are selected such that the section of the sensor data stream overlaps at most half with the section of the sensor data stream selected so far. In other words, the further section of the sensor data stream is only depicted on at least one pattern when half of the predefined period has elapsed. This can ensure that two successive sections of the sensor data stream depicted on the patterns are sufficiently different from each other that they can be assigned to different known traffic scenarios.

[0048] A second aspect of the invention relates to an apparatus for the analysis of a sensor data stream characterizing a vehicle's surroundings with respect to the presence of a traffic scenario. The apparatus preferably comprises a processing module, which is configured to determine a similarity measure indicative of a degree of match between a section of the sensor data stream and at least one pattern stored in a data bank, by mapping the section of the sensor data stream onto the at least one pattern, preferably using a dynamic time warping method, the pattern characterizing a known traffic scenario. Furthermore, the apparatus preferably comprises an assignment module, which is configured to assign the known traffic scenario to the section of the sensor data stream if the similarity measure satisfies a predefined similarity criterion.

[0049] Preferably, the device also comprises a sensor device, which is configured to detect the vehicle surroundings and to supply a sensor data stream. The sensor device may comprise one or more sensors, e.g. a camera, an ultrasonic sensor, a radar sensor, a lidar sensor and / or others, so that relevant variables, such as e.g. the distances between traffic participants, can be detected reliably, preferably redundantly, for the characterization of the existing traffic scenario.

[0050] The device preferably also comprises a databank, the databank being arranged to store at least one pattern, preferably a plurality of patterns, each characterising a known traffic scenario.

[0051] A third aspect of the invention relates to a method for guiding a vehicle on the basis of a sensor data stream using a driver assistance system, the sensor data stream being analyzed using the method according to the first aspect of the invention. Preferably, an output signal is generated based on the assignment of a section of the sensor data stream, which preferably characterizes a current traffic scenario, to a known traffic scenario, which is characterized by a pattern stored in a data bank, and the output signal characterizes the known or identified traffic scenario, in particular a known steering, based on the section of the sensor data stream and is supplied to the driver assistance system. The driver assistance system can thus reliably react to the existing traffic scenario.

[0052] The features and advantages described with respect to the first aspect of the invention and its advantageous configurations are also valid with respect to the second aspect of the invention and its advantageous configurations, and vice versa, at least where technically significant.

[0053] Further features, advantages and applicability of the invention will become apparent from the following description taken in conjunction with the drawings, in which the same reference numerals are used throughout for the same or corresponding elements of the invention, and which show, at least in part and diagrammatically, [Brief description of the drawings]

[0054] [Figure 1] FIG. 1 shows a preferred embodiment of the device according to the present invention. [Diagram 2] FIG. 1 illustrates a preferred embodiment of the method according to the present invention. [Diagram 3] FIG. 2 is a diagram for explaining the allocation of a known traffic scenario, characterized by patterns, to sections of sensor data streams. [Figure 4] FIG. 13 is a diagram for explaining pattern adaptation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] 1 shows a preferred embodiment of an apparatus 1 according to the invention for analyzing a sensor data stream D characterizing the vehicle surroundings, the apparatus 1 comprising a processing module 2, an allocation module 3 and a data bank 4. The apparatus 1 is preferably connected to a sensor device 5 arranged to detect the vehicle surroundings and to supply a corresponding sensor data stream D, and to a driver assistance system 6. The driver assistance system 6 can for example be controlled on the basis of an output signal A generated by the apparatus 1 based on the analysis of the sensor data stream D.

[0056] Alternatively, it is also conceivable that the sensor arrangement 5 and / or the driver assistance system 6 are part of the apparatus 1 .

[0057] The processing module 2 and / or the allocation module 3 are preferably configured as software, for example as program code, and can be executed by means of a data processing unit 7 .

[0058] The processing module 2 is preferably configured to determine a similarity measure characterizing the degree of match between a section of the sensor data stream D and at least one pattern S stored in the data bank 4. In this regard, the processing module 2 provides a section of the sensor data stream D, for example by extraction from sensor data contained in the sensor data stream D within a given time window, which can be mapped onto the at least one pattern S, for example by using dynamic time warping (DTW in English). A similarity measure is preferably obtained as a result of this mapping.

[0059] The processing module 2 may in particular be configured to dynamically, i.e. in particular non-linearly, distort, for example at least partially compress and / or expand, the sections extracted from the sensor data S. As a result, the sensor data contained in the sections can for example be assigned to corresponding pattern data corresponding to the pattern. Preferably, the processing module 2 determines the assignment on the basis of an optimization function, whereby in particular the sections of the sensor data stream D are depicted on the pattern S in a particularly accurate manner, i.e. with minimal differences. The remaining distance between the sections of the sensor data stream D and the pattern S, in particular in the form of differences, preferably forms a similarity.

[0060] In this case, in particular, the distance can characterize the difference between the sensor data contained in the section and the pattern data respectively assigned to said sensor data, in particular, in order to obtain said distance, for example, the differences between the sensor data and the pattern data can be determined for each pair of sensor data and pattern data and summed up.

[0061] The patterns S stored in the data bank 4 preferably characterize each known traffic scenario. The pattern data corresponding to a pattern may for example correspond to the value of a parameter which at least partially describes the traffic scenario, in particular the change in said parameter over time. Said parameter may for example be the distance between two traffic participants.

[0062] The allocation module 3 may correspondingly be configured to allocate known traffic scenarios to sections of the sensor data stream D based on the patterns S stored in the data bank 4 and the similarities determined by the processing module 2, in particular if the similarity satisfies a predetermined similarity criterion, e.g. is smaller than a predetermined similarity threshold.

[0063] In other words, the allocation module 3 is preferably configured to investigate whether the determined similarity satisfies a predefined similarity criterion. For example, the processing module 2 is capable of determining whether the distance between a section of the sensor data stream D and the pattern S, obtained upon dynamic time warping of the section of the sensor data stream D, is below a predefined similarity threshold, i.e. whether the section and the pattern S are similar to each other to the extent that the determined distance between them is smaller than the similarity threshold.

[0064] Depending on the investigation results, in particular if the similarity satisfies a predefined similarity criterion or the distance is below a predefined similarity threshold, the allocation module 3 is able to allocate a section of the sensor data stream D to a known traffic scenario characterized by the pattern S. An output signal A is preferably generated by the allocation module 3, preferably also depending on the investigation results, characterizing the known traffic scenario. The output signal A may for example comprise information about the known traffic scenario.

[0065] If a section of the sensor data stream D cannot be meaningfully depicted in any of the patterns S, i.e. if the similarity does not meet a predefined similarity criterion, the data bank 4 can be extended with a further pattern S, specifically formed, that is characterized by the currently observed section of the sensor data stream D. In particular, in this case the section can be stored in the data bank 4 as a further pattern S.

[0066] FIG. 2 shows a preferred embodiment of a method 100 according to the present invention for analyzing a stream of sensor data characterizing a vehicle's surroundings for the presence of a traffic scenario.

[0067] In method step S1, a sensor data stream is provided, for example by detection of the vehicle's surroundings by means of a sensor device of the vehicle or by simulation. The sensor data stream is preferably formed or has been formed from a time series of sensor data, the sensor data may include values ​​of parameters describing a traffic scenario. For example, the sensor data stream may depict the change over time of the intervals between traffic participants.

[0068] In a further method step S2, a section is selected from the sensor data stream, for example by extracting sensor data from the sensor data stream within a predefined time frame. In particular, the start of a section and / or the end of a section can be selected, the start and end of a section being preferably spaced apart from one another over a predefined period of time. In this case, the end of a section is preferably formed by the last sensor data provided from the sensor data stream.

[0069] In a further method step S3, it is checked whether at least one pattern characterizing the known traffic scenario is stored in the data bank. If no pattern is present in the data bank, the section of the sensor data stream is stored in the data bank as a pattern in a further method step S4. Otherwise, the section of the sensor data stream can be depicted on at least one pattern stored in the data bank in a further method step S5, and a similarity is preferably determined. In this case, the similarity preferably characterizes the degree of match between the section of the sensor data stream and the pattern.

[0070] The rendering of the section on the pattern can be performed on the basis of a dynamic time warping method, and the similarity is preferably formed by a distance between the section and the pattern, which distance can be a measure of the difference between the section rendered on the pattern and the pattern, in particular between the sensor data contained in the section and the pattern data contained in the pattern, whereby the distance or similarity is preferably small if the section and the pattern are very similar to each other and large if the section and the pattern are dissimilar to each other.

[0071] If multiple patterns are already stored in the databank, the section is preferably drawn for each of the patterns and the degree of similarity for each is determined.

[0072] In a further method step S6 it is preferably investigated whether the similarity measure or measures determined in method step S5 satisfy a predefined similarity criterion, e.g. whether the distance obtained using dynamic time warping is smaller than a predefined similarity threshold.

[0073] If this is not the case for any of the patterns, then in method step 4 the section of the sensor data stream can be stored in the data bank as a further pattern, in particular in addition to the patterns already stored in the data bank, so that in the data bank the section of the sensor data stream describes a further, hitherto unknown, traffic scenario.

[0074] On the other hand, if at least one similarity measure satisfies a similarity criterion, e.g. if the distance obtained by the dynamic time warping method is smaller than a predefined similarity threshold, the section of the sensor data stream can preferably be assigned to a known traffic scenario in a further method step S7.

[0075] If there are several similarities identified in method step S5, they are preferably compared with one another and, based on the result of the comparison, a known traffic scenario is assigned to the section. For example, during the comparison, the similarity measure that characterizes the highest degree of match between the section and the pattern corresponding to the similarity measure may be identified. In particular, the smallest distance obtained during dynamic time warping may be selected and the traffic scenario characterized by the pattern corresponding to said distance may be assigned to the section.

[0076] Additionally, in a further method step S8, it is possible to check whether a similarity measure selected from the multiple similarity measures identified in method step S5, as possibly described above, satisfies, in addition to the similarity criterion, also a predefined adaptation criterion, e.g. whether the distance obtained during dynamic time warping is greater than an adaptation threshold.

[0077] If the question is answered in the affirmative, then in a further method step S9 the pattern corresponding to the similarity can be adapted taking into account the sections of the sensor data stream, for example the pattern can be modified by averaging of a number of sections of the sensor data stream to which a corresponding known traffic scenario has been assigned in method step S7.

[0078] 3 is a diagram for explaining the allocation of the sensor data stream to sections B of a known traffic scenario characterized by a pattern S. The traffic scenario is, for example, an entrance maneuver, an exit maneuver, a driving situation one after the other on one lane or a driving situation between two lanes. In this case, the various sections B correspond, for example, to different driving maneuvers and, within the driving maneuvers, also to different implementations or variants of each driving maneuver.

[0079] In this case, the position in the diagram of section B of the sensor data stream shown in FIG. 3 corresponds to the component of the distance between the two vehicles perpendicular to the direction of travel at the start of the steering maneuver, i.e., the start lateral distance dX startand the component of the distance between the two vehicles perpendicular to the direction of travel at the end of the steering maneuver, dX end For a better understanding, section B is shown in FIG. 4, which corresponds to an entry maneuver in which two vehicles are initially traveling on two adjacent lanes, one of which enters in front of the other. In this context, it can be seen from FIG. 4, for example, that the initial lateral distance dX start is in the range of approximately 4 m, and the lateral distance dX at the end of the steering operation when the vehicle is traveling on the same lane is end It can be derived that is generally in the range around 0.

[0080] For each possible steering and for each possible implementation of the steering, i.e. for each section B, there are slightly different start time intervals dX start and end time interval dX end The presence of makes it possible to easily distinguish between different driving maneuvers or possible implementations of steering based on the location of section B in the depiction of FIG.

[0081] As suggested in Fig. 3, the sections B form a cluster C in the selected representation, and each pattern S forms, in the illustrated example, the centroid of a cluster C consisting of several sections B of the sensor data stream. Cluster C is then preferably assigned to all sections B whose temporal evolution (for example as shown in Fig. 4) of the sensor data stream has a certain similarity to one another. In the representation shown in Fig. 3, this similarity is determined by the spatial location, i.e. the similar start time interval dX of sections B. start and end time interval dX end , where similar sections B of the sensor data stream are adjacent to each other within the range. For example, all sections B corresponding to various implementations of the approach maneuver are start ≒ 4 and dX end≈0, but all sections B, which correspond to various implementations of pursuit steering in which one vehicle is pursuing another vehicle in the same lane, are in the range of dX start ≒0 and dX end ≒0.

[0082] In this case, cluster C is preferably assigned all sections B of the sensor data stream for which a determined similarity measure characterizing the degree of match between each section B of cluster C and pattern S satisfies a similarity criterion, the similarity measure being preferably determined by mapping each section B onto each pattern S using a dynamic time warping technique.

[0083] When section B is rendered on each pattern S, it is preferably dynamically, i.e. at least partially, compressed and / or stretched using a dynamic time warping method so that section B at least approximately corresponds to or follows the changes of pattern S. For example, the time-dependent changes of a parameter, such as the lateral spacing between two vehicles, which at least partially characterizes the still unknown traffic scenario to be identified and which is rendered by section B of the sensor data stream, can be adapted by the dynamic time warping method to the time-dependent changes of said parameter, which are characteristic for the known traffic scenario, which are rendered by pattern S. In this case, the similarity preferably represents the difference between the time-dependent changes rendered by section B and the time-dependent changes rendered by pattern S (see FIG. 4 for an example of the time-dependent changes of section B).

[0084] If the similarity thus determined satisfies a predefined similarity criterion, one of the patterns S or one of the clusters C, and thus one of the known traffic scenarios, can be assigned to the further section B. In this case, preferably, the traffic scenario among the known traffic scenarios which, with respect to its pattern S, has been determined to have the highest degree of match with section B is assigned to the further section B.

[0085] In this case, the similarity measure representing the degree of match may correspond, for example, to the distance between the section B of the sensor data stream and each pattern S, as determined within the framework of the dynamic time warping method.

[0086] In this case, fulfilling the similarity criterion may include, for example, that the distance is below a similarity threshold. Alternatively or additionally, fulfilling the similarity criterion may include that the determined distance is smaller than all other distances determined for another pattern S. In other words, section B is preferably assigned to cluster C or to the driving maneuver or traffic scenario corresponding to said cluster if the distance between section B and the pattern S corresponding to cluster C is sufficiently small, and in particular smaller than the distance between section B and all other patterns S.

[0087] It is also possible to check whether the similarity satisfies a predefined adaptation criterion, in which case the corresponding pattern S, preferably characterizing a known traffic scenario assigned to section B, is adapted, as will be further mentioned below, also in connection with FIG.

[0088] Preferably, the adaptation criterion is met if the distance between section B and pattern S determined by the dynamic time warping method is greater than a predefined adaptation threshold. In other words, the adaptation criterion is met if section B is assigned to cluster C but is not located close to the centroid of cluster C, i.e. pattern S. In this case, the difference between section B and pattern S is considered to be sufficient and the adaptation of pattern S, i.e. the recalculation of the centroid of cluster C, results in changes that should be taken into account, which changes may have an impact in particular on future comparisons of the sensor data stream with further sections B.

[0089] If the predetermined similarity criterion is not met by any of the specified similarities, it is possible to form a new cluster C with a further section B as its only element, and a corresponding pattern S is formed by the further section B.

[0090] Figure 4 is a representation to explain the application of pattern S. In this case, the change over time of the lateral distance dX between two vehicles, i.e. the change over time of the distance between the vehicles perpendicular to the direction of travel, is depicted.

[0091] The lateral spacing dX can be considered as a parameter that at least partially describes the traffic scenario, especially the approach maneuver. At the start of the approach maneuver, the two vehicles are in adjacent lanes, so the lateral spacing between the vehicles is, for example, about 4 m. By the end of the maneuver, the two vehicles are in the same lane, so the lateral spacing between the vehicles is approximately 0 m.

[0092] 4 depicts a sensor data stream comprising a time series of sensor data with multiple sections B each characterizing the lateral distance dX for the same traffic scenario, in this case during an approach maneuver. Depending on how violent the driving maneuver is, the sections B are shorter or longer. However, regardless of this, the shape of the change over time is at least similar.

[0093] This similarity is used to assign a section B of the sensor data stream to a known traffic scenario, such as an approach maneuver, and to identify the similarity between section B and a pattern S that characterizes the known traffic scenario, section B is mapped onto pattern S using a dynamic time warping method, as described in more detail in relation to figures 2 and 3.

[0094] In this case, the pattern S is preferably determined on the basis of a plurality of sections B of the sensor data stream, for example previously stored in a data bank. The sections B previously stored in the data bank correspond to a known traffic scenario, here an approach maneuver, and may for example have been recorded during a test drive with the vehicle and correspondingly selected, for example manually, from the sensor data stream. Alternatively, the sections B may be sections extracted in situ from the sensor data stream and stored in a data bank.

[0095] In order to identify a pattern S from these sections B, in particular several sections B of the sensor data stream can be averaged. In this case, the averaging is preferably carried out on the basis of a dynamic time warping method in order to take into account the different time lengths of the sections B. In particular, the method of barycenter averaging of dynamically time warped time series (in English: dynamic time warping barycenter averaging), as described, for example, in Non-Patent Document 2, is preferably used.

[0096] Averaging of section B of the sensor data stream shown in FIG. 4 performed in this way results in pattern S, also shown in FIG. 4, which is comprehensive with respect to the traffic scenario characterized by pattern S, in this case an approach maneuver. [Explanation of symbols]

[0097] 1. Apparatus for analyzing sensor data streams 2 Processing Module 3. Allocation Module 4. Databank 5 Sensor device 6 Driver Assistance Systems 7 Data Processing Unit 100 Method for analyzing a sensor data stream S1~S9 Method steps A Output Signal Section B Sensor Data Stream C Cluster D Sensor Data Stream S-Pattern dX Horizontal Spacing dX start Start horizontal interval dX end End time horizontal interval

Claims

1. A computer-implemented method (100) for analyzing a sensor data stream (D) of a sensor device of a vehicle characterizing the vehicle surroundings or a sensor data stream (D) of a simulation characterizing the vehicle environment for the presence of a traffic scenario, the sensor data of said sensor data stream (D) being set to characterize the vehicle environment at each point in time, the method comprising the following work steps: - a work step (S5) of determining a similarity measure indicative of the degree of match between a section (B) of said sensor data stream (D) and at least one pattern (S) stored in a data bank (4) by mapping said section (B) of said sensor data stream (D) onto at least one pattern (S) using a dynamic time warping method, said pattern (S) representing a centroid of a cluster of the section (B) of said sensor data stream (D), said pattern (S) characterizing a known traffic scenario, said similarity measure being formed by the distance between the section (B) and the pattern (S) obtained by the dynamic time warping method; - assigning (S7) said known traffic scenario to said section (B) of said sensor data stream (D) if said degree of similarity satisfies a predefined similarity criterion; a working step (S9) of adapting said pattern (S) by recalculating said pattern (S) if said similarity simultaneously satisfies said predetermined similarity criterion and a predetermined adaptation criterion; The method (100) comprising:

2. 2. The method (100) of claim 1, wherein said section (B) of said sensor data stream (D) is stored in said data bank (4) upon assignment to said known traffic scenario.

3. an operating step (S4) of storing said section (B) of said sensor data stream (D) as a further pattern in said data bank (4) if said similarity does not satisfy said predefined similarity criterion, The method of claim 2 , further comprising:

4. - a working step (S3) of checking whether at least one said pattern (S) is stored in said data bank (4), - a working step (S4) of adding said section (B) of said sensor data stream (D) as said pattern (S) to said data bank (4) depending on the results of the investigation; The method (100) of any one of claims 1 to 3, further comprising:

5. 2. The method (100) according to claim 1, wherein, when adapting the pattern (S), all sections (B) of the sensor data stream (D) assigned to the known traffic scenario, in particular those previously stored in the data bank (4), are averaged by a dynamic time warping method.

6. The method (100) of any one of claims 1 to 5, wherein the predetermined similarity criterion is met if the degree of similarity is below a predetermined similarity threshold.

7. The method (100) according to claim 6, wherein the predetermined adaptive criterion is met if the degree of similarity is above an adaptive threshold which depends on the similarity criterion, in particular on the similarity threshold.

8. - setting said similarity criterion, in particular said similarity threshold, on the basis of a comparison between a number of, in particular all, similarity measures, which are determined by said sections (B) of said sensor data stream (D), in particular stored in said data bank (4), which have been assigned during mapping, in particular by means of a dynamic time warping method, to at least said known traffic scenario, The method (100) of claim 6, further comprising:

9. 9. The method (100) according to claim 1, wherein a plurality of similarities are determined by mapping the section (B) of the sensor data stream (D) onto one of a plurality of patterns (S) stored in the data bank (4), each of the patterns characterizing a different known traffic scenario, and the section (B) of the sensor data stream (D) is assigned to the known traffic scenario based on a comparison between the determined similarities.

10. The method (100) according to any one of claims 1 to 9, wherein the sensor data stream (D) characterizes at least one spacing between two traffic participants of a traffic scenario.

11. The method (100) according to any one of the preceding claims, wherein the sensor data stream (D) is provided by a sensor device (5) of the vehicle while the vehicle is in operation.

12. - a work step (S2) of selecting the section (B) from the sensor data stream (D), wherein a start of the section (B) of the sensor data stream (D) and / or an end of the section (B) of the sensor data stream (D) are selected such that the start of the section (B) of the sensor data stream (D) and the end of the section (B) of the sensor data stream (D) are spaced apart from one another by a predetermined period of time; The method (100) of any one of claims 1 to 11, further comprising:

13. 13. The method (100) of claim 12, wherein at least the selection of the section (B) of the sensor data stream (D), the determination of similarity, and the allocation of the section (B) of the sensor data stream (D) are performed repeatedly, and in repeatedly selecting the section (B) of the sensor data stream (D), a start and / or an end of the section (B) of the sensor data stream (D) are selected such that the section (B) of the sensor data stream (D) overlaps by at most half with a previously selected section (B) of the sensor data stream (D).

14. 14. An apparatus (1) for analyzing a sensor data stream (D) of a sensor device of a vehicle characterizing the vehicle surroundings for the presence of a traffic scenario, the sensor data of said sensor data stream being set to characterize the vehicle environment at each point in time, said sensor data stream (D) being analyzed using a method according to any one of claims 1 to 13, a processing module (2) configured to determine a similarity measure indicative of the degree of match between a section (B) of said sensor data stream (D) and at least one pattern (S) stored in a data bank (4) by mapping said section (B) of said sensor data stream (D) onto said at least one pattern (S) using a dynamic time warping method, said pattern (S) characterizing a known traffic scenario, said similarity measure being formed by the distance between the section (B) and the pattern (S) obtained by said dynamic time warping method; an allocation module (3) configured to allocate said known traffic scenario to said section (B) of said sensor data stream (D) if said degree of similarity satisfies a predefined similarity criterion; An apparatus (1) comprising:

15. 14. A method for a driver assistance system (6) to guide a vehicle based on a sensor data stream (D), the sensor data stream (D) being analyzed using a method according to any one of claims 1 to 13.

Images