Rapid detection of dangerous or compromised objects in the vehicle's surroundings

Abstract

The invention relates to a method (100) for identifying potentially dangerous or endangered objects (41-43) in the surroundings (2) of a vehicle (1), the method comprising the following steps: detecting (110) a region (2a) of the surroundings (2) using at least one event-based sensor (3), wherein the event-based sensor (3) has light-sensitive pixels (31), and wherein a relative change in the intensity of light incident on the pixel (31) by at least one predetermined percentage causes the sensor (3) to output an event (31a) assigned to the pixel (31); The invention also relates to a computer program for the present invention.

Description

Technical Field

[0001] The invention relates to a method with which objects in the surroundings of a vehicle can be rapidly analyzed in order to determine whether they could become dangerous for the vehicle or could otherwise be endangered by the vehicle. Background Art

[0002] Analysis of past traffic accidents has shown that a large percentage of these accidents could have been avoided if only slightly more time had been available for prevention. One second or even less of extra time has made a difference in many cases.

[0003] Vehicles are increasingly equipped with assistance systems that react automatically in the event of a collision and, for example, initiate emergency braking. In addition, such an automatic reaction is necessary if the vehicle is to be moved at least partially automatically in public road traffic.

[0004] In order to free up as much reaction time as possible, US 2016 / 096 477 A1 proposes using a sensor with event-based light-sensitive pixels. As soon as the light intensity incident on such an event-based pixel changes by at least a predetermined percentage, the sensor outputs an event assigned to this pixel. Compared to camera-based sensors, there is no need to wait for the capture of a complete image. Summary of the invention

[0005] Within the scope of the present invention, a method for detecting potentially dangerous or endangered objects in the surroundings of a vehicle is developed. In this context, "dangerous" should be understood to mean, in particular, that the object is at risk of colliding with the vehicle and could cause damage to the vehicle, to the vehicle's occupants and / or to other road users. "Endangered" should be understood to mean, in particular, that the object is at risk of colliding with the vehicle and could cause damage to the object itself.

[0006] An area of ​​the surroundings is detected by at least one event-based sensor, wherein the event-based sensor has light-sensitive pixels and wherein a relative change in the intensity of the light incident on a pixel with an amplitude of at least one predetermined percentage causes the sensor to output an event assigned to this pixel. The detected area can be, for example, the area towards which the vehicle is currently traveling. In particular, the surroundings of the vehicle can be covered by a plurality of event-based sensors, of which sensors are used at each moment that are closely related to the current direction of travel (in particular forward travel and reverse driving). However, it is also possible, for example, to monitor lateral areas. For vehicles parked on the shoulder of the road, it is also possible, for example, to subsequently monitor the traffic space behind to see whether any vehicle is about to leave the right lane in the direction of the shoulder of the road.

[0007] The event output by the sensor is assigned to at least one object in the area. In particular, the object can be obtained from the data stream of the event itself. For example, objects that can be distinguished from each other and move independently of each other can be detected in an abstract manner first, so as to be classified according to the object type (such as pedestrians, cyclists or other motor vehicles) in the next step. However, as an alternative or in combination, any other source of information about which objects exist in the surrounding environment of the vehicle can be considered. For example, it is also possible to assign events to objects that were previously detected and classified when using a camera sensor. In addition, it is possible to assign the same event to multiple objects, such as if the objects overlap each other from the perspective of the event-based sensor. However, it is not mandatory to assign each event output by the sensor to at least one object. The event can also remain in an unassigned state.

[0008] For at least one object to which an event is assigned, the event assigned to the object is analyzed with respect to the current movement of the object. These current movements can be, for example, intrinsic movements of the object or also other movements that take place on a shorter time scale than the typical frame rate of the camera sensor. In this context, the term intrinsic movement includes in particular movements that leave the position of the object as a whole unchanged in space. This is in particular a movement in which at least one point of the object remains stationary in space.

[0009] The upcoming movement of the object and / or the upcoming change of state of the object is derived from the current movement. The upcoming movement can be, for example, in particular the forward movement of the object as a whole. However, it is also possible, for example, to firstly determine the upcoming second intrinsic movement from the first intrinsic movement and to infer the upcoming forward movement of the object as a whole only from the second intrinsic movement. The concept of the change of state includes, for example, any change of the object that is not the forward movement of the object as a whole, but is equally relevant from the perspective of the vehicle. For example, the movement of the trunk lid on the leading coach may indicate that the load will fall from the trunk onto the roadway. Similarly, for example, the bending or breaking of the bracket used to fix the top box or other luggage to the leading coach may indicate that the luggage is about to fall.

[0010] It has been found that many movements or changes of state of objects only seem completely surprising, but in fact indicate corresponding intrinsic movements of these objects or also other movements that take place on a time scale shorter than the frame rate of the camera sensor. Thus, for example, in the case of a walking person, the length of the stride changes or the frequency of the swinging movement of the arm changes before the person's speed increases significantly. A cyclist stops pedaling before he brakes his bicycle and increases his pedaling speed before the bicycle as a whole accelerates significantly. In addition, in order to be able to walk in another direction, a person usually has to rotate his own axis in order to align in the desired direction and then be able to continue his journey in this direction. It is advantageous to select at least one intrinsic movement as the current movement of the object.

[0011] This knowledge is not limited to living objects. Thus, for example, the falling of a vehicle part or a load on a highway is very dangerous for the following traffic, which is often not an unexpected event when examined in detail. Rather, the corresponding fastening structure usually gradually loosens before it finally fails completely, so that the vehicle part or the load first has play and is then stimulated by the driving dynamics to move or vibrate relative to the rest of the vehicle.

[0012] Furthermore, the detection is not limited to unexpectedly occurring dangerous situations. Thus, for example, it is also possible to detect an intrinsic movement with which a person swings his arms in order to throw a stone at a moving vehicle.

[0013] The detection of an impending movement or an impending change of state may therefore include, in particular, the detection of the respectively underlying intention of the object, without being restricted thereto.

[0014] The intrinsic movements mentioned can be very subtle and at the same time fast movements. It is therefore particularly advantageous to detect these movements physically with event-based sensors. The event-based sensors provide a continuous data stream: each event is also reported at the exact moment it occurs. It is not a discretization of time that is fixedly predetermined in the form of a frame rate as with camera sensors. This has the result, on the one hand, that the intrinsic movements can be tracked in significantly more detail. On the other hand, it is possible to react significantly faster overall, i.e. also in a time that is shorter than the time that elapses between the capture of two complete images (frames) for a camera sensor. In addition to reporting events more quickly, this is also contributed to by the fact that the events only relate to changes in the current situation from the outset. Time does not have to be spent on evaluating what has changed, for example by comparing two images.

[0015] Compared to the video data stream of a camera sensor, the data stream of events provided by an event-based sensor is greatly compressed. Therefore, less bandwidth is required for data transmission within the vehicle and for further processing. If, for example, a CAN bus with a maximum bandwidth of 1 Mbit / s is used as a transmission channel, the CAN bus can already fully transmit the video data stream in HD quality.

[0016] For example, a set of rules based on primitives of predetermined rules of intrinsic motion can establish a connection between the intrinsic motion and the predicted motion or state change. Such a set of rules can be expressed explicitly, but can also be trained, for example, using machine learning. Such a rule can include, for example, that for a pedestrian walking on the sidewalk, the combination of a changing step frequency and a turn toward the roadway can predict with a high probability that the pedestrian will cross the roadway.

[0017] Advantageously, in response to a situation in which an upcoming movement of an object conflicts with the current or planned trajectory of the vehicle, the object is evaluated as dangerous or endangered. A criterion in this respect can be, for example, that the upcoming movement of the object intersects the current or planned trajectory of the vehicle or comes closer to the trajectory than a predefined safety distance. If the upcoming movement does not affect the vehicle in this way, the upcoming movement can be classified as not critical. For example, other resources for tracking such a movement can then be freed up in order to focus these resources on the closely related movement of other objects.

[0018] Advantageously, an object is evaluated as dangerous or endangered in response to a situation in which an impending change of state of the object has the possibility of causing at least one part of the object to collide with the current or planned trajectory of the vehicle. This is the case, for example, with the aforementioned open trunk lid or a broken bracket on a preceding passenger vehicle, since the load of this passenger vehicle could then enter the trajectory of the own vehicle.

[0019] In another particularly advantageous embodiment, rigid objects are excluded from the analysis of the intrinsic movement and the subsequent evaluation of the intrinsic movement. In this case, a rigid object can be in particular an object for which the intrinsic movement does not normally occur or is not of great relevance from the perspective of the vehicle, since it is not suitable for endangering the vehicle. Thus, for example, a passenger car should generally be considered a fixed object, except for the described special case of an imminent fall of a load. If, for example, the rear window wiper of a passenger car in front moves, this is not suitable for having a negative impact on the safety of the own vehicle, and therefore it is not necessary to bind resources for tracking this movement.

[0020] In particular, the exclusion of stationary objects from the analysis of the intrinsic motion and the subsequent evaluation of the intrinsic motion can be activated as a function of the road type of the road currently traveled by the vehicle. In this way, available processing resources can be given priority.

[0021] Thus, for example, in city traffic, there is less risk of a load being lost and of serious damage to the vehicle behind. Instead, it is more common for pedestrians to run right in front of the vehicle, whereby damage to the pedestrians could be serious. Thus, for example, it may be advantageous to activate the aforementioned exclusion for stationary objects in city traffic.

[0022] On the highway, pedestrians are rarely seen. However, loads often fall there at high speeds. Therefore, it can be advantageous on the highway to monitor the internal movement of stationary objects, such as buses or trucks, even under normal circumstances.

[0023] In the summary of the previously described embodiments, a person or an animal is advantageously selected as the object whose intrinsic motion is to be detected. The impending motion and / or state change of the person or animal is detected as the intrinsic motion from the change in amplitude and / or frequency of the periodic motion of at least one body part of the person or animal and / or from the rotation of the person or animal or its body parts. In particular, the analysis of periodic motions has the advantage that these motions can be better distinguished from noise.

[0024] In another advantageous embodiment, a person is selected as the object whose intrinsic movement is to be analyzed. From the communicative behavior of the person, which indicates the person's movement in a specific spatial direction, the person's impending movement in this spatial direction is determined as the intrinsic movement.

[0025] The analysis as an object may be particularly relevant in the following situation, in which it is to be foreseen whether a person moving on the sidewalk in the direction of the roadway is staying at the edge of the roadway or preparing to cross the road. In this regard, for example, not only the presence of a certain intrinsic movement but also the clear absence of a certain intrinsic movement can be used as a criterion for whether a collision is imminent. If a pedestrian moves, for example, towards the edge of the roadway and has previously turned his head in the direction of an approaching vehicle, then the probability is high that he has seen the vehicle and will stop, so that emergency braking is not required. However, if the pedestrian does not turn his head and, for example, other intrinsic movements that indicate the operation of a smartphone can be seen, it should be assumed that the pedestrian is not aware of the traffic activity at all. In this case, emergency braking is displayed.

[0026] The analysis of animals as objects can be particularly relevant in situations where animals are often seen. An animal at the edge of the roadway may enter the light cone of the driving lights only moments before the vehicle arrives, leaving only a short time for evaluating the animal's reaction to the light. The light can warn the animal not to enter the roadway, but can just as well trigger a flight reaction that causes the animal to enter the roadway.

[0027] According to the above description, the loosening of a part from an object is advantageously detected as an impending change in state of the object from the intrinsic movement of the object, wherein the intrinsic movement of the object comprises a movement or vibration of a part of the object relative to the rest of the object. In particular, vibrations triggered by driving dynamics often occur at a time scale that is significantly faster than the typical frame rate of a camera sensor. With the aid of the image provided by the camera sensor, the vibrations can thus be determined without a doubt. Furthermore, not only an impending loosening of a vehicle component or load can be detected. For example, the movement of ice accumulated on the awning of a truck moving ahead can also be detected, and it can be concluded from this that the truck may soon lose such an ice cake.

[0028] In a particularly advantageous embodiment, in response to the fact that an object is assessed as dangerous or endangered, a warning device that is physically perceptible to the driver of the vehicle is activated and / or the drive system, steering system and / or braking system of the vehicle are manipulated by intervention in order to avoid a collision or mitigate the consequences of a collision. For example, a braking maneuver and / or an evasive maneuver can be performed. If the vehicle ahead loses a load, it may be sufficient to increase the distance to the vehicle ahead, for example. However, if the traffic situation permits, it may also be sensible, for example, to overtake the vehicle ahead in order not to be hit by a load that may have fallen.

[0029] In another particularly advantageous embodiment, a camera image of the area is additionally taken into account for the purpose of identifying objects in the area detected by the event-based sensor and / or for the purpose of dividing the object into individual parts. In this way, the advantages of the two data records can be combined with one another. Although it is therefore possible in principle to divide the object into a plurality of segments solely on the basis of the data stream of the event, this is complicated by the fact that such a data stream always only shows the area of ​​the object that is just moving. In contrast, the entire object can be seen in the camera image, and, for example, the color contrast between different parts of the object can be utilized. In addition, camera sensors have a higher position resolution than event-based sensors for system reasons.

[0030] In this context, it should be noted that a more precise extraction of features of an object and a possible division of the object into individual parts do not necessarily have to precede an analysis of the current movement. The analysis can, for example, already be performed at a point in time when feature extraction or segmentation has not yet been completed, based on a worst-case estimate of the object type and can be corrected later, when the object has been more precisely classified or segmented. In this way, additional information from the camera image that is only available later can also be meaningfully incorporated.

[0031] In principle, the combination of event-based sensors and camera-based sensors offers the advantage of introducing diversity in data acquisition and data processing, especially for safety-related requirements in road traffic. In particular, the corresponding acquired or evaluated information can verify each other. Thus, for example, it may happen that a tree is incorrectly classified as a pedestrian based on the camera image. If, for example, it is now determined that the current movement pattern is typical for a tree and not typical for a pedestrian, the classification can be corrected accordingly.

[0032] The analysis of events with respect to the current movement and / or the evaluation of the current movement with respect to an upcoming movement or state change can be performed in particular using artificial neural networks. Here, in particular, convolutional neural networks, CNNs or recurrent neural networks, RNNs can be used. However, the analysis and evaluation can also be performed, for example, on a model basis.

[0033] The time span for predicting an upcoming movement or state change of an object can be between 1 and 3 seconds, for example. The computing power required for this can be provided by a conventional controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Below together with the description of the preferred embodiment of the present invention, further measures for improving the present invention are described in detail with the aid of the accompanying drawings. It is shown therein:

[0035] Figure 1 An embodiment of method 100 is shown;

[0036] Figure 2 An exemplary traffic situation in surroundings 2 of vehicle 1 is shown, in which method 100 functions advantageously. DETAILED DESCRIPTION

[0037] according to Figure 1 In step 110 of the method 100, the event-based sensor 3 is first used to detect the Figure 1 The area 2a of the surroundings 2 of the vehicle 1 (not shown) is detected. The events 31a reported by the sensor 3 are assigned in step 120 to objects 41-43 in the area 2a, wherein in an optional step 115 the camera image of the area 2a can also be taken into account for the recognition and / or classification of the objects 41-43.

[0038] In step 130 , the current movement 41 a - 43 a of the objects 41 - 43 is evaluated from the events 31 a respectively assigned to the objects 41 - 43 . This can be done in particular according to step 135 using an artificial neural network, KNN.

[0039] In step 140, the upcoming movement 41b-43b and / or the upcoming state change 41c-43c of the objects 41-43 are evaluated by the current movement 41a-43a, such as intrinsic movement. In this case, for example, according to step 141, a periodic movement of at least one body part of a person or animal 42 and / or a rotation of the person or animal 42 or its body parts can be used as intrinsic movement 42a. As an alternative or in combination, for example, according to step 142, the intrinsic movement 42a of the person 42 involving communicative behavior can be used. As an alternative or in combination, for example, according to step 143, the object 41 can be used as a periodic movement of at least one body part of a person or animal 42 and / or a rotation of the person or animal 42 or its body parts. Figure 1 The movement or vibration of the portion 41d not shown in the figure relative to the rest of the object 41 serves as the intrinsic movement 41a.

[0040] According to optional step 125 , stationary objects can be excluded from the determination 130 of the intrinsic movements 41 a - 43 a and their subsequent evaluation 140 , wherein this exclusion can be activated according to optional step 126 as a function of the road type of the road traveled by the vehicle 1 .

[0041] According to step 150, an object 41-43 is evaluated as dangerous if its upcoming movement 41b-43b is in conflict with the current or planned movement of the vehicle 1. Figure 1 According to step 160 , an object 41 - 43 is evaluated as dangerous if its upcoming state change 41 c - 43 c has the possibility of causing at least one part of the object 41 - 43 to conflict with the current or planned trajectory 1 a of the vehicle 1 .

[0042] In response to the determination that objects 41 - 43 are dangerous or compromised, a warning device 11 of the vehicle can be activated according to step 170 , and / or an intervention 180 a can be used to control a drive system 12 , a steering system 13 and / or a braking system 14 of the vehicle 1 according to step 180 .

[0043] Figure 2 An exemplary traffic situation is shown in the surroundings 2 of a vehicle 1. The vehicle 1 moves along a trajectory 1a on a road 21 which is bordered on both sides by sidewalks 22 and 23. An area 2a of the surroundings 2 is detected by an event-based sensor 3 which is divided into light-sensitive pixels 31. In the exemplary situation shown, three objects 41-43 are closely related.

[0044] The vehicle 41 in front has a bumper 41d, the lower end of which is broken off from its support. Due to the driving dynamics, the bumper 41d performs an overly exaggerated oscillating movement, which is to be regarded as an intrinsic movement 41a of the vehicle 41 as a whole. From this intrinsic movement 41a as an impending state change 41c of the vehicle 1, it can be inferred that the bumper 41d will soon be completely detached from the vehicle 41 and will fall onto the road 21 (at Figure 2 shown by dotted lines).

[0045] The first person 42 on the sidewalk 23 performs a turn around his own axis as an intrinsic movement 42a. From this, it can be inferred as the upcoming movement 42b of this person 42 that this person will run onto the road 21 soon.

[0046] The second person 43 on the sidewalk 22 performs a swinging motion with the stone 43d as an intrinsic motion 43a. It can be inferred from this as an impending state change 43c of the person 43 that he will throw the stone 43d so that it falls on the road 21 or even hits the vehicle 1 (in Figure 2 indicated by dotted lines).

Claims

1. A method (100) for detecting potentially dangerous or endangered objects (41-43) in the surroundings (2) of a vehicle (1), the method comprising the following steps: detecting (110) a region (2a) of the surroundings (2) using at least one event-based sensor (3), wherein the event-based sensor (3) has light-sensitive pixels (31) and wherein a relative change in the intensity of light incident on the pixels (31) by at least one predetermined percentage causes the sensor (3) to output an event (31a) assigned to the pixels (31); assigning (120) an event (31a) output by the sensor (3) to an object (41-43) in the area (2a); For at least one object (41-43) to which an event (31a) is assigned, analyzing (130) the event (31a) assigned to the object (41-43) with respect to a current movement (41a-43a) of the object (41-43); An upcoming movement (41b-43b) of the object (41-43) and / or an upcoming change of state (41c-43c) of the object (41-43) is obtained (140) from the current movement (41a-43a), wherein at least one intrinsic movement is selected as the current movement (41a-43a) of the object (41-43), the intrinsic movement being a movement that allows the position of the object (41-43) as a whole in space to remain unchanged, wherein in response to the fact that the upcoming movement (41b-43b) of the object (41-43) intersects the current or planned trajectory (1a) of the vehicle (1) or comes closer to the trajectory than a predetermined safety distance, the object (41-43) is identified (150) as dangerous or endangered.

2. A method (100) according to claim 1, wherein the object (41-43) is identified as (160) dangerous or endangered in response to an impending change in state (41c-43c) of the object (41-43) having the possibility of causing at least one part of the object (41-43) to intersect with the current or planned trajectory (1a) of the vehicle (1) or to come closer to the trajectory than a predetermined safety distance.

3. The method (100) according to claim 2, wherein the stationary object (41) is excluded (125) from the analysis (130) of the intrinsic movement (41a) and the subsequent evaluation (140) of the intrinsic movement (41a).

4. A method (100) according to claim 3, wherein the exclusion (125) of the stationary object (41) from the analysis (130) of the intrinsic movement (41a) and the subsequent evaluation (140) of the intrinsic movement (41a) is activated (126) depending on the road type of the road traveled by the vehicle (1).

5. A method (100) according to claim 1, wherein a person or an animal is selected as an object (42) whose intrinsic motion (42a) is to be acquired (130), and wherein the impending motion (42b) and / or state change of the person or animal (42) is acquired (141) as the intrinsic motion (42a) from changes in the amplitude and / or frequency of the periodic motion of at least one body part of the person or animal (42) and / or from the rotation of the person or animal (42) or its body part.

6. A method (100) according to any one of claims 1 to 5, wherein a person is selected as an object (42) whose intrinsic movement (42a) is to be analyzed, and wherein the impending movement (42b) of the person (42) in a specific spatial direction is obtained (142) as the intrinsic movement (42a) from the movement indicating the communicative behavior of the person (42) in this spatial direction.

7. A method (100) according to any one of claims 1 to 5, wherein the loosening of the part (41d) from the object (41) is detected (143) as an impending state change (41c) of the object (41) from an intrinsic movement (41a) of the object (41), wherein the intrinsic movement (41a) of the object (41) includes a movement or vibration of the part (41d) of the object (41) relative to the rest of the object (41).

8. A method (100) according to any one of claims 1 to 5, wherein in response to an object (41-43) being identified as (150, 160) dangerous or endangered, a warning device (11) that is physically perceptible to the driver of the vehicle (1) is activated (170), and / or an intervention (180a) is used to manipulate (180) the drive system (12), steering system (13) and / or braking system (14) of the vehicle in order to avoid a collision or mitigate the consequences of a collision.

9. A method (100) according to any one of claims 1 to 5, wherein a camera image of the area (2a) is additionally taken into account (115) in order to identify objects (41-43) located in the area (2a) and / or to divide the objects (41-43) into individual parts.

10. A method (100) according to any one of claims 1 to 5, wherein the analysis (130) of the object (41-43) with respect to the current movement (41a-43a) and / or the evaluation of the current movement (41a-43a) with respect to an upcoming movement (41b-43b) or a state change (41c-43c) is performed (135) using an artificial neural network.

11. A computer program product comprising machine-readable instructions which, when executed on a computer and / or a controller, cause the computer and / or the controller to implement the method (100) according to any one of claims 1 to 10.

Images