VEHICLE SYSTEM FOR A VEHICLE TO PREDICTE THE FUTURE PRESENCE OF A PEDESTRIAN ON A STREET

Abstract

Vehicle system (100) for a vehicle (200) for predicting a future presence of a pedestrian (350) on a road (330, 335), wherein the vehicle system (100) comprises: at least one sensor (120) configured to capture one or more images of the pedestrian (350) located near the road (330, 335); and a control module (102) that communicates with the at least one sensor (120), wherein the control module (102) is configured to: Determine one or more properties associated with the pedestrian (350) located near the road (330, 335) based on the one or more images taken; Creating a trajectory prediction for the pedestrian (350); Overlaying the pedestrian trajectory prediction (350) onto a road segmentation of the road (330, 335); and Creating a road crossing prediction for the pedestrian (350) based on one or more properties and the superimposed trajectory prediction, wherein the road crossing prediction predicts whether the pedestrian (350) will be on or next to the road (330, 335), characterized by the fact that the one or more properties associated with the pedestrian (350) include a crossing intention of the pedestrian (350), indicating whether the pedestrian (350) intends to cross the road (330, 335), and a motion state estimate of the pedestrian (350), indicating whether the pedestrian (350) is moving or not moving, and the control module (102) is configured to generate the road crossing prediction, which predicts that the pedestrian (350) will be on the road (330, 335), only if the pedestrian (350) intends to cross the road (330, 335), the motion state estimation is “in motion”, the speed of the vehicle (200) is below a defined threshold, and the trajectory prediction overlaps at least part of the road (330, 335).

Description

INTRODUCTION

[0001] The invention relates to a vehicle system for a vehicle for predicting the future presence of a pedestrian on a road.

[0002] The present disclosure relates to the prediction of pedestrians crossing roads and, in particular, to vehicle systems and methods for predicting pedestrians crossing roads based on the integration of multiple sources.

[0003] German patent application DE 10 2018 104 270 A1 discloses a vehicle system according to the preamble of claim 1. German patent applications DE 10 2020 121 865 A1 and DE 11 2020 002 666 T5 disclose related systems.

[0004] Vehicles can be fully autonomous, semi-autonomous, or non-autonomous. A fully autonomous or semi-autonomous vehicle may be equipped with a driver assistance system based on blind spot sensors, adaptive cruise control, lane departure warning, and other features. In some cases, the sensors may include radar (e.g., long-range or short-range radar), cameras, and other devices. The sensor data is processed and analyzed to detect objects near the vehicle and is then used by the driver assistance system to control the vehicle.

[0005] One objective of the invention is to provide a vehicle system that can accurately predict whether a pedestrian will be on or next to a road in the future. SUMMARY

[0006] The aforementioned problem is solved by the features of claim 1. Advantageous further developments are defined in the dependent claims.

[0007] A vehicle system for predicting the future presence of a pedestrian on a road is disclosed. The vehicle system comprises at least one sensor configured to capture one or more images of the pedestrian near the road, and a control module that communicates with the sensor. The control module is configured to determine one or more properties associated with the pedestrian near the road based on the one or more captured images, generate a trajectory prediction for the pedestrian, overlay the trajectory prediction onto a road segmentation, and generate a road crossing prediction for the pedestrian based on the one or more properties and the overlaid trajectory prediction.Street crossing prediction forecasts whether the pedestrian will be on or next to the road.

[0008] The properties associated with the pedestrian include the pedestrian's intention to cross, which indicates whether the pedestrian intends to cross the road.

[0009] In other features, the control module is configured to determine the pedestrian's intention to cross the road using a machine learning module based on one or more behavioral features associated with the pedestrian in one or more captured images.

[0010] The one or more properties associated with the pedestrian include a motion state estimation of the pedestrian, indicating whether the pedestrian is moving or not moving.

[0011] In other features, the control module is configured to determine the pedestrian's motion state estimation using a machine learning module based on one or more captured images.

[0012] In other features, the control module is configured to receive position data of the pedestrian in order to detect and locate the pedestrian, and the control module includes a prediction model that is configured to create the trajectory prediction based on the position data.

[0013] In other features, the prediction model is configured to generate the trajectory prediction based on the pedestrian's motion state estimation.

[0014] In other features, the control module is configured to create the road crossing prediction based on the question of whether the trajectory prediction overlaps at least part of the road.

[0015] In other features, the control module is configured to generate the road crossing prediction, which indicates whether the pedestrian will be on or beside the road for a certain period of time.

[0016] Other features of the vehicle system include a sensor configured to detect the vehicle's speed. The control module is configured to generate a pedestrian crossing prediction based on the vehicle's speed.

[0017] The control module is configured to generate the road crossing prediction, which forecasts that the pedestrian is on the road, only if the pedestrian intends to cross the road, the motion state estimation is "in motion", the vehicle speed is below a defined threshold, and the trajectory prediction overlaps at least part of the road.

[0018] According to a non-inventive embodiment, the control module is configured to set confidence values ​​for the pedestrian's intention to cross, for the pedestrian's motion state estimation, and for the prediction that the trajectory overlaps at least part of the road, and to generate the road crossing prediction, which forecasts that the pedestrian is on the road, only if the sum of the confidence values ​​exceeds a defined threshold.

[0019] In other characteristics, the confidence values ​​are weighted values.

[0020] Other features of the vehicle system include a vehicle control module, which is connected to the control module. The vehicle control module is configured to receive one or more signals from the control module indicating the generated road crossing prediction.

[0021] In other features, the vehicle control module is configured to control at least one vehicle control system based on one or more signals.

[0022] Other features of the vehicle control system include an autonomous braking system.

[0023] A method for predicting the future presence of a pedestrian on a road where a vehicle is located is disclosed. The method comprises: determining one or more properties associated with the pedestrian located near the road, based on one or more images captured by at least one sensor; determining one or more properties associated with the pedestrian located near the road, based on the one or more captured images; generating a trajectory prediction for the pedestrian; superimposing the pedestrian trajectory prediction with a road segmentation of the road; and generating a road crossing prediction for the pedestrian based on the one or more properties and the superimposed trajectory prediction.Street crossing prediction forecasts whether the pedestrian will be on or next to the road.

[0024] Other features include the one or more properties associated with the pedestrian, a pedestrian's crossing intention, which indicates whether the pedestrian intends to cross the road, and a pedestrian's motion state estimate, which indicates whether the pedestrian is moving or not moving.

[0025] In other features, creating the pedestrian road crossing prediction includes creating the road crossing prediction that predicts the pedestrian will be on the road only if the pedestrian intends to cross the road, the motion state estimation is "in motion", the vehicle speed is below a defined threshold, and the trajectory prediction overlaps at least part of the road.

[0026] In other features, the procedure further includes setting confidence values ​​for the pedestrian's intention to cross, for the estimate of the pedestrian's state of motion, and for the prediction that the trajectory will overlap at least part of the road; and generating the road crossing prediction for the pedestrian includes generating the road crossing prediction that predicts the pedestrian will be on the road only if a sum of the confidence values ​​exceeds a defined threshold.

[0027] In other features, the method further includes the creation of one or more signals indicating the generated road crossing prediction and the control of at least one vehicle control system based on the one or more signals.

[0028] Other features of the vehicle control system include an autonomous braking system.

[0029] Further applications of this disclosure will become apparent from the detailed description, the claims, and the drawings. The detailed description and specific examples serve only for illustration and are not intended to limit the scope of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present disclosure will become more fully apparent from the detailed description and the accompanying drawings, whereby the following applies: Fig. Figure 1 is a block diagram of an example of a vehicle system for predicting the future presence of a pedestrian on a road according to the present disclosure; Fig. 2 is a vehicle with parts of the vehicle system from Fig. 1, according to the present revelation; Fig. Figure 3 is a block diagram of an example prediction scenario in which the vehicle consists of Fig. 2 approaches a zebra crossing near which a pedestrian is located, according to the present revelation; and Fig. Figures 4-6 are flowcharts of exemplary control processes for predicting the future presence of a pedestrian on a road, according to the present disclosure.

[0031] Reference numbers can be reused in the drawings to designate similar and / or identical elements. DETAILED DESCRIPTION

[0032] Autonomous vehicles are equipped with a driver assistance system that relies on sensors for vehicle control. These sensors (e.g., radar, cameras, etc.) provide data that can be analyzed to detect the presence and / or future location of objects, such as pedestrians. Predicting whether pedestrians will cross a road in the near future where a vehicle is present is a critical aspect of autonomous (e.g., fully and partially autonomous) driving. However, such predictions are complex and demanding, requiring, for example, an understanding of the scene, the interactions between pedestrians and the road, the movements of pedestrians and vehicles, and other factors.

[0033] The vehicle systems and methods described herein accurately predict whether a pedestrian will be on or beside a road in the future. This can be achieved by combining information from various sources, such as estimated pedestrian characteristics, estimated pedestrian trajectories, vehicle characteristics, and road characteristics, as explained below. By integrating such information, the vehicle systems and methods described herein can predict the future presence of a pedestrian on a road (e.g., within a short timeframe). Based on this prediction, autonomous vehicles can then react quickly and change their course (e.g., turn, brake, etc.) to avoid the pedestrian.

[0034] In Fig. Figure 1 shows a block diagram of an example of a vehicle system 100 that predicts the presence of a pedestrian on a road. As explained below, the vehicle system 100 accurately predicts, for example, whether a person will be on a road within a future time interval (e.g., within the next 3 seconds, between 0.5 seconds and 2 seconds, between 1 second and 3 seconds, etc.).

[0035] As in Fig. As shown in Figure 1, the vehicle system 100 generally comprises a control module 102, a vehicle control module 104, one or more vehicle control systems 106, and various sensors. The control module 102 generally comprises a crossing intention module 108, a motion state estimation module 110, a trajectory prediction module 112, a trajectory overlay module 114, and a road crossing prediction module 116. Although in Fig. 1 where the vehicle system 100 is represented with several separate modules, any combination of the modules (e.g. the control module 102, the vehicle control module 104, the modules in the control module 102, etc.) and / or their functionality can be integrated into one or more modules.

[0036] The sensors of Fig. 1. These may include one or more devices for capturing images of the vehicle's surroundings and / or for recording vehicle parameters. In the example of Fig. 1. The sensors can, for example, include one or more cameras 118 that capture a single image (e.g., a single picture) and / or multiple images over time (e.g., a video) of the vehicle's surroundings. Additionally, the sensors can include a speed sensor 120 for detecting the vehicle's speed. In such examples, the speed sensor 120 can be an inertial measurement unit (IMU), a wheel speed sensor (WSS), a vehicle speed sensor (VSS), or another suitable sensor for generally detecting the vehicle's speed. As in Fig. As shown in 1, the sensors are located at Fig. 1 in communication with the control module 102. As explained below, for example, the camera(s) 118 are connected to the crossing intention module 108 and the motion state estimation module 110, and the speed sensor 120 is connected to the road crossing prediction module 116.

[0037] Although in Fig. Not shown in Figure 1, the modules and sensors of the vehicle system can exchange up to 100 parameters via a network, such as a Controller Area Network (CAN). In such examples, parameters can be shared via one or more data buses of the network. This allows various parameters from a specific module and / or sensor to be made available to other modules and / or sensors via the network.

[0038] The vehicle system 100 of Fig. 1 can be used in various embodiments in any suitable vehicle, such as an electric vehicle (e.g., a pure electric vehicle, a plug-in hybrid electric vehicle, etc.), a vehicle with an internal combustion engine, etc. Furthermore, the vehicle system 100 can be used for autonomous vehicles, including fully autonomous and semi-autonomous vehicles. Fig. Figure 2, for example, shows a vehicle 200, which includes the control module 102 and the vehicle control module 104. Fig. 1 and one or more sensors 204 (e.g. one or more of the cameras 118 of Fig. 1, the speed sensor 120 from Fig. 1 etc.) contains modules that communicate with control module 102.

[0039] As in Fig. As shown in Figure 1, the control module 102 can determine one or more properties of the pedestrian who is near or beside the road. These properties can be based on one or more images captured by the camera(s) 118. In such examples, each camera 118 can capture one or more images of a pedestrian who is near a road on which the vehicle is located (e.g., stopped, parked, moving, etc.) or toward which it is turning. The captured image(s) can be a single frame or a video of any duration (e.g., 0.2 seconds, 0.3 seconds, 0.4 seconds, 0.5 seconds, 0.6 seconds, 0.7 seconds, etc.).

[0040] Fig. Figure 3, for example, shows a block diagram of an example forecast scenario 300, in which the vehicle 200 from Fig. 2. A vehicle is traveling along a road 330 and approaching a zebra crossing 370. In this example, the vehicle 200 can turn onto another road 335 with a zebra crossing 380, as shown in... Fig. Figure 3 shows that at least one camera on the vehicle 200 can capture one or more images of a pedestrian 350 who is located near roads 330, 335. The camera can, for example, have a field of view 340 represented by lines with a dash-dot-dash configuration. In the example of Fig. 3. In the future, pedestrian 350 can begin to cross road 330 (e.g. on or next to the zebra crossing 370) or road 335 (e.g. on or next to the zebra crossing 380).

[0041] In the example of Fig. 1. The properties determined by the control module 102 can be any suitable predictive and / or estimated pedestrian attribute. For example, the control module 102 can determine a pedestrian's crossing intention, indicating whether the pedestrian intends to cross the street (e.g., Street 330, Street 335, etc.). In such examples, the crossing intention module 108 of the control module 102 receives the captured image(s) or representative data from at least one of the cameras 118, generates a pedestrian crossing intention, and then outputs the crossing intention to the street crossing prediction module 116.

[0042] In such examples, the crossing intention represents what the pedestrian intends to do in the near future. The crossing intention module 108 can, for example, determine whether the pedestrian intends to cross the street or not. In such examples, the crossing intention module 108 can output a positive signal (e.g., a "1" or another suitable indicator) to the street crossing prediction module 116 if the determined intention is to cross the street, and a negative signal (e.g., a "0" or another suitable indicator) to the street crossing prediction module 116 if the determined intention is not to cross the street.

[0043] In various embodiments, the crossing intention module 108 can determine the pedestrian's crossing intention based on one or more behavioral features attributed to the pedestrian in the captured images. For example, the crossing intention module 108 can include or be associated with a machine learning module (e.g., a neural network or other suitable machine learning module) that analyzes the pedestrian's behavioral features (from the images) and then determines the crossing intention based on these features. In such examples, the machine learning module can be trained to distinguish between crossing and non-crossing intentions. For example, tagged datasets can be created based on one or more analysts examining input images (e.g., 0.5-second video clips, etc.).In such examples, analysts can identify characteristics / properties of pedestrians and their relationship to the road and the vehicle, such as the distance between a pedestrian and the road, whether the pedestrian's head is pointing up or down, where the pedestrian is looking (e.g., towards the road, towards the vehicle, away from the road, etc.), whether the pedestrian's eyes are closed or open, etc.

[0044] Furthermore, one or more of the properties that the control module 102 of Fig. 1 determines the motion state of the pedestrian. For example, the motion state estimation module 110 of the control module 102 receives the captured image(s) or representative data from at least one of the cameras 118, generates an estimated motion state for the pedestrian, and then outputs the motion state to the road crossing prediction module 116. In such examples, the captured image(s) or data used by the motion state estimation module 110 may be the same or different images / data used by the crossing intention module 108.

[0045] In the example of Fig. 1. The motion state estimation represents the current state of the pedestrian. For example, the motion state estimation can fall into one of two suitable categories. For example, the motion state estimation can indicate whether the pedestrian is moving or not. In other examples, the motion state estimation can represent whether the pedestrian is walking (or running, etc.) or stationary (e.g., standing, sitting, etc.). Regardless of the categories, the motion state estimation module 110 can output a positive signal (e.g., a "1" or another suitable indicator) to the road crossing prediction module 116 if the current pedestrian state falls into one category (e.g., in motion), and a negative signal (e.g., a "0" or another suitable indicator) to the road crossing prediction module 116 if the current pedestrian state falls into another category (e.g., in non-movement).

[0046] In various embodiments, the motion state estimator 110 can determine the pedestrian's motion state using a machine learning module that analyzes the pedestrian's behavioral characteristics from the received images. For example, the motion state estimator 110 can include or communicate with a machine learning module (e.g., a neural network or other suitable machine learning module) trained to categorize a pedestrian's current state. In such examples, the machine learning module for the motion state estimator 110 can be trained on labeled datasets created based on one or more analysts reviewing input images (e.g., 0.5-second input video clips, etc.).In such examples, the input images can be cropped around the position of a pedestrian for training and / or determining the state of movement in order to reduce processing requirements.

[0047] With continued reference to Fig. 1. The control module 102 can generate a trajectory prediction for the pedestrian. In such examples, the trajectory prediction can be determined using the trajectory prediction module 112 based on the pedestrian's position data. In Fig. For example, the trajectory prediction module 112 receives a signal 122 indicating the pedestrian's position. In some examples, the signal 122 may include object detection data for identifying and locating the pedestrian in a 3D or 2D environment based on various properties such as the pedestrian's shape, location, orientation, etc. In other words, the trajectory prediction module 112 can receive 3D or 2D object detection data. In such examples, the 3D or 2D object detection can be performed using conventional techniques and based on input from one or more sensors on the vehicle (e.g., the camera(s) 118, radar sensors, lidar sensors, etc.).

[0048] In various embodiments, the trajectory prediction module 112 can generate the trajectory prediction using a prediction model that receives pedestrian position data (e.g., the 3D or 2D object acquisition data) as input. For example, the trajectory prediction module 112 can include or be associated with a Kalman filter or another suitable prediction model that predicts the pedestrian's next position based on the received position data (e.g., previous coordinates) of the pedestrian. After generation, the trajectory prediction, or data representative of it, can be output to the trajectory overlay module 114, as shown in Fig. 1 shown.

[0049] In some examples, the trajectory prediction module 112 can optionally generate the trajectory prediction based on other inputs. As in Fig. As shown in Figure 1, the trajectory prediction module 112 receives, for example, an input signal 124 from the motion state estimator module 110. In such examples, the input signal 124 can be the motion state estimator (e.g., a positive or negative signal) output by the motion state estimator module 110 as explained above. The trajectory prediction module 112 can then generate the trajectory prediction based on the pedestrian's motion state estimator. For example, the input signal 124 (e.g., the motion state estimator) can be used as a control signal for the trajectory prediction module 112. In such examples, the trajectory prediction module 112 can generate the trajectory prediction only if the input signal 124 is positive (e.g., the pedestrian is moving). In other examples, the trajectory prediction module 112 can take the motion state estimator into account when generating the trajectory prediction.For example, the generated trajectory prediction may differ depending on whether the pedestrian is moving or not.

[0050] The control module 102 of Fig. 1. The generated pedestrian trajectory prediction can be overlaid with a street segmentation. As in Fig. As shown in Figure 1, the trajectory overlay module 114 receives, for example, the trajectory prediction from the trajectory prediction module 112 and an input signal 126 that specifies a road segmentation of the road on which the vehicle is located or into which it is turning. In such examples, the road segmentation can be an image or a map.

[0051] In various embodiments, the trajectory prediction can be overlaid with a machine learning module. In such examples, the trajectory overlay module 114 can contain or communicate with a machine learning module (e.g., a neutral network or another suitable machine learning module) that is designed and trained to segment road pixels on the received image or map, which already contains road segmentations. For example, a received image (e.g., a satellite image) is segmented with respect to the road. In such examples, each pixel of the image has coordinates and can be classified based on these coordinates as to whether it is part of the road or not. The trajectory overlay module 114 then overlays the trajectory prediction (e.g., a specific area, an arrow, etc.) onto the segmented image.In such examples, the pixels of the trajectory prediction are overlaid on the pixels of the segmented image.

[0052] The trajectory overlay module 114 then determines whether the trajectory prediction overlaps the road. After the trajectory prediction module 112 has been overlaid onto the segmented image (or map), the trajectory overlay module 114 can, for example, determine whether any of the trajectory prediction pixels overlap pixels classified as part of the road, and then provide an output to the road crossing prediction module 116 indicating this determination. For example, if some of the trajectory prediction pixels overlap the road pixels, the trajectory overlay module 114 can output a positive signal (e.g., a "1" or other suitable indicator) to the road crossing prediction module 116 to indicate that the pedestrian trajectory is on the road. However, if no of the trajectory prediction pixels overlap the road pixels, the trajectory overlay module 114 can output a negative signal (e.g.,a “0” or another suitable indicator) can be output to the road crossing prediction module 116 to signal that the pedestrian trajectory does not follow the road. In other examples, the trajectory overlay module 114 can output a positive signal if a defined set (e.g., a threshold) of road pixels overlaps with the trajectory prediction pixels, and a negative signal if this is not the case.

[0053] As in Fig. As shown in Figure 1, the control module 102 can then generate a road crossing prediction for the pedestrian, predicting whether the pedestrian will be on or off the road. The road crossing prediction module 116, for example, can generate a road crossing prediction indicating whether the pedestrian will be on or beside the road. In various embodiments, the road crossing prediction module 116 can generate the road crossing prediction and predict whether the pedestrian will be on or beside the road for a specific period of time. In such examples, the road crossing prediction can specify when the pedestrian will be on or beside the road. For instance, the road crossing prediction module 116 can determine that the pedestrian will be on the road at time X, between time Y and time Z, and / or at another future time.The road crossing prediction might indicate, for example, that the pedestrian will be on the road within 3 seconds. In other examples, the road crossing prediction might indicate that the pedestrian will be on the road at time t + T, where t is the current time (e.g., 0 seconds) or a later time (e.g., 0.5 seconds) and T is a time after t (e.g., 1.5 seconds, 2 seconds, 3 seconds, 3.5 seconds, etc.).

[0054] In various embodiments, the road crossing prediction module 116 can utilize multiple inputs to generate the road crossing prediction. In such examples, the road crossing prediction module 116 can leverage a comprehensive view of the scene / environment to understand pedestrian-road interaction and gain knowledge about pedestrian and vehicle movements. For instance, the road crossing prediction module 116 can generate the road crossing prediction based on at least the properties associated with the pedestrian and the superimposed trajectory prediction. In such examples, the road crossing prediction can be generated based on whether the trajectory prediction overlaps at least a portion of the road (e.g., as indicated by the trajectory overlay module 114).

[0055] In some embodiments, the road crossing prediction module 116 can also take the vehicle's speed into account when generating the road crossing prediction. For example, the road crossing prediction module 116 can receive the vehicle's speed from the speed sensor 120, as shown in Fig. Figure 1 shows the pedestrian crossing situation, and then the road crossing prediction is generated based on speed. For example, if the vehicle is traveling at a high speed (e.g., the speed is greater than a threshold), the probability of the pedestrian crossing the road may be low. However, if the vehicle is traveling at a low speed (e.g., below a threshold), the probability of the pedestrian crossing the road may be higher.

[0056] Control module 102 can generate the road crossing prediction by predicting that the pedestrian will cross the road under various conditions. For example, control module 102's road crossing prediction module 116 can generate the road crossing prediction that forecasts the pedestrian will be on the road (and in some cases within a possible time span) only if the pedestrian intends to cross the road, the motion state estimate is "in motion," the vehicle's speed is below a certain threshold, and the trajectory prediction overlaps at least part of the road. In other words, each input fed to road crossing prediction module 116 may be required to satisfy a specific condition. For example, the output of crossing intention module 108 may need to be positive (e.g.,the pedestrian's intention is to cross the road), that the output of the motion state estimation module 110 is positive (e.g. the pedestrian is moving), that the vehicle's speed is below a defined threshold (e.g. 6 m / s, 5.5 m / s, 5 m / s, 4.5 m / s, 4 m / s, etc.), and that the output of the trajectory overlay module 114 indicates that the pedestrian's trajectory is on the road (e.g. the pedestrian's trajectory overlaps the road).

[0057] In other examples, the road crossing prediction module 116 can generate a road crossing prediction that the pedestrian will be on the road if a majority of the inputs meet certain conditions. In such examples, if any three of the inputs meet certain conditions, as explained above, the road crossing prediction can be generated stating that the pedestrian will be on the road. In some examples, specific inputs may be required to meet the conditions explained above. For example, if the three inputs that meet certain conditions include the pedestrian's state of motion (moving) and the pedestrian's trajectory (being on the road), the road crossing prediction module 116 can generate a road crossing prediction stating that the pedestrian will be on the road.

[0058] In other examples, the road crossing prediction module 116 can generate a road crossing prediction that the pedestrian will be on the road, based on a scoring system. For example, the road crossing prediction module 116 can set or receive confidence values ​​for the pedestrian's crossing intention (from the road crossing prediction module 116), for the pedestrian's motion state estimation (from the motion state estimation module 110), for the prediction that the trajectory will overlap at least part of the road (from the trajectory overlay module 114), and / or for the vehicle's speed. The road crossing prediction module 116 can then generate the road crossing prediction only if the sum of the confidence values ​​exceeds a certain threshold. In some examples, the confidence values ​​can be weighted, if desired.In such examples, each confidence value associated with a particular input can be weighted based on its importance relative to the other inputs.

[0059] The control module 102 can be configured in various embodiments. Fig. 1. Generate a signal indicating the generated road crossing prediction and transmit it to the vehicle control module 104. In such examples, the control module 102 can generate signals to indicate the prediction that the pedestrian will be on the road, and in some cases within a possible time period, as explained above. Upon receipt, the vehicle control module 104 can generate one or more control signals to control the vehicle control system(s) 106 based on the road crossing prediction.

[0060] For example, the vehicle control module 104 can use the road crossing prediction signal to control driver assistance systems in the vehicle 200. Fig. 2 - 3 to control. In such examples, the driver assistance systems may include, for example, an autonomous braking system (e.g., autonomous emergency braking (AEB), assisted steering (e.g., assisted evasive steering), and / or any other suitable driver assistance system that can be used to brake the vehicle, change the vehicle's direction, etc. In other examples, the vehicle control system(s) 106 may include pedestrian and / or driver notification systems. In such examples, the vehicle control module 104 may trigger an alarm (e.g., an audible warning, a visual warning, etc.) inside and / or outside the vehicle to warn the pedestrian and / or the driver of the vehicle.

[0061] In the Fig. Figures 4-6 illustrate exemplary control processes 400, 500, and 600 for predicting the future presence of a pedestrian on a street. These examples... Fig. 4 - 6 can use the control methods 400, 500, 600 with the vehicle system 100 of the Fig. 1 will be realized, which is in vehicle 200 of the Fig. 2 - 3 can be used. Although the exemplary control methods 400, 500, 600 with regard to the vehicle system 100 of Fig. While the control methods 400, 500, and 600 can be described using control module 102, each of these can also be used in any other suitable vehicle system. Although the steps of control methods 400, 500, and 600 are presented and described below in a specific order, they can also be performed in a different suitable sequence if desired. For example, the decision steps of Fig. 5. These steps must be carried out in a different order than shown.

[0062] As in Fig. As shown in Figure 4, the control process 400 begins with module 402, where the control module 102 receives one or more images. As explained above, the control module 102 (more precisely, the crossing intention module 108 and the motion state estimation module 110) receives, for example, an image and / or a video (e.g., multiple still images or images) that was captured by one or more cameras of the vehicle 200. The control process then proceeds to module 404.

[0063] In the 404, the control module 102 determines one or more properties associated with a pedestrian based on the received images. In various embodiments, the pedestrian properties may include, for example, the pedestrian's intention to cross the road (e.g., the pedestrian's intention to cross the road or not) and a motion state estimation of the pedestrian (e.g., in motion or not in motion), as explained above. In such examples, the crossing intention module 108 and the motion state estimation module 110 of Fig. 1. Generate the crossing intention or motion state estimation using separate machine learning modules (e.g., neural networks, etc.), as explained herein. The control then proceeds to 406.

[0064] At 406, control module 102 generates a trajectory prediction for the pedestrian. As explained above, for example, the trajectory prediction module 112 of control module 102 can generate the trajectory prediction based on pedestrian position data (e.g., 3D or 2D object capture data). In some examples, the trajectory prediction can be generated using a Kalman filter or another suitable prediction model. The control then proceeds to 408.

[0065] At 408, the control module 102 overlays the generated trajectory prediction with a road segmentation of the road. In various embodiments, and as explained above, the overlay of the trajectory prediction can be performed using a machine learning module (e.g., a neutral network, etc.) trained to segment road pixels on an image or map that already contains road segmentations. As explained above, for example, each pixel of the segmented image or map has coordinates and can be classified as part of the road or not based on these coordinates. The trajectory prediction pixels can then be overlaid on the pixels of the segmented image or map based on these coordinates. The control then proceeds to 410.

[0066] At 410, control module 102 generates a pedestrian crossing prediction and forecasts whether the pedestrian will be on or beside the road. In some examples, the pedestrian crossing prediction can be combined with pedestrian crossing prediction module 116. Fig. 1. Based on at least the determined pedestrian characteristics and the superimposed trajectory prediction, as explained herein, the road crossing prediction can be generated. In other examples, additional inputs (e.g., the vehicle speed 200) can be used when generating the road crossing prediction. The control then proceeds to 412.

[0067] At 412, the control module 102 determines whether the road crossing prediction indicates or predicts that the pedestrian will be on the road at a specific time, e.g., at time X, between time Y and time Z, and / or at another future time interval. If not, the control returns to 402, as in Fig. 4 is shown. If so, the control continues with 414.

[0068] At 414, control module 102 generates a signal indicating the generated road crossing prediction. In some examples, control module 102 (e.g., the road crossing prediction module 116) can send the signal to a vehicle control module (e.g., the vehicle control module 104). Fig. 1) transmitted. The control then proceeds to 416, where a vehicle action is initiated based on the signal. For example, the vehicle control module or control module 102 can generate a control signal to control one or more vehicle control systems based on the generated road crossing prediction, as explained above. The control can then, as in Fig. 4 is displayed, can be terminated, or return to 402 if desired.

[0069] In Fig. 5. The control process 500 with 502 begins, with the control module 102 receiving a vehicle speed 200. The road crossing prediction module 116 of the control module 102 can, for example, obtain the speed (or representative data) from the speed sensor 120. Fig. 1 or another suitable sensor of the vehicle receives 200. The control then proceeds to 404.

[0070] At 504, the control module 102 determines whether the vehicle's speed is below a certain threshold. As previously explained, the road crossing prediction module 116 can compare the speed to any threshold, e.g., 6 m / s, 5.5 m / s, 5 m / s, 4.5 m / s, 4 m / s, etc. If the speed is not below the threshold at 504, the control returns to 502. However, if the speed is below the threshold at 504, the control continues with 506.

[0071] At 506, the control module 102 determines whether a crossing intention indicates that the pedestrian intends to cross the street. In such examples, the street crossing prediction module 116 can receive a signal from the crossing intention module 108 indicating whether the pedestrian intends to cross or not. As previously explained, the crossing intention module 108 can, for example, use a machine learning module that analyzes behavioral characteristics of the pedestrian and then determines the crossing intention based on these characteristics. If 506 indicates no, the control returns to 502. If 506 indicates yes, the control proceeds to 508.

[0072] At 508, the control module 102 determines whether the pedestrian's motion state is "in motion." For example, the road crossing prediction module 116 might receive a signal from the motion state estimator module 110 indicating the pedestrian's current motion state. In such examples, the motion state estimator module 110 might use a machine learning module that analyzes the pedestrian's behavior and then determines the motion state based on the analysis, as explained above. The pedestrian's current motion state, as determined by the motion state estimator module 110, can be "in motion" (e.g., walking, running, etc.) or "stationary" (e.g., standing still, etc.). If 508 is no, the control returns to 502. If 508 is yes, the control proceeds to 510.

[0073] At 510, the control module 102 determines whether a pedestrian trajectory prediction overlaps the road. As explained above, the control module 102's trajectory prediction module 112, for example, can generate the pedestrian trajectory prediction based on 3D or 2D object capture data of the pedestrian. Then, the control module 102's trajectory overlay module 114 can overlay the pedestrian trajectory prediction onto a road segmentation, as explained above. In such examples, the trajectory overlay module 114 can overlay the trajectory prediction (e.g., a specific area, an arrow, etc.) onto the segmented image and then determine whether any pixels of the trajectory prediction are placed over pixels representing the road in the segmented image. If so, the trajectory prediction overlaps the road. If not, at 510, the control returns to 502. If yes at 510, the control continues to 512.

[0074] At 512, the control module 102 generates a street crossing prediction for the pedestrian and forecasts whether the pedestrian will be on or beside the road. In some examples, the street crossing prediction can be performed using the street crossing prediction module 116. Fig. 1. A predicted timeframe in which the pedestrian is expected to be on the road is created, as explained here. For example, the road crossing prediction module 116 can predict that the pedestrian will be on the road at a specific time, e.g., at time X, between time Y and time Z, and / or at another future time interval.

[0075] The control then proceeds to 514, where control module 102 determines whether vehicle action is required. This determination can be based on the road crossing prediction and other suitable inputs (e.g., vehicle speed, vehicle trajectory, etc.). For example, control module 102 may determine that at a given time, vehicle 200 is not in the section of the road that the pedestrian is expected to cross. In such examples, control module 102 may determine that no vehicle action is required. In other examples, control module 102 may determine that vehicle 200 is on a section of the road that the pedestrian is expected to cross at a given time. If this is the case, control module 102 may determine that vehicle action is required. If 514 determines that no vehicle action is required, the control returns to 502.If this is the case at 514, the control continues with 414, 416, as above in relation to . Fig. 4 explained. The control can then be, as in Fig. 5 is displayed, will be terminated, or return to 502 if desired.

[0076] In various embodiments, decision steps 504, 506, 508, 510 of Fig. 5. They function differently. For example, each decision step 504, 506, 508, 510 can continue with the next step instead of going to step 502 of Fig. 5 to return if the result is "No". Then the road crossing prediction module 116 can create the road crossing prediction at 512, which predicts that the pedestrian will be on the road if a majority or certain of the decision steps 504, 506, 508, 510 result in a Yes.

[0077] In Fig. In step 6, the control process 600 begins with step 602, where the control module 102 sets values ​​for various generated parameters. Specifically, the control module 102 sets a confidence value for the pedestrian's intention to cross the road (from the road crossing prediction module 116), a confidence value for the pedestrian's motion state estimation (from the motion state estimation module 110), a confidence value for the prediction that the trajectory will overlap at least part of the road (from the trajectory overlay module 114), and / or a confidence value for the vehicle's speed. In such examples, the road crossing prediction module 116 can be used by Fig. 1. Set the confidence values ​​and / or receive the values ​​from the road crossing prediction module 116, the motion state estimation module 110, the trajectory overlay module 114, and the speed sensor 120. Furthermore, in some examples, each associated confidence value can be weighted based on its significance, as explained above. Control then proceeds to 604.

[0078] At 604, the control module 102 combines the confidence values ​​into an overall value. This can be done by the road crossing prediction module 116 or another suitable module in Fig. 1. The control then proceeds to module 606, where control module 102 determines whether the total value (e.g., the combined confidence values) is greater than a defined threshold. As explained above, the road crossing prediction module 116, for example, can compare the combined confidence values ​​with any suitable threshold. If the threshold is not met at module 606, the control can return to module 602, as described in Fig. Figure 6 illustrates this. In some examples, the road crossing prediction module 116 can generate a road crossing prediction stating that the pedestrian will not be on the road before returning to module 602. If this occurs at module 606, the control continues to modules 512 and 514, as described above. Fig. 5 explained, and then with 414, 416, as above in relation to Fig. 4 explained. The control can then be, as in Fig. 6 is displayed, can be terminated, or return to 602 if desired.

[0079] The systems and methods described here accurately predict, based on information from multiple sources, whether a pedestrian will be on or beside a road in the future. In various embodiments, the systems and methods can, for example, generate a road-crossing prediction for a pedestrian, forecasting whether the pedestrian will be on or beside the road (i.e., crossing or not crossing) within 0.5 to 2 seconds. In such examples, the accuracy for crossing roads can be 80.3% or higher, the accuracy for not crossing roads can be 92% or higher, and the balanced accuracy for both crossing and not crossing roads can be 86.6% or higher.

[0080] The foregoing description is for illustrative purposes only. Although each of the embodiments above is described with certain features, any one or more of these features described in relation to any embodiment of the disclosure may be implemented in one of the other embodiments and / or combined with features of another embodiment, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with each other remain within the scope of this disclosure.

[0081] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, such as "connected," "interlocking," "coupled," "adjacent," "next to," "on," "above," "below," and "arranged." If a relationship between a first and a second element is not explicitly described as "direct" in the above disclosure, this relationship may be a direct relationship in which no other intervening elements exist between the first and the second element, or it may be an indirect relationship in which one or more intervening elements (either spatial or functional) exist between the first and the second element.As used herein, the phrase “at least one of A, B and C” should be interpreted as a logical (A OR B OR C) using a non-exclusive logical OR and not as “at least one of A, at least one of B and at least one of C”.

[0082] In the diagrams, the direction of an arrow, as indicated by the arrowhead, generally shows the flow of information (e.g., data or instructions) that is relevant to the illustration. For example, if Element A and Element B exchange a variety of information, but the information transferred from Element A to Element B is relevant to the illustration, the arrow may point from Element A to Element B. This unidirectional arrow does not imply that no further information is transferred from Element B to Element A. Furthermore, Element B may send requests for or acknowledgments of information to Element A in return for information sent from Element A to Element B.

[0083] In this application, including the definitions below, the term "module" or the term "controller" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include: an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor circuit (common, dedicated, or group) that executes code; a memory circuit (common, dedicated, or group) that stores the code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, e.g., in a system-on-a-chip.

[0084] The module may contain one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the internet, a wide area network (WAN), or combinations thereof. The functionality of any module of this disclosure may be distributed across multiple modules connected via interface circuits. For example, multiple modules may enable load balancing. In another example, a server module (also called a remote or cloud module) may perform some functions on behalf of a client module.

[0085] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" refers to a single processor circuit that executes some or all of the code of multiple modules. The term "group processor circuit" refers to a processor circuit that, in combination with other processor circuits, executes some or all of the code of one or more modules. References to "multiple processor circuits" include multiple processor circuits on discrete chips, multiple processor circuits on a single chip, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above.The term "shared memory circuit" refers to a single memory circuit that stores some or all of the code from multiple modules. The term "group memory circuit" refers to a memory circuit that, in combination with other memory devices, stores some or all of the code from one or more modules.

[0086] The term "memory circuit" is a subset of the term "computer-readable medium." The term "computer-readable medium," as used here, does not include transitory electrical or electromagnetic signals that propagate through a medium (e.g., on a carrier wave); the term "computer-readable medium" can therefore be considered tangible / material and non-transient. Non-restrictive examples of a non-transient, tangible, computer-readable medium are non-volatile memory circuits (e.g., a flash memory circuit, a erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (e.g., a static random-access memory circuit or a dynamic random-access memory circuit), magnetic storage media (e.g., an analog or digital magnetic tape or a hard disk drive), and optical storage media (e.g.,a CD, a DVD or a Blu-ray Disc).

[0087] The devices and methods described in this application can be implemented partially or completely by a specialized computer formed by configuring a general-purpose computer to perform one or more specific functions embodied in computer programs. The functional blocks, flowchart components, and other elements described above serve as software specifications that can be translated into computer programs through the routine work of an experienced technician or programmer.

[0088] The computer programs contain processor-executable instructions stored on at least one non-transient, tangible, machine-readable medium. The computer programs may also contain or access stored data. The computer programs may include a basic input / output system (BIOS) that interacts with the hardware of the specialized computer, device drivers that interact with specific devices of the specialized computer, one or more operating systems, user applications, background services, background applications, etc.

[0089] The computer programs can contain: (i) descriptive text to be parsed, e.g., HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from the source code by a compiler; (iv) source code for execution by an interpreter; (v) source code for compilation and execution by a just-in-time compiler, etc. The source code can, for example, use the syntax of languages ​​such as C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5th revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, Simulink, and others. It must be written in Python®.

Claims

[1] Vehicle system (100) for a vehicle (200) for predicting a future presence of a pedestrian (350) on a road (330, 335), wherein the vehicle system (100) comprises: at least one sensor (120) configured to capture one or more images of the pedestrian (350) located near the road (330, 335); and a control module (102) that communicates with the at least one sensor (120), wherein the control module (102) is configured to: Determine one or more properties associated with the pedestrian (350) located near the road (330, 335) based on the one or more images taken; Creating a trajectory prediction for the pedestrian (350); Overlaying the pedestrian trajectory prediction (350) onto a road segmentation of the road (330, 335); and Creating a road crossing prediction for the pedestrian (350) based on one or more properties and the superimposed trajectory prediction, wherein the road crossing prediction predicts whether the pedestrian (350) will be on or next to the road (330, 335), characterized by , that the one or more properties associated with the pedestrian (350) include a crossing intention of the pedestrian (350), indicating whether the pedestrian (350) intends to cross the road (330, 335), and a motion state estimate of the pedestrian (350), indicating whether the pedestrian (350) is moving or not moving, and the control module (102) is configured to generate the road crossing prediction, which predicts that the pedestrian (350) will be on the road (330, 335), only if the pedestrian (350) intends to cross the road (330, 335), the motion state estimation is “in motion”, the speed of the vehicle (200) is below a defined threshold, and the trajectory prediction overlaps at least part of the road (330, 335). [2] Vehicle system (100) according to claim 1, wherein the control module (102) is configured to determine the pedestrian's (350) intention to cross using a machine learning module based on one or more behavioral features associated with the pedestrian (350) in the one or more recorded images. [3] Vehicle system (100) according to claim 1, wherein the control module (102) is configured to determine the motion state estimation of the pedestrian (350) using a machine learning module based on the one or more recorded images. [4] Vehicle system (100) according to claim 1, wherein: the control module (102) is configured to receive position data from the pedestrian (350) in order to detect and locate the pedestrian (350); and the control module (102) contains a prediction model that is configured to generate the trajectory prediction based on the position data. [5] Vehicle system (100) according to claim 1, further comprising a sensor (120) configured to detect the speed of the vehicle (200), wherein the control module (102) is configured to generate the road crossing prediction for the pedestrian (350) based on the speed of the vehicle (200). [6] Vehicle system (100) according to claim 1, further comprising a vehicle control module (104) which is connected to the control module (102), wherein the vehicle control module (104) is configured to receive one or more signals from the control module (102) indicating the generated road crossing prediction. [7] Vehicle system (100) according to claim 6, wherein the vehicle control module (104) is configured to control at least one vehicle control system based on one or more signals.

Images