Vehicle systems and methods for pedestrian road cross prediction

Abstract

A vehicle system includes at least one sensor configured to capture one or more images of a pedestrian located near a road; and a control module in communication with the at least one sensor. The control module is configured to: determine, based on the one or more captured images, one or more characteristics associated with a pedestrian located near the road; generating a trajectory prediction for the pedestrian; covering a road segment of the road with the trajectory prediction for the pedestrian; and generating a road crossing prediction for the pedestrian based on the one or more characteristics and the covered trajectory prediction, the road crossing prediction predicting whether the pedestrian will be on the road or outside the road. Other example vehicle systems and methods for predicting future presence of pedestrians on a road are also disclosed.

Description

Background Art

[0001] The information provided in this section is for the purpose of presenting the context of the present disclosure generally. The work of the presently named inventors, to the extent described in this section, and aspects of the description that may not have been eligible as prior art at the time of filing are neither expressly nor implicitly admitted as prior art against the present disclosure.

[0002] The present disclosure relates to pedestrian road crossing prediction, and more particularly to vehicle systems and methods for performing pedestrian road crossing prediction based on the integration of multiple sources.

[0003] A vehicle can be fully autonomous, semi-autonomous, or non-autonomous. When fully or semi-autonomous, the vehicle can include driver assistance systems that rely on sensors for blind spot detection, adaptive cruise control, lane departure warning, etc. In some cases, the sensors can include radar devices (e.g., long-range or short-range radar devices), cameras, etc. Data from the sensors is processed and analyzed to detect objects near the vehicle, and then the data from the sensors is utilized by the driver assistance system to control the vehicle. Summary of the Invention

[0004] Disclosed is a vehicle system for a vehicle to predict a future presence of a pedestrian on a road. The vehicle system includes: at least one sensor configured to capture one or more images of a pedestrian located near the road; and a control module in communication with the at least one sensor. The control module is configured to: determine one or more characteristics associated with the pedestrian based on one or more captured images; generate a trajectory prediction for the pedestrian; overlay the trajectory prediction for the pedestrian on a road segment of the road; and generate a road crossing prediction for the pedestrian based on the one or more characteristics and the overlaid trajectory prediction. The road crossing prediction forecasts whether the pedestrian will be on the road or off the road.

[0005] In other features, one or more characteristics associated with the pedestrian include the pedestrian's crossing intention indicating whether the pedestrian intends to cross the road.

[0006] In other features, the control module is configured to: utilize a machine learning module to determine the pedestrian's crossing intention based on one or more behavioral characteristics associated with the pedestrian in one or more captured images.

[0007] In other features, one or more characteristics associated with the pedestrian include an estimation of the pedestrian's motion state indicating whether the pedestrian is moving or not moving.

[0008] Among other features, the control module is configured to: utilize a machine learning module to determine an estimated motion state of the pedestrian based on one or more captured images.

[0009] Among other features, the control module is configured to receive position data of the pedestrian to detect and locate the pedestrian, and the control module includes: a prediction model configured to generate the trajectory prediction based on the position data.

[0010] Among other features, one or more characteristics associated with the pedestrian include the estimated motion state of the pedestrian indicating whether the pedestrian is moving or not moving, and the prediction model is configured to: generate the trajectory prediction based on the estimated motion state of the pedestrian.

[0011] Among other features, the control module is configured to: generate the road crossing prediction based on whether the trajectory prediction overlaps at least a portion of the road.

[0012] Among other features, the control module is configured to: generate a road crossing prediction predicting whether the pedestrian will be on the road or off the road within a period of time.

[0013] Among other features, the vehicle system further includes: a sensor configured to detect the speed of the vehicle. The control module is configured to: generate a road crossing prediction for the pedestrian based on the speed of the vehicle.

[0014] Among other features, one or more characteristics associated with the pedestrian include: the crossing intention of the pedestrian indicating whether the pedestrian intends to cross the road; and the estimated motion state of the pedestrian indicating whether the pedestrian is moving or not moving. The control module is configured to: generate a road crossing prediction predicting that the pedestrian will be on the road only if the pedestrian intends to cross, the estimated motion state is moving, the speed of the vehicle is below a defined threshold, and the trajectory prediction overlaps at least a portion of the road.

[0015] Among other features, one or more characteristics associated with the pedestrian include: the crossing intention of the pedestrian indicating whether the pedestrian intends to cross the road; and the estimated motion state of the pedestrian indicating whether the pedestrian is moving or not moving. The control module is configured to: set confidence values for the crossing intention of the pedestrian, the estimated motion state of the pedestrian, and the trajectory prediction overlapping at least a portion of the road; and generate a road crossing prediction predicting that the pedestrian will be on the road only if the sum of the confidence values exceeds a defined threshold.

[0016] Among other features, the confidence value is a weighted value.

[0017] Among other features, the vehicle system further includes: a vehicle control module in communication with the control module. The vehicle control module is configured to: receive from the control module an indication that the generated road crosses one or more predicted signals.

[0018] Among other features, the vehicle control module is configured to: control at least one vehicle control system based on the one or more signals.

[0019] Among other features, the vehicle control system includes an autonomous braking system.

[0020] A method for predicting a future presence of a pedestrian on a road on which a vehicle is located is disclosed. The method includes: determining one or more characteristics associated with a pedestrian located near the road based on one or more images captured by at least one sensor; determining one or more characteristics associated with a pedestrian located near the road based on the one or more captured images; generating a trajectory prediction for the pedestrian; overlaying the trajectory prediction for the pedestrian on a road segment of the road; and generating a road crossing prediction for the pedestrian based on the one or more characteristics and the overlaid trajectory prediction. The road crossing prediction predicts whether the pedestrian will be on or off the road.

[0021] Among other features, the one or more characteristics associated with the pedestrian include: the crossing intention of the pedestrian, indicating whether the pedestrian intends to cross the road; and the motion state estimate of the pedestrian, indicating whether the pedestrian is moving or not moving.

[0022] Among other features, generating a road crossing prediction for the pedestrian includes: generating a road crossing prediction predicting that the pedestrian will be on the road only if the pedestrian intends to cross, the motion state estimate is moving, the speed of the vehicle is below a defined threshold, and the trajectory prediction overlaps at least a portion of the road.

[0023] Among other features, the method further includes setting confidence values for the crossing intention of the pedestrian, the motion state estimate of the pedestrian, and the trajectory prediction that overlaps at least a portion of the road, and generating a road crossing prediction for the pedestrian includes generating a road crossing prediction predicting that the pedestrian will be on the road only if the sum of the confidence values exceeds a defined threshold.

[0024] Among other features, the method further includes: generating an indication that the generated road crosses one or more predicted signals; and controlling at least one vehicle control system based on the one or more signals.

[0025] Among other features, the vehicle control system includes an autonomous braking system.

[0026] Further applicable fields of the present disclosure will become apparent from the detailed description, the claims, and the drawings. The detailed description and the specific examples are intended for illustrative purposes only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present disclosure will be more fully understood from the detailed description and the drawings, in which: Figure 1 is a block diagram of an example vehicle system for predicting a future presence of a pedestrian on a road according to the present disclosure; Figure 2 is a vehicle including a portion of a vehicle system according to the present disclosure including Figure 1 ; Figure 3 is a block diagram of an example prediction scenario according to the present disclosure, in which Figure 2 the vehicle approaches a crosswalk with a pedestrian standing nearby; and Figures 4 - 6 is a flowchart of an example control process for predicting a future presence of a pedestrian on a road according to the present disclosure.

[0028] In the drawings, reference numerals may be reused to identify similar and / or identical elements. DETAILED DESCRIPTION

[0029] Autonomous vehicles include driver assistance systems that rely on sensors for vehicle control. Sensors (such as radar devices, cameras, etc.) provide data that can be analyzed to detect the presence of objects and / or the future positions of objects (such as pedestrians). Anticipating whether a pedestrian will cross the road on which the vehicle will be operating in the near future is a key aspect of autonomous (e.g., fully autonomous and semi-autonomous) driving. However, such prediction is complex and challenging due to, for example, scene understanding, pedestrian-road interaction, pedestrian and vehicle movement, etc.

[0030] A vehicle system and method according to the present disclosure accurately predict whether a pedestrian will be on or off the road in the future. This can be achieved by combining information from multiple sources, such as estimated pedestrian characteristics, estimated pedestrian trajectories, vehicle characteristics, and road characteristics, as further explained below. Through this integration of information, the vehicle systems and methods herein can predict the future presence of a pedestrian on the road (e.g., within a short time frame). In doing so, an autonomous vehicle can react quickly based on the prediction, thereby changing the vehicle's heading (e.g., turning, braking, etc.) to avoid the pedestrian.

[0031] Now referring to Figure 1 , a block diagram of an example vehicle system 100 is presented for a vehicle to predict the future presence of a pedestrian on the road. For example, and as further explained below, the vehicle system 100 accurately predicts whether a person will be on the road within a future time interval (e.g., within the next 3 seconds, between 0.5 seconds and 2 seconds, between 1 second and 3 seconds, etc.).

[0032] As Figure 1 shown, the vehicle system 100 generally includes a control module 102, a vehicle control module 104, one or more vehicle control systems 106, and various sensors. The control module 102 generally includes a crossing intent module 108, a motion state estimation module 110, a trajectory prediction module 112, a trajectory coverage module 114, and a road crossing prediction module 116. Although Figure 1 the vehicle system 100 is illustrated as including multiple separate modules, it should be appreciated that any combination of modules (e.g., control module 102, vehicle control module 104, modules within control module 102, etc.) and / or their functions can be integrated into one or more modules.

[0033] Figure 1 The sensors can include one or more devices for capturing images of the environment around the vehicle and / or detecting or sensing vehicle parameters. For example, in Figure 1 the example of Figure 1 , the sensors can include: one or more cameras 118 that capture a single frame (e.g., a single image) of the environment around the vehicle and / or multiple frames over time (e.g., video). Additionally, the sensors can include a speed sensor 120 for detecting the speed of the vehicle. In such an example, the speed sensor 120 can be an inertial measurement unit (IMU), a wheel speed sensor (WSS), a vehicle speed sensor (VSS), or another suitable sensor for generally detecting the speed of the vehicle. As Figure 1The sensors communicate with the control module 102. For example, and as further explained below, one or more cameras 118 communicate through the intention module 108 and the motion state estimation module 110, and the speed sensor 120 communicates with the road crossing prediction module 116.

[0034] Although not shown in Figure 1 , the modules and sensors of the vehicle system 100 can share parameters via a network (such as a Controller Area Network (CAN)). In such an example, parameters can be shared via one or more data buses of the network. Thus, various parameters can be made available to other modules and / or sensors by a given module and / or sensor via the network.

[0035] In various embodiments, Figure 1 the vehicle system 100 can be employed in any suitable vehicle (such as an electric vehicle (e.g., a pure electric vehicle, a plug-in hybrid electric vehicle, etc.), an internal combustion engine vehicle, etc.). Additionally, the vehicle system 100 can be applicable to autonomous vehicles, including fully autonomous vehicles and semi-autonomous vehicles. For example, Figure 2 depicts a vehicle 200, including Figure 1 the control module 102 and the vehicle control module 104 and one or more sensors 204 (e.g., Figure 1 one or more of the one or more cameras 118 of Figure 1 the speed sensor 120 of

[0036] Continuing to refer to Figure 1 , the control module 102 can determine one or more characteristics associated with a pedestrian located near or adjacent to a road. Such characteristics can be based on one or more captured images from one or more cameras 118. In such an example, each camera 118 can capture one or more images of a pedestrian located near a road on which the vehicle is located (e.g., stopped on it, parked, traveling, etc.) or turning onto. The one or more captured images can be a single image or a video of any suitable duration (e.g., 0.2 seconds, 0.3 seconds, 0.4 seconds, 0.5 seconds, 0.6 seconds, 0.7 seconds, etc.).

[0037] For example, Figure 3 depicts a block diagram of an example prediction scenario 300, in which Figure 2 the vehicle 200 of Figure 3As shown. At least one camera on the vehicle 200 can capture one or more images of the pedestrian 350 located near the roads 330, 335. For example, the camera can have a field of view 340 represented by a line with a dashed line configuration. In Figure 3 the example, the pedestrian 350 can start moving across the road 330 (e.g., on or outside the crosswalk 370) or across the road 335 (e.g., on or outside the crosswalk 380) in the future.

[0038] In Figure 1 the example, one or more characteristics determined by the control module 102 can be any suitable prediction and / or estimation of pedestrian attributes. For example, the control module 102 can determine the crossing intention of the pedestrian indicating whether the pedestrian intends to cross the road (e.g., road 330, road 335, etc.). In such an example, the crossing intention module 108 of the control module 102 receives the captured image or its data representative from at least one of the cameras 118, generates the crossing intention of the pedestrian, and then outputs the crossing intention to the road crossing prediction module 116.

[0039] In such an example, the crossing intention represents what the pedestrian wants to do in the near future. For example, the crossing intention module 108 can determine whether the intention of the pedestrian is to cross the road or not to cross the road. In such an example, if the determined intention is to cross, the crossing intention module 108 can output an affirmative signal (e.g., "1" or another suitable indicator) to the road crossing prediction module 116, and if the determined intention is not to cross, the crossing intention module 108 can output a negative signal (e.g., "0" or another suitable indicator) to the road crossing prediction module 116.

[0040] In various embodiments, the crossing intention module 108 can determine the crossing intention of the pedestrian based on one or more behavioral characteristics associated with the pedestrian in the captured image. For example, the crossing intention module 108 can include a machine learning module (e.g., a neural network or another suitable machine learning module) that analyzes the behavioral characteristics (from the image) of the pedestrian and then determines the crossing intention based on such characteristics, or communicates with the machine learning module. In such an example, the machine learning module can be trained to discriminate between crossing and non-crossing intentions. For example, a labeled dataset can be generated based on one or more analysts examining the input image (e.g., a 0.5-second input video clip, etc.). In such an example, the analyst can note the characteristics / features of the pedestrian and their relationship to the road and the vehicle, such as the distance between the pedestrian and the road, whether the pedestrian's head is up or down, where the pedestrian is looking (e.g., towards the road, towards the vehicle, away from the road, etc.), whether the pedestrian's eyes are closed or open, etc.

[0041] In addition, by Figure 1One or more characteristics determined by the control module 102 may be the estimated motion state of a pedestrian. For example, the motion state estimation module 110 of the control module 102 receives (one or more) captured images or data representatives thereof from at least one of the cameras 118, generates the estimated motion state for the pedestrian, and then outputs the motion state to the road crossing prediction module 116. In such an example, the (one or more) captured images or data representatives used by the motion state estimation module 110 may be the same or different (one or more) images / data as those used by the crossing intention module 108.

[0042] In Figure 1 the example, the motion state estimation represents the current state of the pedestrian. For example, the motion state estimation may fall into one of any two suitable categories. For example, the motion state estimation may indicate whether the pedestrian is moving or not moving. In other examples, the motion state estimation may represent whether the pedestrian is walking (or running, etc.) and stationary (e.g., standing, sitting, etc.). Regardless of the category, if the current pedestrian state falls into one category (e.g., moving), the motion state estimation module 110 may output an affirmative signal (e.g., "1" or another suitable indicator) to the road crossing prediction module 116, and if the current pedestrian state falls into another category (e.g., not moving), the motion state estimation module 110 may output a negative signal (e.g., "0" or another suitable indicator) to the road crossing prediction module 116.

[0043] In various embodiments, the motion state estimation module 110 may utilize a machine learning module that analyzes the behavioral characteristics of the pedestrian from the received images to determine the pedestrian motion state. For example, the motion state estimation module 110 may include a machine learning module (e.g., a neural network or another suitable machine learning module) trained to classify the current pedestrian state or communicate with the machine learning module. In such an example, the machine learning module for the motion state estimation module 110 may be trained according to a labeled dataset generated based on one or more analysts' examination of the input images (e.g., a 0.5-second input video clip, etc.). In such an example, the input images for training and / or motion state determination may be cropped around the location of the behavior to reduce processing requirements.

[0044] Continuing to refer to Figure 1 , the control module 102 may generate a trajectory prediction for the pedestrian. In such an example, the trajectory prediction may be determined by the trajectory prediction module 112 based on the pedestrian's position data. For example, in Figure 1In [the figure], the trajectory prediction module 112 receives a signal 122 indicating the position of a pedestrian. In some examples, the signal 122 may include object detection data for identifying and locating the pedestrian in a 3D or 2D environment based on various characteristics of the pedestrian, such as shape, position, orientation, etc. In other words, the trajectory prediction module 112 may receive 3D or 2D object detection data. In such an example, 3D or 2D object detection may be accomplished by conventional techniques and based on inputs from one or more sensors on the vehicle (e.g., one or more cameras 118, radar sensors, lidar sensors, etc.).

[0045] In various embodiments, the trajectory prediction module 112 may use a prediction model that receives pedestrian position data (e.g., 3D or 2D object detection data) as input to generate a trajectory prediction. For example, the trajectory prediction module 112 may include or communicate with a Kalman filter or another suitable prediction model that predicts the next position of the pedestrian based on the received position data of the pedestrian (e.g., previous coordinates). Once generated, the trajectory prediction or its data representation may be output to the trajectory coverage module 114, as Figure 1 shown in [the figure].

[0046] In some examples, the trajectory prediction module 112 may optionally generate a trajectory prediction based on other inputs. For example, and as Figure 1 shown in [the figure], the trajectory prediction module 112 receives an input signal 124 from the motion state estimation module 110. In such an example, the input signal 124 may be a motion state estimate (e.g., an affirmative signal or a negative signal) output from the motion state estimation module 110, as explained above. The trajectory prediction module 112 may then generate a trajectory prediction based on the motion state estimate of the pedestrian. For example, the input signal 124 (e.g., the motion state estimate) may be employed as a control signal for the trajectory prediction module 112. In such an example, the trajectory prediction module 112 may generate a trajectory prediction only if the input signal 124 is affirmative (e.g., the pedestrian is moving). In other examples, the trajectory prediction module 112 may factor in the motion state estimate when generating a trajectory prediction. For example, if the motion state of the pedestrian is moving, as opposed to not moving, the generated trajectory prediction may be different.

[0047] Figure 1 The control module 102 of [the figure] may overlay the generated trajectory prediction for the pedestrian on a road segment of the road. For example, and as Figure 1 shown in [the figure], the trajectory coverage module 114 receives the trajectory prediction from the trajectory prediction module 112 and an input signal 126 indicating the road segment of the road on which the vehicle is located or turning onto. In such an example, the road segment may be an image or a map.

[0048] In various embodiments, the coverage of the trajectory prediction can be implemented using a machine learning module. In such an example, the trajectory coverage module 114 can include, or communicate with, a machine learning module (e.g., a neural network or another suitable machine learning module) that is designed and trained to segment road pixels on a received image or map that already includes road segments. For example, segment the received image (e.g., a satellite image) with respect to the road. In such an example, each pixel of the image has coordinates and can be classified as part of the road or not based on the coordinates. Then, the trajectory coverage module 114 overlays or superimposes the trajectory prediction (e.g., the determined area, arrow, etc.) on the segmented image. In such an example, the pixels of the trajectory prediction are placed on the pixels of the segmented image.

[0049] Then, the trajectory coverage module 114 determines whether the trajectory prediction overlaps with the road. For example, after the trajectory prediction module 112 is overlaid on the segmented image (or map), the trajectory coverage module 114 can determine whether any of the pixels of the trajectory prediction overlap with the pixels classified as part of the road, and then provide an output indicating that determination to the road crossing prediction module 116. For example, if any part of the trajectory prediction pixel overlaps with the road pixel, the trajectory coverage module 114 can output an affirmative signal (e.g., "1" or another suitable indicator) to the road crossing prediction module 116 to indicate that the pedestrian trajectory is on the road. However, if no part of the trajectory prediction pixel overlaps with the road pixel, the trajectory coverage module 114 can output a negative signal (e.g., "0" or another suitable indicator) to the road crossing prediction module 116 to indicate that the pedestrian trajectory is not on the road. In other examples, if a defined quantity (e.g., a threshold) of road pixels overlaps with the trajectory prediction pixels, the trajectory coverage module 114 can output an affirmative signal, and if not, the trajectory coverage module 114 can output a negative signal.

[0050] Continue to refer to Figure 1, the control module 102 can then generate a road crossing prediction for the pedestrian predicting whether the pedestrian will be on or off the road. For example, the road crossing prediction module 116 can generate a road crossing prediction indicating whether the pedestrian will be on or off the road. In various embodiments, the road crossing prediction module 116 can generate a road crossing prediction predicting whether the pedestrian will be on or off the road over a period of time. In such an example, the road crossing prediction can indicate when the pedestrian will be on or off the road. For example, the road crossing prediction module 116 can determine that the pedestrian will be on the road at time X, between times Y and Z, and / or at another future time interval. For example, the road crossing prediction can indicate that the pedestrian will be on the road within 3 seconds. In other examples, the road crossing prediction can indicate that the pedestrian will be on the road at time t+T, where t is the current time (e.g., 0 seconds) or a later time (e.g., 0.5 seconds), and T is some time after t (e.g., 1.5 seconds, 2 seconds, 3 seconds, 3.5 seconds, etc.).

[0051] In various embodiments, the road crossing prediction module 116 can rely on multiple inputs to generate the road crossing prediction. In such an example, the road crossing prediction module 116 can utilize a comprehensive view of the scene / environment to understand the pedestrian-road interaction and obtain knowledge related to both pedestrian and vehicle movement. For example, the road crossing prediction module 116 can generate the road crossing prediction based at least on the characteristics associated with the pedestrian and the predicted covered trajectory. In such an example, the road crossing prediction can be generated based on whether the predicted trajectory overlaps at least part of the road (e.g., as indicated by the trajectory coverage module 114).

[0052] Additionally, in some embodiments, the road crossing prediction module 116 can consider the speed of the vehicle when generating the road crossing prediction. For example, the road crossing prediction module 116 can receive the speed of the vehicle from the speed sensor 120, as Figure 1 shown, and then generate the road crossing prediction based on the speed. For example, if the vehicle is traveling at a high rate (e.g., speed greater than a threshold), the probability that the pedestrian will cross the road may be low. However, if the vehicle is traveling at a low rate (e.g., speed less than a threshold), the probability that the pedestrian will cross the road can be higher.

[0053] The control module 102 may generate a road crossing prediction predicting that a pedestrian will be on a road based on different conditions. For example, the road crossing prediction module 116 of the control module 102 may generate a road crossing prediction predicting that a pedestrian will be on a road (and in some cases, at a possible time period) only if the pedestrian intends to cross, the motion state estimation is moving, the speed of the vehicle is below a defined threshold, and the trajectory prediction overlaps at least a portion of the road. In other words, it may be necessary that each input provided to the road crossing prediction module 116 satisfies a certain condition. For example, it may be required that the output from the crossing intention module 108 is affirmative (e.g., the intention of the pedestrian is to cross), it may be required that the output from the motion state estimation module 110 is affirmative (e.g., the pedestrian is moving), the speed of the vehicle is less than a defined threshold (e.g., 6 m / s, 5.5 m / s, 5 m / s, 4.5 m / s, 4 m / s, etc.), and the output from the trajectory coverage module 114 indicates that the pedestrian trajectory is on the road (e.g., the pedestrian trajectory overlaps the road).

[0054] In other examples, if most of the inputs satisfy certain conditions, the road crossing prediction module 116 may generate a road crossing prediction predicting that a pedestrian will be on a road. In this case, if any three of the inputs satisfy certain conditions as explained above, a road crossing prediction predicting that a pedestrian will be on a road may be generated. In some examples, it may be required that specific inputs among the inputs satisfy the conditions explained above. For example, if three of the inputs that satisfy certain conditions include a moving pedestrian motion state and the pedestrian trajectory is on the road, the road crossing prediction module 116 may generate a road crossing prediction predicting that a pedestrian will be on a road.

[0055] In still other examples, the road crossing prediction module 116 may generate a road crossing prediction predicting that a pedestrian will be on a road based on a scoring system. For example, the road crossing prediction module 116 may set or receive confidence values for the crossing intention of the pedestrian (from the road crossing prediction module 116), the motion state estimation of the pedestrian (from the motion state estimation module 110), the trajectory prediction that overlaps at least a portion of the road (from the trajectory coverage module 114), and / or the speed of the vehicle. Then, the road crossing prediction module 116 may generate a road crossing prediction only if the sum of the confidence values exceeds a defined threshold. In some examples, the confidence values may be weighted values, if desired. In such an example, each confidence value associated with a specific input may be weighted based on its importance relative to other inputs.

[0056] In various embodiments, Figure 1The control module 102 can generate a signal indicating that the generated road crosses the predicted signal and transmit the signal to the vehicle control module 104. In such an example, the control module 102 can generate a signal for indicating that a pedestrian will be on the road and, in some cases, on the road during the possible time periods as explained above. Once received, the vehicle control module 104 can generate one or more control signals for controlling the vehicle control system(s) 106 based on the road crossing prediction.

[0057] For example, the vehicle control module 104 can use the road crossing prediction signal to control Figures 2 - 3 the driver assistance system in the vehicle 200. In such an example, the driver assistance system can include, for example, an autonomous braking system (such as autonomous emergency braking (AEB), etc.), assisted steering (such as assisted evasive steering, etc.), and / or any other suitable driver assistance system that can be employed to decelerate the vehicle, change the direction of the vehicle, etc. In other examples, the vehicle control system(s) 106 can include a pedestrian and / or driver notification system. In such an example, the vehicle control module 104 can activate an alarm (such as an audible alarm, a visual alarm, etc.) inside and / or outside the vehicle to attempt to warn pedestrians and / or vehicle drivers.

[0058] Figures 4 - 6 Illustrated are example control processes 400, 500, 600 for predicting the future presence of pedestrians on a road. In Figures 4 - 6 the example, the control processes 400, 500, 600 can be implemented using the Figures 2 - 3 vehicle system 100 that can be employed in the Figure 1 vehicle 200. Although the example control processes 400, 500, 600 are described with respect to the vehicle system 100 including the control module 102, Figure 1 any one of the control processes 400, 500, 600 can be employed by any suitable vehicle system. Additionally, although the steps of the control processes 400, 500, 600 are shown and described in a particular order below, it should be appreciated that the steps of the control processes 400, 500, 600 can be implemented in another suitable order when desired. For example, Figure 5 the decision steps can be implemented in an order different from the order shown.

[0059] As Figure 4As shown in FIG. 0, the control process 400 begins at 402, where the control module 102 receives one or more images. For example, and as explained above, the control module 102 receives (and more specifically, through the intention module 108 and the motion state estimation module 110) images and / or videos (e.g., multiple frames or images) captured by one or more cameras on the vehicle 200. Control then continues to 404.

[0060] At 404, the control module 102 determines one or more characteristics associated with the pedestrian based on the received images. In various embodiments, the pedestrian characteristics may include, for example, the crossing intention of the pedestrian (e.g., the intention of the pedestrian is to cross the road or not to cross the road) and the motion state estimation of the pedestrian (e.g., moving or not moving), as explained above. In such an example, Figure 1 the crossing intention module 108 and the motion state estimation module 110 of FIG. 0 may respectively use separate machine learning modules (e.g., neural networks, etc.) to generate the crossing intention and the motion state estimation, as explained herein. Control then continues to 406.

[0061] At 406, the control module 102 generates a trajectory prediction for the pedestrian. For example, and as explained above, the trajectory prediction module 112 of the control module 102 may generate a trajectory prediction based on pedestrian position data (e.g., 3D or 2D object detection data). In some examples, the trajectory prediction may be generated using a Kalman filter or another suitable prediction model. Control then continues to 408.

[0062] At 408, the control module 102 overlays the generated trajectory prediction on a road segment of the road. In various embodiments, and as explained above, the overlay of the trajectory prediction may be implemented using a machine learning module (e.g., neural network, etc.) trained for segmenting road pixels on an image or map that already contains the road segment. For example, and as explained above, each pixel of the segmented image or map has coordinates and can be classified as part of the road or not based on the coordinates, and the pixels of the trajectory prediction can be overlaid on the pixels of the segmented image or map based on the coordinates. Control then continues to 410.

[0063] At 410, the control module 102 generates a road crossing prediction for the pedestrian predicting whether the pedestrian will be on or off the road. In some examples, the road crossing prediction may be generated by Figure 1 the road crossing prediction module 116 of FIG. 0 based at least on the determined pedestrian characteristics and the overlaid trajectory prediction, as explained herein. In other examples, additional inputs (e.g., the speed of the vehicle 200) may be relied upon when generating the road crossing prediction. Control then continues to 412.

[0064] At 412, the control module 102 determines whether the road crossing prediction indicates or forecasts that a pedestrian will be on the road at a given time, such as at time X, between time Y and time Z, and / or at another future time interval. If not, the control returns to 402, as Figure 4 shown. If so, then the control continues to 414.

[0065] At 414, the control module 102 generates a signal indicating the generated road crossing prediction. In some examples, the control module 102 (e.g., the road crossing prediction module 116) may transmit the signal to a vehicle control module (e.g., Figure 1 the vehicle control module 104). The control then continues to 416, where a vehicle action is initiated based on the signal. For example, the vehicle control module or the control module 102 may generate control signals for controlling one or more vehicle control systems based on the generated road crossing prediction, as explained above. The control can then end as Figure 4 shown or return to 402 when desired.

[0066] In Figure 5 , the control process 500 starts at 502, where the control module 102 receives the speed of the vehicle 200. For example, the road crossing prediction module 116 of the control module 102 may receive the speed (or a data representative thereof) from the Figure 1 speed sensor 120 or another suitable sensor associated with the vehicle 200. The control then continues to 404.

[0067] At 504, the control module 102 determines whether the speed of the vehicle 200 is below a defined threshold. For example, and as explained above, the road crossing prediction module 116 may compare the speed with any suitable threshold (such as 6 m / s, 5.5 m / s, 5 m / s, 4.5 m / s, 4 m / s, etc.). If the speed is not less than the threshold at 504, the control returns to 502. However, if the speed is less than the threshold at 504, the control continues to 506.

[0068] At 506, the control module 102 determines whether the crossing intent indicates that a pedestrian intends to cross the road. In such an example, the road crossing prediction module 116 may receive a signal indicating whether the pedestrian intends to cross or not from the crossing intent module 108. For example, and as explained above, the crossing intent module 108 may employ a machine learning module that analyzes the behavioral characteristics of the pedestrian and then determines the crossing intent based on such characteristics. If the answer is no at 506, the control returns to 502. If the answer is yes at 506, the control continues to 508.

[0069] At 508, the control module 102 determines whether the motion state of the pedestrian is moving. For example, the road crossing prediction module 116 may receive a signal indicating the current motion state of the pedestrian from the motion state estimation module 110. In such an example, the motion state estimation module 110 may employ a machine learning module that analyzes the behavioral characteristics of the pedestrian and then determines the motion state based on the analysis, as explained above. The current motion state of the pedestrian as determined by the motion state estimation module 110 may be moving (e.g., walking, running, etc.) or not moving (e.g., stationary, etc.). If the answer at 508 is no, the control returns to 502. If the answer at 508 is yes, the control continues to 510.

[0070] At 510, the control module 102 determines whether the trajectory prediction for the pedestrian overlaps onto the road. For example, and as explained above, the trajectory prediction module 112 of the control module 102 may generate a trajectory prediction for the pedestrian based on, for example, 3D or 2D object detection data of the pedestrian. Then, the trajectory overlay module 114 of the control module 102 may overlay the trajectory prediction of the pedestrian onto the road segments of the road, as explained above. In such an example, the trajectory overlay module 114 may overlay the trajectory prediction (e.g., the determined area, arrow, etc.) onto the segmented image and then determine whether any pixels of the trajectory prediction are placed on the pixels for the road in the segmented image. If so, the trajectory prediction overlaps onto the road. If the answer at 510 is no, the control returns to 502. If the answer at 510 is yes, the control continues to 512.

[0071] At 512, the control module 102 generates a road crossing prediction for the pedestrian predicting whether the pedestrian will be on the road or off the road. In some examples, the road crossing prediction may be generated by Figure 1 the road crossing prediction module 116 and includes the predicted time frame in which the pedestrian is predicted to be on the road, as explained herein. For example, the road crossing prediction module 116 may predict that the pedestrian will be on the road at a given time, such as at time X, between time Y and time Z, and / or at another future time interval.

[0072] Control then proceeds to 514, where control module 102 determines whether vehicle action is required. This determination can be made based on road crossing predictions and other suitable inputs (such as the speed of the vehicle, the trajectory of the vehicle, etc.). For example, control module 102 may determine that vehicle 200 will be away from the road section where pedestrians are predicted to cross at a given time. In such an example, control module 102 may determine that no vehicle action is required. In other examples, control module 102 may determine that vehicle 200 will be on the road section where pedestrians are predicted to cross at a given time. If so, control module 102 may determine that vehicle action is required. If the answer is no at 514, control returns to 502. If the answer is yes at 514, control then proceeds to 414, 416, as explained above with respect to Figure 4 as explained. Control may then end as shown in Figure 5 or return to 502 when desired.

[0073] In various embodiments, Figure 5 the decision steps 504, 506, 508, 510 of Figure 5 may function in different ways. For example, instead of returning to Figure 5 step 502, if the result is no, each decision step 504, 506, 508, 510 may proceed to the next step. Then, road crossing prediction module 116 may generate a road crossing prediction at 512 predicting the following: if the majority of decision steps 504, 506, 508, 510 or a specific decision step results in yes, then pedestrians will be on the road.

[0074] In Figure 6 control process 600 begins as 602, where control module 102 sets values for various generated parameters. Specifically, control module 102 sets a confidence value for the crossing intention of the pedestrian (from road crossing prediction module 116), a confidence value for the motion state estimate of the pedestrian (from motion state estimation module 110), a confidence value for the trajectory prediction that at least partially overlaps with the road (from trajectory coverage module 114), and / or a confidence value for the speed or rate of the vehicle. In such an example, Figure 1 road crossing prediction module 116 of Figure 1 may set confidence values and / or receive values from road crossing prediction module 116, motion state estimation module 110, trajectory coverage module 114, and speed sensor 120. Additionally, in some examples, each associated confidence value may be weighted based on its importance, as explained above. Control then proceeds to 604.

[0075] At 604, control module 102 combines the confidence values into a total value. This may be done by Figure 1The road crossing prediction module 116 or another suitable module in implements this. Control then proceeds to 606, where the control module 102 determines whether the total value (e.g., the combined confidence value) is greater than a defined threshold. For example, and as explained above, the road crossing prediction module 116 can compare the combined confidence value with any suitable threshold. If the answer is no at 606, the control can return to 602, as Figure 6 shown in. In some examples, before returning to 602, the road crossing prediction module 116 can generate a road crossing prediction that the predicted pedestrian will not be on the road. If the answer is yes at 606, the control proceeds to 512, 514, as explained above with respect to Figure 5 and then proceeds to 414, 416, as explained above with respect to Figure 4 and then the control can end as shown in Figure 6 or return to 602 when desired.

[0076] The systems and methods described herein accurately predict whether a pedestrian will be on or off the road in the future based on information from multiple sources. For example, in various embodiments, the systems and methods can generate a road crossing prediction for a pedestrian that predicts whether the pedestrian will be on or off the road (e.g., will cross or not cross the road) in 0.5 seconds to 2 seconds. In such an example, the road crossing accuracy can be 80.3% or higher, the non-crossing accuracy can be 92% or higher, and the balanced accuracy for both road crossing and non-crossing can be 86.6% or higher.

[0077] The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or its use. The broad teachings of the disclosure can be implemented in a variety of forms. Thus, although the disclosure includes specific examples, the true scope of the disclosure should not be so limited because other modifications will become apparent upon a study of the drawings, the specification, and the appended claims. It should be understood that one or more steps within a method can be performed in a different order (or simultaneously) without changing the principles of the disclosure. Further, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of the disclosure can be implemented in and / or combined with the features of any one of the other embodiments, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the arrangement of one or more of the embodiments with respect to each other remains within the scope of the disclosure.

[0078] A variety of terms are used to describe the spatial and functional relationships between components (e.g., between modules, circuit elements, semiconductor layers, etc.), including "connected," "joined," "coupled," "adjacent," "immediately adjacent to," "on top of," "above," "below," and "disposed." Unless explicitly described as "direct," when the relationship between a first and a second component is described in the foregoing disclosure, the relationship can be a direct relationship in which no other intervening component exists between the first and second components, but can also be an indirect relationship in which one or more intervening components (spatially or functionally) exist between the first and second components. As used herein, the phrase "at least one of A, B, and C" should be understood to mean the logical (A or B or C) using non-exclusive logic "or," and should not be understood to mean "at least one of A, at least one of B, and at least one of C."

[0079] In the drawings, as indicated by the arrows, the direction of the arrows generally depicts the flow of information (such as data or instructions) of interest to the illustration. For example, when components A and B exchange various information but the information transmitted from component A to component B is relevant to the illustration, the arrow can point from component A to component B. This one-way arrow does not imply that no other information is transmitted from component B to component A. Further, for the information sent from component A to component B, component B can send a request for that information or an affirmative acknowledgment of the receipt of that information to component A.

[0080] In the present application, which includes the following definitions, the term "circuit" can be used in place of the term "module" or the term "controller." The term "module" can refer to, be part of, or include the following: application specific integrated circuit (ASIC); digital, analog, or mixed analog / digital discrete circuit; digital, analog, or mixed analog / digital integrated circuit; combinational logic circuit; field programmable gate array (FPGA); processor circuit (shared, dedicated, or group) that executes code; memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the foregoing, such as in a system on a chip.

[0081] A module can include one or more interface circuits. In some examples, the interface circuit can include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of the present disclosure can be distributed among multiple modules connected via the interface circuit. For example, multiple modules can allow load balancing. In a further example, a server (also referred to as remote or cloud) module can perform a certain functionality on behalf of a client module.

[0082] As used above, the term "code" can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" encompasses a single processor circuit that executes some or all of the code from multiple modules. The term "group of processor circuits" encompasses processor circuits that, in conjunction with additional processor circuits, execute some or all of the code from one or more modules. References to multiple processor circuits encompass multiple processor circuits on separate die, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or combinations of the above. The term "shared memory circuit" encompasses a single memory circuit that stores some or all of the code from multiple modules. The term "group of memory circuits" encompasses memory circuits that, in conjunction with additional memory, store some or all of the code from one or more modules.

[0083] The term "memory circuit" is a subset of the term "computer-readable medium". As used herein, the term "computer-readable medium" does not encompass transitory electrical or electromagnetic signals propagated through a medium (such as on a carrier wave); the term "computer-readable medium" can thus be considered tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital tape or a hard disk drive), and optical storage media (such as a CD, DVD, or Blu-ray disc).

[0084] The devices and methods described in this application can be implemented in part or in whole by a special-purpose computer created by configuring a general-purpose computer to execute one or more specific functions embodied in a computer program. The functional blocks, flowchart components, and other elements described above serve as software specifications, which can be translated into a computer program by routine work of a skilled technician or programmer.

[0085] A computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. A computer program can also include or rely on stored data. A computer program can encompass a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, and the like.

[0086] A computer program may include: (i) descriptive text to be parsed, such as HTML (HyperText Markup Language), XML (eXtensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated by a compiler from source code; (iv) source code for execution by an interpreter; (v) source code for compilation and execution by a just-in-time compiler; and so on. By way of example only, source code may be written using the syntax of languages including the following: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (5th revision of the HyperText Markup Language), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK, and

Claims

1. A vehicle system for predicting the future presence of pedestrians on a road, the vehicle system comprising: At least one sensor configured to capture one or more images of a pedestrian located near the road; And A control module in communication with the at least one sensor, the control module being configured to: Determine one or more characteristics associated with a pedestrian located near the road based on one or more of the captured images; Generate a trajectory prediction for the pedestrian; Overlay the trajectory prediction for the pedestrian on a road segment of the road; And Generate a road crossing prediction for the pedestrian based on the one or more characteristics and the overlaid trajectory prediction, the road crossing prediction predicting whether the pedestrian will be on or off the road.

2. The vehicle system according to claim 1, wherein one or more characteristics associated with the pedestrian include the pedestrian's crossing intention indicating whether the pedestrian intends to cross the road.

3. The vehicle system according to claim 2, wherein the control module is configured to: use a machine learning module to determine the pedestrian's crossing intention based on one or more behavioral characteristics associated with the pedestrian in one or more of the captured images.

4. The vehicle system according to claim 1, wherein: One or more characteristics associated with the pedestrian include an estimate of the pedestrian's motion state indicating whether the pedestrian is moving or not moving; And The control module is configured to: use a machine learning module to determine the estimate of the pedestrian's motion state based on one or more of the captured images.

5. The vehicle system according to claim 1, wherein: The control module is configured to receive position data of the pedestrian to detect and locate the pedestrian; And The control module includes: a prediction model configured to generate the trajectory prediction based on the position data.

6. The vehicle system according to claim 1, further comprising: A sensor configured to detect the speed of the vehicle, wherein the control module is configured to: generate a road crossing prediction for the pedestrian based on the speed of the vehicle.

7. The vehicle system according to claim 1, wherein: One or more characteristics associated with the pedestrian include: the pedestrian's crossing intention indicating whether the pedestrian intends to cross the road; and an estimate of the pedestrian's motion state indicating whether the pedestrian is moving or not moving; and The control module is configured to: generate a road crossing prediction predicting that the pedestrian will be on the road only if the pedestrian intends to cross, the estimate of the motion state is moving, the speed of the vehicle is below a defined threshold, and the trajectory prediction overlaps at least part of the road.

8. The vehicle system according to claim 1, wherein: One or more characteristics associated with the pedestrian include: the pedestrian's crossing intention indicating whether the pedestrian intends to cross the road; and an estimate of the pedestrian's motion state indicating whether the pedestrian is moving or not moving; and The control module is configured to: set confidence values for the crossing intention of the pedestrian, the motion state estimation of the pedestrian, and the trajectory prediction that overlaps at least a portion of the road; and generate a road crossing prediction that the pedestrian will be on the road only if the sum of the confidence values exceeds a defined threshold.

9. The vehicle system according to claim 1, further comprising: A vehicle control module, in communication with the control module, the vehicle control module being configured to: receive one or more signals from the control module indicating the generated road crossing prediction.

10. The vehicle system according to claim 9, wherein the vehicle control module is configured to: control at least one vehicle control system based on the one or more signals.

Images