Early-stage alarm and collision avoidance

Abstract

To provide a method of assisting in collision avoidance by giving an early-stage alarm against a dangerous situation.SOLUTION: A wireless communication device transmits an alarm against a dangerous situation at or near an intersection to one device out of terrestrial traffic entities. There are provided a processor and a storage that stores instructions that the processor can execute to perform actions including the one described below. Current action data received from a sensor and related to a terrestrial traffic entity at or near an intersection is applied to a machine learning model in order to predict an imminent behavior of the terrestrial traffic entity. An imminent dangerous situation of one or more terrestrial traffic entities at or near an intersection is predicted, and an alarm is transmitted to the wireless communication device.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] This application is entitled to the benefit of the filing dates of U.S. patent application Ser. Nos. 15 / 994,568, filed May 31, 2018, 15 / 994,826, filed May 31, 2018, 15 / 994,702, filed May 31, 2018, 15 / 994,915, filed May 31, 2018, 16 / 222,536, filed December 17, 2018, and 15 / 994,850, filed May 31, 2018, all of which claim priority to and the benefit of U.S. Provisional Patent Application No. 62 / 644,725, filed March 19, 2018, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] This description relates to early warning and collision avoidance.

[0003] Collision avoidance systems are on the rise. King et al. (US Patent Application Publication No. 2007 / 0276600(A1), 2007) described, for example, placing sensors in front of an intersection and applying physics-based decision rules to predict whether two vehicles are about to collide at the intersection based on their heading and speed.

[0004] In Aoude et al. (U.S. Pat. No. 9,129,519(B2), 2015, the entire contents of which are incorporated herein by reference), driver behavior is monitored and modeled to enable prediction and prevention of violations in traffic situations at intersections. Summary of the Invention [Problem to be solved by the invention]

[0005] Collision avoidance is the primary defense against injury and loss of life and property in ground transportation. Providing early warning of dangerous situations assists in collision avoidance. [Means for solving the problem]

[0006] In general, in one aspect, an apparatus is located at an intersection of a transportation network. The apparatus includes an input for receiving data from a sensor oriented to monitor ground traffic entities at or near the intersection. The wireless communication device transmits a warning regarding a dangerous situation at or near the intersection to a device of one of the ground traffic entities, and there is a processor and storage for instructions executable by the processor for performing operations including: A machine learning model capable of predicting a behavior of the ground traffic entity at the intersection or near the intersection at a current time is stored. The machine learning model is based on training data regarding previous operations and associated behaviors of the ground traffic entity at the intersection or near the intersection. Current operation data received from the sensor regarding the ground traffic entity at the intersection or near the intersection is applied to the machine learning model to predict an impending behavior of the ground traffic entity. An impending dangerous situation for one or more of the ground traffic entities at the intersection or near the intersection is inferred from the predicted impending behavior. The wireless communication device transmits a warning regarding the dangerous situation to a device of one of the ground traffic entities.

[0007] An embodiment may include one or a combination of two or more of the following features: The wireless communication device transmits a warning about the hazardous situation to a sign or other infrastructure presentation device. The warning includes an instruction or command capable of controlling a particular ground traffic entity. The equipment includes a roadside equipment. There is a housing for the equipment, and a sensor is mounted to the housing. The warning is transmitted by broadcasting the warning for reception by any of the ground traffic entities at or near the intersection. The machine learning model includes an artificial intelligence model. The training data and the behavior data include at least one of speed, position, or heading. The training data and the behavior data may further include intent, posture, looking direction, or interaction with other vulnerable road users, for example in a group. The processor is configured to enable generating the machine learning model in the equipment. The training data is stored in the equipment. The intersection includes an intersection without a traffic light. The intersection includes an intersection with a traffic light. The traffic network includes a road network. The ground traffic entity includes a vulnerable road user. The ground traffic entity includes a vehicle. The imminent dangerous situation includes a collision or a near miss. The ground traffic entity includes a pedestrian crossing a road at a crosswalk. There is another communication device that communicates with the central server. One device of the ground traffic entity includes a mobile communication device.

[0008] In general, in one aspect, an apparatus is located at an intersection of a transportation network. The apparatus includes an input for receiving data from a sensor oriented to monitor a ground traffic entity at or near the intersection. The wireless communication device transmits a warning regarding a dangerous situation at or near the intersection to a device of one of the ground traffic entities. There is a processor and a storage for instructions executable by the processor for storing a machine learning model capable of predicting a behavior of the ground traffic entity at or near the intersection at a current time. The machine learning model is based on training data regarding previous operations and associated behaviors of the ground traffic entity at or near the intersection. Current operation data received from the sensor regarding the ground traffic entity at or near the intersection is applied to the machine learning model to predict an impending behavior of the ground traffic entity, including a ground traffic entity for which an associated device (a device of the ground traffic entity) cannot receive a warning from a wireless communication device. An impending dangerous situation is inferred for a ground traffic entity for which an associated device (a device of the ground traffic entity) may receive a warning from a wireless communication device. The impending hazardous situation is a result of a predicted impending behavior of a ground transportation entity that is unable to receive the warning. A warning about the hazardous situation is transmitted to a device of the ground transportation entity that can receive the warning from the wireless communication device.

[0009] An embodiment may include one or a combination of two or more of the following features: The equipment includes a roadside equipment. There is a housing for the equipment, with a sensor mounted to the housing. The alert is sent by broadcasting an alert for reception by any of the ground traffic entities at or near the intersection that may receive the alert. The machine learning model includes an artificial intelligence model. The intersection includes an intersection without a traffic light. The intersection includes an intersection with a traffic light. The traffic network includes a road network. The ground traffic entity includes a road user that may be harmed. The ground traffic entity includes a vehicle. The imminent hazard situation includes a collision. The ground traffic entity whose associated device cannot receive an alert from the wireless communication device includes a vehicle. The ground traffic entity whose associated device may receive an alert from the wireless communication device includes a pedestrian crossing a road at a crosswalk. There is another communication device in communication with a central server. A device of one of the ground traffic entities includes a mobile communication device.

[0010] Generally, in one aspect, messages and data are received at a road vehicle traveling within a ground transportation network, including messages from external sources regarding the location, operation, and status of other ground transportation entities, data from on-board sensors regarding road and driving conditions and regarding the location of static objects and moving ground transportation entities in the vehicle's vicinity, data regarding the quality of driving by the driver of the road vehicle, and basic safety messages from other ground transportation entities and personal safety messages from potential road users. The received data and messages are fused and applied to an artificial intelligence model to predict the behavior of the driver of the road vehicle or of the potential road users and / or collision risk to the road vehicle.

[0011] Implementations may include one or a combination of two or more of the following features: The road vehicle generates a map of static objects and moving ground traffic entities in the vicinity of the road vehicle. The driver of the road vehicle is alerted to a collision risk. The collision risk is identified based on probabilities of predicted trajectories of other nearby moving ground traffic entities. Basic safety messages and personal safety messages are filtered to reduce the number of alerts provided to the driver of the road vehicle.

[0012] In general, in one aspect, electronic sensors located near a crosswalk across a roadway are used to monitor an area within or near the crosswalk. The electronic sensors generate motion data related to vulnerable roadway users within or near the crosswalk. The generated motion data is applied to a machine learning model operating on a device located near the crosswalk to predict when one of the vulnerable roadway users is about to enter the crosswalk. Before the vulnerable roadway user enters the crosswalk, a warning is wirelessly transmitted to at least one of a device associated with the vulnerable roadway user or a device associated with another ground transportation entity approaching the crosswalk on the roadway.

[0013] An embodiment may include one or a combination of two or more of the following features: The equipment includes a roadside equipment. The potentially vulnerable roadway users include pedestrians, animals, or cyclists. The potentially vulnerable roadway users associated devices include a smart watch or other wearable device, a smartphone, or another mobile device. The other ground transportation entities include motorized vehicles. The other ground transportation entities associated devices include a smartphone or another mobile device. The machine learning model is provided to the equipment located near the crosswalk by a remote server over the Internet. The machine learning model is generated in the equipment located near the crosswalk. The machine learning model is trained using motion data generated by a sensor located near the crosswalk. The motion data generated by the sensor located near the crosswalk is transmitted to the server for use in training the machine learning model. The motion data generated by the sensor located near the crosswalk is segmented based on a corresponding zone in the vicinity of the crosswalk. The electronic sensors are used to generate motion-related data representative of physical properties of the potentially vulnerable road users. Trajectory information regarding the potentially vulnerable road users is derived from the motion data generated by the sensors.

[0014] In general, in one aspect, electronic sensors located near intersections are used to monitor intersections and approaches to the intersections of a ground transportation network. The electronic sensors generate motion data regarding ground transportation entities moving on the approaches or moving at the intersection. One or more of the ground transportation entities are unable to transmit basic safety messages to other ground transportation entities in the vicinity of the intersection. Based on the motion data generated by the electronic sensors, virtual basic safety messages are transmitted to one or more of the ground transportation entities capable of receiving the messages. The virtual basic safety messages incorporate information regarding one or more of the ground transportation entities unable to transmit the basic safety messages. The incorporated information in each of the virtual basic safety messages includes at least one of a position, a heading, a speed, and a predicted future trajectory of one of the ground transportation entities unable to transmit the basic safety message.

[0015] Implementations may include one or a combination of two or more of the following features: The equipment includes a roadway equipment. The incorporated information includes a subset of information that would be incorporated into a basic safety message generated by a ground transportation entity if the ground transportation entity were capable of transmitting the basic safety message. The generated operational data is applied to a machine learning model operating on the equipment located near the intersection to predict a trajectory of a ground transportation entity that is unable to transmit the basic safety message. One of the ground transportation entities includes a motorized vehicle. The machine learning model is provided to the equipment located near the intersection by a remote server over the Internet. The machine learning model is generated on the equipment located near the intersection. The machine learning model is trained using operational data generated by a sensor located near the intersection. The operational data generated by the sensor located near the intersection is transmitted to the server for use in training the machine learning model.

[0016] In general, in one aspect, electronic sensors located near the intersections are used to monitor intersections and approaches to the intersections of a ground transportation network. The electronic sensors generate motion data for ground transportation entities moving at the approaches or at the intersections. Separate virtual zones are defined at the intersections and the approaches to the intersections. The motion data is segmented according to the corresponding virtual zones to which the generated motion data relates. The generated motion data is applied to a machine learning model operating on equipment located near the intersections to predict, for each respective segment, an imminent hazardous situation at the intersection or one of the approaches involving one or more of the ground transportation entities. An alert is wirelessly transmitted to a device associated with at least one of the involved ground transportation entities before the imminent hazardous situation becomes an actual hazardous situation.

[0017] An embodiment may include one or a combination of two or more of the following features: The equipment includes a roadside equipment. The device associated with each of the ground traffic entities includes a wearable device, a smartphone, or another mobile device. One of the ground traffic entities includes a motorized vehicle. The machine learning model is provided to the equipment located near the intersection by a remote server over the Internet. The machine learning model is generated at the equipment located near the intersection. The machine learning model is trained using motion data generated by a sensor located near the intersection. The motion data generated by the sensor located near the intersection is transmitted to the server for use in training the machine learning model. Electronic sensors are used to monitor an area in or near a crosswalk crossing one of the approaches to the intersection. The electronic sensors are used to generate motion-related data representative of physical properties of vulnerable road users in the vicinity of the crosswalk. Trajectory information regarding the vulnerable road users is derived from the motion data generated by the sensor. There is a machine learning model for each of the approaches to the intersection. A decision is made whether to send an alert further based on motion data generated by a sensor associated with another nearby intersection. A determination is made whether to send a warning further based on information received from a ground traffic entity moving on the approach road or a ground traffic entity moving on the intersection. The intersection is signalized and information regarding a state of the signal is received. The intersection is not signalized and is controlled by one or more signs. The defined virtual zone includes one or more approach roads controlled by signs. The signs include a stop sign or a yield sign. One of the ground traffic entities includes a rail vehicle.

[0018] In general, in one aspect, an apparatus is located in or on a ground transportation entity. The apparatus includes an input for receiving data from a sensor in or on the ground transportation entity oriented to monitor nearby features of the ground transportation network and other information related to a context in which the ground transportation entity is traversing the ground transportation network. A wireless communication device receives the information related to the context. A signal processor applies signal processing to the data from the sensor and other information related to the context. There is a processor and storage for instructions executable by the processor to perform operations including storing a machine learning model capable of predicting the behavior of an operator of the ground transportation entity and the intent and movement of other ground transportation entities in the vicinity, and applying the currently received data from the sensor and other information related to the context to predict the behavior of the operator and the intent and movement of other ground transportation entities in the vicinity.

[0019] An embodiment may include one or a combination of two or more of the following features: The equipment includes roadside equipment. The instructions are executable by the processor to monitor users or occupants of the ground transportation entity. The other information related to the context includes emergency broadcasts, roadside equipment traffic and safety messages, and messages related to safety, location, and other operational information from other ground transportation entities. The sensors include cameras, distance sensors, vibration sensors, microphones, seat sensors, hydrocarbon sensors, volatile organic compounds and other toxic substances sensors, and kinematic sensors, or combinations thereof. The instructions are executable by the processor to filter received alerts received by the vehicle by applying the alerts to machine learning models to predict which alerts are significant in the current location, environmental conditions, driver behavior, vehicle health and status, and kinematics.

[0020] In general, in one aspect, operational data is obtained for unconnected ground transportation entities moving in a transportation network, and a virtual safety message incorporating information about the operational data for the unconnected ground transportation entities is transmitted to connected ground transportation entities in proximity to the unconnected ground transportation entities.

[0021] Implementations may include one or a combination of two or more of the following features: The virtual safety message is a substitute for a safety message sent by the unconnected ground traffic entity if the unconnected ground traffic entity were connected. The unconnected ground traffic entity comprises a vehicle and the virtual safety message is a substitute for a basic safety message. The unconnected ground traffic entity comprises a vulnerable road user and the virtual safety message is a substitute for a personal safety message. The operational data is detected by infrastructure sensors.

[0022] In general, in one aspect, equipment located at an intersection of a transportation network includes an input for receiving data from sensors oriented to monitor ground traffic entities at or near the intersection. The data from each of the sensors represents at least one position or operational parameter of at least one of the ground traffic entities. The data from each of the sensors is represented in a native format. The data received from at least two of the sensors does not match with respect to the position or operational parameter, or with respect to the native format, or with respect to both. There is storage for processor-executable instructions for converting the data from each of the sensors into data having a common format independent of the native format of the sensor's data. The data having the common format is incorporated into a global integrated representation of ground traffic entities monitored at or near the intersection. The global integrated representation includes the position, speed, and heading of each of the ground traffic entities. Relationships between the positions and operations of two of the ground traffic entities are identified using the global integrated representation. A dangerous situation involving the two ground traffic entities is predicted, and a message is transmitted to the at least one ground traffic entity of the two ground traffic entities alerting the at least one ground traffic entity of the two ground traffic entities about the dangerous situation.

[0023] Implementations may include one or a combination of two or more of the following features: The sensor includes at least two of a radar, a lidar, and a camera. The data received from one of the sensors includes image data of a field of view at successive instants. The data received from one of the sensors includes reflection points in 3D space. The data received from one of the sensors includes a distance from the sensor, and a speed. The global joint representation represents the position of the ground traffic entity in a common reference frame. The two sensors from which the data is received are mounted at fixed positions at or near the intersection and have at least partially non-overlapping fields of view. One of the sensors includes a radar, and the transformation of the data includes determining a position of the ground traffic entity from a known position of the radar and a distance from the radar to the ground traffic entity. One of the sensors includes a camera, and the transformation of the data includes determining a position of the ground traffic entity from a known position, a direction of view, and a tilt of the camera, and a position of the ground traffic entity within an image frame of the camera.

[0024] In general, in one aspect, an apparatus is located at an at-grade intersection of a transportation network including road intersections, crosswalks, and railroad tracks. The apparatus includes inputs for receiving data from sensors oriented to monitor road vehicles and pedestrians at or near the at-grade intersection, and for receiving phase and timing data for signals on the roads and on the railroad tracks. A wireless communication device is included for transmitting a warning to one of a device of a ground traffic entity, a pedestrian, or a rail vehicle on the railroad tracks regarding a dangerous situation at or near the at-grade intersection. There is storage for instructions executable by a processor to store a machine learning model capable of predicting a behavior of a ground traffic entity at or near the at-grade intersection at a current time. The machine learning model is based on training data regarding previous actions and associated behaviors of road vehicles and pedestrians at or near the at-grade intersection. Current motion data received from the sensors regarding road vehicles and pedestrians at or near the at-grade intersection is applied to the machine learning model to predict impending behavior of the road vehicles and pedestrians. An imminent dangerous situation for a rail vehicle on the rail track at or near the intersection is inferred from the predicted imminent behavior, and the wireless communication device is caused to transmit a warning about the dangerous situation to at least one of a road vehicle, a pedestrian, and a rail vehicle device.

[0025] Implementations may include one or a combination of two or more of the following features: The warning is transmitted to on-board equipment of the rail vehicle. The rail track is on a segregated rail right-of-way. The rail track is not on a segregated rail right-of-way. The equipment comprises roadside equipment. The warning is transmitted by broadcasting a warning for reception by any of the ground traffic entities, pedestrians, or rail vehicles at or near the grade crossing. The impending hazardous situation comprises a collision or near miss.

[0026] In general, in one aspect, data representative of the location and behavior of road vehicles operating in a ground transportation network or walking pedestrians is received from infrastructure sensors. The data is received in virtual basic safety messages and virtual personal safety messages regarding the status of the road vehicles and pedestrians. The received data is applied to a machine learning model that is trained to identify risky driving or walking behaviors of one of the road vehicles or pedestrians. The risky driving or walking behaviors are reported to authorities.

[0027] Implementations may include one or a combination of two or more of the following features: The road vehicle is identified based on plate number recognition. The pedestrian is identified based on biometric recognition. The road vehicle or pedestrian is identified based on social networking.

[0028] These and other aspects, features, and implementations may be expressed as methods, apparatus, systems, components, program products, ways of doing business, means or steps for performing a function, and other manners.

[0029] These and other aspects, features, and embodiments will become apparent from the following description, including the claims. [Brief description of the drawings]

[0030] [Figure 1] FIG. [Diagram 2] FIG. [Diagram 3] FIG. [Figure 4] FIG. 1 is a schematic diagram of a road network from above. [Diagram 5] FIG. 1 is a schematic diagram of a road network from above. [Figure 6] FIG. 2 is an annotated perspective view of an intersection. [Figure 7] FIG. 2 is an annotated perspective view of an intersection. [Figure 8] FIG. 1 is a schematic diagram of a road network from above. [Figure 9] FIG. 1 is a schematic diagram of a road network from above. [Figure 10] FIG. 1 is a schematic diagram of a road network from above. [Figure 11] FIG. 1 is a schematic diagram of a road network from above. [Figure 12] FIG. 1 is a schematic side view of a road network. [Figure 13] FIG. 1 is a schematic diagram of a road network from above. [Figure 14] FIG. 1 is a schematic diagram of a road network from above. [Figure 15] FIG. [Figure 16] FIG. 1 is a schematic perspective view of a road network. [Figure 17] FIG. 1 is a schematic diagram of a road network from above. [Figure 18] FIG. 1 is a schematic diagram of a road network from above. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] With advances in sensor technology and computers, it has become feasible to predict hazardous situations (and provide early warning of hazardous situations) and thus prevent collisions and near misses between ground traffic entities in ground traffic operations (i.e., enabling collision avoidance).

[0032] The term "ground transportation" is used broadly herein to encompass, for example, any mode or medium of moving from place to place involving contact with land or water on the surface of the Earth, such as walking or running (or performing other pedestrian actions), non-motorized vehicles, motorized vehicles (autonomous, semi-autonomous, and non-autonomous), and rail vehicles.

[0033] The term "ground transportation entity" (or sometimes simply "entity") is used broadly herein to encompass, for example, a person or a discrete motorized or non-motorized vehicle engaged in a mode of ground transportation, such as, among others, a pedestrian, a bicycle rider, a boat, a car, a truck, a tram, a streetcar, or a train. In some cases, the term "vehicle" or "road user" is used herein as a shorthand reference to a ground transportation entity.

[0034] The term "hazardous situation" is used broadly herein to encompass any event, occurrence, sequence, context, or other circumstance that may result in, for example, imminent property damage or personal injury or death, and that may be reducible or avoidable. The term "hazard" is sometimes used interchangeably herein with "hazardous situation." The phrases "violation" or "violating" are sometimes used herein with respect to behavior of an entity that results in, may lead to, or will lead to a dangerous situation.

[0035] In some implementations of the technology discussed herein, a ground transportation network is used by a mix of ground transportation entities that do not have or do not use transportation connectivity, and ground transportation entities that do have and use transportation connectivity.

[0036] The term "connectivity" is used broadly herein to encompass any capability of a ground traffic entity to, for example, (a) be aware of and act on knowledge of the ground traffic entity's surroundings, other ground traffic entities in the vicinity of the ground traffic entity, and traffic conditions associated with the ground traffic entity, (b) broadcast or otherwise transmit data regarding the state of the ground traffic entity, or (c) do both (a) and (b). The transmitted data may include its location, heading, speed, or internal state of its components related to the traffic situation. In some examples, the ground traffic entity's awareness is based on wirelessly received data regarding other ground traffic entities or traffic conditions associated with the operation of the ground traffic entity. The received data may originate from other ground traffic entities, or from infrastructure devices, or both. Typically, connectivity involves transmitting or receiving data in real time, or substantially in real time, or in time for one or more of the ground traffic entities to act on the data in the traffic situation.

[0037] The term "traffic situation" is used broadly in this specification to encompass any situation in which two or more ground transportation entities operate in proximity to one another and in which the actions or status of each of the entities may affect or be related to the actions or status of the others.

[0038] At times herein, ground transportation entities that do not have connectivity or connectivity aspects or that do not use connectivity or connectivity aspects are referred to as “disconnected ground transportation entities” or simply “disconnected entities.” At times herein, ground transportation entities that have connectivity or connectivity aspects and that use connectivity or connectivity aspects are referred to as “connected ground transportation entities” or simply “connected entities.”

[0039] In some cases herein, the term "associated entity" is used to refer to a ground transportation entity that broadcasts data to its environment, including, for example, location, heading, speed, or the status of on-board safety systems (e.g., brakes, lights, and wipers).

[0040] In some cases herein, the term "uncoordinated entity" is used to refer to a ground transportation entity that does not broadcast one or more types of data, such as the ground transportation entity's position, speed, heading, or status, to the ground transportation entity's environment.

[0041] In some cases, the term "neighborhood" of a ground transportation entity is used broadly herein to encompass, for example, the area where a broadcast by the entity may be received by other ground transportation entities or infrastructure devices. In some instances, the neighborhood varies with the location of the entity and the number and characteristics of obstacles around the entity. An entity traveling on an open road in the desert has a very large neighborhood because there are no obstacles that prevent a broadcast signal from the entity from reaching long distances. Conversely, the neighborhood in an urban canyon is made smaller by buildings around the entity. Additionally, there may be electromagnetic noise sources that degrade the quality of the broadcase signal, thus shortening the reception distance (neighborhood).

[0042] 14, the neighborhood of entity 7001 traveling along road 7005 may be represented by concentric circles, with outermost circle 7002 representing the outermost extent of the neighborhood. Any other entity that is within circle 7002 is in the neighborhood of entity 7001. Any other entity located outside circle 7002 is outside the neighborhood of entity 7001 and cannot receive broadcasts by entity 7001. Entity 7001 is invisible to all entities and infrastructure devices outside its neighborhood.

[0043] Typically, a coordinated entity continuously broadcasts the coordinated entity's status data. Connected entities in the vicinity of the broadcasting entity can receive these broadcasts, process the received data, and act on the received data. For example, if a vulnerable road user carries a wearable device that can receive a broadcast from an entity, such as an approaching truck, the wearable device can process the received data and inform the vulnerable user when it is safe to cross the road. As long as the user's device can receive the broadcast, i.e., is within the vicinity of the coordinated entity, this action occurs without considering the location of the coordinated entity or the vulnerable user relative to the "smart" intersection.

[0044] The terms "vulnerable road user" or "vulnerable road user" are used broadly herein to encompass any user of a roadway or other feature of a road network, for example, not using a motorized vehicle. If the vulnerable road user is struck by a motorized vehicle, the vulnerable road user is generally not protected against injury or death or property damage. In some examples, the vulnerable road user may be a person walking, running, bicycling, or performing any type of activity that exposes the vulnerable road user to the risk of direct physical contact by a vehicle or other ground traffic entity in the event of a collision.

[0045] In some embodiments, the collision avoidance techniques and systems described herein (sometimes referred to herein simply as "the systems") use sensors mounted on infrastructure facilities to monitor, track, detect, and predict the actions (e.g., speed, direction, and location), behavior (e.g., high speeds), and intentions (e.g., about to violate a stop sign) of ground transportation entities and their drivers and operators. The information provided by the sensors ("sensor data") enables the system to predict dangerous situations and provide early warnings to the entities to increase the chances of collision avoidance.

[0046] The term "collision avoidance" is used broadly in this specification to encompass any situation in which a collision or near miss between two or more ground traffic entities, or between a ground traffic entity and another object in the environment, that may result from, for example, a dangerous situation, is prevented, or the opportunity for such an interaction is reduced.

[0047] The term "early warning" is used broadly herein to encompass any notification, alarm, instruction, command, broadcast, transmission, or other transmission or receipt of information useful, for example, for identifying, suggesting, or in some manner indicating a dangerous situation and for collision avoidance.

[0048] Road intersections are the main locations where dangerous situations can occur. The technology described herein may include intersections that include infrastructure devices, including sensors, computing hardware, and intelligence, that allow for simultaneous monitoring, detection, and prediction of dangerous situations. Data from these sensors are normalized to one reference coordinate system and then processed. Artificial intelligence models of traffic flow along different approaches to the intersection are constructed. These models serve, for example, entities that are likely to violate traffic rules. The models are configured to detect dangerous situations before the actual violation and thus may be considered predictive. Based on the prediction of the dangerous situation, an alert is sent from the infrastructure device at the intersection to all connected entities in the vicinity of the intersection. Each entity that receives the alert processes the data in the alert and performs alert filtering. Alert filtering is the process of discarding or ignoring alerts that are not useful to the entity. If an alert, such as an alert of an impending collision, is deemed useful (i.e., not ignored as a result of filtering), the entity automatically reacts to the alert (e.g., by applying the brakes) or a notification is presented to the driver, or both.

[0049] The system may be used on, but is not limited to, roadways, waterways, and railroads. These and other similar transportation contexts are sometimes referred to herein as "surface transportation networks."

[0050] Although the system is often discussed herein in the context of intersections, the system may also be applied in other contexts.

[0051] The term "intersection" is used broadly herein to encompass any actual arrangement of roads, rails, bodies of water, or other paths of travel where, for example, two or more ground transportation entities traveling along a route in a ground transportation network may occupy the same location at some time and location and thus cause a collision.

[0052] Ground transportation entities using the ground transportation network may move at various speeds and arrive at a given intersection at different speeds and times. If the speed and distance of an entity from an intersection are known, dividing the distance by the speed (both expressed in the same unit system) gives the arrival time to the intersection. However, the expected arrival time to an intersection changes continuously as the intended speed changes due to, for example, traffic conditions, speed limits on the route, traffic signals, and other factors. This dynamic change in the expected arrival time makes it impossible to predict the actual arrival time with 100% confidence.

[0053] Considering factors that influence the behavior of an entity requires applying multiple relationships between the entity's velocity and various influencing factors. The absolute state of an entity's behavior can be observed by sensors that track the entity from the entity or from an external location. The data captured by these sensors can be used to model patterns of the entity's movements, behavior, and intent. Machine learning can be used to generate complex models from vast amounts of data. Patterns that cannot be directly modeled using the entity's kinematics can be captured using machine learning. The trained model can predict whether an entity is going to move or stop at a particular point by using the entity's tracking data from the sensors that track them.

[0054] In other words, in addition to detecting information about ground traffic entities directly from the sensor data, the system uses artificial intelligence and machine learning to process vast amounts of sensor data to learn patterns of ground traffic entity actions, behaviors, and intents, for example, at intersections of the ground traffic network, on approaches to such intersections, and at crosswalks of the ground traffic network. Based on the direct use of current sensor data and based on the results of applying artificial intelligence and machine learning to the current sensor data, the system generates early warnings, for example, warnings of dangerous situations, and thus aids in collision avoidance. In connection with early warnings in the form of instructions or commands, the commands or instructions may be targeted to a particular autonomous or human-driven entity to directly control the vehicle. For example, the instructions or commands may slow down or stop an entity driven by a malicious person that is determined to be running a red light with the intent of injuring a person.

[0055] The system may be tuned to make predictions for that particular intersection and to send alerts to entities in the vicinity of the device broadcasting the alert. For this purpose, the system derives data on dangerous entities using sensors and passes current readings from the sensors through a trained model. The output of the model may thus predict a dangerous situation and broadcast a corresponding alert. The alert received by connected entities in the vicinity contains information about the dangerous entity, and as a result, the receiving entity may analyze the information to assess the threat posed to the receiving entity by the dangerous entity. If a threat is present, the receiving entity may take action by itself (e.g., slow down) or notify the driver of the receiving entity using a human-machine interface based on visual, audio, tactile, or any kind of sensory stimuli. The autonomous entity may take action by itself to avoid the dangerous situation.

[0056] The alert may also be transmitted directly through a cellular or other network to a mobile phone or other device equipped to receive the alert and carried by the pedestrian. The system identifies potentially dangerous entities at an intersection and broadcasts (or transmits directly) an alert to the pedestrian's personal device that includes a communication unit. The alert may, for example, prevent the pedestrian from entering the crosswalk, thus avoiding a potential accident.

[0057] The system may further track pedestrians and broadcast information related to the pedestrian's status (location, speed, and other parameters) to other entities so that they may take action to avoid dangerous situations.

[0058] As shown in FIG. 1, the system includes at least the following types of components:

[0059] 1. Roadside Equipment (RSE) 10 that includes or uses sensors 12 to monitor, track, detect, and predict the motion (e.g., speed, heading, and location), behavior (e.g., high speed), and intent (e.g., violating a stop sign) of ground traffic entities 14. The RSE may also include or use data processing units 11 and data storage 18. Ground traffic entities exhibit a wide range of behaviors that depend on the infrastructure of the ground traffic network, as well as the state of the entity itself, the state of the driver, and the state of other ground traffic entities. To capture the entity's behavior, the RSE collects information from sensors, other RSEs, OBEs, OPEs, local or central servers, and other data processing units. The RSE may also store data received by the RSE and store data processed at some or all of the steps in the pipeline.

[0060] The RSE may store data in a local storage device or a remote storage. The collected data is processed in real time using a predefined logic or using logic based on the collected data dynamically, which means that the RSE may automatically update its own logic. The data may be processed in one processing unit or in a cluster of processing units to get faster results. The data may be processed in a local or remote processing unit or a local or remote cluster of processing units. The RSE may use simple logic or advanced models trained on the collected data. The models may be trained locally or remotely.

[0061] The RSE may pre-process the data before using the trained model to filter outliers. Outliers may appear due to noise or reflections in the sensor or due to some other artifacts. The resulting outliers may result in false alarms that may affect the performance of the entire RSE. The filtration method may be based on data collected by the RSE, OBE, OPE, or online resources. The RSE may interface with other controllers, such as traffic light controllers at intersections or other locations, to extract information for use in the data processing pipeline.

[0062] The RSE may further include or use communication equipment 20 for communicating, wired or wirelessly, with other RSEs and with the OBE, OPE, local or central servers, and other data processing units. The RSE may use any available standard for communicating with other equipment. The RSE may use a wired or wireless Internet connection for downloading and uploading data to other equipment, a cellular network for sending messages and receiving messages from other cellular devices, and a dedicated radio device for communicating to infrastructure devices and other RSEs at intersections or other locations.

[0063] The RSE may be installed in close proximity to different types of intersections. For example, at a signalized intersection (e.g., an intersection where traffic is controlled by signal lights), the RSE 10 is installed in the same enclosure near the traffic light controller 26 or in a nearby enclosure. Data (e.g., traffic light phase and timing) is intended to flow 28 between the traffic light controller and the RSE. At non-signalized intersections, the RSE 10 is usually located to simplify connecting the RSE 10 to sensors 12 used to monitor other features of the road or ground traffic network in the vicinity of the intersection. The proximity of the RSE to the intersection helps maintain a low latency system, which is important to provide the receiving ground unit with the maximum time to respond to an alert.

[0064] 2. Onboard Equipment (OBE) 36 mounted on, supported by, or within the ground traffic entity 14 that includes sensors 38 that determine the entity's position and kinematics (motion data) as well as safety-related data about the entity. The OBE further includes a data processing unit 40, data storage 42, and communication equipment 44 that may wirelessly communicate with other OBEs, OPEs, RSEs, and possibly servers and computing units.

[0065] 3. On Person Equipment (OPE) 46, which may be, but is not limited to, a mobile phone, a wearable device, or any other device capable of being worn by, held by, attached to, or otherwise associated with a person or animal. The OPE may include or be coupled to a data processing unit 48, a data storage 50, and a communication device 52, as needed. In some embodiments, the OPE serves as a dedicated communication unit for vulnerable road users that are not vehicles. In some instances, the OPE may be used for other purposes as well. The OPE may include components for providing visual, audio, or tactile alerts to vulnerable road users.

[0066] Potentially affected road users may include pedestrians, cyclists, road workers, people in wheelchairs, scooters, self-balancing devices, or battery-powered personal mobility devices, animal-powered carriages, guide or police animals, livestock, herds, and pets.

[0067] Typically, an OPE is carried by a vulnerable road user and is capable of sending and receiving messages. An OPE may be attached to or integrated into a mobile phone, tablet, personal mobility device, bicycle, wearable device (e.g., watch, bracelet, anklet), or attached to a pet's collar.

[0068] The message transmitted by the OPE may include kinematic information related to the potential road user, including but not limited to time, 3D position, heading, speed, and acceleration. The transmitted message may further convey data representing the alert level, current behavior, and future intentions of the potential road user, e.g., a potential road user currently crossing a crosswalk, listening to music, or about to cross a crosswalk. Among other possibilities, the message may convey the blob size or data size of the potential road user, whether there is an external device (e.g., a stroller, cart, or other device) with the potential road user, whether the potential road user has a physical disability or is using any personal aids. The message may convey the category of the potential road user if the potential road user is a worker, and may further describe the type of action being performed by the worker. If a cluster of similar potential road users (e.g., a group of pedestrians) have similar characteristics, one message may be transmitted to avoid multiple message broadcasts.

[0069] Typically, messages received by OPE are warning messages from roadside equipment or entities. OPE may act on the received messages by issuing warnings to vulnerable road users. Warning messages convey data that is useful in providing customized warnings for vulnerable road users. For example, warnings to vulnerable road users may indicate the type of dangerous situation and suggest possible actions to be taken. OPE may apply warning filtering to all received messages and present only relevant messages to vulnerable road users.

[0070] The alert filtering is based on the results of applying a learning algorithm to historical data associated with the OPE, which allows the alert filtering to be custom tailored to each of the vulnerable road users. The OPE learning algorithm tracks the responses of the vulnerable road users to received alerts and tailors future alerts to elicit the best response times and attention from the vulnerable road users. The learning algorithm may also be applied to data conveyed in transmitted messages.

[0071] 4. A data storage server 54, which may be, but is not limited to, cloud storage, local storage, or any other storage facility allowing storage and retrieval of data. The data storage server is accessible by the RSE, by the computing unit, and possibly by the OBE, the OPE, and the data server, for example to store data related to early warning and collision avoidance. The data storage server is accessible from the RSE, and possibly from the OBE, the OPE, and the data server, for fetching the stored data. The data may be raw sensor data, data processed by the processing unit, or any other information generated by the RSE, the OBE, and the OPE.

[0072] Sensors at intersections that continuously monitor ground traffic entities may generate large amounts of data daily. The volume of this data depends on the number and type of sensors. The data is both processed in real time and stored for future analysis, which requires, for example, a data storage unit (e.g., hard disk drive, solid state drive, and other mass storage device) locally at the intersection. The local storage device fills up after a period of time depending on the storage capacity of the local storage device, the volume of data generated, and the rate at which the data is generated. To store the data for future use, the data is uploaded to a remote server with a larger capacity. The remote server may upgrade its storage capacity on demand as needed. The remote server may use a data storage device similar to the local storage (e.g., hard disk drive, solid state drive, or other mass storage device) accessible through a network connection.

[0073] Data stored locally and on the server for future analysis may include data broadcast by ground transportation entities and received by the RSE that is saved for future analysis. Stored data may be downloaded from the server or other remote sources for processing at the RSE. For example, a machine learning model of an intersection where the RSE is located may be stored on the server or other remote storage and downloaded by the RSE for use in analyzing current data received at the RSE from local sources.

[0074] 5. Computing unit 56, a powerful computing machine located in the cloud or locally (e.g., as part of the RSE), or a combination thereof. Among other functions, the computing unit processes the available data to generate predictions and machine learning-based models of the actions, behaviors, and intentions of vehicles, pedestrians, or other ground traffic entities using the traffic network. Each computing unit may include dedicated hardware for processing the corresponding type of data (e.g., a graphics processing unit for processing images). In the case of heavy processing load, the computing unit in the RSE may become overloaded. This may occur, for example, when additional data generating units (e.g., sensors) are added to the system causing a computational overload. Overload may also occur if logic running in the computing unit is replaced with more computationally intensive logic. Overload may occur due to an increase in the number of ground traffic entities being tracked. In the case of local computational overload, the RSE may offload some of the tasks to another computing unit. Such other computing unit may be near the RSE or remote, e.g., a server. Computational tasks may be prioritized, and non-time-critical tasks may be performed on such other computing units, with results available to the local computing unit. For example, a computing unit in the RSE may request another computing unit to run a job to analyze stored data and to train a model using the data. The trained model is then downloaded by the computing unit in the RSE for storage and use therein.

[0075] Computing units in the RSE may save time by using other smaller computing units to perform computationally intensive jobs more efficiently. Available computing units are used wisely to perform most tasks in the shortest time, for example, by dividing tasks between the RSE computing units and other available computing units. Computing units may also be attached to the RSE as external devices to add more computing power to the computing units in the RSE. The externally attached computing units may include the same or different architecture compared to the computing units in the RSE. The externally attached computing units may communicate with existing computing units using any available communication ports. The RSE computing units may request more computing power from the external computing units as needed.

[0076] The remainder of this specification describes in detail, among other things, the roles and functions of the above-mentioned components in the system.

[0077] Roadside Equipment (RSE)

[0078] As shown in FIG. 2, the RSE may include, but is not limited to, the following components:

[0079] 1. One or more communication units 103, 104 that enable receipt and / or transmission of operational and other data related to ground transportation entities, and traffic safety data from and to nearby vehicles or other ground transportation entities, infrastructure, and remote servers and data storage systems 130. In some examples, this type of communication is known as infrastructure-to-everything (I2X), where I2X includes, but is not limited to, infrastructure-to-vehicles (I2V), infrastructure-to-pedestrians (I2P), infrastructure-to-infrastructure (I2I), and infrastructure-to-devices (I2D), and combinations thereof. Communications may be wireless or wired and may conform to a wide variety of communication protocols.

[0080] 2. Communication unit 103 is used for communication with ground transportation entities, and unit 104 is used for communication with a remote server and data storage system 130 over the Internet.

[0081] 3. Local storage 106 for storing programs, intersection models, and behavior and traffic models. The local storage 106 may also be used for temporary storage of data collected from the sensors 101.

[0082] 4. Sensors 101 and sensor controllers 107 that enable monitoring (e.g., generating data regarding) moving objects, such as ground traffic entities typically near the RSE. Sensors may include, but are not limited to, cameras, radar, lidar, ultrasonic detectors, or any other hardware that can detect or infer from detected data, the distance to the ground traffic entity, or the speed, heading, or location of the ground traffic entity, or a combination thereof, among other things. Sensor fusion is performed using the collection or combination of data from two or more sensors 101.

[0083] 5. A position receiver (102) (eg, a GPS receiver) that provides positioning data (eg, coordinates of the RSE's location) and helps correct positioning errors in the positioning of ground traffic entities.

[0084] 6. A processing unit 105 that acquires and uses data generated from the sensors and incoming data from the communication units 103, 104. The processing unit processes and stores the data locally and, in some embodiments, transmits the data for remote storage and further processing. The processing unit also generates messages and alerts that are broadcast or otherwise transmitted over wireless communication facilities to nearby pedestrians, motor vehicles, or other ground transportation entities, and in some examples to signs or other infrastructure presentation devices. The processing unit also periodically reports the health and status of all RSE systems to a remote server for monitoring.

[0085] 7. Expansion connector 108 allowing control and communication between the RSE and other hardware or other components, such as temperature and humidity sensors, traffic light controllers, other computing units as mentioned above, and other electronic devices that may become available in the future.

[0086] Onboard equipment (OBE)

[0087] The on-board equipment may typically be original equipment for the ground transportation entity or may be added to the entity by a third-party supplier. As shown in FIG. 3, the OBE may include, but is not limited to, the following components:

[0088] 1. A communication unit 203 that enables transmission and / or reception of data to and from nearby vehicles, pedestrians, cyclists, or other ground traffic entities, and infrastructure, and combinations thereof. The communication unit further enables transmission and / or reception of data between vehicles or other ground traffic entities and a local or remote server 212 for machine learning purposes and for remote monitoring of the ground traffic entities by the server. In some examples, this type of communication is known as vehicle-to-everything (V2X), which includes, but is not limited to, vehicles-to-vehicles (V2V), vehicles-to-pedestrians (V2P), vehicle-to-infrastructure (V2I), vehicle-to-devices (V2D), and combinations thereof. Communications may be wireless or wired and may conform to a wide variety of communication protocols.

[0089] The communication unit 204 enables the OBE to communicate with remote servers over the Internet for program updates, data storage, and data processing.

[0090] 2. Local storage 206 for storing programs, intersection models, and traffic models. The local storage 206 may be used for temporary storage of data collected from the sensors 201.

[0091] 3. Sensors 201 and sensor control unit 207, which may include, but are not limited to, external cameras, lidar, radar, ultrasonic sensors, or any device that may be used to detect nearby objects or people or other ground traffic entities. Sensors 201 may include additional kinematic sensors, global positioning receivers, and internal and local microphones and cameras.

[0092] 4. A position receiver 202 (eg, a GPS receiver) that provides position location data (eg, coordinates of the location of a ground transportation entity).

[0093] 5. A processing unit 205 for acquiring, using, generating, and transmitting data, including consuming data from the communication unit and transmitting data to the communication unit and consuming data from sensors in or on the ground traffic entity.

[0094] 6. Expansion connector 208 to allow control of the OBE and other hardware and communication between the OBE and other hardware.

[0095] 7. An interface unit that may be retrofitted or integrated into the head unit, steering wheel, or driver mobile device in one or more ways, for example using visual, audible, or tactile feedback.

[0096] Smart OBE (SOBE: Smart OBE)

[0097] In a world where all vehicles and other ground traffic entities are connected entities, each vehicle or other ground traffic entity may be a coordinated entity and may report its current location, safety status, intent, and other information to others. Currently, nearly all vehicles are not connected entities, are not capable of reporting such information to other ground traffic entities, and are operated by humans with different levels of skill, happiness, stress, and behavior. Without such connectivity and communication, it becomes difficult to predict the next move of a vehicle or the next move of a ground traffic entity, resulting in a reduced ability to implement collision avoidance and provide early warnings.

[0098] The smart OBE monitors the environment and the user or occupant of the ground transportation entity. The smart OBE also monitors the health and status of the different systems and subsystems of the entity. The SOBE monitors the outside world, for example, by listening to radio transmissions from emergency broadcasts, traffic and safety messages from nearby RSEs, and messages regarding safety, location, and other operational information from other connected vehicles or other ground transportation entities. The SOBE also interfaces with on-board sensors that can view the road and driving conditions, for example, cameras, distance sensors, vibration sensors, microphones, or any other sensors that enable such monitoring. The SOBE also monitors the immediate environment and generates a map of all stationary and moving objects.

[0099] The SOBE may further monitor the behavior of users or occupants of a vehicle or other ground transportation entity. The SOBE may use microphones to monitor conversation quality. The SOBE may further use other sensors, such as seat sensors, cameras, hydrocarbon sensors, and sensors for volatile organic compounds and other toxic substances. The SOBE may further use kinematic sensors to measure driver reactions and behavior and infer driving quality therefrom.

[0100] SOBE also receives vehicle-to-vehicle messages from other ground transportation entities, such as basic safety messages (BSM), and vehicle-to-pedestrian messages from potential road users, such as personal safety messages (PSM).

[0101] SOBE then fuses data from this array of sensors, sources, and messages. SOBE then applies the fused data to an artificial intelligence model that can predict not only the next action or reaction of the driver or user of the vehicle or other ground traffic entity, or of the vulnerable road user, but also predict the intent and future trajectory and the associated near miss or collision risk due to other vehicles, ground traffic entities, and nearby vulnerable road users. For example, SOBE may use the BSM received from a nearby vehicle to predict that the nearby vehicle is about to enter a lane-changing maneuver that creates a risk to its own host vehicle, and may alert the driver of the impending risk. The risk is calculated by SOBE based on the various future predicted trajectories of the nearby vehicle (e.g., going straight, changing lanes to the right, changing lanes to the left) and the probability of the associated collision risk with the host vehicle for each of those trajectories. If the collision risk is higher than a certain threshold, a warning is displayed to the driver of the host vehicle.

[0102] Due to the complexity of modeling human driver behavior, which is also influenced by external factors (e.g., changing environmental and weather conditions), machine learning is typically required to predict intent and future trajectories.

[0103] SOBE is characterized by powerful computing power capable of processing multiple data feeds, some of which provide several megabytes of data per second. The amount of available data is, in turn, proportional to the level of detail required from each sensor.

[0104] SOBE further includes powerful signal processing equipment that can extract useful information from environments known to have high (signal) noise levels and low signal-to-noise ratios. SOBE also protects the driver from the overwhelming number of alerts the vehicle receives by providing smart alert filtering. Alert filtering is the result of machine learning models that can distinguish which alerts are important in the context of current location, environmental conditions, driver behavior, vehicle health and status, and kinematics.

[0105] Smart OBEs are important for collision avoidance and early warning, and for achieving a safer transportation network for all users, not just for the occupants or users of the vehicles that contain the SOBE. SOBEs can detect and predict the movements of different entities on the road, and thus aid in collision avoidance.

[0106] Person equipment (OPE)

[0107] As mentioned above, Person On Board Equipment (OPE) includes any device that may be held, worn, or otherwise directly associated with a pedestrian, jogger, or other person that is a ground transportation entity or is otherwise present on or using a ground transportation network. Such a person may be, for example, a road user that may be vulnerable to being struck by a vehicle. OPE may include, but is not limited to, mobile devices (e.g., smartphones, tablets, digital assistants), wearable devices (e.g., eyewear, watches, bracelets, anklets), and implants. Existing components and features of OPE may be used to track and report location, speed, and heading. OPE may be used to receive and process data and display alerts to the user through various modes (e.g., visual, audio, tactile).

[0108] Honda has developed a communication system and method for V2P applications that focuses on direct communication between vehicles and pedestrians using OPE. In one example, the vehicle is equipped with an OBE for broadcasting messages to the OPE of surrounding pedestrians. The messages convey the current status of the vehicle, including, for example, vehicle parameters, speed, and direction of travel. For example, the messages can be basic safety messages (BSMs). When necessary, the OPE presents the pedestrian with warnings about predicted dangerous situations, tailored to the pedestrian's distraction level, to avoid a collision. In another example, the pedestrian's OPE broadcasts a message (e.g., personal safety message (PSM)) to the OBE of surrounding vehicles that the pedestrian may cross the vehicle's intended path. When necessary, the vehicle's OBE displays warnings to the vehicle user about the predicted danger to avoid a collision. See Strickland, Richard Dean, et al., "Vehicle to pedestrian communication system and method." U.S. Pat. No. 9,421,909.

[0109] The system described herein uses an I2P or I2V approach that uses sensors external to vehicles and pedestrians (mainly in infrastructure) to track and collect data on pedestrians and other vulnerable road users. For example, sensors may track pedestrians crossing streets and vehicles operating at or near crossing locations. The collected data is then used to build predictive models of pedestrian and vehicle driver intent and behavior on roads using rule-based and machine learning methods. These models help analyze the collected data and make predictions of pedestrian and vehicle paths and intents. If a danger is predicted, a message is broadcast from the RSE to the OBE and / or OPE, alerting each entity in the other's intended path, allowing each of them to take preemptive action with enough time to avoid a collision.

[0110] Remote Computing (Cloud Computing and Storage)

[0111] Data collected from sensors connected to or embedded in the RSE, OBE, and OPE need to be processed so that effective mathematical machine learning models can be generated. This processing requires a lot of data processing power to shorten the time required to generate each model. The required processing power is much more than is typically available locally in the RSE. To solve this, data can be sent to a remote computing facility that provides the required power and can be scaled on demand. In this specification, the remote computing facility is referred to as a "remote server," in keeping with the terminology used in the computing literature. In some instances, it may be possible to perform some or all of the processing in the RCE by equipping the RCE with high-powered computing power.

[0112] Rule-Based Processing

[0113] Unlike artificial intelligence and machine learning techniques, rule-based processing can be applied at any time without the need for data collection, training, and model building. Rule-based processing can be deployed from the beginning of the system's operation, typically until sufficient training data is acquired to generate a machine learning model. After a new installation, rules are set up to process the incoming sensor data. This is not only useful for improving road safety but also a suitable test case to ensure that all components of the system work as expected. Rule-based processing can be further added and used later as an additional layer to capture rare cases where machine learning cannot make accurate predictions. The rule-based approach is based on simple associations between collected data parameters (e.g., speed, range, etc.). The rule-based approach can further provide a baseline for evaluation of the performance of machine learning algorithms.

[0114] In rule-based processing, a vehicle or other ground traffic entity crossing a portion of a ground traffic network is monitored by sensors. If its current speed and acceleration exceed thresholds that prevent it from stopping before a stop bar (line) on the road, for example, an alert is generated. A variable area is assigned to each vehicle or other ground traffic entity. The area is labeled as a dilemma zone, within which the vehicle has not yet been labeled as a violating vehicle. If the vehicle crosses the dilemma zone into a danger zone because its speed or acceleration, or both, becomes higher than a predefined threshold, the vehicle is labeled as a violating entity and an alert is generated. The thresholds for speed and acceleration are based on physics and kinematics and vary depending on each ground traffic entity approaching, for example, an intersection.

[0115] Two traditional rule-based approaches are 1) static TTI (Time-To-Intersection) and 2) static RDP (Required Deceleration Parameter). See Aoude, Georges S. et al., "Driver behavior classification at intersections and validation on large naturalistic data set." IEEE Transactions on Intelligent Transportation Systems 13.2(2012):724-736.

[0116] Static TTI (Time to Intersection) uses the estimated time to reach an intersection as a classification criterion. In its simplest form, TTI is:

number

[0117] The static RDP (required deceleration parameter) calculates the deceleration required for a vehicle to come to a safe stop, given the vehicle's current speed and position on the road. The RDP is

number

number

[0118] Similar to the static TTI algorithm, RDP alert The parameter reflects the conservatism level of the rule-based algorithm.

[0119] Rule-based approaches are used herein as a baseline for evaluating the performance of the machine learning algorithms of the present disclosure, and in some cases they are run in parallel with the machine learning algorithms to capture rare cases that machine learning may not be able to predict.

[0120] Machine Learning

[0121] Modeling driver behavior has been shown to be a complex task given the complexity of human behavior. See HM Mandalia and DDDalvucci, "Using Support Vector Machines for Lane-Change Detection," Human Factors and Ergonomics Society Annual Meeting Proceedings, vol. 49, 1965 {p. 1969, 2005. Machine learning techniques are well suited to model human behavior, but they require "learning" using training data to operate properly. To provide good detection and prediction results, machine learning is used herein to model traffic detected at intersections, or other features of the ground traffic network, during a training period before alert processing is applied to current traffic during the deployment phase. Machine learning can be further used to model driver responses using in-vehicle data from on-board equipment (OBE), and can be further based on in-vehicle sensors and driving record history and preferences. Machine learning models are further used herein to detect and predict trajectories, behaviors, and intentions of potentially vulnerable road users (e.g., pedestrians). Machine learning can further be used to model the responses of vulnerable road users from the Person On Board Equipment (OPE). These models can include interactions between entities, between vulnerable road users, and between one or more entities and one or more vulnerable road users.

[0122] Machine learning techniques may further be used to model the behavior of non-autonomous ground traffic entities. By observing and / or communicating with the non-autonomous ground traffic entities, machine learning may be used to predict the intentions of the non-autonomous ground traffic entities, and to communicate with the non-autonomous ground traffic entities when near misses, or accidents, or other dangerous situations are predicted, and to communicate with other involved entities.

[0123] The machine learning mechanism works well in two phases: 1) training, and 2) deployment.

[0124] Training Phase

[0125] After installation, the RSE starts to collect data from sensors that the RSE can access. Since AI model training requires powerful computing power, AI model training is usually performed on powerful servers that contain multiple parallel processing modules to speed up the training phase. For this reason, data acquired at the RSE's location in the ground transportation network can be packaged and sent to a remote powerful server immediately after acquisition. This is done using an Internet connection. The data is then prepared automatically or with the help of a data scientist. An AI model is then built to capture key characteristics of the traffic flow of vehicles and other ground transportation entities relative to that intersection or other aspects of the ground transportation network. The captured data features may encompass the location, direction, and movement of the vehicles or other ground transportation entities, which can then be translated into intent and behavior. Knowing the intent, the actions and future behavior of vehicles or other ground transportation entities approaching the traffic location can be predicted with high accuracy using the AI ​​model. The trained AI model is tested on a subset of the data that was not included in the training phase. If the performance of the AI ​​model is in line with expectations, the training is considered complete. This phase is repeated iteratively using different model parameters until a satisfactory performance of the model is achieved.

[0126] Deployment Phase

[0127] In some embodiments, the completed and tested AI model is then communicated to the RSE at the traffic location in the ground transportation network through the Internet. The RSE is then ready to process new sensor data and perform prediction and detection of dangerous situations, such as traffic light violations. If a dangerous situation is predicted, the RSE generates an appropriate warning message. Before the predicted dangerous situation occurs, the dangerous situation can be predicted, a warning message can be generated, and the warning message can be broadcast to and received by vehicles and other ground traffic entities in the vicinity of the RSE. This gives the operators of the vehicles or other ground traffic entities sufficient time to react and perform collision avoidance. The output of the AI ​​model from various intersections where the corresponding RSE is located can be recorded and made available online in a dashboard incorporating all the data generated and displayed in an intuitive and user-friendly manner. Such a dashboard can be used as an interface with the customer of the system (e.g., city traffic engineers or planners). One example of a dashboard is a map with markers showing the locations of monitored intersections, violation events that have occurred, and statistics and analytics based on AI predictions and actual results.

[0128] Smart RSE (SRSE) and connected / unconnected entity bridging

[0129] As already alluded to, there are gaps between the capabilities and behaviors of connected and unconnected entities. For example, connected entities are typically coordinated entities that continuously advertise their location and safety system status, e.g., speed, heading, brake status, and headlight status, to the world. Unconnected entities are not able to coordinate and communicate in these ways. Thus, even connected entities are not aware of unconnected entities that are not in the vicinity of the connected entities or that are outside their detection range due to interference, distance, or lack of vantage point.

[0130] With the right equipment and configuration, RSEs can be enabled to detect all entities using the ground transportation network in their vicinity, including unconnected entities. Special sensors can be used to detect different kinds of entities. For example, radars are suitable for detecting moving metal objects, e.g., cars, buses, and trucks. Such road entities are most likely moving in one direction towards an intersection. Cameras are suitable for detecting vulnerable road users who may be pondering around the intersection waiting for a safe time to cross.

[0131] Placing sensors on surface transportation network components has at least the following advantages:

[0132] - Vantage points: infrastructure poles, beams, and support cables usually include high vantage points. High vantage points allow a more general view of the intersection. This is similar to a control tower at an airport, where the controller has a full view of most of the important and vulnerable users on the ground. For ground traffic entities, in contrast, the view from the vantage point of the sensor (camera, lidar, radar, etc., ... or other) may be obstructed or obstructed by trucks in nearby lanes, direct sunlight, or other interference. Sensors at intersections may be selected to be resistant to and less susceptible to such interference. For example, radar is not affected by sunlight and remains effective during the evening commute. Thermal cameras are more likely to detect pedestrians in bright light conditions where the view of optical cameras is obstructed.

[0133] - Fixed position: A sensor located at an intersection can be adjusted and fixed to detect in a specific direction that may be optimal for detecting important targets. This helps the processing software to detect objects better. As an example, if a camera has a fixed view, the background information (of stationary objects and structures) in the fixed view can be easily detected and used to improve the identification and classification of relatively important moving entities.

[0134] Fixed sensor locations also allow for easier placement of each entity in a joint global view of the intersection. Because the sensor view is fixed, measurements from the sensors can be easily mapped to a joint global location map of the intersection. Such a joint map is useful when performing a global analysis of traffic movement from all directions to investigate the interactions and dependencies of one traffic flow on another. An example is detecting near misses (hazardous situations) before they occur. When two entities are traveling along intersecting paths, the global and joint view of the intersection allows for the calculation of the arrival time of each entity to the intersection of their respective paths. If the time is within a certain limit or tolerance, the near miss can be flagged (e.g., subject to a warning message) before the near miss occurs.

[0135] With the help of sensors installed on infrastructure components, Smart RSE (SRSE) may bridge this gap and enable connected entities to recognize "dark" or unconnected entities.

[0136] FIG. 8 shows a scenario illustrating how strategically placed sensors can help a connected entity identify the speed and location of an unconnected entity.

[0137] A connected entity 1001 travels along a path 1007. Entity 1001 has a green light 1010 on. An unconnected entity 1002 travels along a path 1006. Unconnected entity 1002 is subject to a red light 1009, but is about to turn right along path 1006 on the red light. This places unconnected entity 1002 directly in the path of entity 1001. A dangerous situation is imminent, as entity 1001 does not see entity 1002. Because entity 1002 is an unconnected entity, it cannot broadcast (e.g., advertise) its location and heading to other entities sharing the intersection. Furthermore, even if entity 1001 were connected, entity 1001 would not be able to "see" entity 1002, who is obscured by building 1008. There is a risk of entity 1001 driving straight through the intersection and colliding with entity 1002.

[0138] If the intersection is configured as a smart intersection, a radar 1004 mounted on a beam 1005 above the road at the intersection detects the entities 1002 and the speed and distance of the entities 1002. This information can be relayed to the connected entities 1001 through the SRSE 1011, which acts as a bridge between the unconnected entities 1002 and the connected entities 1001.

[0139] Artificial Intelligence and Machine Learning

[0140] The smart RSE further relies on learning traffic patterns and entity behavior to better predict and prevent dangerous situations and to avoid collisions. As shown in FIG. 8, the radar 1004 is constantly detecting and providing data about each entity moving along the approach road 1012. This data is collected and communicated to the cloud, directly or through the RSE, for example, for analysis and to build and train a model that adequately represents the traffic along the approach road 1012. When the model is complete, it is downloaded to the SRSE 1011. This model can then be applied to each entity moving along the approach road 1012. If an entity is classified by the model as violating (or about to violate) a traffic rule, a warning (alert) can be broadcast by the SRSE to all connected entities in the vicinity. This warning, known as an intersection collision avoidance warning, can be received by the connected entities and can affect their consideration of dangerous situations and their avoidance of collisions. Using an appropriate traffic model, violating entities may be detected in advance, giving connected entities using the intersection sufficient time to react and avoid a dangerous situation.

[0141] With the help of multiple sensors (some mounted high on the infrastructure components of the ground transportation network), artificial intelligence models, and accurate traffic models, the SRSE can obtain a virtual overview of the ground transportation network and recognize each entity in its field of view, including unconnected entities in its field of view that are not "visible" to connected entities in its field of view. The SRSE can use this data to feed the AI ​​models and provide alerts to connected entities on behalf of the unconnected entities. Otherwise, the connected entities would not know that there are unconnected entities sharing the road.

[0142] The SRSE has available mass computing power at the SRSE's location, either in the same housing or by connection to a nearby unit, or by connection through the Internet to a server. The SRSE can process data received directly from the sensors or by broadcast from nearby SRSEs, emergency and weather information, and other data. The SRSE also includes mass storage to help store and process data. High bandwidth connectivity is also required to help transfer raw data and AI models between the SRSE and more powerful remote servers. The SRSE enhances other AI-based traffic hazard detection technologies to achieve high accuracy and provide additional time to react and avoid collisions.

[0143] The SRSE can still be compatible with current and new standardized communication protocols, so the SRSE can seamlessly interface with equipment already deployed in the field.

[0144] The SRSE may further reduce network congestion by sending messages only when necessary.

[0145] Global and integrated intersection topology

[0146] Effective traffic monitoring and control at an intersection benefits from a bird's-eye view of the intersection that is unobstructed by obstacles, lighting, or any other interference.

[0147] As described above, different types of sensors may be used to detect different types of entities. The information from these sensors may differ, e.g., not match with respect to the location or motion parameters that the data represents, or with respect to the native format of the data, or both. For example, radar data typically includes speed, distance, and sometimes additional information, e.g., the number of moving and stationary entities in the radar's field of view. Camera data, in contrast, may represent an image of the field of view at any point in time. Lidar data may provide the location of a point in 3D space corresponding to the reflection point of a laser beam emitted from the lidar at a particular time and heading. In general, each sensor provides data in a native format that adequately represents the physical quantity that the sensor measures.

[0148] To obtain a unified view (representation) of the intersection, it is useful to fuse data from different kinds of sensors. For the purpose of fusion, the data from the various sensors is converted into a common (unified) format that is independent of the sensor used. The data contained in the unified format from all of the sensors includes the global position, speed, and heading of each entity using the intersection, independent of how it was detected.

[0149] By using this integrated global data, the smart RSE can not only detect and predict the movements of entities, but also identify the relative positions and headings of different entities with respect to each other. Thus, the SRSE can achieve improved detection and prediction of dangerous situations.

[0150] For example, in the scenario shown in FIG. 9, a motorized entity 2001 and a vulnerable road user 2002 share the same pedestrian crossing. The entity 2001 is moving along the road 2007 and is detected by the radar 2003. A vulnerable road user 2002 walking along the sidewalk 2006 is detected by the camera 2004. The vulnerable road user 2002 may decide to cross the road 2007 using the crosswalk 2005. Doing so results in the road user 2002 in the path of the entity 2001 creating a potentially dangerous situation. Since each of the sensors may only detect entities in its respective field of view, if the data from each of the sensors 2003 and 2004 is considered independently and the other information is not taken into account, the dangerous situation will not be identified. Furthermore, each of the sensors may not be able to detect objects that the sensor is not designed to detect. However, when a synthetic view is considered by the SRSE, the positions and dynamics of the entity 2001 and of the potentially affected road users 2002 may be located in the same reference frame, i.e. a geographical coordinate system, e.g. a map projection, or other coordinate system. When considered in a common reference frame, the fused data from the sensors may be used to detect and predict potentially dangerous situations between the two entities 2001 and 2002. In the following paragraphs, the transformation between the sensor space and the synthetic space is discussed.

[0151] Transforming radar data into a synthetic standard

[0152] As shown in FIG. 10, the radar 3001 is used to monitor road entities moving along a road including two lanes 3005 and 3008 including centerlines 3006 and 3007, respectively. A stop bar 3003 indicates the ends of the lanes 3005 and 3008. The T 3006 may be defined by a set of markers 3003 and 3004. Although FIG. 10 shows only two markers, in general, the centerline is a piecewise linear function. The global positions of the markers 3003 and 3004 (and other markers not shown) are predefined by the roadway design and are known to the system. The exact global position of the radar 3001 may be further determined. Thus, the distances 3009 and 3010 of the markers 3003 and 3004 from the radar 3001 may be calculated. The distance 3011 of the entity 3002 from the radar 3001 may be measured by the radar 3001. Using simple geometry, the system can determine the location of the entity 3002 using the measured distance 3011. The result is a global position since it is derived from the global positions of the markers 3003, 3004 and the radar 3001. The above method can be applied to any roadway that can be monitored by radar, since each roadway can be approximated by a generalized piecewise linear function.

[0153] 11 shows a similar scenario on a curved road. Radar 4001 monitors an entity moving along road 4008. Markers 4003 and 4004 represent linear segments 4009 (of a piecewise linear function) of centerline 4007. Distances 4005 and 4006 typically represent the normal distances between the plane 4010 of radar 4001 and markers 4003 and 4004, respectively. Distance 4007 is the measured distance of entity 4002 from radar plane 4010. Following this, given the global positions of radar 4001 and markers 4003 and 4004, the global position of entity 4002 can be calculated using simple ratio arithmetic.

[0154] Transforming camera data into a unified standard

[0155] By knowing the camera's height, global position, orientation, tilt, and field of view, computing the global position of each pixel in the camera image becomes straightforward using existing 3D geometric laws and transformations. As a result, when an object is identified in an image, its global position can be easily estimated by knowing the pixels it occupies. It is instructive to note that if the camera specifications are known, such as sensor size, focal length, or field of view, or a combination thereof, the type of camera is irrelevant.

[0156] 12 shows a side view of a camera 5001 looking at an entity 5002. The height 5008 and tilt angle 5006 of the camera 5001 can be determined at installation. The field of view 5007 can be known from the specifications of the camera 5001. The global position of the camera 5001 can also be specified at installation. From the known information, the system can determine the global positions of points 5003 and 5004. The distance between points 5003 and 5004 is further divided into pixels in the image generated by the camera 5001. This number of pixels is known from the specifications of the camera 5001. The pixels occupied by the entity 5002 can be determined. Thus, the distance 5005 can be calculated. The global position of the entity 5002 can also be calculated.

[0157] By fusing information from various sensors, a global, integrated view of any intersection can be put together. FIG. 13 shows a top view of a four-way intersection. Each leg of the intersection is divided by the median 6003. The intersection in the figure is monitored by two different types of sensors, radar and camera, and the principles described herein can be generalized to other types of sensors. In this example, the radar monitoring area 6001 overlaps the camera monitoring area 6002. By using a unified global view, each entity moving between the areas keeps being tracked in the unified global view. This easily allows the SRSE to identify, for example, associations between the actions of different entities. Such information allows a truly universal, bird's-eye view of the intersection and roadways. The integrated data from the sensors can then be fed to an artificial intelligence program as described in the following paragraphs.

[0158] FIG. 2 above showed the components of the RSE. In addition, in the SRSE, the processing unit may further include one or more specialized processing units that may process data in parallel. An example of such a unit is a graphics processing unit or GPU. With the assistance of a GPU or similar hardware, machine learning algorithms may run much more efficiently in the SRSE and provide results in real time. Such a processing architecture allows for real-time prediction of dangerous situations and thus allows for sending early warnings to provide entities with enough time to react and avoid collisions. In addition, since the SRSE may perform processing that may use data from different sensors and different types of sensors, the SRSE may build an integrated view of the intersection that is useful for analyzing traffic flow and detecting and predicting dangerous situations.

[0159] Usage example

[0160] A wide variety of vehicles may benefit from the system and the early warning it may provide for collision avoidance. Examples are provided herein.

[0161] Example 1: Vulnerable Ground Transportation Entities

[0162] As shown in FIG. 4, a roadway crossing a typical intersection 409 may include a crosswalk that includes specific crossing areas 401, 402, 403, 404 that pedestrians and other vulnerable road users (vulnerable road users) may use to walk across the roadway. Sensors suitable for detecting such crossings or other vulnerable users are located at one or more vantage points that allow monitoring of the crosswalk and the surroundings of the crosswalk. During the training phase, the collected data may be used to train an artificial intelligence model to learn about the behavior of vulnerable road users at the intersection. Then, during the deployment phase, the AI ​​model may use current data about the vulnerable road users to, for example, predict that the vulnerable road users are about to cross the roadway and may make that prediction before the vulnerable road users start to cross. If the behavior and intentions of pedestrians and other vulnerable road users, drivers, vehicles, and other people and ground traffic entities can be predicted in advance, early warnings (e.g., alerts) may be sent to any or all of them. Early warnings may allow vehicles to stop, slow down, reroute, or a combination thereof, and may enable vulnerable road users to refrain from crossing the road if a dangerous situation is predicted to be imminent.

[0163] Generally, sensors are used to monitor all areas of possible movement of road users and vehicles that may be affected in the vicinity of the intersection. The type of sensor used depends on the type of object being monitored and tracked. Some sensors are better at tracking people and bicycles or other non-motorized vehicles. Some sensors are better at monitoring and tracking motorized vehicles. The solution described herein is sensor and hardware agnostic, since the type of sensor is irrelevant if the sensor provides suitable data at a sufficient data rate, which may depend on the type of object being monitored and tracked. For example, Doppler radar is a suitable sensor for monitoring and tracking the speed and distance of vehicles. The data rate, or sampling rate, is the rate at which the radar can provide successive new data values. The data rate must be fast enough to capture the dynamics of the motion of the object being monitored and tracked. The higher the sampling rate, the more details are captured and the more robust and accurate the representation of the motion by the data becomes. If the sampling rate is too low, and if the vehicle travels a significant distance between two sample instances, it becomes difficult to model the behavior due to details that are missed during the intervals where no data is generated.

[0164] For crosswalks, sensors monitor the intersection and areas near the intersection for crossing pedestrians and other vulnerable road users (e.g., cyclists). Data from these sensors can be segmented to represent conditions using different virtual zones to aid in detection and localization. Zones can be selected to correspond to critical areas where dangerous situations may be expected, such as sidewalks, sidewalk entrances, and road approaches 405, 406, 407, 408 to the intersection. Behavior and other conditions in each zone are recorded. Recordings can include, but are not limited to, kinematics (e.g., position, heading, speed, and acceleration), and facial and body features (e.g., eyes, posture).

[0165] The number of sensors, the number of zones, and the shape of the zones are specific to each intersection and to each approach to the intersection.

[0166] FIG. 5 shows a floor plan of a typical exemplary setup illustrating different zones used to monitor and track the movements and behavior of pedestrians or other vulnerable road users, motorized and non-motorized vehicles, and other ground traffic entities.

[0167] The sensors are configured to monitor a crosswalk across a road. Virtual zones (301, 302) may be located on the sidewalk and along the crosswalk. Other sensors are positioned to monitor vehicles and other ground traffic entities traveling on the path leading to the crosswalk, and virtual zones (303, 304) are strategically placed to help detect, for example, incoming vehicles and other ground traffic entities, their distance from the crosswalk, and their speed.

[0168] The system (e.g., the RSE or SRSE associated with the sensors) collects streams of data from all the sensors. When the system is first operationalized, an initial rule-based model may be deployed to aid in the calibration and functioning of the equipment. Meanwhile, sensor data (e.g., speed and distance from the radar unit, images and videos from the cameras) is collected and stored locally on the RSE in preparation, and in some embodiments, communicated to a remote computer powerful enough to use this collected data to build AI models of the behavior of different entities at the intersection. In some examples, the RSE itself is an SRSE capable of generating AI models.

[0169] Thus, data is prepared and trajectories are constructed for each ground traffic entity passing through the intersection. For example, trajectories can be derived from radar data by connecting together points of different distances that belong to the same entity. Pedestrian trajectories and behaviors can be derived, for example, from camera and video recordings. By implementing video and image processing techniques, pedestrian movements can be detected in images and videos and their respective trajectories can be estimated.

[0170] For human behavior, intelligent machine learning-based models typically outperform simple rules based on simple physics because human intent is difficult to capture and because large data sets are required to be able to detect patterns.

[0171] When the machine learning (AI) model is completed in the server, the AI ​​model is downloaded to the RSE, e.g., through the Internet. The RSE then applies current data captured from the sensors to the AI ​​model resulting in it predicting intent and behavior, identifying when a dangerous situation is imminent, and triggering a corresponding alert that is disseminated (e.g., broadcast) to vehicles and other ground traffic entities as an early warning and to potential road users and drivers in time to enable them to take collision avoidance steps.

[0172] This example setup can be combined with any other use case, for example traffic at a signalized intersection or at grade intersection.

[0173] Example 2: Intersection with traffic lights

[0174] For a signalized intersection (e.g., controlled by traffic lights), the overall setup of the system is as in Example 1. One difference may be the type of sensors used to monitor or track the vehicle's speed, direction, distance, and location. The setup for a pedestrian crossing in Example 1 may also be combined with the signalized intersection setup for a more general solution.

[0175] The operating concept for the traffic light intersection use case is to track road users around the intersection using external sensors that collect data about the users or data communicated by the users themselves, predict the user's behavior, and broadcast warnings through different communication means about approaching dangerous situations, typically due to violation of the traffic rules at the intersection, e.g. violating a red light.

[0176] Data on road users can be collected using (a) entity data broadcast by each entity itself, for example through BSM or PSM, regarding its current state, and (b) sensors installed externally on the infrastructure or on the vehicle, for example Doppler radar, ultrasonic sensors, visual or thermal cameras, lidar, etc. As mentioned above, the type of sensor selected and the position and orientation of the selected sensor at the intersection must provide the most extensive coverage of the intersection or the part under investigation, and furthermore the collected data on the entities approaching the intersection is the most accurate. The collected data thus allows the reconstruction of the current state of the road users and the generation of accurate, timely and useful VBSMs (virtual basic safety messages) or VPSMs (virtual personal safety messages). The frequency with which data must be collected depends on the potential danger of each type of road user and the criticality of the potential violations. For example, a motorized vehicle moving at high speed at an intersection typically requires 10 data updates per second to achieve real-time collision avoidance, while a pedestrian crossing the intersection at a much slower speed may require as few as 1 data update per second.

[0177] As explained above, FIG. 4 shows an example of a plan view of a signalized intersection with detection virtual zones. These zones may segment each approach to the intersection into separate lanes 410, 411, 412, 413, 405, 406, 407, 408, and may further divide each lane into areas corresponding to a general range of distances from the stop bar. The selection of these zones may generally be performed empirically to suit the characteristics of the particular approaches and intersections. Segmenting the intersection allows for more accurate identification of the relative heading, speed, acceleration, and position for each road user, which in turn allows for a better assessment of the potential hazards that the road users present to other ground traffic entities.

[0178] To determine whether an observed traffic situation is a dangerous situation, the system further needs to compare the results of the predicted situation with the state of the traffic lights and take into account local traffic rules (e.g., left turn lanes, right turns on red, etc.). Therefore, it is necessary to collect and use signal phase and timing (SPaT) information of the intersection. SPaT data can be collected by directly interfacing with the traffic light controller at the intersection, generally by reading the data through a wired connection, or by interfacing with a traffic management system to receive the requested data, for example through an API. In order to ensure that the state of the road users is always synchronized with the state of the traffic lights, it is important to collect SPaT data at a rate as close as possible to the rate at which the road user data is collected. An additional complication to the requirement of knowing SPaT information is that modern traffic control techniques used to regulate the flow of traffic near an intersection are not based on fixed timing, but use algorithms that can dynamically adapt to real-time traffic conditions. Therefore, it is important to incorporate SPaT data prediction algorithms to ensure the highest accuracy in violation prediction. These SPaT data prediction algorithms can be built using rule-based or machine learning methods.

[0179] For each approach to an intersection, data is collected by the RSE (or SRSE) and a machine learning (AI) model is constructed to explain vehicle behavior corresponding to the collected data. Current data collected at the intersection is then applied to the AI ​​model to generate an early prediction of whether a vehicle or other ground traffic entity traveling on one of the approaches to the intersection is about to violate, for example, a traffic light. If a violation is imminent, a message is relayed (e.g., broadcast) from the RSE to ground traffic entities in the vicinity. Vehicles (including the violating vehicle) and pedestrians or other potentially affected road users receive the message and have time to take appropriate preemptive measures to avoid a collision. The message may be delivered to the ground traffic entities by one or more of the following techniques: flashing lights, signs, or radio signals, among other possibilities.

[0180] If a vehicle or other entity approaching the intersection is equipped with an OBE or OPE, it can receive a broadcast message from the RSE that a potential hazard has been predicted at the intersection. This allows the user to be alerted and to take appropriate preemptive measures to avoid a collision. If a violation road user at the intersection is also equipped with an OBE or OPE, the user also receives the broadcast alert. An algorithm in the OBE or OPE can then match the user's violation behavior with the message and warn the user appropriately.

[0181] The decision to send an alert does not only depend on the vehicle's behavior as indicated by data collected by sensors at the intersection. While sensors play a large role in the decision, other inputs are also considered. These inputs may include, but are not limited to, information from nearby intersections (if a vehicle has run a red light at a nearby intersection, there is a higher probability that it will do the same at this intersection), information from other associated vehicles, or even information from the vehicle itself, for example if the vehicle reports that it is experiencing an anomaly.

[0182] Example 3: Intersection without traffic lights

[0183] Controlled intersections without traffic lights, e.g., intersections controlled by stop signs or yield signs, may be monitored as well. Sensors are used to monitor approaches controlled by traffic signs, and predictions can be made for incoming vehicles similar to predictions for incoming vehicles at approaches to signalized intersections. The rules of the road at controlled intersections without traffic lights are typically clearly defined. Ground traffic entities at approaches controlled by stop signs must come to a complete stop. At multi-way stop intersections, the order in which ground traffic entities reach the intersection determines the right-of-way. Special cases with one-way stops may be considered. A set of sensors may monitor approaches without stop signs as well. Such a configuration may assist in stop sign gap negotiation. For intersections controlled by yield signs, ground traffic entities at approaches controlled by yield signs must reduce their speed to grant right-of-way to other ground traffic entities at the intersection.

[0184] A major challenge is that due to internal factors (e.g., driver distraction) or external factors (e.g., lack of visibility), ground transportation entities violate the rules of the road and put other ground transportation entities at risk.

[0185] In the general example of a stop sign controlled intersection (i.e., each approach is controlled by a stop sign), the overall setup of the system is as in Example 1. One difference may be the type of sensors used to monitor or track the vehicle's speed, direction, distance, and position. Another difference is that the rules of the road are indicated by road signs, without including a traffic light control device. The setup for the pedestrian crossing in Example 1 may also be combined with the setup for a non-traffic light controlled intersection for a more general solution.

[0186] 4 may also be understood to show an example of a plan view of a four-way stop intersection including detection virtual zones. These zones may segment each approach to the intersection into separate lanes 410, 411, 412, 413, 405, 406, 407, 408, and may further divide each lane into areas corresponding to common ranges of distances from the stop bar. The selection of these zones may generally be done empirically to suit the characteristics of the particular approaches and intersection.

[0187] In a similar manner as described above for Figure 4, current data collected at the intersection is applied to an AI model to generate an early prediction of whether a vehicle or other ground traffic entity traveling on one of the approaches to the intersection is about to violate a stop sign. If a violation is imminent, the message may be treated similarly to the previous example involving a traffic light violation.

[0188] Further, as discussed above, the decision to send an alert may be based on the factors discussed above, as well as other information, such as whether the vehicle has run a stop sign at a nearby intersection (suggesting that the vehicle is more likely to do the same at this intersection).

[0189] Figure 18 shows a use case for an intersection without controlled traffic lights. It illustrates how an SRSE containing strategically placed sensors can alert connected entities to impending dangerous situations originating from unconnected entities.

[0190] A connected entity 9106 is traveling along path 9109. Entity 9106 has the right-of-way. An unconnected entity 9107 is traveling along path 9110. Entity 9107 is subject to a yield sign 9104, which is attempting to merge with path 9109 without granting entity 9106 the right-of-way, and is attempting to place entity 9107 directly into entity 9106's path. A dangerous situation is imminent because entity 9106 does not see entity 9107. Because entity 9107 is an unconnected entity, entity 9107 cannot advertise (broadcast) its location and heading to other entities sharing the intersection. Furthermore, entity 9106 may not be able to "see" entity 9107 that is not in direct line of sight of entity 9106. If entity 9106 proceeds along its path, entity 9106 may eventually collide with entity 9107.

[0191] As the intersection is a smart intersection, a radar 9111 mounted on a beam 9102 above the road detects the entity 9107. The radar 9111 also detects the speed and distance of the entity 9107. This information can be relayed as an alert through the SRSE 9101 to the connected entities 9106. The SRSE 9101 contains a machine learning model for entities moving along the approach road 9110. The entity 9107 is classified by the model as a potential violator of traffic rules and a warning (alert) is broadcast to the connected entities 9106. This warning is sent in advance to give the entity 9106 enough time to react to and prevent a dangerous situation.

[0192] Example 4: At-grade intersection

[0193] At-grade intersections are dangerous because they can carry motorized vehicles, pedestrians, and rail vehicles. Often, the roads leading to the at-grade intersection are in the blind spots of train or other rail vehicle operators (e.g., conductors). Because rail vehicle operators operate primarily based on line-of-sight information, this increases the likelihood of accidents when road users violate the right-of-way of rail vehicles and when road users cross an at-grade intersection when they are not authorized to cross.

[0194] The behavior of the grade crossing use case is similar to the signalized intersection use case in the sense that grade crossings are often a collision point between road and rail traffic regulated by traffic rules and signals. Therefore, this use case also requires collision avoidance warnings to increase safety near grade crossings. Rail traffic may have a planned separated rail right-of-way (e.g., high speed rail) or not (e.g., light urban rail or tram). When using light rail and tram, this use case becomes even more important since these rail vehicles operate on more active roads and need to follow the same traffic rules as road users.

[0195] Figure 6 shows a common use case for an at-grade intersection where a road and pedestrian crossing cross a railroad. As with the pedestrian crossing use case, sensors are deployed to collect data on pedestrian movements and intent. Other sensors are used to monitor and predict the movements of road vehicles approaching the intersection. Data on road users may also be collected from road user broadcasts (e.g., BSM or PSM). Data from nearby intersections, vehicles, and a remote command and control center may be used to make decisions to trigger alerts.

[0196] In order to properly assess possible violations, data on SPaT for road and rail approaches also needs to be collected.

[0197] Similar to the signalized intersection use case, the collected data allows for the generation of predictive models using rule-based and machine learning algorithms.

[0198] In this use case, the rail vehicle is equipped with an OBE or OPE to receive collision avoidance warnings. If a violation of the rail vehicle's right-of-way is predicted, the RSE broadcasts an alarm message, alerting the rail vehicle driver that a road user is on its intended path, allowing the rail vehicle driver to take preemptive action with enough time to avoid a collision.

[0199] If the violation road user is further equipped with an OBE or an OPE, the message broadcast by the RSE is further received by the violation road user. The algorithm in the OBE or OPE can then match the user's violation behavior with the received message and warn the user appropriately.

[0200] (Bridge the Gap) Virtual Connected Ground Transportation Environment

[0201] As described thus far, a useful application of the system is to generate a virtual connected environment in place of unconnected ground traffic entities. Obstacles to the adoption of connected technologies are not only the lack of infrastructure installations, but also the near-existence of connected vehicles, connected vulnerable road users, and other connected ground traffic entities.

[0202] In relation to connected vehicles, in some regulatory regimes, such vehicles constantly transmit what are called basic safety messages (BSMs). The BSMs include, among other information, the vehicle's location, direction of travel, speed, and future route. Other connected vehicles can pay attention to these messages and use them to generate a map of the vehicles present in their environment. By knowing where the surrounding vehicles are, vehicles have information that is useful to maintain a high level of safety, whether they are autonomous or not. For example, an autonomous vehicle can maneuver to avoid a vehicle connected to its path. Similarly, the driver can be alerted if there is any other vehicle on the path the driver plans to follow, such as in the case of a sudden lane change.

[0203] Until all ground traffic entities are equipped to transmit and receive traffic safety messages and information, some road entities will be "dark" or invisible to the rest of the road entities. Dark road entities pose a risk of hazardous conditions.

[0204] Dark road entities do not advertise (e.g., broadcast) their location, and therefore are not visible to connected entities that may expect all road entities to broadcast their information (i.e., being connected entities). On-board sensors may detect obstacles and other road entities, but the range of these sensors tends to be too short to be effective in preventing dangerous situations and collisions. Thus, there is a gap between the connectivity of connected vehicles and the lack of connectivity of unconnected vehicles. The technology described below aims to bridge this gap by using intelligence about the infrastructure that can detect all vehicles or other components of the ground transportation network at an intersection and send messages on behalf of unconnected vehicles.

[0205] The system may establish a virtual connected ground traffic environment that may bridge the gap between a future where most vehicles (and other ground traffic entities) are envisioned to be connected entities and the present where most vehicles and other ground traffic entities have no connectivity, for example at intersections. In the virtual connected ground traffic environment, smart traffic lights and other infrastructure installations may use sensors to track all vehicles and other ground traffic entities (connected, disconnected, semi-autonomous, autonomous, non-autonomous) and (in the case of vehicles) generate virtual BSM messages (VBSMs) on their behalf.

[0206] A VBSM message may be considered a subset of a BSM. It may not contain all the fields required to generate a BSM, but may include all the localization information including position, heading, speed, and trajectory. Because V2X communications are standardized and anonymous, VBSM and BSM cannot be easily distinguished and follow the same message structure. The main difference between the two messages is the availability of the source of the information contained in these messages. VBSM may not include data and information that is not easily generated by external sensors, such as steering angle, brake status, tire pressure, or wiper activation.

[0207] With the appropriate sensors installed, an intersection including a smart RSE may detect all road entities moving through the intersection. The SRSE may further convert all data from the multiple sensors into a global integrated coordinate system. This global integrated system is represented by the geographic location, speed, and direction of travel of each road entity. Each road entity is detected by the intersection equipment, whether it is connected or not, and a global integrated position is generated on behalf of each road entity. Thus, standard safety messages may be broadcast on behalf of road entities. However, if the RSE broadcasts safety messages for all entities it detects, the RSE may send messages on behalf of connected road entities. To resolve conflicts, the RSE may filter connected road entities from its list of dark entities. This may be accomplished because the RSE continuously receives safety messages from connected vehicles and because the RSE sensors are continuously detecting road entities passing through the intersection. If the detected road entity location matches a location from which a safety message was received by the RSE receiver, then the road entity is inferred to be connected and no safety messages are broadcast by the RSE on behalf of that road entity. This is shown in Figure 15.

[0208] By creating a bridge between connected and non-connected vehicles, connected entities (including autonomous vehicles) may safely maneuver through intersections with full awareness of all nearby road entities.

[0209] This aspect of the technology is illustrated in Figure 17. An intersection 9001 contains multiple road entities at a given time. Some of these entities are unconnected 9004, 9006 and others are connected 9005, 9007. Vulnerable road users 9004, 9007 are detected by a camera 9002. Motorized road entities 9005, 9006 are detected by a radar 9003. The location of each road entity is calculated. Broadcasts from connected road entities are further received by an RSE 9008. The location of the entity from which a message was received is compared to the location at which the entity was detected. If the two entities match within a predefined tolerance, the entity at that location is considered connected and no safety messages are sent on behalf of that entity. The rest of the road entities without a matching receiving location are considered dark. Safety messages are broadcast on their behalf.

[0210] For collision warning and intersection violation warning, which are an integrated part of the V2X protocol, each entity needs to be connected for the system to be effective. That requirement poses a hurdle in the deployment of V2X devices and systems. Intersections with smart RSEs solve that concern by providing a virtual bridge between connected and non-connected vehicles.

[0211] The U.S. Department of Transportation (DOT) and National Highway Traffic Safety Administration (NHTSA) have identified many connected vehicle applications that use BSM to help substantially reduce non-impaired crashes and fatalities. These applications include, but are not limited to, Forward Collision Warning (FCW), Intersection Movement Assist (IMA), Left Turn Assist (LTA), Do Not Pass Warning (DNPW), and Blind Spot / Lane Change Warning (BS / LCW). The U.S. DOT and NHTSA have defined these applications as follows:

[0212] FCW resolves rear-end collisions and warns drivers of stopped, slowing, or slower vehicles ahead. IMA is designed to avoid intersection crossing collisions and warns drivers of vehicles approaching an intersection from the side, covering two main scenarios: same- or opposite-direction turn-arounds and straight-line crossings. LTA resolves collisions when one involved vehicle turns left at an intersection and another vehicle moves straight-line from the opposite direction, and warns drivers of the presence of oncoming opposite-direction traffic when attempting to make a left turn. DNPW assists drivers to avoid reverse collisions caused by overtaking maneuvers and warns drivers of oncoming opposite-direction vehicles when attempting to overtake a slower vehicle on an undivided two-lane roadway. BS / LCW resolves collisions when vehicles perform lane-changing / merging maneuvers before a collision and alerts drivers of oncoming vehicles or the presence of vehicles in their blind spots in nearby lanes.

[0213] The V2X protocol specifies that these applications must be accomplished using vehicle-to-vehicle (V2V) communications, where one connected remote vehicle broadcasts basic safety messages to a connected host vehicle. The host vehicle's OBE then attempts to match its own vehicle parameters, e.g., speed, heading, and trajectory, with these BSMs to determine whether there is a potential danger or threat posed by the remote vehicle, as described herein thus far. Furthermore, autonomous vehicles particularly benefit from such applications, as they allow surrounding vehicles to communicate their intent, a key piece of information not included in the data collected from their on-board sensors.

[0214] However, current vehicles are not connected, and as noted above, it will take a very long time before there is a high percentage of connected vehicles for the BSM to function properly as described above. Thus, in an environment with a small percentage of connected vehicles, there is no need for connected vehicles to receive and analyze the large number of BSMs that they would receive if the percentage of connected vehicles was high enough to enable the above-described applications and to fully benefit from V2X communications.

[0215] VBSMs may help bridge the gap between a current environment with mostly unconnected entities and a future environment with mostly connected entities, enabling the above-mentioned applications in the interim. In the technology described herein, a connected vehicle that receives a VBSM processes the VBSM as a regular BSM in that application. Because VBSMs and BSMs follow the same message structure, and because VBSMs contain substantially the same basic information as BSMs, e.g., speed, acceleration, heading, past trajectory, and predicted trajectory, the results of applying the message to a given application are substantially the same.

[0216] For example, consider an intersection with an unprotected left turn where a connected host vehicle is attempting to make a left turn at the moment when an unconnected remote vehicle is moving in a straight line from the opposite direction with the right-of-way. This is a situation where the execution of the maneuver depends on the host vehicle's driver's judgment of the situation, and an improper assessment of the situation may result in conflict and a potential collisional approach or collision. External sensors installed in the surrounding infrastructure may detect and track the remote vehicle or even both vehicles, collect basic information such as speed, acceleration, heading, and past trajectory, and transmit them to the RSE, which may then use rule-based and / or machine learning algorithms to build a predicted trajectory for the remote vehicle, fill in the required fields for the VBSM, and broadcast it on behalf of the unconnected remote vehicle. The host vehicle's OBE receives the VBSM containing information about the remote vehicle and processes the VBSM in its LTA application to determine whether the driver's maneuver poses a potential hazard and whether the OBE should display a warning to the host vehicle's driver to take proactive or corrective action to avoid the collision. A similar result can be achieved if a remote vehicle is connected and receives data from the RSE and sensors that an oncoming vehicle is about to make a left turn that predicts a collision.

[0217] VBSM can also be used in lane-changing maneuvers. Such maneuvers can be dangerous if the vehicle changing lanes does not take the necessary steps to ensure the maneuver is safe, e.g., by checking the rearview and side mirrors and blind spots. New advanced driver assistance systems, e.g., blind spot warnings using on-board ultrasonic sensors, have been developed to help prevent vehicles from performing dangerous lane changes. However, these systems can have shortcomings when the sensors are dirty or have an obstructed view. Furthermore, existing systems do not attempt to warn the endangered vehicle about another vehicle that is about to change lanes. While V2X communication can help solve this problem through applications such as BS / LCW using BSM, the vehicle that is about to change lanes may be an unconnected vehicle and thus unable to communicate its intentions. VBSM can help achieve that goal. Similar to the LTA use case, external sensors installed in the surrounding infrastructure can detect and track unconnected vehicles that are about to perform lane-changing maneuvers, collect basic information such as speed, acceleration, heading and past trajectory, and transmit them to the RSE. The RSE then uses rule-based and machine learning algorithms to build a predicted trajectory for the lane-changing vehicle, fills in the required fields for the VBSM, and broadcasts it instead of the unconnected, distant vehicle. The OBE of the at-risk vehicle then receives the VBSM containing information about the vehicle attempting to merge into the same lane, processes the VBSM, and determines whether the maneuver poses a potential hazard and whether it should display a lane-changing warning to the vehicle's driver. If the lane-changing vehicle is a connected vehicle, its OBE may similarly receive VBSMs from the RSE for vehicles in its blind spot and determine whether the lane-changing maneuver poses a potential hazard to surrounding traffic and whether it should display a blind spot warning to the vehicle's driver.When both vehicles are connected, they can broadcast their BSM to each other, enabling BS / LCW applications, which would still benefit from applying the same rule-based and / or machine learning algorithms to the BSM data as described above for early prediction of the lane-changing vehicle's intent, with the OBE deciding whether to display a warning or not.

[0218] Autonomous Vehicles

[0219] The connectivity that unconnected road entities lack impacts autonomous vehicles. Sensors in autonomous vehicles have a short range or narrow field of view. They cannot detect vehicles coming near buildings, for example at street corners. They also cannot detect vehicles that may be hidden behind delivery trucks. If these hidden vehicles are unconnected entities, they are not visible to autonomous vehicles. These circumstances impact the ability of autonomous vehicle technology to achieve the level of safety required for large-scale adoption of the technology. Smart intersections can help mitigate this gap and aid in the acceptance of autonomous vehicles by the public. An autonomous vehicle is only as good as the sensors it has. An intersection with a smart RSE can extend the coverage of on-board sensors around poorly visible corners or beyond large trucks. Such extensions allow autonomous entities and other connected entities to coexist with traditional unconnected vehicles. Such coexistence can accelerate the adoption of autonomous vehicles and the benefits they bring.

[0220] The virtual connected ground traffic environment includes VBSM messages that enable the implementation of vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), and vehicle-to-device (V2D) applications that would be difficult to implement using other approaches.

[0221] The system may use machine learning to quickly and accurately generate fields of data required for various safety messages, pack them into a VBSM message structure, and transmit the messages to nearby ground transportation entities using various mediums, such as, but not limited to, DSRC, Wi-Fi, cellular, or traditional road signs.

[0222] Virtual personal safety messages (VPMS)

[0223] The ground traffic environment may include not only disconnected vehicles, but also disconnected people and other potentially vulnerable road users.

[0224] In some regulatory regimes, connected vulnerable ground transportation entities continuously transmit Personal Safety Messages (PSMs). The PSMs include, among other information, the location, heading, speed, and future route of the vulnerable ground transportation entity. Connected vehicles and infrastructure can receive these messages and use them to generate maps that include vulnerable entities and to increase the level of safety in the ground transportation network.

[0225] Thus, a virtual connected ground traffic environment may bridge the gap between a future in which most vulnerable ground traffic entities are assumed to be connected and a present in which most vulnerable ground traffic entities have no connectivity. In a virtual connected ground traffic environment, smart traffic lights and other infrastructure installations may use sensors to track all vulnerable ground traffic entities (connected and unconnected) and generate VPSMs on their behalf.

[0226] A VPSM message may be considered a subset of a PSM. A VPSM need not include all fields required to generate a PSM, but may include data necessary for safety assessment and prevention of unsafe situations, and may include location specific information including position, heading, speed, and trajectory. In some instances, non-standard PSM fields may also be included in a VPSM, such as the driver's intent, posture, or looking direction.

[0227] The system can use machine learning to quickly and accurately generate these fields, pack them into a VPSM message structure, and transmit it to nearby ground traffic entities using a variety of mediums, such as, but not limited to, DSRC, Wi-Fi, cellular, or traditional road signs.

[0228] VPSM messages enable the implementation of pedestrian to vehicle (P2V), pedestrian to infrastructure (P2I), pedestrian to devices (P2D), vehicle to pedestrian (V2P), infrastructure to pedestrians (I2P), and devices to pedestrians (D2P) applications that were difficult to implement using other approaches.

[0229] FIG. 16 shows a pedestrian 8102 crossing a crosswalk 8103. The crosswalk 8103 may be at an intersection or a mid-block crosswalk that spans a stretch of road between intersections. A camera 8101 is used to monitor the walkway 8104. The global position of the boundary of the field of view 8105 of the camera 8101 may be determined at installation. The field of view 8105 is covered by a certain number of pixels reflected by the specifications of the camera 8101. A road entity 8102 may be detected within the field of view of the camera and its global position may be calculated. The speed and direction of the road entity 8102 may further be determined from its displacement at time s. The path of the road entity 8102 may be represented by a breadcrumb 8106, which is a sequence of positions crossed by the entity 8102. This data may be used to construct a virtual PSM message. The PSM message may then be broadcast to all entities near the intersection.

[0230] Traffic and behavior enforcement at unsignalized intersections

[0231] Another useful application of the system is traffic enforcement at intersections without signals (eg, stop signs, yield signs) and enforcement of good driving behavior anywhere on the ground transportation network.

[0232] As a by-product of generating the VBSM and VPSM, the system may track and detect road users who do not comply with traffic laws and increase the probability of unsafe situations and collisions. Prediction of unsafe situations may be extended to include enforcement. Unsafe situations do not necessarily result in collisions. Near misses are common and may increase driver stress levels and lead to subsequent accidents. The frequency of near misses is positively correlated to lack of enforcement.

[0233] Additionally, using the VBSM, the system may detect inappropriate driving behaviors, such as abrupt lane changes and other forms of reckless driving. Data collected by the sensors may be used to train and enable machine learning models to flag ground traffic entities that are engaging in risky driving behaviors.

[0234] A law enforcement agency typically enforces the rules of the road against ground traffic entities that contain vulnerable road users, but the agency needs to be near an intersection to monitor, detect, and report violations. By tracking unconnected ground traffic entities that contain vulnerable road users using VBSM and VPSM, a smart RSE can act as a law enforcement agency and enforce the rules of the road at an intersection. For example, an unconnected vehicle tracked by a smart RSE can be detected, identified, and reported to authorities for violating a stop sign or a yield sign. Similarly, a vulnerable road user near an intersection tracked by a smart RSE can be detected, identified, and reported to authorities for illegally crossing the intersection.

[0235] For enforcement and other purposes, ground transportation entities may be identified using unique identification information, including but not limited to plate number recognition. Potential victim road users may be identified using biometric recognition, including but not limited to facial identification, retinal identification, and voice waveform identification. In special cases, including civil investigations or criminal investigations, social media networks (e.g., Facebook, Instagram, Twitter) may further be used to support the identification of violating ground transportation entities or potential victim road users. An example of a social network being utilized is uploading a captured photo of a violator to the social network and requesting users of the social network who recognize the violator to provide information to law enforcement authorities that will help identify the violator.

[0236] Other embodiments are within the scope of the following claims. (Additional note 1) An apparatus comprising a device located at an intersection of a transportation network, The device, an input for receiving data from a sensor oriented to monitor ground traffic entities at or near the intersection; a wireless communication device that transmits a warning to a device of one of the ground transportation entities regarding a hazardous condition at or near the intersection; A processor; Storage and Equipped with The storage, storing a machine learning model capable of predicting a behavior of a ground traffic entity at or near the intersection at a current time, the machine learning model being based on training data relating to previous actions and associated behaviors of a ground traffic entity at or near the intersection; applying current motion data received from the sensors regarding ground traffic entities at or near the intersection to the machine learning model to predict impending behavior of the ground traffic entities; inferring from the predicted impending behavior an imminent hazardous situation for one or more of the ground traffic entities at or near the intersection; causing the wireless communication device to transmit the warning regarding the hazardous situation to the device of one of the ground transportation entities; the storage for instructions executable by the processor to Device. (Additional note 2) the equipment comprises a roadside equipment; 2. The device according to claim 1. (Additional note 3) the apparatus comprising a housing for the device; The sensor is mounted in the housing. 2. The device according to claim 1. (Additional note 4) transmitting the alert by broadcasting the alert for receipt by any of the ground traffic entities at or near the intersection; 2. The device according to claim 1. (Additional note 5) the machine learning model comprises an artificial intelligence model; 2. The device according to claim 1. (Additional note 6) the training data and the motion data include at least one of speed, position, heading, intent, attitude, or looking direction; 2. The device according to claim 1. (Additional note 7) the processor is configured to enable generation of the machine learning model at the device; 2. The device according to claim 1. (Additional note 8) The machine learning model is generated on the device. 2. The device according to claim 1. (Additional note 9) the instructions are executable by the processor to store the training data at the device. 2. The device according to claim 1. (Additional note 10) The intersection includes an intersection without a traffic light. 2. The device according to claim 1. (Additional note 11) The transportation network includes a road network. 2. The device according to claim 1. (Additional note 12) the ground transportation entities include road users that may be affected; 2. The device according to claim 1. (Additional note 13) the ground transportation entity includes a vehicle; 2. The device according to claim 1. (Additional note 14) The imminent dangerous situation includes a collision or near miss, 2. The device according to claim 1. (Additional note 15) the ground traffic entities include pedestrians crossing a roadway at a crosswalk; 2. The device according to claim 1. (Additional note 16) a separate communication device for communicating with the central server; 2. The device according to claim 1. (Additional note 17) the device of one of the ground transportation entities includes a mobile communications device; 2. The device according to claim 1. (Additional note 18) An apparatus comprising equipment located in or on a ground transportation entity, The device, an input for receiving data from a sensor in or on the ground transportation entity oriented to monitor nearby features of a ground transportation network and other information related to the context in which the ground transportation entity traverses the ground transportation network; a wireless communication device for receiving information regarding the context; a signal processor that applies signal processing to data from the sensor and other information related to the context; A processor; Storage and Equipped with The storage, storing a machine learning model capable of predicting the behavior of an operator of the ground transportation entity and the intent and movements of other ground transportation entities in the vicinity; applying the currently received data from the sensors and other information about the context to predict the behavior of the operator and the intent and movements of other ground traffic entities in the vicinity; a storage for instructions executable by the processor to Device. (Additional note 19) the instructions are executable by the processor to monitor users or occupants of the ground transportation entity. 19. The device according to claim 18. (Additional note 20) the other information relating to the context includes emergency broadcasts, traffic and safety messages from roadside equipment, and messages relating to safety, location, and other operational information from other ground transportation entities; 19. The device according to claim 18. (Additional note 21) The sensors include cameras, distance sensors, vibration sensors, microphones, seat sensors, hydrocarbon sensors, volatile organic compounds and other toxic substances sensors, and kinematic sensors, or combinations thereof; 19. The device according to claim 18. (Additional note 22) and instructions executable by the processor to filter the alerts received at the vehicle by applying the alerts to the machine learning models to predict which alerts are significant in terms of current location, environmental conditions, driver behavior, vehicle health and status, and kinematics. 19. The device according to claim 18. (Additional note 23) the wireless communication device transmitting a warning to a sign or other infrastructure presentation device regarding a dangerous situation; 19. The device according to claim 18. (Additional note 24) the warning includes an instruction or command capable of controlling a particular ground transportation entity; 19. The device according to claim 18. (Additional note 25) storing a machine learning model capable of predicting a behavior of a ground transportation entity at a current intersection of a transportation network or near the intersection of the transportation network, the machine learning model being based on training data related to previous actions and associated behaviors of the ground transportation entity at the intersection or near the intersection; applying current motion data received from sensors relating to ground traffic entities at or near the intersection to the machine learning model to predict impending behavior of the ground traffic entities; inferring from the predicted impending behavior an imminent hazardous situation for one or more of the ground traffic entities at or near the intersection; causing a wireless communication device to transmit a warning regarding the hazardous condition to a device of one of the ground transportation entities; The method includes: (Additional note 26) transmitting the alert by broadcasting the alert for receipt by any of the ground traffic entities at or near the intersection; The method according to claim 25. (Additional note 27) the machine learning model comprises an artificial intelligence model; The method according to claim 25. (Additional note 28) the training data and the motion data include at least one of speed, position, heading, intent, attitude, or looking direction; The method according to claim 25. (Additional note 29) generating the machine learning model on a device. The method according to claim 25. (Additional note 30) storing the training data in a device. The method according to claim 25. (Additional note 31) receiving data from a sensor in or on a ground transportation entity oriented to monitor nearby features of a ground transportation network and other information related to a context in which the ground transportation entity traverses the ground transportation network; receiving information regarding the context; storing a machine learning model capable of predicting the behavior of an operator of the ground transportation entity and the intent and movements of other ground transportation entities in the vicinity; applying the currently received data from the sensors and other information about the context to predict the operator's behavior and the intent and movements of other ground traffic entities in the vicinity; A method comprising: (Additional note 32) monitoring users or occupants of said ground transportation entity; The method according to claim 31. (Additional note 33) the other information relating to the context includes emergency broadcasts, traffic and safety messages from roadside equipment, and messages relating to safety, location, and other operational information from other ground transportation entities; The method according to claim 31. (Additional note 34) The sensors include cameras, distance sensors, vibration sensors, microphones, seat sensors, hydrocarbon sensors, volatile organic compounds and other toxic substances sensors, and kinematic sensors, or combinations thereof; The method according to claim 31. (Additional note 35) filtering the alerts received at the vehicle by applying machine learning models to predict which alerts are significant in terms of current location, environmental conditions, driver behavior, vehicle health and status, and kinematics; The method according to claim 31. (Additional note 36) including transmitting warnings about hazardous conditions to signs or other infrastructure presentation devices; The method according to claim 31. (Additional note 37) the warning includes an instruction or command capable of controlling a particular ground transportation entity; 32. The device according to claim 31. (Additional note 38) In road vehicles moving within a ground transportation network, receiving messages from external sources regarding the location, operation, and status of other ground transportation entities; receiving data from on-board sensors regarding road and driving conditions and regarding positions of static objects and moving ground traffic entities in the vicinity of said vehicle; receiving data relating to a quality of driving by a driver of said road vehicle; receiving basic safety messages from other ground transportation entities and personal safety messages from vulnerable road users; aggregating the received data and the message; and applying the fused data and the messages to an artificial intelligence model to predict a behaviour of a driver of the road vehicle or a behaviour of a vulnerable road user, or a collision risk to the road vehicle, or both a behaviour of a driver of the road vehicle or a behaviour of a vulnerable road user and a collision risk to the road vehicle; A method comprising: (Additional note 39) generating, at the road vehicle, a map of the stationary objects and the moving ground traffic entities in a vicinity of the road vehicle; The method according to claim 38. (Additional note 40) alerting the driver of the road vehicle of a collision risk. The method according to claim 38. (Additional note 41) determining said collision risk based on probabilities of predicted trajectories of other nearby moving ground traffic entities; The method according to claim 38. (Additional note 42) and filtering received basic safety messages and personal safety messages to reduce a number of alerts provided to the driver of the road vehicle. The method according to claim 38. (Additional note 43) acquiring motion data for unconnected ground transportation entities moving in the transportation network; transmitting a virtual safety message to a connected ground transportation entity in the vicinity of the unconnected ground transportation entity, the virtual safety message incorporating information regarding the operational data for the unconnected ground transportation entity; A method comprising: (Additional note 44) the virtual safety message is in lieu of a safety message that would be transmitted by the unconnected ground transportation entity if the unconnected ground transportation entity were connected; The method according to claim 43. (Additional note 45) the unconnected ground transportation entities include vehicles; the virtual safety message is a substitute for a primary safety message; The method according to claim 43. (Additional note 46) the unconnected ground transportation entities include vulnerable road users; the virtual safety message is a substitute for a personal safety message; The method according to claim 43. (Additional note 47) the operational data is acquired by infrastructure sensors; 44. The method according to claim 43. (Additional note 48) An apparatus comprising a device located at an intersection of a transportation network, The device, an input for receiving data from a sensor oriented to monitor ground traffic entities at or near the intersection; a wireless communication device that transmits a warning to a device of one of the ground transportation entities regarding a hazardous condition at or near the intersection; A processor; Storage and Equipped with The storage, storing a machine learning model capable of predicting a behavior of a ground traffic entity at or near the intersection at a current time, the machine learning model being based on training data relating to previous actions and associated behaviors of a ground traffic entity at or near the intersection; applying current operational data received from the sensors regarding ground traffic entities at or near the intersection to the machine learning model to predict impending behavior of the ground traffic entities, including ground traffic entities whose participating devices are unable to receive warnings from the wireless communication device; inferring an imminent hazardous situation for a ground traffic entity whose involved devices are able to receive a warning from the wireless communication device, the imminent hazardous situation being a result of a predicted imminent behavior of the ground traffic entity that is unable to receive the warning; transmitting the warning regarding the hazardous situation to the device of the ground transportation entity that may receive the warning from the wireless communication device; a storage for instructions executable by the processor to Device. (Additional note 49) the equipment comprises a roadside equipment; 49. The apparatus according to claim 48. (Additional note 50) the apparatus comprising a housing for the device; The sensor is mounted in the housing. 49. The apparatus according to claim 48. (Additional note 51) transmitting the alert by broadcasting the alert for receipt by any of the ground traffic entities at or near the intersection that can receive the alert. 49. The apparatus according to claim 48. (Additional note 52) the machine learning model comprises an artificial intelligence model; 49. The apparatus according to claim 48. (Additional note 53) The intersection includes an intersection without a traffic light. 49. The apparatus according to claim 48. (Additional note 54) The intersection includes a traffic light intersection. 49. The apparatus according to claim 48. (Additional note 55) The transportation network includes a road network. 49. The apparatus according to claim 48. (Additional note 56) the ground transportation entities include road users that may be affected; 49. The apparatus according to claim 48. (Additional note 57) the ground transportation entity includes a vehicle; 49. The apparatus according to claim 48. (Additional note 58) the imminent dangerous situation includes a collision; 49. The apparatus according to claim 48. (Additional note 59) the ground transportation entity, the involved device of which is unable to receive the alert from the wireless communication device, comprises a vehicle; the ground transportation entities whose participating devices may receive the alert from the wireless communication device include a pedestrian crossing a road at a crosswalk; 49. The apparatus according to claim 48. (Additional note 60) a separate communication device for communicating with the central server; 49. The apparatus according to claim 48. (Additional note 61) the device of one of the ground transportation entities comprises a mobile communications device; 49. The apparatus according to claim 48. (Additional note 62) Using electronic sensors located near intersections of a ground transportation network to monitor the intersections and approaches to the intersections, the electronic sensors generating operational data regarding ground transportation entities moving on the approaches or at the intersections, one or more of the ground transportation entities being unable to transmit safety messages to other ground transportation entities in the vicinity of the intersections; transmitting a virtual safety message to one or more of the ground transportation entities capable of receiving the message based on the operational data generated by the electronic sensors; Incorporating information about one or more of the ground traffic entities that are unable to transmit safety messages into the virtual safety messages, the incorporated information in each of the virtual safety messages including at least one of a location, a heading, a speed, and a predicted future trajectory of one of the ground traffic entities that are unable to transmit safety messages; A method comprising: (Additional note 63) the embedded information includes a subset of information that would be embedded in a basic safety message or a personal safety message generated by the ground transportation entity if the ground transportation entity were capable of transmitting the basic safety message or the personal safety message; 63. The method according to claim 62. (Additional note 64) applying the generated operational data to a machine learning model operating on equipment located near the intersection to predict a trajectory of the ground transportation entity that is unable to transmit a safety message. 63. The method according to claim 62. (Additional note 65) at least one of the ground transportation entities includes a motorized vehicle; 63. The method according to claim 62. (Additional note 66) The machine learning model is provided by a remote server over the Internet to a device located near the intersection. 63. The method according to claim 62. (Additional note 67) The machine learning model is generated on a device located near the intersection. 63. The method according to claim 62. (Additional note 68) training a machine learning model using operational data generated by the sensors located near the intersection. 63. The method according to claim 62. (Additional note 69) transmitting operational data generated by the sensors located near the intersection to a server for use in training a machine learning model. 63. The method according to claim 62. (Additional note 70) using electronic sensors located near a crosswalk across a roadway to monitor an area within or near the crosswalk, the electronic sensors generating motion data related to vulnerable roadway users within or near the crosswalk; applying the generated motion data to a machine learning model operating on a device located near the crosswalk to predict when one of the vulnerable roadway users is about to enter the crosswalk; wirelessly transmitting a warning to at least one of a device associated with the vulnerable roadway user or a device associated with another ground transportation entity approaching the crosswalk on the roadway before the vulnerable roadway user enters the crosswalk; A method comprising: (Additional note 71) The potentially affected roadway users include pedestrians, animals, or cyclists; 71. The method according to claim 70. (Additional note 72) The device associated with the potentially vulnerable roadway user includes a smart watch or other wearable device, a smartphone, or another mobile device; 71. The method according to claim 70. (Additional note 73) the other ground transportation entities include motorized vehicles; 71. The method according to claim 70. (Additional note 74) the device associated with the other ground transportation entity includes a smartphone or another mobile device; 71. The method according to claim 70. (Additional note 75) the machine learning model is provided to the device located near the crosswalk by a remote server over the Internet; 71. The method according to claim 70. (Additional note 76) the machine learning model is generated on the device located near the crosswalk; 71. The method according to claim 70. (Additional note 77) training the machine learning model using motion data generated by the sensor located near the crosswalk. 71. The method according to claim 70. (Additional note 78) transmitting motion data generated by the sensor located near the crosswalk to a server for use in training the machine learning model. 71. The method according to claim 70. (Additional note 79) segmenting the motion data generated by the sensors located near the crosswalk based on corresponding zones in the vicinity of the crosswalk. 71. The method according to claim 70. (Additional note 80) using said electronic sensors to generate motion related data representative of a physical characteristic of said vulnerable road users; 71. The method according to claim 70. (Additional note 81) deriving trajectory information about the vulnerable road users from the motion data generated by the sensors. 71. The method according to claim 70. (Additional note 82) An apparatus comprising equipment located at a grade intersection of a transportation network, the at-grade intersection includes a road intersection, a pedestrian crossing, and a railroad track; The device, inputs for receiving data from sensors oriented to monitor road vehicles and pedestrians at or near the grade crossing and for receiving phase and timing data for signals on the road and signals on the railroad tracks; a wireless communication device that transmits a warning regarding a hazardous condition at or near the grade crossing to one of a ground transportation entity, a pedestrian, or a rail vehicle on the railroad track; A processor; Storage and Equipped with The storage, storing a machine learning model capable of predicting a behavior of ground traffic entities at or near the grade intersection at a current time, the machine learning model being based on training data relating to previous actions and associated behaviors of road vehicles and pedestrians at or near the grade intersection; applying current motion data received from the sensors relating to road vehicles and pedestrians at or near the grade intersection to the machine learning model to predict impending behavior of the road vehicles and pedestrians; inferring from the predicted impending behavior an imminent dangerous situation for a rail vehicle on the rail track at the intersection or on the rail track near the intersection; causing the wireless communication device to transmit the warning regarding the dangerous situation to at least one of the road vehicle, the pedestrian, and the rail vehicle devices; the storage for instructions executable by the processor to Device. (Additional note 83) The warning is transmitted to an on-board device of the rail vehicle. 83. The apparatus according to claim 82. (Additional note 84) the railroad tracks are on isolated railroad right-of-way; 83. The apparatus according to claim 82. (Additional note 85) The railroad tracks are not located on separate railroad rights-of-way; 83. The apparatus according to claim 82. (Additional note 86) the equipment comprises a roadside equipment; 83. The apparatus according to claim 82. (Additional note 87) the warning is transmitted by broadcasting the warning for reception by any of the ground traffic entities, the pedestrians, or the rail vehicles at or near the grade crossing; 83. The apparatus according to claim 82. (Additional note 88) The imminent dangerous situation includes a collision or near miss, 83. The apparatus according to claim 82. (Additional note 89) Receiving data from infrastructure sensors representative of a location and movement of a road vehicle being driven or a pedestrian being walked in a ground transportation network; receiving data in virtual basic safety messages and virtual personal safety messages relating to the state of the road vehicle and the pedestrian; applying the received data to a trained machine learning model to identify risky driving or walking behavior of one of the road vehicles or the pedestrians; reporting said unsafe driving or walking behavior to authorities; and A method comprising: (Additional note 90) identifying said road vehicle based on plate number recognition. 89. The method according to claim 89. (Additional note 91) identifying the pedestrian based on biometric recognition. 89. The method according to claim 89. (Additional note 92) identifying said road vehicles or said pedestrians based on social networking. 89. The method according to claim 89. (Additional note 93) using electronic sensors located near intersections of a ground transportation system to monitor the intersections and approaches to the intersections, the electronic sensors generating operational data regarding ground transportation entities moving on the approaches or at the intersections; defining separate virtual zones at the intersection and at the approaches to the intersection; segmenting the motion data according to the corresponding virtual zone to which the generated motion data relates; applying the generated operational data for each respective segment to a machine learning model operating on a device located near the intersection to predict an imminent hazardous situation at the intersection or at one of the approaches involving one or more of the ground traffic entities; wirelessly transmitting a warning to a device associated with at least one of the involved ground transportation entities before the imminent hazardous situation becomes an actual hazardous situation; A method comprising: (Additional note 94) the device associated with each of the ground transportation entities includes a wearable device, a smartphone, or another mobile device; 94. The method according to claim 93. (Additional note 95) at least one of the ground transportation entities includes a motorized vehicle; 94. The method according to claim 93. (Additional note 96) the machine learning model is provided to the device located near the intersection by a remote server over the Internet; 94. The method according to claim 93. (Additional note 97) the machine learning model is generated on the device located near the intersection; 94. The method according to claim 93. (Additional note 98) training the machine learning model using operational data generated by the sensor located near the intersection. 94. The method according to claim 93. (Additional note 99) transmitting the motion data generated by the sensors located near the intersection to a server for use in training the machine learning model. 94. The method according to claim 93. (Additional note 100) using the electronic sensor to monitor an area in or near a crosswalk across one of the approaches to the intersection. 94. The method according to claim 93. (Additional note 101) using said electronic sensors to generate motion related data representative of physical characteristics of vulnerable road users in the vicinity of a crosswalk; 94. The method according to claim 93. (Additional note 102) deriving trajectory information about vulnerable road users from the motion data generated by the sensors; 94. The method according to claim 93. (Additional note 103) a machine learning model exists for each of the approaches to the intersection; 94. The method according to claim 93. (Additional note 104) determining whether to transmit the alert further based on operational data generated by a sensor associated with another nearby intersection. 94. The method according to claim 93. (Additional note 105) determining whether to transmit the alert further based on information received from a ground traffic entity operating on the approach or a ground traffic entity operating at the intersection. 94. The method according to claim 93. (Additional note 106) The intersection is equipped with a traffic light; Information regarding the state of the signal is received; 94. The method according to claim 93. (Additional note 107) the intersection is not signalized and is controlled by one or more signs; 94. The method according to claim 93. (Additional note 108) The defined virtual zone includes one or more of the access routes controlled by the sign; The method according to claim 107. (Additional note 109) The sign includes a stop sign or a yield sign; The method according to claim 107. (Supplementary Note 110) one of the ground transportation entities includes a rail vehicle; The method according to claim 107. (Additional note 111) An apparatus comprising a device located at an intersection of a transportation network, The device, an input for receiving data from sensors oriented to monitor ground traffic entities at or near the intersection, the data from each of the sensors representing at least one location or operational parameter of at least one of the ground traffic entities, the data from each of the sensors being represented in a native format, and the data received from at least two of the sensors being inconsistent with respect to the location or operational parameter, or with respect to the native format, or with respect to both the location or operational parameter and the native format; A processor; Storage and Equipped with The storage, converting the data from each of the sensors into data having a common format that is independent of the native format of the data of the sensors; incorporating the data having the common format into a global, integrated representation of the ground traffic entities monitored at or near the intersection, the global, integrated representation including a position, a speed, and a heading of each of the ground traffic entities; determining a relationship between positions and operations of two of the ground transportation entities using the global integrated representation; and predicting a hazardous situation involving the two ground transportation entities; and sending a message to at least one of the two ground traffic entities alerting the at least one of the two ground traffic entities about the hazardous situation; the storage for instructions executable by the processor to Device. (Additional Note 112) the sensors include at least two of a radar, a lidar, and a camera; 12. The apparatus according to claim 111. (Additional note 113) the data received from one of the sensors includes image data of a field of view at successive instants; 12. The apparatus according to claim 111. (Additional note 114) the data received from one of the sensors includes reflection points in 3D space; 12. The apparatus according to claim 111. (Additional note 115) the data received from one of the sensors includes distance and velocity from the sensor; 12. The apparatus according to claim 111. (Additional note 116) the global integrated representation represents the positions of the ground transportation entities in a common reference frame; 12. The apparatus according to claim 111. (Additional note 117) the device comprises at least two of said sensors; The data is received from the at least two sensors; The two sensors are mounted at fixed locations at or near the intersection and have at least partially non-overlapping fields of view. 12. The apparatus according to claim 111. (Additional note 118) one of the sensors includes a radar; and converting the data includes determining a location of a ground traffic entity from a known location of the radar and a distance of the ground traffic entity from the radar. 18. The apparatus according to claim 117. (Additional note 119) one of the sensors includes a camera; converting the data includes determining a position of a ground traffic entity from a known position, a direction of view, and a tilt of the camera, and a position of the ground traffic entity within an image frame of the camera; 18. The apparatus according to claim 117. (Supplementary Note 120) receiving data from sensors oriented to monitor ground transportation entities at or near an intersection of a ground transportation network, the data from each of the sensors representing at least one position or operational parameter of at least one of the ground transportation entities, the data from each of the sensors being represented in a native format, and the data received from at least two of the sensors being inconsistent with respect to the position or operational parameter, with respect to the native format, or with respect to both the position or operational parameter and the native format; converting the data from each of the sensors into data having a common format that is independent of the native format of the data of the sensors; incorporating the data having the common format into a global, integrated representation of the ground traffic entities monitored at or near the intersection, the global, integrated representation including a position, a speed, and a heading of each of the ground traffic entities; determining a relationship between positions and operations of two of the ground transportation entities using the global integrated representation; and predicting a hazardous situation involving the two ground transportation entities; and sending a message to at least one of the two ground traffic entities alerting the at least one of the two ground traffic entities about the hazardous situation; A method comprising: (Additional note 121) the sensors include at least two of a radar, a lidar, and a camera; The method according to claim 120. (Additional note 122) the data received from one of the sensors includes image data of a field of view at successive instants; The method according to claim 120. (Additional note 123) the data received from one of the sensors includes reflection points in 3D space; The method according to claim 120. (Additional note 124) the data received from one of the sensors includes distance and velocity from the sensor; 121. The method according to claim 120. (Additional note 125) the global integrated representation represents the positions of the ground transportation entities in a common reference frame; 121. The method according to claim 120. (Additional note 126) the data is received from at least two of the two sensors mounted at fixed locations at or near the intersection and having at least partially non-overlapping fields of view; 121. The method according to claim 120. (Additional note 127) one of the sensors includes a radar; the method including transforming the data, the transforming including determining a position of a ground traffic entity from a known position of the radar and a distance of the ground traffic entity from the radar; The method according to claim 126. (Additional note 128) one of the sensors comprises a camera; the method includes transforming the data, including determining a position of a ground traffic entity from a known position, a direction of view, and a tilt of the camera, and a position of the ground traffic entity within an image frame of the camera; The method according to claim 126. (Aspect 1) using electronic sensors located near a crosswalk across a roadway to monitor an area within and near the crosswalk, the electronic sensors generating motion data regarding vulnerable roadway users within or near the crosswalk, the motion data including a location and a direction of the vulnerable roadway users; applying the generated motion data to a machine learning model operating on a device located near the crosswalk to make a prediction of one of the vulnerable roadway users intending to enter the road within the crosswalk or the road near the crosswalk, the prediction being made before the vulnerable roadway user enters the road within the crosswalk or the road near the crosswalk, the machine learning model being trained using motion data generated in the vicinity of the crosswalk, the motion data indicative of intent or behavior of a vulnerable roadway user who was previously in or near the crosswalk, the motion data including position, speed, acceleration, and orientation; sending a warning to at least one of a device associated with the vulnerable roadway user, a device associated with another ground transportation entity approaching the crosswalk on the roadway, and a road sign configured to alert the vulnerable roadway user or a driver before the vulnerable roadway user enters the roadway within the crosswalk or near the crosswalk; A method comprising: (Aspect 2) The potentially affected roadway users include pedestrians, animals, or cyclists; The method according to embodiment 1. (Aspect 3) The device associated with the potentially vulnerable roadway user includes a smart watch or other wearable device, a smartphone, or another mobile device; The method according to embodiment 1. (Aspect 4) the other ground transportation entities include motorized vehicles; The method according to embodiment 1. (Aspect 5) the device associated with the other ground transportation entity includes a smartphone or another mobile device; The method according to embodiment 1. (Aspect 6) the machine learning model is provided to the device located near the crosswalk by a remote server over the Internet; The method according to embodiment 1. (Aspect 7) the machine learning model is generated on the device located near the crosswalk; The method according to embodiment 1. (Aspect 8) training the machine learning model using motion data generated by the sensor located near the crosswalk. The method according to embodiment 1. (Aspect 9) transmitting motion data generated by the sensor located near the crosswalk to a server for use in training the machine learning model. The method according to embodiment 1. (Aspect 10) segmenting the motion data generated by the sensors located near the crosswalk based on corresponding zones in the vicinity of the crosswalk. The method according to embodiment 1. (Aspect 11) using the electronic sensors to generate motion-related data representative of a physical characteristic of the vulnerable roadway users; The method according to embodiment 1. (Aspect 12) deriving trajectory information about the vulnerable roadway users from the motion data generated by the sensors. The method according to embodiment 1. (Aspect 13) An apparatus comprising equipment located at a grade intersection of a transportation network, the at-grade intersection includes a road intersection, a pedestrian crossing, and a railroad track; The device, inputs for receiving data from sensors oriented to monitor road vehicles and pedestrians at or near the grade crossing and for receiving phase and timing data for signals on the road and signals on the railroad tracks; a wireless communication device that transmits a warning regarding a hazardous condition at or near the grade crossing to one of a ground transportation entity, a pedestrian, or a rail vehicle on the railroad track; A processor; Storage and Equipped with The storage, storing a machine learning model capable of predicting a behavior of ground traffic entities at or near the grade intersection at a current time, the machine learning model being based on training data relating to previous actions and associated behaviors of road vehicles and pedestrians at or near the grade intersection and previous phase and timing data for signals on the road and signals on the railroad tracks; applying current motion data received from the sensors relating to road vehicles and pedestrians at or near the grade crossing and current phase and timing data for signals on the road and signals on the railway line to the machine learning model to predict the impending behavior of the road vehicles and pedestrians; inferring from the predicted impending behavior an imminent dangerous situation for a rail vehicle on the rail track at the intersection or on the rail track near the intersection; causing the wireless communication device to transmit the warning regarding the dangerous situation to at least one of the road vehicle, the pedestrian, and the rail vehicle; the storage for instructions executable by the processor to Device. (Aspect 14) The warning is transmitted to an on-board device of the rail vehicle. 14. The apparatus according to embodiment 13. (Aspect 15) the railroad tracks are on isolated railroad right-of-way; 14. The apparatus according to embodiment 13. (Aspect 16) The railroad tracks are not located on separate railroad rights-of-way; 14. The apparatus according to embodiment 13. (Aspect 17) the equipment comprises a roadside equipment; 14. The apparatus according to embodiment 13. (Aspect 18) the warning is transmitted by broadcasting the warning for reception by any of the ground traffic entities, the pedestrians, or the rail vehicles at or near the grade crossing; 14. The apparatus according to embodiment 13. (Aspect 19) The imminent dangerous situation includes a collision or near miss, 14. The apparatus according to embodiment 13. (Aspect 20) Receiving data from infrastructure sensors representative of a location and movement of a road vehicle being driven or a pedestrian being walked in a ground transportation network; receiving data in a basic safety message or a virtual basic safety message, a personal safety message and a virtual personal safety message relating to the state of the road vehicle and the pedestrian, the basic safety message, the virtual basic safety message, the personal safety message and the virtual personal safety message including position, heading and speed information for the road vehicle or the pedestrian, the basic safety message and the personal safety message received from the road vehicle or the pedestrian, the virtual basic safety message and the virtual personal safety message including position, heading and speed information for the road vehicle and the pedestrian reconstructed from the data received from the infrastructure sensors; applying the data received from the infrastructure sensors and data from the basic safety message, the virtual basic safety message, the personal safety message and the virtual personal safety message to a trained machine learning model to identify risky driving or walking behavior of one of the road vehicles or the pedestrians; automatically reporting said unsafe driving or walking behavior to authorities; A method comprising: (Aspect 21) identifying said road vehicle based on plate number recognition. The method according to embodiment 20. (Aspect 22) identifying the pedestrian based on biometric recognition. The method according to embodiment 20. (Aspect 23) identifying said road vehicles or said pedestrians based on social networking. The method according to embodiment 20.

Claims

1. receiving data from infrastructure sensors representative of locations and movements of driven road vehicles or pedestrian vulnerable road users in a ground transportation network, the road vehicles including connected and unconnected road vehicles and the vulnerable road users including connected and unconnected vulnerable road users; receiving data in a basic safety message or a virtual basic safety message, a personal safety message and a virtual personal safety message relating to the condition of the road vehicle and the vulnerable road user, the basic safety message, the virtual basic safety message, the personal safety message and the virtual personal safety message including position, heading and speed information for the road vehicle or the vulnerable road user, the basic safety message and the personal safety message being received from the connected road vehicle or the connected vulnerable road user, the virtual basic safety message and the virtual personal safety message including position, heading and speed information for the non-connected road vehicle and the non-connected vulnerable road user, the virtual basic safety message and the virtual personal safety message being reconstructed from the data received from the infrastructure sensors; applying the data received from the infrastructure sensors and data from the basic safety message, the virtual basic safety message, the personal safety message and the virtual personal safety message to a trained machine learning model to identify risky driving or walking behavior of one of the unconnected road vehicles or the unconnected vulnerable road users; broadcasting the virtual basic safety messages and the virtual personal safety messages to the connected road vehicles and the connected vulnerable road users regarding the risky driving or walking behavior of the unconnected road vehicles or the unconnected vulnerable road users; automatically reporting said unsafe driving or walking behavior to authorities; A method comprising:

2. identifying said road vehicle based on plate number recognition. The method of claim 1.

3. identifying the pedestrian based on biometric recognition. The method of claim 1.

Images