Early Warning and Collision Avoidance

Abstract

To provide early warning of a dangerous situation for assisting collision avoidance.SOLUTION: An apparatus is located at an intersection and receives data from a sensor oriented so as to monitor ground transportation entities at or near the intersection. A wireless communication device transmits an alert regarding a dangerous situation at or near the intersection to one device of the ground transportation entities. A machine learning model is stored that may predict behavior of the ground transportation entity at or near the intersection at a current time. Current operation data received from the sensors is applied to the machine learning model in order to predict imminent behaviors of the ground transportation entities. The wireless communication device transmits the alert regarding the dangerous situation to one device of the ground transportation entities.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] This application has the right to enjoy the benefits of the filing dates of U.S. Patent Application No. 15 / 994,568, filed on May 31, 2018; U.S. Patent Application No. 15 / 994,826, filed on May 31, 2018; U.S. Patent Application No. 15 / 994,702, filed on May 31, 2018; U.S. Patent Application No. 15 / 994,915, filed on May 31, 2018; U.S. Patent Application No. 16 / 222,536, filed on December 17, 2018; and U.S. Patent Application No. 15 / 994,850, filed on May 31, 2018, and claims priority and benefits based on U.S. Provisional Patent Application 62 / 644,725, filed on March 19, 2018, and the entire contents of the above applications are incorporated herein by reference.

Background Art

[0002] This description relates to early warning and collision avoidance.

[0003] Collision avoidance systems are increasing. King et al. (U.S. Patent Application Publication No. 2007 / 0276600 (A1), 2007), for example, described placing sensors in front of intersections and applying physics-based decision rules to predict whether two vehicles are about to collide at an intersection based on their direction of travel and speed.

[0004] Aoude et al. (U.S. Patent No. 9,129,519 (B2), 2015, the entire contents of which are incorporated herein by reference) monitor and model the behavior of drivers to enable prediction and prevention of violations in traffic situations at intersections.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Collision avoidance is a major defense against damage and death in ground transportation and assets. Providing early warning of dangerous situations will assist in collision avoidance.

Means for Solving the Problem

[0006] Generally, in one aspect, the device is located at an intersection of a transportation network. The device includes an input that receives data from sensors oriented to monitor surface transportation entities at or near the intersection. The wireless communication device transmits a warning regarding a dangerous situation at the intersection or a dangerous situation near the intersection to one of the surface transportation entities, and there is a processor and storage for executable instructions by the processor for performing operations including the following. A machine learning model that can predict the behavior of surface transportation entities at the intersection or near the intersection is stored. The machine learning model is based on training data regarding the behavior associated with the previous behavior of surface transportation entities at the intersection or near the intersection. The current motion data received from the sensors regarding the surface transportation entities at the intersection or near the intersection is applied to the machine learning model to predict the imminent behavior of the surface transportation entities. An imminent dangerous situation for one or more of the surface transportation entities at the intersection or near the intersection is inferred from the predicted imminent behavior. The wireless communication device transmits a warning regarding the dangerous situation to one of the surface transportation entities.

[0007] An embodiment may include one or a combination of two or more of the following features. The wireless communication device transmits a warning regarding a dangerous situation to a signage or other infrastructure presenting device. The warning includes an instruction or command capable of controlling a specific surface transportation entity. The device includes devices along the road. There is a housing for the device, and a sensor is attached to the housing. The warning is transmitted by broadcasting the warning for reception by any of the surface transportation entities at an intersection or the surface transportation entities near the intersection. The machine learning model includes an artificial intelligence model. The training data and the operation data include at least one of speed, position, or direction of travel. The training data and the operation data may further include intent, posture, direction of view, or interaction with other road users who may be subject to other damages, for example, within a group. The processor is configured to enable generating a machine learning model in the device. The training data is stored in the device. The intersection includes an intersection without a traffic signal. The intersection includes an intersection with a traffic signal. The transportation network includes a road network. The surface transportation entity includes a road user who may be subject to damage. The surface transportation entity includes a vehicle. The imminent dangerous situation includes a collision or a near miss. The surface transportation entity includes a pedestrian crossing the road at a crosswalk. There is another communication device that communicates with a central server. One of the devices of the surface transportation entities includes a mobile communication device.

[0008] Generally, in one aspect, the device is located at an intersection of a transportation network. The device includes an input for receiving data from sensors oriented to monitor surface transportation entities at the intersection or surface transportation entities near the intersection. The wireless communication device transmits a warning regarding a dangerous situation at the intersection or a dangerous situation near the intersection to a device of one of the surface transportation entities. There is a processor and storage for executable instructions by the processor for storing a machine learning model capable of predicting the behavior of surface transportation entities at the current intersection or surface transportation entities near the intersection. The machine learning model is based on training data regarding the behavior associated with previous actions of surface transportation entities at the intersection or surface transportation entities near the intersection. The current motion data received from the sensors regarding the surface transportation entities at the intersection or surface transportation entities near the intersection is applied to the machine learning model to predict the imminent behavior of the surface transportation entities, including surface transportation entities for which the associated device (the device of the surface transportation entity) cannot receive a warning from the wireless communication device. An imminent dangerous situation is inferred for surface transportation entities for which the associated device (the device of the surface transportation entity) can receive a warning from the wireless communication device. The imminent dangerous situation is a result of the predicted imminent behavior of surface transportation entities that cannot receive a warning. A warning regarding the dangerous situation is transmitted to the device of the surface transportation entity that can receive a warning from the wireless communication device.

[0009] Embodiments may include one or a combination of two or more of the following features. The device includes devices along the road. There is a housing for the device, and the sensor is mounted on the housing. Warnings are sent by broadcasting a warning for reception by any of the ground traffic entities at intersections where warnings can be received or ground traffic entities near intersections. The machine learning model includes an artificial intelligence model. The intersection includes intersections without traffic lights. The intersection includes intersections with traffic lights. The transportation network includes a road network. The ground traffic entity includes road users who may be victimized. The ground traffic entity includes vehicles. The imminent dangerous situation includes collisions. The ground traffic entities for which related devices cannot receive warnings from wireless communication devices include vehicles. The ground traffic entities for which related devices can receive warnings from wireless communication devices include pedestrians crossing the road at crosswalks. There is another communication device that communicates with the central server. One of the devices of the ground traffic entities includes a mobile communication device.

[0010] Generally, in one aspect, in a road vehicle moving within a ground transportation network, messages from external sources regarding the position, movement, and state of other ground traffic entities, regarding the road and driving conditions, and regarding the position of static objects and moving ground traffic entities in the vicinity of the vehicle, data from in-vehicle sensors, data regarding the quality of driving by the driver of the road vehicle, and messages and data including basic safety messages from other ground traffic entities and personal safety messages from road users who may be victimized are received. The received data and messages are fused and applied to an artificial intelligence model to predict the behavior of the driver of the road vehicle, or of a road user who may be victimized, or the risk of collision with the road vehicle, or both.

[0011] An embodiment may include one or a combination of two or more of the following features. The road vehicle generates a map of static objects and moving ground traffic entities in the vicinity of the road vehicle. The driver of the road vehicle receives an alert about a collision risk. The collision risk is identified based on the probability of the predicted trajectory of other nearby moving ground traffic entities. Basic safety messages and personal safety messages are filtered to reduce the number of alerts provided to the driver of the road vehicle.

[0012] Generally, in one aspect, an electronic sensor located near a crosswalk that crosses a road is used to monitor an area within the crosswalk or an area near the crosswalk. The electronic sensor generates motion data regarding a lane user who may be harmed within the crosswalk or a lane user who may be harmed near the crosswalk. The generated motion data is applied to a machine learning model operating in a device located near the crosswalk to predict that one of the lane users who may be harmed is about to enter the crosswalk. Before a lane user who may be harmed enters the crosswalk, a warning is wirelessly transmitted to at least one of a device associated with the lane user who may be harmed or a device associated with another ground traffic entity approaching the crosswalk on the road.

[0013] Embodiments may include one or a combination of two or more of the following features. The device includes devices along the road. Lane users who may be victimized include pedestrians, animals, or cyclists. Devices related to lane users who may be victimized include smartwatches or other wearable devices, smartphones, or other mobile devices. Other surface transportation entities include motor-driven vehicles. Devices related to other surface transportation entities include smartphones or other mobile devices. The machine learning model is provided to a device located near a crosswalk by a remote server through the Internet. The machine learning model is generated in a device located near a crosswalk. The machine learning model is trained using motion data generated by sensors located near a crosswalk. Motion data generated by sensors located near a crosswalk is transmitted to a server for use in training the machine learning model. Motion data generated by sensors located near a crosswalk is segmented based on corresponding zones in the vicinity of the crosswalk. An electronic sensor is used to generate motion-related data representing the physical properties of lane users who may be victimized. Trajectory information regarding lane users who may be victimized is derived from motion data generated by the sensors.

[0014] Generally, in one aspect, to monitor intersections of a ground transportation network and access roads to the intersections, electronic sensors located in the vicinity of the intersections are used. The electronic sensors generate motion data regarding ground transportation entities moving on the access roads or ground transportation entities moving at the intersections. One or more of the ground transportation entities may not be able to send basic safety messages to other ground transportation entities in the vicinity of the intersection. Based on the motion data generated by the electronic sensors, virtual basic safety messages are sent to one or more of the ground transportation entities capable of receiving messages. The virtual basic safety messages incorporate information regarding one or more of the ground transportation entities that cannot send basic safety messages. The incorporated information in each of the virtual basic safety messages includes at least one of the position, travel direction, speed, and predicted future trajectory of one of the ground transportation entities that cannot send basic safety messages.

[0015] Embodiments may include one or a combination of two or more of the following features. The devices include devices along the road. The incorporated information includes a subset of the information incorporated in the basic safety messages that would be generated by the ground transportation entity if it were assumed to be capable of sending basic safety messages. The generated motion data is applied to a machine learning model operating in a device located in the vicinity of the intersection to predict the trajectory of a ground transportation entity that cannot send basic safety messages. One of the ground transportation entities includes a motor-driven vehicle. The machine learning model is provided to a device located in the vicinity of the intersection by a remote server through the Internet. The machine learning model is generated in a device located in the vicinity of the intersection. The machine learning model is trained using the motion data generated by sensors located in the vicinity of the intersection. The motion data generated by sensors located in the vicinity of the intersection is sent to a server for use in training the machine learning model.

[0016] Generally, in one aspect, to monitor intersections of a ground transportation network and access roads to the intersections, electronic sensors located in the vicinity of the intersections are used. The electronic sensors generate motion data regarding ground transportation entities moving on the access roads or ground transportation entities moving at the intersections. Separate virtual zones are defined at the intersections and the access roads to the intersections. The generated motion data is segmented according to the corresponding virtual zones to which the generated motion data relates. The generated motion data is applied to a machine learning model operating in a device located in the vicinity of the intersection to predict an imminent dangerous situation in one of the intersections or one of the access roads, with one or more of the ground transportation entities. A warning is wirelessly transmitted to a device associated with at least one of the ground transportation entities involved before the imminent dangerous situation becomes an actual dangerous situation.

[0017] The embodiment may include one or a combination of two or more of the following features. The device includes a device along the road. The devices associated with each of the surface transportation entities include wearable devices, smartphones, or other mobile devices. One of the surface transportation entities includes a motor-driven vehicle. The machine learning model is provided to a device located near an intersection by a remote server through the Internet. The machine learning model is generated in a device located near an intersection. The machine learning model is trained using operation data generated by sensors located near the intersection. The operation data generated by the sensors located near the intersection is transmitted to a server for use in training the machine learning model. The electronic sensor is used to monitor an area within a crosswalk or an area near the crosswalk that crosses one of the access roads to the intersection. The electronic sensor is used to generate operation-related data representing the physical properties of road users who may be damaged in the vicinity of the crosswalk. Trajectory information regarding road users who may be damaged is derived from the operation data generated by the sensors. There is a machine learning model for each of the access roads to the intersection. A determination is made as to whether to send a warning based further on operation data generated by sensors associated with another nearby intersection. A determination is made as to whether to send a warning based further on information received from a surface transportation entity moving on the access road or a surface transportation entity moving at the intersection. The intersection is signalized and information regarding the state of the signal is received. The intersection is not signalized and is controlled by one or more signs. The defined virtual zone includes one or more access roads controlled by signs. The signs include a stop sign or a yield sign. One of the surface transportation entities includes a railway vehicle.

[0018] Generally, in one aspect, the device is located within or on a ground transportation entity. The device includes an input that receives data from sensors within or on the ground transportation entity that are oriented to monitor features near the ground transportation network and other information regarding the context in which the ground transportation entity is crossing the ground transportation network. A wireless communication device receives information regarding this context. A signal processor applies signal processing to the data from the sensors and other information regarding this context. There is storage for processor-executable instructions for a processor to perform operations including storing a machine learning model that can predict the behavior of an operator of the ground transportation entity and the intentions and movements of other ground transportation entities in the vicinity, and applying current received data from the sensors and other information regarding the context to predict the behavior of the operator and the intentions and movements of other ground transportation entities in the vicinity.

[0019] Embodiments may include one or a combination of two or more of the following features. The device includes roadside devices. The instructions are processor-executable to monitor a user or occupant of the ground transportation entity. Other information regarding the context includes emergency broadcasts, traffic and safety messages from roadside devices, and messages regarding safety, location, and other operational information from other ground transportation entities. The sensors include cameras, distance sensors, vibration sensors, microphones, seat sensors, hydrocarbon sensors, sensors for volatile organic compounds and other toxic substances, and kinematic sensors, or combinations thereof. The instructions are processor-executable to filter received alerts received by a vehicle by applying the alerts to a machine learning model to predict which alerts are important in terms of current location, environmental conditions, driver behavior, vehicle condition and status, and kinematics.

[0020] Generally, in one aspect, the motion data is acquired for unconnected ground traffic entities moving in the transportation network. A virtual safety message incorporating information regarding the motion data for the unconnected ground traffic entities is transmitted to connected ground traffic entities in the vicinity of the unconnected ground traffic entities.

[0021] The embodiment may include one or a combination of two or more of the following features. The virtual safety message is in place of a safety message that would be transmitted by an unconnected ground traffic entity assuming the unconnected ground traffic entity were connected. The unconnected ground traffic entities include vehicles and the virtual safety message is in place of a basic safety message. The unconnected ground traffic entities include road users who may be victims and the virtual safety message is in place of a personal safety message. The motion data is detected by infrastructure sensors.

[0022] Generally, in one aspect, a device located at an intersection of a transportation network includes an input for receiving data from sensors oriented to monitor surface traffic entities at the intersection or surface traffic entities in the vicinity of the intersection. Data from each of the sensors represents at least one position or motion parameter of at least one of the surface traffic entities. The data from each of the sensors is represented in a native format. Data received from at least two of the sensors do not match with respect to the position or motion parameters, or with respect to the native format, or both. There is storage for instructions executable by a processor to convert the data from each of the sensors into data having a common format independent of the native format of the sensor data. The data having the common format is incorporated into a global integrated representation of the surface traffic entities being monitored at the intersection or in the vicinity of the intersection. The global integrated representation includes the position, speed, and direction of travel of each of the surface traffic entities. The relatedness of the positions and motions of two of the surface traffic entities is identified using the global integrated representation. A dangerous situation involving the two surface traffic entities is predicted, and a message alerting at least one of the two surface traffic entities about the dangerous situation is sent to at least one of the two surface traffic entities.

[0023] An embodiment may include one or a combination of two or more of the following features. The sensors include at least two of radar, lidar, and camera. Data received from one of the sensors includes image data of the field of view at consecutive instants. Data received from one of the sensors includes reflection points in 3D space. Data received from one of the sensors includes the distance from the sensor and the speed. The global integrated representation represents the position of ground traffic entities in a common reference coordinate system. Two sensors from which data is received are provided at fixed positions at or near an intersection and have non-overlapping fields of view at least partially. One of the sensors includes radar, and the data conversion includes identifying the position of a ground traffic entity from the known position of the radar and the distance from the radar to the ground traffic entity. One of the sensors includes a camera, and the data conversion includes identifying the position of a ground traffic entity from a known position, the viewing direction, and the tilt of the camera, and the position of the ground traffic entity within the image frame of the camera.

[0024] Generally, in one aspect, the device is located at a planar intersection of a transportation network including road intersections, crosswalks, and railroad tracks. The device includes an input for receiving data from sensors oriented to monitor road vehicles and pedestrians at the planar intersection or road vehicles and pedestrians in the vicinity of the planar intersection, and for receiving phase and timing data for signals on the road and on the railroad tracks. The wireless communication device is included to send a warning regarding a dangerous situation at the planar intersection or a dangerous situation in the vicinity of the planar intersection to one of the devices of a ground transportation entity, a pedestrian, or a railroad vehicle on the railroad track. There is storage for executable instructions executable by a processor to store a machine learning model capable of predicting the behavior of a ground transportation entity at the current planar intersection or a ground transportation entity in the vicinity of the planar intersection. The machine learning model is based on training data regarding the behavior associated with the previous actions of road vehicles and pedestrians at the intersection or road vehicles and pedestrians in the vicinity of the intersection. Current motion data received from the sensors regarding road vehicles and pedestrians at the planar intersection or road vehicles and pedestrians in the vicinity of the planar intersection is applied to the machine learning model to predict the imminent behavior of the road vehicles and pedestrians. An imminent dangerous situation for a railroad vehicle on the railroad track at the intersection or a railroad vehicle on the railroad track in the vicinity of the intersection is inferred from the predicted imminent behavior. The wireless communication device causes a warning regarding the dangerous situation to be sent to at least one of the devices of a road vehicle, a pedestrian, and a railroad vehicle.

[0025] Embodiments may include one or a combination of two or more of the following features. A warning is sent to on-vehicle equipment of a railway vehicle. The railway line is on a separate railway land. The railway line is not on a separate railway land. The equipment includes roadside equipment. The warning is sent by broadcasting a warning for reception by any of a ground traffic entity, a pedestrian, or a railway vehicle at a level crossing, or a ground traffic entity, a pedestrian, or a railway vehicle near the level crossing. An imminent dangerous situation includes a collision or a near miss.

[0026] Generally, in one aspect, data representing the position and movement of a road vehicle being driven or a pedestrian walking in a ground traffic network is received from an infrastructure sensor. The data is received in a virtual basic safety message and a virtual personal safety message regarding the state of the road vehicle and the pedestrian. The received data is applied to a machine learning model trained to identify a dangerous driving or walking behavior of one of the road vehicle or the pedestrian. The dangerous driving or walking behavior is reported to the authorities.

[0027] Embodiments may include one or a combination of two or more of the following features. A road vehicle is identified based on license plate recognition. A pedestrian is identified based on biometric recognition. A road vehicle or a pedestrian is identified based on social networking.

[0028] These and other aspects, features, and embodiments may be represented as a method, an apparatus, a system, a component, a program product, a method of doing business, means or steps for performing a function, and other approaches.

[0029] These and other aspects, features, and embodiments will become apparent from the following description including the claims.

Brief Description of the Drawings

[0030]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Mode for Carrying Out the Invention

[0031] With the progress of sensor technology and computers, it has become possible to predict dangerous situations (and provide early warnings of dangerous situations), and thus prevent collisions and near misses of ground traffic entities in the implementation of ground traffic (that is, enable collision avoidance).

[0032] As used herein, the term "ground transportation" is used broadly to include, for example, any mode or medium of moving from place to place with contact with land or water on the surface of the Earth, such as walking, or running (or performing other walking actions), non-motorized vehicles, motorized vehicles (autonomous, semi-autonomous, and non-autonomous), and rail vehicles.

[0033] As used herein, the term "ground transportation entity" (or, in some cases, simply "entity") is used broadly to include, for example, a person or a discrete motorized or non-motorized vehicle involved in a mode of ground transportation, such as, among others, a pedestrian, a cyclist, a boat, an automobile, a truck, a tram, a streetcar, or a train. In some cases, the terms "vehicle" or "road user" are used herein as a concise reference to a ground transportation entity.

[0034] As used herein, the term "hazardous situation" is used broadly to include any event, occurrence, sequence, context, or other situation that can result in imminent property damage or personal injury or death and that can be mitigated or avoided. In some cases, the term "hazard" is used interchangeably with "hazardous situation" herein. In some cases, the terms "violation" or "violating" are used with respect to the behavior of an entity that causes, can lead to, or will lead to a hazardous situation.

[0035] In some embodiments of the technology discussed herein, the ground transportation network is used by a mixture of ground transportation entities that do not have or do not use traffic connectivity and ground transportation entities that do have and do use traffic connectivity.

[0036] As used herein, the term "connectivity" is used broadly to encompass any capabilities of a ground transportation entity, such as, for example, (a) to recognize knowledge information about the surrounding environment of a ground transportation entity, other ground transportation entities in the vicinity of the ground transportation entity, and traffic conditions associated with the ground transportation entity, and to act based on that knowledge information, (b) to broadcast or otherwise transmit data regarding the state of the ground transportation entity, or (c) to do both (a) and (b). The data transmitted may include its location, direction of travel, speed, or the internal state of its components related to traffic conditions. In some examples, the recognition of a ground transportation entity is based on wirelessly received data regarding other ground transportation entities or traffic conditions related to the operation of the ground transportation entity. The received data may be from other ground transportation entities, or from infrastructure devices, or both. Typically, connectivity involves transmitting or receiving data in real time, or substantially in real time, or in time for one or more of the ground transportation entities to act based on the data in a traffic situation.

[0037] As used herein, the term "traffic situation" is used broadly to encompass any situation in which two or more ground transportation entities operate in the vicinity of each other, and any situation in which the operation or status of each entity may affect or be related to the operation or status of the others.

[0038] In this specification, in some cases, a ground transportation entity that does not have connectivity or an aspect of connectivity, or does not use connectivity or an aspect of connectivity, is referred to as an "unconnected ground transportation entity" or simply an "unconnected entity". In this specification, in some cases, a ground transportation entity that has connectivity or an aspect of connectivity, and that uses connectivity or an aspect of connectivity, is referred to as a "connected ground transportation entity" or simply a "connected entity".

[0039] In this specification, in some cases, the term "cooperating entity" is used to represent a ground transportation entity that broadcasts data, such as data including the position, direction of travel, speed, or the state of in-vehicle safety systems (e.g., brakes, lights, and wipers), into the environment of a ground transportation entity.

[0040] In this specification, in some cases, the term "non - cooperating entity" is used to represent a ground transportation entity that does not broadcast one or more types of data, such as the position, speed, direction of travel, or state, of the ground transportation entity into the environment of the ground transportation entity.

[0041] In this specification, in some cases, the term "vicinity" of a ground transportation entity is widely used to include, for example, an area in which broadcasts by the entity can be received by other ground transportation entities or infrastructure devices. In some examples, the vicinity varies with the position of the entity and the number and characteristics of obstacles in the vicinity of the entity. An entity moving on an open road in the desert has a very wide vicinity because there are no obstacles to prevent broadcast signals from the entity from reaching long distances. Conversely, the vicinity in an urban valley is reduced by buildings in the vicinity of the entity. Further, there may be sources of electromagnetic noise that degrade the quality of the broadcast case signal and thus shorten the reception distance (vicinity).

[0042] As shown in FIG. 14, the vicinity of entity 7001 moving along road 7005 can be represented by concentric circles, with the outermost circle 7002 representing the outermost range of the vicinity. Any other entity existing within circle 7002 is in the vicinity of entity 7001. Any other entity located outside of circle 7002 is outside the vicinity of entity 7001 and cannot receive a broadcast from entity 7001. Entity 7001 is invisible to all entities and infrastructure devices outside its vicinity.

[0043] Typically, a collaborating entity continuously broadcasts the state data of the collaborating entity. Connected entities in the vicinity of the broadcasting entity can receive these broadcasts, process the received data, and act based on the received data. For example, if a road user who may be affected possesses a wearable device that can receive broadcasts from an entity, such as an approaching truck, the wearable device can process the received data and inform the user who may be affected of when it is safe to cross the road. This operation occurs regardless of the position of the collaborating entity or the user who may be affected with respect to the "smart" intersection as long as the user's device can receive the broadcast, i.e., as long as it is within the vicinity of the collaborating entity.

[0044] As used herein, the term "vulnerable road user" or "vulnerable road user" is used broadly to include any user of a roadway, or other features of a road network, for example, who is not using a motorized vehicle. When a vulnerable road user collides with a motorized vehicle, the vulnerable road user is generally not protected against injury or death or damage to property. In some examples, a vulnerable road user can be a person walking, running, riding a bicycle, or any person performing any kind of activity that exposes the vulnerable road user to the risk of direct physical contact by a vehicle or other ground transportation entity in the event of a collision.

[0045] In some embodiments, the collision avoidance techniques and systems described herein (sometimes simply referred to herein as "systems") use sensors mounted on infrastructure facilities to monitor, track, detect, and predict the actions (e.g., speed, direction of travel, and position), behavior (e.g., speeding), and intentions (e.g., attempting to violate a stop sign) of ground transportation entities, and the drivers and operators of ground transportation entities. The information provided by the sensors ("sensor data") enables the system to predict dangerous situations and provide early warnings to entities to increase the opportunity for collision avoidance.

[0046] As used herein, the term "collision avoidance" is used broadly to include any situation in which a collision or near miss between two or more ground transportation entities, or between a ground transportation entity and another object in the environment, that could be caused by a dangerous situation is prevented, or any situation in which the opportunity for such an interaction is reduced.

[0047] As used herein, the term "early warning" is used broadly to include, for example, any notification, alert, instruction, command, broadcast, transmission, or other sending or receiving that identifies, proposes, or otherwise indicates a dangerous situation and useful information for collision avoidance.

[0048] Road intersections are prime locations where dangerous situations can occur. The techniques described herein may include intersections with infrastructure devices, including sensors, computing hardware, and intelligence, that enable simultaneous monitoring, detection, and prediction of dangerous situations. Data from these sensors is processed after being normalized to a single reference coordinate system. Artificial intelligence models of traffic flow along different approaches to the intersection are constructed. These models, for example, serve entities that are likely to violate traffic rules. The models are configured to detect dangerous situations prior to an actual violation and can thus be considered predictive. Based on the prediction of a dangerous situation, an alert is transmitted from the infrastructure devices at the intersection to all connected entities in the vicinity of the intersection. Each entity that receives the alert processes the data in the alert and performs alert filtering. Alert filtering is the process of discarding or ignoring alerts that are not beneficial to the entity. If an alert, such as an alert of an imminent collision, is considered beneficial (i.e., not ignored as a result of the filtering process), the entity automatically responds to the alert (e.g., by applying the brakes), or a notification is presented to the driver, or both.

[0049] The system can be used on roadways, waterways, and railways, but is not limited thereto. In some cases herein, these and other similar traffic contexts are referred to as the "surface transportation network."

[0050] Although often discussed herein in the context of intersections, the system can also be applied to other contexts.

[0051] As used herein, the term "intersection" is used broadly to include any physical arrangement of roads, rails, waterways, or other travel paths where two or more surface transportation entities moving along the paths of a surface transportation network can occupy the same position at a given point in time and location and cause a collision.

[0052] Surface transportation entities using a surface transportation network move at various speeds and can reach a given intersection at different speeds and times. If the speed and distance of an entity from an intersection are known, dividing the distance by the speed (both expressed in the same unit system) gives the time of arrival at the intersection. However, the expected speed can change due to, for example, traffic conditions, speed limits on the route, traffic signals, and other factors, so the expected time of arrival at the intersection changes continuously. This dynamic change in the expected time of arrival makes it impossible to predict the actual time of arrival with 100% reliability.

[0053] Accounting for factors that affect the operation of an entity requires applying a number of relationships between the speed of the entity and various influencing factors. The absolute value of the state of the operation of an entity can be observed by sensors that track the entity from the entity itself or from an external location. The data captured by these sensors can be used to model the motion, behavior, and intention patterns of the entity. Machine learning can be used to generate complex models from large amounts of data. Patterns that cannot be directly modeled using the kinematics of the entity can be captured using machine learning. A trained model can predict whether an entity is about to move or stop at a particular point by using the tracking data of that entity from sensors that track it.

[0054] In other words, in addition to directly detecting information about surface traffic entities from sensor data, the system uses artificial intelligence and machine learning to process vast amounts of sensor data to learn patterns of operation, behavior, and intent of surface traffic entities, for example, at intersections of the surface transportation network, on approaches to such intersections, and at crosswalks of the surface transportation network. Based on the direct use of current sensor data and on the results of applying artificial intelligence and machine learning to current sensor data, the system generates early warnings, such as warnings of dangerous situations, and thereby assists in collision avoidance. In connection with early warnings in the form of instructions or commands, the commands or instructions may be directed at specific autonomous or human-driven entities to directly control the vehicle. For example, the command or instruction can cause an entity being driven by a malicious person determined to be attempting to run a red light with the intent of harming someone to decelerate or stop.

[0055] The system can be adjusted to predict for that particular intersection and to send warnings to entities in the vicinity of the device broadcasting the warning. For this purpose, the system uses sensors to derive data about dangerous entities and passes the current readings from the sensors through a trained model. Thus, the output of the model can predict dangerous situations and broadcast corresponding warnings. Warnings received by connected entities in the vicinity contain information about the dangerous entity, and as a result, the receiving entity can analyze that information to assess the threat posed to the receiving entity by the dangerous entity. If a threat exists, the receiving entity can act on its own (e.g., decelerate) or, based on visual, audible, tactile, or any type of sensory stimulus, use a human-machine interface to notify the driver of the receiving entity. An autonomous entity can act on its own to avoid a dangerous situation.

[0056] The alert can also be directly transmitted to mobile phones or other devices that are equipped to receive the alert through a cellular network or other network and are carried by pedestrians. The system identifies potential dangerous entities at intersections and broadcasts (or directly transmits) the alert to the personal devices of pedestrians that include communication units. The alert can, for example, prevent a pedestrian from entering a crosswalk and thus avoid a potential accident.

[0057] The system can further track pedestrians and broadcast information (position, speed, and other parameters) related to the state of the pedestrians to other entities so that the other entities can take actions to avoid dangerous situations.

[0058] As shown in FIG. 1, the system includes at least the following types of components.

[0059] 1. Roadside Equipment (RSE) 10 that includes or uses sensors 12 for monitoring, tracking, detecting, and predicting the operation (e.g., speed, direction of travel, and position), behavior (e.g., high speed), and intent (e.g., violating a stop sign) of ground traffic entities 14. The RSE can further include or use a data processing unit 11 and a data storage 18. Ground traffic entities exhibit a wide range of behaviors that depend on the infrastructure of the ground traffic network as well as the state of the entity itself, the state of the driver, and the state of other ground traffic entities. To capture the behavior of the entities, the RSE collects information from sensors, other RSEs, OBEs, OPEs, local servers or central servers, and other data processing units. The RSE can further store the data received by the RSE and can store the processed data in some or all of the steps in the pipeline.

[0060] The RSE can store data in a local storage device or remote storage. The collected data is processed in real time using default logic or logic based on dynamically collected data, which means the RSE can automatically update its own logic. The data can be processed in one processing unit or in a cluster of processing units to obtain results faster. The data can be processed in local or remote processing units or in local or remote clusters of processing units. The RSE can use simple logic or an advanced model trained on the collected data. The model can be trained locally or remotely.

[0061] The RSE can preprocess the data before using the trained model to filter out outliers. Outliers can appear due to noise in the sensor, or reflections, or some other artifact. The resulting outliers can lead to false alarms that can affect the overall performance of the RSE. The filtration method can be based on data collected by the RSE, OBE, OPE, or online resources. The RSE can mediate between other control devices, such as traffic signal control devices at intersections or other locations, to extract information for use in the data processing pipeline.

[0062] The RSE may further include, or be used with, a communication device 20 for communicating, either wired or wirelessly, with other RSEs, and with OBEs, OPEs, local or central servers, and other data processing units. The RSE may use any available standard for communication with other devices. The RSE may use a wired or wireless Internet connection for downloading and uploading data to and from other devices, a cellular network for sending and receiving messages from other cellular devices, and a dedicated radio device for communicating with infrastructure devices and other RSEs at intersections or other locations.

[0063] The RSE can be installed in the vicinity of different types of intersections. For example, at a signalized intersection (e.g., an intersection where traffic is controlled by traffic lights), the RSE 10 is installed within the same enclosure or a nearby enclosure near the traffic signal control device 26. Data (e.g., the phases and timings of traffic lights) is intended to flow 28 between the traffic signal control device and the RSE. At an intersection without traffic lights, the RSE 10 is typically positioned to simplify connecting the RSE 10 to sensors 12 that are used to monitor other features of the road or surface transportation network in the vicinity of the intersection. The proximity of the RSE to the intersection helps maintain a critical low-latency system to provide the receiving-side surface unit with maximum time to respond to an alert.

[0064] An on-board equipment (OBE) 36 supported by, mounted on, or within a surface transportation entity 14, the surface transportation entity 14 including sensors 38 that identify the position and kinematics (motion data) of the entity in addition to safety-related data regarding the entity. The OBE further includes a data processing unit 40, a data storage 42, and a communication device 44 that can communicate wirelessly with other OBEs, OPEs, RSEs, and, optionally, servers and computing units.

[0065] 3. Although not limited to, it can be a mobile phone, a wearable device, or any other device that can be worn, held, attached to, or otherwise associated with a person or an animal, namely, an on-person equipment (OPE) 46. The OPE can include, if necessary, a data processing unit 48, a data storage 50, and a communication device 52, or can be coupled to the data processing unit 48, the data storage 50, and the communication device 52. In some embodiments, the OPE functions as a dedicated communication unit for road users who can be victimized and are not in vehicles. In some examples, the OPE can also be used for other purposes. The OPE can include components for providing visual, audio, or tactile alerts to road users who can be victimized.

[0066] Road users who can be victimized can include pedestrians, cyclists, road workers, wheelchair users, scooter riders, self-balancing devices, or people on battery-powered personal mobility devices, carriages driven by animals, guide or police animals, livestock, herds of animals, and pets.

[0067] Typically, the OPE is possessed by road users who can be victimized and is capable of sending and receiving messages. The OPE can be worn or integrated into a mobile phone, a tablet, a personal mobility device, a bicycle, a wearable device (e.g., a wristwatch, a bracelet, an anklet), or can be attached to a pet collar.

[0068] Messages sent by the OPE may include kinematic information related to road users who may be affected, including but not limited to time, 3D position, direction of travel, speed, and acceleration. The messages sent may further convey data representing the alert level, current behavior, and future intentions of the road users who may be affected, such as a road user who is currently crossing a crosswalk, listening to music, or about to cross a crosswalk. Among other possibilities, the message may convey the blob size or data size of the road user who may be affected, whether an external device (e.g., a stroller, cart, or other device) is present with the road user who may be affected, whether the road user who may be affected has a physical disability, or whether any personal assistance is being used. The message may convey the category of the worker if the road user who may be affected is a worker, and may further describe the type of action being performed by the worker. If clusters of similar road users who may be affected (e.g., groups of pedestrians) have similar characteristics, one message may be sent to avoid multiple message broadcasts.

[0069] Typically, messages received by the OPE are alert messages from roadside devices or entities. The OPE may operate based on the received messages by alerting road users who may be affected. The alert messages convey data useful in providing custom alerts for road users who may be affected. For example, the alert to a road user who may be affected may indicate the type of dangerous situation and suggest possible actions. The OPE may apply an alert filtering process to all received messages and present only relevant messages to road users who may be affected.

[0070] The warning filter process is based on the result of applying a learning algorithm to historical data related to OPE, which enables customizing the warning filter process for each road user who may be affected. The OPE learning algorithm tracks the responses of road users who may be affected to received warnings and adjusts future warnings to elicit the best response times and best attention from road users who may be affected. The learning algorithm may also be applied to the data transmitted in the messages sent.

[0071] 4. A data storage server 54 that can be, but is not limited to, cloud storage, local storage, or any other storage facility that enables data storage and access. The data storage server is accessible by the RSE, by the computing unit, and optionally by the OBE, OPE, and the data server, for example for the purpose of storing data related to early warning and collision avoidance. The data storage server is accessible from the RSE and optionally from the OBE, OPE, and the data server for the purpose of fetching the stored data. The data can be raw sensor data, data processed by the processing unit, or any other information generated by the RSE, OBE, and OPE.

[0072] Sensors at intersections that continuously monitor ground traffic entities can generate large amounts of data every day. The volume of this data depends on the number and type of sensors. The data is both processed in real time and stored for future analysis, for example, locally at the intersection, requiring a data storage unit (e.g., hard disk drive, solid state drive, and other large-capacity storage devices). Local storage devices will become full after a period of time, depending on the storage capacity of the local storage device, the volume of the generated data, and the speed at which the data is generated. To store data for future use, the data is uploaded to a remote server with a larger capacity. The remote server can upgrade the storage capacity on demand as needed. The remote server may use a data storage device similar to the local storage (e.g., hard disk drive, solid state drive, or other large-capacity storage device) that is accessible through a network connection.

[0073] Data stored locally and on the server for future analysis may include data broadcast by ground traffic entities, received by the RSE, and saved for future analysis. The stored data can be downloaded from the server or other remote sources for processing at the RSE. For example, a machine learning model of the intersection where the RSE is located may be stored on the server or other remote storage and downloaded by the RSE for use in analyzing the current data received at the RSE from local sources.

[0074] 5. The computing unit 56, which is a powerful computing machine located in the cloud, or locally (e.g., as part of the RSE), or a combination thereof. Although there may be other functions, the computing unit processes available data to generate predictions and generate machine learning-based models of the behavior, behavior, and intent of vehicles, pedestrians, or other ground traffic entities using the transportation network. Each of the computing units may include dedicated hardware for processing corresponding types of data (e.g., a graphics processing unit for processing images). In the case of a heavy processing load, the computing unit in the RSE can become overloaded. For example, this can occur when an additional data generation unit (e.g., a sensor) is added to a system that is causing a computational overload. Overloading can also occur when the logic operating in the computing unit is replaced with more computationally intensive logic. An increase in the number of ground traffic entities being tracked can cause an overload. If a local computing overload occurs, the RSE can delegate some of the tasks to another computing unit. Such other computing units can be near the RSE or remote, e.g., a server. The computing tasks can be prioritized, and tasks that are not time-critical can be performed in such other computing units, and the results can be obtained by the local computing unit. For example, the computing unit in the RSE can request another computing unit to execute jobs to analyze stored data and train a model using the data. The trained model is then downloaded by the computing unit in the RSE for storage and use there.

[0075] The arithmetic unit in the RSE can reduce time by using other small arithmetic units to more efficiently perform computationally intensive jobs. The available arithmetic units are wisely used to perform most tasks in the shortest time, for example, by dividing tasks between the RSE arithmetic unit and other available arithmetic units. The arithmetic unit can also be attached as an external device to the RSE to add more computing power to the arithmetic unit in the RSE. The externally attached arithmetic unit may include the same or a different architecture compared to the arithmetic unit in the RSE. The externally attached arithmetic unit can communicate with the existing arithmetic unit using any available communication port. The RSE arithmetic unit can request more computing power from the external arithmetic unit as needed.

[0076] The following part of this specification details, among other things, the roles and functions of the above components in the system.

[0077] Roadside Equipment (RSE)

[0078] As shown in Figure 2, the RSE may include, but is not limited to, the following components.

[0079] 1. One or more communication units 103, 104 that enable the reception and / or transmission of operational data and other data related to a ground transportation entity, and traffic safety data, from nearby vehicles or other ground transportation entities, infrastructure, and remote servers, and data storage system 130, and to nearby vehicles or other ground transportation entities, infrastructure, and remote servers, and data storage system 130. In some examples, this type of communication is known as infrastructure-to-everything (I2X), which includes, but is not limited to, infrastructure-to-vehicles (I2V), infrastructure-to-pedestrians (I2P), infrastructure-to-infrastructure (I2I), and infrastructure-to-devices (I2D), and combinations thereof. The communication may be wireless or wired and may conform to a wide variety of communication protocols.

[0080] 2. Communication unit 103 is used for communication with a ground transportation entity, and unit 104 is used for communication through the Internet with remote servers and data storage system 130.

[0081] 3. A local storage 106 for storing programs, intersection models, and behavior and traffic models. Local storage 106 may also be used for temporary storage of data collected from sensor 101.

[0082] 4. Sensors 101 and sensor control device 107 that enable the monitoring of moving objects (e.g., the generation of data regarding moving objects), such as typically ground traffic entities near an RSE. The sensors may include, but are not limited to, cameras, radars, lidars, ultrasonic detectors, or any other hardware that can detect or infer from the detected data the distance to a ground traffic entity, or the speed, direction of travel, or position of a ground traffic entity, or a combination thereof. Sensor fusion is performed using the collection or combination of data from two or more sensors 101.

[0083] 5. A position receiver (102) (e.g., a GPS receiver) that provides location data (e.g., the coordinates of the position of an RSE) and that helps correct location errors in the location determination of ground traffic entities.

[0084] 6. A processing unit 105 that acquires and uses data generated by sensors and incoming data from communication units 103, 104. The processing unit processes and stores data locally and, in some embodiments, transmits data for remote storage and further processing. The processing unit further generates messages and alerts that are broadcast or otherwise transmitted via wireless communication facilities to nearby pedestrians, motor vehicles, or other ground traffic entities and, in some examples, to signage or other infrastructure presentation devices. The processing unit further periodically reports the condition and status of all RSE systems to a remote server for monitoring.

[0085] 7. An expansion connector 108 that enables control and communication between an RSE and other hardware, or other components, such as temperature and humidity sensors, traffic signal control devices, other computing units as described above, and other electronic devices that may become available in the future.

[0086] On-Board Equipment (OBE)

[0087] The in-vehicle device may typically be original equipment for surface transportation entities or may be added to the entity by a third-party supplier. As shown in FIG. 3, the OBE may include, but is not limited to, the following components.

[0088] 1. A communication unit 203 that enables the transmission and reception, or both, of data to and from nearby vehicles, pedestrians, cyclists, or other surface transportation entities, and infrastructure, and combinations thereof. The communication unit further enables the transmission or reception (or both) of data between the vehicle or other surface transportation entity and a local or remote server 212 for the purpose of machine learning and for the remote monitoring of surface transportation entities by the server. In some examples, this type of communication is known as vehicle-to-everything (V2X), which includes, but is not limited to, vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), vehicle-to-infrastructure (V2I), vehicle-to-device (V2D), and combinations thereof. The communication may be wireless or wired and may comply with a wide variety of communication protocols.

[0089] The communication unit 204 enables the OBE to communicate with a remote server through the Internet for program updates, data storage, and data processing.

[0090] 2. A local storage 206 for storing programs, intersection models, and traffic models. The local storage 206 may be used for temporary storage of data collected from the sensors 201.

[0091] 3. Sensors 201 and sensor control devices 207 that may include, but are not limited to, external cameras, lidars, radars, ultrasonic sensors, or any device that can be used to detect nearby objects, people, or other ground traffic entities. The sensors 201 may include additional kinematic sensors, global positioning system receivers, and internal and local microphones and cameras.

[0092] 4. A position receiver 202 (e.g., a GPS receiver) that provides location data (e.g., coordinates of the position of a ground traffic entity).

[0093] 5. A processing unit 205 that acquires, uses, generates, and transmits data, including consuming data from a communication unit, transmitting data to a communication unit, and consuming data from sensors within or on a ground traffic entity.

[0094] 6. An expansion connector 208 that enables control of the OBE and other hardware and communication between the OBE and other hardware.

[0095] 7. An interface unit that can be additionally introduced or integrated into a head unit, a handle, or a driver mobile device by one or more methods, for example, using visual, audible, or tactile feedback.

[0096] Smart OBE (SOBE)

[0097] In a world where all vehicles and other ground transportation entities are connected entities, each vehicle or other ground transportation entity can be an entity that coordinates with others and can report its current location, safety status, intentions, and other information to others. At present, almost all vehicles are not connected entities and cannot report such information to other ground transportation entities, and are operated by people with different levels of skill, happiness, stress, and behavior. Without such connectivity and communication, it becomes difficult to predict the next movement of a vehicle or the next movement of a ground transportation entity, resulting in a decline in the ability to implement collision avoidance and the ability to provide early warnings.

[0098] The Smart OBE monitors the environment and the users or passengers of ground transportation entities. The Smart OBE further monitors the condition and status of different systems and subsystems of the entity. The SOBE monitors the external world by listening to, for example, radio wave transmissions from emergency broadcasts, traffic and safety messages from nearby RSEs, and messages regarding safety, location, and other operation information from other connected vehicles or other ground transportation entities. The SOBE further interfaces with in-vehicle sensors that can see the road and driving conditions, such as cameras, distance sensors, vibration sensors, microphones, or any other sensors that enable such monitoring. The SOBE further monitors the immediate environment and generates a map of all stationary and moving objects.

[0099] The SOBE can further monitor the behavior of the users or passengers of a vehicle or other ground transportation entity. The SOBE uses a microphone to monitor the quality of conversations. The SOBE can further use other sensors, such as seat sensors, cameras, hydrocarbon sensors, and sensors for volatile organic compounds and other toxic substances. The SOBE can further use kinematic sensors to measure the driver's reactions and behavior and infer the quality of driving therefrom.

[0100] SOBE also receives vehicle-to-vehicle messages (e.g., basic safety messages (BSMs)) from other ground transportation entities and vehicle-to-pedestrian messages (e.g., personal safety messages (PSMs)) from vulnerable road users.

[0101] SOBE then fuses the data from this array of sensors, sources, and messages. SOBE can then not only predict the next actions or reactions of the driver or user of the vehicle or other ground transportation entity, or of the vulnerable road user, but also predict intentions and future trajectories, and apply the data fused into an artificial intelligence model that can also predict related near misses or collision risks due to other vehicles, ground transportation entities, and nearby vulnerable road users. For example, SOBE may use the BSMs received from nearby vehicles to predict that a nearby vehicle is about to initiate a lane change manoeuvre that poses a risk to its host vehicle, and can alert the driver about the impending risk. The risk is calculated by SOBE based on various predicted future trajectories of the nearby vehicle (e.g., going straight, changing lanes to the right, changing lanes to the left), and the probability of the associated collision risk with the host vehicle for each of those trajectories. If the collision risk is higher than a certain threshold, a warning is displayed to the driver of the host vehicle.

[0102] Due to the complexity of modelling human driver behavior, which is also affected by external factors (e.g., changing environmental and weather conditions), machine learning is typically required to predict intentions and future trajectories.

[0103] SOBE is characterized by its ability to process a large number of data feeds, some of which provide data at several megabytes per second, and by its powerful computing capabilities. The amount of data available is further proportional to the level of detail required from each sensor.

[0104] SOBE further includes powerful signal processing equipment that can extract useful information from environments known to have high (signal) noise levels and low signal-to-noise ratios. SOBE also protects the driver from the vast number of warnings received by the vehicle by providing smart alert filtering. The alert filtering is the result of a machine learning model that can distinguish which warnings are important in terms of the current location, environmental conditions, driver behavior, vehicle condition and status, and kinematics.

[0105] Smart OBE is important for collision avoidance and early warning, and is important not only for the people or users riding in the vehicle containing the SOBE, but also for all users to achieve a safer transportation network. SOBE can detect and predict the movements of different entities on the road and thus assist in collision avoidance.

[0106] Over-Person Equipment (OPE)

[0107] As described above, an on-person device (OPE) includes any device that can be held, worn, or otherwise directly associated by a pedestrian, jogger, or other person who is a ground transportation entity, or who is otherwise present on or using a ground transportation network. Such a person can be, for example, a road user who is vulnerable to being hit by a vehicle. The OPE can include, but is not limited to, mobile devices (e.g., smartphones, tablets, digital assistants), wearable devices (e.g., eyewear, watches, bracelets, anklets), and implants. Existing components and features of the OPE can be used to track and report position, speed, and direction of travel. The OPE can also be used to receive and process data and to display alerts to the user in various modes (e.g., visual, audio, tactile).

[0108] Honda has developed a communication system and method for V2P applications that focuses on direct communication between vehicles and pedestrians using an OPE. In one example, a vehicle is equipped with an OBE for broadcasting messages to the OPEs of surrounding pedestrians. The messages convey the current status of the vehicle, including, for example, vehicle parameters, speed, and direction of travel. For example, the message can be a basic safety message (BSM). If necessary, the OPE presents an alert to the pedestrian that is adjusted according to the pedestrian's level of inattentiveness regarding a predicted dangerous situation in order to avoid a collision. In another example, the pedestrian's OPE broadcasts a message (e.g., a personal safety message (PSM)) to the OBEs of surrounding vehicles that the pedestrian may cross the intended path of a vehicle. If necessary, the vehicle's OBE displays an alert to the vehicle user regarding a predicted danger to avoid a collision. See Strickland, Richard Dean, et al., "Vehicle to pedestrian communication system and method." U.S. Patent No. 9,421,909.

[0109] The systems described herein use an I2P or I2V approach that uses sensors external to vehicles and pedestrians (mainly in infrastructure) to track and collect data regarding pedestrians and other vulnerable road users. For example, the sensors can track pedestrians crossing a street and vehicles operating at or near the crossing location. The data collected is subsequently used to build predictive models of the intentions and behaviors of pedestrians and vehicle drivers on the road using rule-based and machine learning methods. These models help in analyzing the data collected and predicting the paths and intentions of pedestrians and vehicles. When a danger is predicted, a message is broadcast from the RSE to the OBE or OPE, or both, alerting each entity of the intended path of the other and enabling each of them to take action with sufficient time to avoid a collision.

[0110] Remote computing (cloud computing and storage)

[0111] Data collected from sensors connected to or incorporated in the RSE, OBE, and OPE needs to be processed such that an effective mathematical machine learning model can be generated. This processing requires a lot of data processing power to reduce the time required to generate each model. The processing power required is much more than the processing power typically available locally at the RSE. To address this, the data can be sent to a remote computing facility that provides the required power and can scale on demand. In this document, the remote computing facility is referred to as a "remote server" in line with the terms used in the literature regarding computing. In some examples, it may be possible to perform some or all of the processing at the RCE by equipping the RCE with high-powered computing capabilities.

[0112] Rule-based processing

[0113] Unlike artificial intelligence and machine learning techniques, rule-based processing can be applied at any point without the need for data collection, training, and model building. Rule-based processing can be deployed from the very beginning of the system's operation and typically occurs until sufficient training data is obtained to generate a machine learning model. After a new installation, rules are set up to process incoming sensor data. This is not only useful for improving road safety but also serves as an appropriate test case to ensure that all components of the system operate as expected. Rule-based processing can be further added and used later as an additional layer to capture rare cases where machine learning cannot make accurate predictions. The rule-based approach is based on simple correlations between the collected data parameters (e.g., speed, range, etc.). The rule-based approach can further provide a baseline for evaluating the performance of machine learning algorithms.

[0114] In rule-based processing, vehicles or other ground traffic entities crossing a part of the ground transportation network are monitored by sensors. If their current speed and acceleration exceed a threshold that would prevent them from stopping before a stop bar (line) on the road, for example, an alert is generated. A variable area is assigned to each vehicle or other ground traffic entity. The area is labeled as a dilemma zone, and within the dilemma zone, the vehicle is not yet labeled as a violating vehicle. If a vehicle enters the danger zone beyond the dilemma zone because its speed or acceleration, or both, are higher than a predefined threshold, the vehicle is labeled as a violating entity and an alert is generated. The thresholds for speed and acceleration are based on physics and kinematics and vary, for example, according to each ground traffic entity approaching an intersection.

[0115] Two conventional rule - based approaches are: 1) static TTI (Time - To - Intersection), and 2) static RDP (Required Deceleration Parameter). See Aoude, Georges S. et al., "Driver behavior classification at intersections and validation on large naturalistic data set.", IEEE Transactions on Intelligent Transportation Systems 13.2 (2012): 724 - 736.

[0116] Static TTI (Time - To - Intersection) uses the estimated time to reach an intersection as a classification criterion. In its simplest form, TTI is

Number

[0117] Static RDP (Required Deceleration Parameter) is given the current speed and position of the vehicle and calculates the deceleration required for the vehicle to stop safely. RDP is

Number

Number

[0118] Similar to the static TTI algorithm, the RDP alert parameter reflects the level of conservatism of the rule-based algorithm.

[0119] In this specification, a rule-based approach is used as a baseline for evaluating the performance of the machine learning algorithms of the present disclosure, and in some cases, they are executed in parallel with the machine learning algorithms to capture rare cases that the machine learning may not be able to predict.

[0120] Machine learning

[0121] Modeling a driver's behavior has been shown to be a complex task considering the complexity of human behavior. See H.M. Mandalia and D.D. Dalvucci, "Using Support Vector Machines for Lane-Change Detection", Human Factors and Ergonomics Society Annual Meeting Proceedings, vol. 49, pages 1965{1969, 2005. Machine learning techniques are well-suited to modeling human behavior but require "learning" using training data to operate properly. In order to provide excellent detection and prediction results, herein, during the training period before the current traffic is applied with warning processing during the deployment phase, machine learning is used to model the traffic detected at intersections or other features of the ground transportation network. Machine learning can be further used to model a driver's response using in-vehicle data from an on-board equipment (OBE), and can be further based on the history and preferences of in-vehicle sensors and driving records. Herein, machine learning models are further used to detect and predict the trajectories, behaviors, and intentions of road users who may be victims (e.g., pedestrians). Machine learning can be further used to model the responses of road users who may be victims from an on-board person equipment (OPE). These models can include interactions between entities, between road users who may be victims, and between one or more entities and one or more road users who may be victims.

[0122] Machine learning techniques can be further used to model the behavior of non-autonomous ground transportation entities. By observing or communicating with non-autonomous ground transportation entities, or both, machine learning can be used to predict the intentions of non-autonomous ground transportation entities and to communicate with non-autonomous ground transportation entities and other involved entities when a near miss, or an accident, or other dangerous situation is predicted.

[0123] The machine learning mechanism functions properly in two phases, namely, 1) training, and 2) deployment.

[0124] Training phase

[0125] After installation, the RSE begins to collect data from sensors accessible to the RSE. Since AI model training requires powerful computing capabilities, AI model training is typically performed on a powerful server that includes multiple parallel processing modules to speed up the training phase. For this reason, the data obtained at the location of the RSE in the ground transportation network can be packaged and sent to a remote powerful server shortly after acquisition. This is done using an internet connection. Next, the data is prepared automatically or with the help of a data scientist. Next, an AI model is constructed to capture important characteristics of the traffic flow of vehicles and other ground transportation entities for that intersection or other aspects of the ground transportation network. The captured data features may include the position, direction, and movement of vehicles or other ground transportation entities, and the position, direction, and movement of vehicles or other ground transportation entities can then be converted into intent and behavior. By knowing the intent, the behavior and future behavior of vehicles or other ground transportation entities approaching the traffic location can be predicted with high accuracy using the AI model. The trained AI model is tested on a subset of data not included in the training phase. If the performance of the AI model meets expectations, the training is considered complete. This phase is repeatedly iterated using different model parameters until a satisfactory performance of the model is achieved.

[0126] Deployment phase

[0127] In some embodiments, next, the completed and tested AI model is transmitted over the Internet to the RSE at the traffic location in the ground transportation network. Next, the RSE is ready to process new sensor data and to perform predictions and detections of dangerous situations, such as traffic signal violations. If a dangerous situation is predicted, the RSE generates an appropriate warning message. Before the predicted dangerous situation occurs, the dangerous situation can be predicted, a warning message can be generated, and the warning message can be broadcast to vehicles and other ground transportation entities in the vicinity of the RSE and received by the vehicles and other ground transportation entities in the vicinity of the RSE. This gives the operator of the vehicle or other ground transportation entity sufficient time to react and to perform collision avoidance. The outputs of the AI model from the various intersections where the corresponding RSEs are located can be recorded and made available online in a dashboard that incorporates all the data generated and displayed in an intuitive and user-friendly manner. Such a dashboard can be used as an interface to the customers of the system (e.g., city traffic engineers or planners). An example of a dashboard is a map that shows the location of the intersections being monitored, the violation events that have occurred, and markers indicating statistical data and analysis results based on the AI predictions and the actual results.

[0128] Smart RSE (SRSE: Smart RSE) and Bridge for Connected / Unconnected Entities

[0129] As already suggested, there is a gap between the capabilities and behaviors of connected entities and unconnected entities. For example, connected entities are typically collaborative entities that continuously advertise their location and safety system status, such as speed, direction of travel, brake status, and headlight status, to the world. Unconnected entities cannot collaborate and communicate by these means. Therefore, they do not recognize unconnected entities that are outside the detection range due to not being in the vicinity of connected entities or due to interference, distance, or lack of a clear line of sight.

[0130] With appropriate equipment and configuration, an RSE can be enabled to detect all entities using the ground transportation network in its vicinity, including unconnected entities. Special sensors can be used to detect different types of entities. For example, radar is suitable for detecting moving metallic objects, such as cars, buses, and trucks. Such road entities are most likely moving in one direction towards an intersection. Cameras are suitable for detecting road users who may be contemplating potential hazards around an intersection, timing their crossing for safety.

[0131] Placing sensors on components of the ground transportation network has at least the following advantages.

[0132] - Good visibility points: Infrastructure poles, beams, and support cables typically include high visibility points. High visibility points enable a more comprehensive view of intersections. This is similar to the control tower at an airport, where the controller has an overview of important and potentially vulnerable users on the ground. In contrast, for ground traffic entities, the view from a good visibility point of sensors (such as cameras, lidars, radars, etc., or others) can be obstructed or interfered with by trucks in adjacent lanes, direct sunlight, or other interferences. Sensors at intersections can be selected to be resistant to and less susceptible to such interferences. For example, radar is not affected by sunlight and remains effective during evening commuting. Thermal cameras are likely to detect pedestrians in bright light situations where the view of an optical camera is obstructed.

[0133] - Fixed position: Sensors located at intersections can be adjusted and fixed to detect in specific directions that may be optimal for detecting important targets. This helps the processing software to more appropriately detect objects. As an example, if a camera has a fixed view, information about the background (stationary objects and structures) in the fixed view can be easily detected and used to improve the identification and classification of relatively important moving entities.

[0134] A fixed sensor position further enables easier placement of each entity in an integrated global view of the intersection. Since the sensor's view is fixed, the measurements from the sensor can be easily mapped to an integrated global position map of the intersection. Such an integrated map is useful when performing a global analysis of traffic movement from all directions to investigate interactions and dependencies of one traffic flow on another. An example is detecting a near miss (hazardous situation) before it occurs. When two entities are moving along a path where they will cross, the global and integrated view of the intersection enables calculation of the arrival time of each entity at the intersection for each path. If the time is within a certain limit or tolerance, the near miss can be flagged (e.g., targeted by an alert message) before it occurs.

[0135] With the help of sensors installed on infrastructure components, a Smart RSE (SRSE) can bridge this gap and enable connected entities to recognize "dark" or unconnected entities.

[0136] Figure 8 shows a scenario explaining how strategically placed sensors can help a connected entity identify the speed and position of an unconnected entity.

[0137] The connected entity 1001 moves along path 1007. The entity 1001 has its blue signal 1010 on. The unconnected entity 1002 is moving along path 1006. The unconnected entity 1002 is the target of the red signal 1009 and is trying to turn right along path 1006 at the red signal. This positions the entity 1002 directly in the path of the entity 1001. Since the entity 1001 does not recognize the entity 1002, a dangerous situation is imminent. Since the entity 1002 is an unconnected entity, it cannot broadcast (e.g., advertise) its position and direction of travel to other entities sharing the intersection. Further, even if the entity 1001 were connected, the entity 1001 cannot "see" the entity 1002 that is made invisible by the building 1008. There is a risk that the entity 1001 will go straight through the intersection and collide with the entity 1002.

[0138] When the intersection is configured as a smart intersection, the radar 1004 mounted on the beam 1005 above the road at the intersection detects the entity 1002, as well as the speed and distance of the entity 1002. This information can be relayed to the connected entity 1001 through the SRSE 1011 that functions as a bridge between the unconnected entity 1002 and the connected entity 1001.

[0139] Artificial Intelligence and Machine Learning

[0140] The Smart RSE further relies on learning traffic patterns and the behavior of entities in order to more appropriately predict and prevent dangerous situations and to avoid collisions. As shown in FIG. 8, radar 1004 constantly detects and provides data regarding each entity moving along approach path 1012. This data is collected and transmitted directly or through the RSE to the cloud for, e.g., analysis and to build and train a model that accurately represents the traffic along approach path 1012. When the model is complete, the model is downloaded to the SRSE 1011. This model can then be applied to each entity moving along approach path 1012. If an entity is classified by the model as violating (or attempting to violate) traffic rules, a warning (alert) can be broadcast by the SRSE to all connected entities in the vicinity. This warning, known as an intersection collision avoidance warning, can be received by the connected entities and can influence taking into account dangerous situations and avoiding collisions. By using an appropriate traffic model, violating entities can be detected in advance and sufficient time can be given to the connected entities using the intersection to react and avoid dangerous situations.

[0141] With the help of multiple sensors (some mounted high on components of the ground transportation network infrastructure), an artificial intelligence model, and an accurate traffic model, the SRSE can obtain a virtual overview of the ground transportation network and recognize each entity within its field of view, including unconnected entities within its field of view that are not "visible" to the connected entities within its field of view. The SRSE can use this data to supply an AI model and can provide warnings to connected entities instead of unconnected entities. Without doing so, the connected entities would not know that there are unconnected entities sharing the road.

[0142] The SRSE can utilize high-capacity computing power at the location of the SRSE within the same housing, or by connection to nearby units, or through an Internet connection to a server. The SRSE can process data directly received from sensors, or data received by broadcast from nearby SRSEs, emergency and weather information, and other data. The SRSE further comprises high-capacity storage useful for storing and processing data. High-bandwidth connectivity is required to further facilitate the transfer of raw data and AI models between the SRSE and a more powerful remote server. The SRSE enhances other traffic hazard detection technologies using AI to achieve high accuracy and provide additional time to react and avoid collisions.

[0143] The SRSE can still conform to current and new standardized communication protocols, and thus the SRSE can be seamlessly interfaced with equipment already deployed on site.

[0144] The SRSE can further reduce network congestion by sending messages only when necessary.

[0145] Global and integrated intersection topology

[0146] Effective traffic monitoring and control of intersections benefits from an aerial view of the intersection that is not obstructed by obstacles, lighting, or any other interference.

[0147] As described so far, different types of sensors can be used to detect different types of entities. The information from these sensors can be different, for example, they do not match in terms of the position or motion parameters represented by the data, or in terms of the native format of the data, or both. For example, radar data typically includes speed, distance, and in some cases, additional information such as the number of moving and stationary entities within the radar's field of view. In contrast, camera data can represent an image of the field of view at any given point in time. Lidar data can provide the positions of points in 3D space corresponding to the reflection points of laser beams emitted from the lidar at a specific time and direction of travel. Generally, each sensor provides data in a native format that appropriately represents the physical quantity measured by the sensor.

[0148] To obtain an integrated view (representation) of the intersection, the fusion of data from different types of sensors is useful. For the purpose of fusion, data from various sensors is converted into a common (integrated) format that is independent of the sensors used. The data included in the integrated format from all of the sensors includes the global position, speed, and direction of travel of each entity using the intersection, regardless of how it was detected.

[0149] By using this integrated global data, the smart RSE can not only detect and predict the movement of entities, but can also further identify the relative positions and directions of travel of different entities with respect to each other. Therefore, the SRSE can achieve improved detection and prediction of dangerous situations.

[0150] For example, in the scenario shown in FIG. 9, the motor-driven entity 2001 and the potentially affected road user 2002 share the same crosswalk. The entity 2001 is moving along the road 2007 and is detected by the radar 2003. The potentially affected road user 2002 walking along the sidewalk 2006 is detected by the camera 2004. The potentially affected road user 2002 may decide to cross the road 2007 using the crosswalk 2005. Doing so would create a potentially dangerous situation for the road user 2002 in the path of the entity 2001. Since each of the sensors may only detect entities within their respective fields of view, if the data from each of the sensors 2003 and 2004 is considered independently and no other information is considered, the dangerous situation will not be identified. Furthermore, each of the sensors may not be able to detect objects that are not designed to be detected by the sensor. However, when an integrated view is considered by the SRSE, the positions and dynamics of the entity 2001 and the potentially affected road user 2002 can be placed in the same reference coordinate system, i.e., a geographic coordinate system, such as a map projection, or another coordinate system. When considered within a common reference system, the fused data from the sensors can be used to detect and predict potentially dangerous situations that may occur between the two entities 2001 and 2002. The conversion between the sensor space and the integrated space is discussed in the following paragraphs.

[0151] Conversion from Radar Data to an Integrated Reference

[0152] As shown in FIG. 10, radar 3001 is used to monitor road entities moving along a road that includes two lanes 3005 and 3008, each including center lines 3006 and 3007. Stop bar 3003 indicates the ends of lanes 3005 and 3008. T3006 can be defined by a set of markers 3003 and 3004. FIG. 10 shows only two markers, but generally, the center line is a piecewise linear function. The global positions of markers 3003 and 3004 (and other markers not shown) are predefined by the lane design and are known to the system. The exact global position of radar 3001 can be further specified. Thus, distances 3009 and 3010 from radar 3001 to markers 3003 and 3004 can be calculated. Distance 3011 from radar 3001 to entity 3002 can be measured by radar 3001. Using simple geometry, the system can identify the position of entity 3002 using the measured distance 3011. Since it is derived from the global positions of markers 3003, 3004, and radar 3001, the result is a global position. Since each lane can be approximated by a generalized piecewise linear function, the above method can be applied to any lane that can be monitored by radar.

[0153] FIG. 11 shows a similar scenario on a curved road. Radar 4001 monitors entities moving along road 4008. Markers 4003 and 4004 represent linear segments 4009 of the center line 4007 (piecewise linear function). Distances 4005 and 4006 typically represent the normal distances between the plane 4010 of radar 4001 and markers 4003 and 4004, respectively. Distance 4007 is the measured distance of entity 4002 from radar plane 4010. Continuing with the above explanation, given the global positions of radar 4001 and markers 4003 and 4004, the global position of entity 4002 can be calculated using simple arithmetic of ratios.

[0154] Conversion from Camera Data to Integrated Criteria

[0155] By knowing the height, global position, orientation, tilt, and field of view of the camera, calculating the global position of each pixel in the camera image simplifies using existing 3D geometric laws and transformations. As a result, when an object is identified in the image, its global position can be easily estimated by knowing the pixels the object occupies. It is beneficial to note that the type of camera is irrelevant when the camera specifications, such as sensor size, focal length, or field of view, or combinations thereof, are known.

[0156] FIG. 12 shows a side view of a camera 5001 looking at an entity 5002. The height 5008 and tilt angle 5006 of the camera 5001 can be determined during installation. The field of view 5007 can be ascertained from the specifications of the camera 5001. The global position of the camera 5001 can also be specified during installation. From the known information, the system can identify the global positions of points 5003 and 5004. The distance between point 5003 and point 5004 is further divided into pixels in the image generated by the camera 5001. This number of pixels is known from the specifications of the camera 5001. The pixels occupied by the entity 5002 can be identified. Thus, the distance 5005 can be calculated. The global position of the entity 5002 can also be calculated.

[0157] By fusing information from various sensors, a global integrated view of any intersection can be pieced together. FIG. 13 shows a top view of a four-way intersection. Each leg of the intersection is divided by the median 6003. The intersection in the figure is monitored by two different types of sensors, radar and camera, and the principles described herein can be generalized to other types of sensors. In this example, the radar monitoring area 6001 overlaps the camera monitoring area 6002. By using a unified global view, each entity moving between areas is continuously tracked within the unified global view. This enables, for example, the SRSE to easily identify the relevance between the actions of different entities. Such information enables a truly universal bird's-eye view of intersections and roadways. The integrated data from the sensors can then be fed into an artificial intelligence program as described in the following paragraphs.

[0158] FIG. 2 above showed the components of the RSE. Additionally, in the SRSE, the processing unit may further include one or more special processing units that can process data in parallel. An example of such a unit is a graphics processing unit or GPU. With the assistance of a GPU or similar hardware, machine learning algorithms can operate much more efficiently in the SRSE and provide results in real time. Such a processing architecture enables real-time prediction of dangerous situations and thus enables early warnings to be sent to provide entities with sufficient time to react and avoid collisions. Additionally, since the SRSE can perform processing that uses data from different sensors and different types of sensors, the SRSE can construct an integrated view of the intersection that is useful for analyzing traffic flow and detecting and predicting dangerous situations.

[0159] Usage examples

[0160] A wide variety of examples can benefit from the system and from the early warnings it can provide for collision avoidance. Examples are provided herein.

[0161] Example 1: Ground transportation entities that can be victimized

[0162] As shown in FIG. 4, the lanes crossing a typical intersection 409 may include a crosswalk that includes specific crosswalk areas 401, 402, 403, 404 that can be used by pedestrians and other road users who can be victimized (road users who can be victimized) to cross the lanes on foot. Sensors suitable for detecting such crossers or other users who can be victimized are located at one or more vantage points that enable monitoring of the crosswalk and the area surrounding the crosswalk. During the training phase, the data collected can be used to train an artificial intelligence model to learn about the behavior of road users who can be victimized at the intersection. Next, during the deployment phase, the AI model can use current data regarding road users who can be victimized to predict, for example, that a road user who can be victimized is about to cross a lane, and can make that prediction before the road user who can be victimized begins to cross. If the behavior and intentions of pedestrians, and other road users who can be victimized, drivers, vehicles, and other people and ground transportation entities can be predicted in advance, an early warning (e.g., an alarm) can be sent to any or all of them. The early warning can enable a vehicle to stop, slow down, change route, or a combination thereof, and can enable a road user who can be victimized to refrain from crossing the road if a dangerous situation is predicted to be imminent.

[0163] Generally, sensors are used to monitor all areas of possible movement of road users and vehicles that may be damaged in the vicinity of an intersection. The type of sensor used depends on the type of object being monitored and tracked. Some sensors are better at tracking people and bicycles or other non-motorized vehicles. Some sensors are better at monitoring and tracking motorized vehicles. The type of sensor is irrelevant if the sensor provides appropriate data at a sufficient data rate that may depend on the type of object being monitored and tracked, so the solutions described herein are regardless of the sensor and hardware. For example, Doppler radar is a suitable sensor for monitoring and tracking the speed and distance of vehicles. The data rate, or sampling rate, is the rate at which the radar can provide successive new data values. The data rate must be fast enough to capture the dynamics of the movement of the object being monitored and tracked. The higher the sampling rate, the more detailed is captured and the representation of the movement by the data is more robust and accurate. If the sampling rate is overly low and the vehicle moves a significant distance between two sample instances, it becomes difficult to model the behavior due to missed details during the intervals where no data is generated.

[0164] For a crosswalk, the sensors monitor pedestrians crossing and other road users (e.g., cyclists) who can be victims in the area of the intersection and in the vicinity of the intersection. Data from these sensors can be segmented to represent the state using different virtual zones so as to be useful for detection and location identification. The zones can be selected to correspond to each critical area where a dangerous situation can be assumed, such as, for example, the sidewalk, the entrance to the sidewalk, and the incoming access roads 405, 406, 407, 408 to the intersection. The actions and other conditions in each zone are recorded. The records can include, but are not limited to, kinematics (e.g., position, direction of travel, speed, and acceleration), and facial and body features (e.g., eyes, posture).

[0165] The number of sensors, the number of zones, and the shape of the zones are specific to each intersection and to each access road to the intersection.

[0166] FIG. 5 shows a plan view of a typical exemplary setup showing different zones used to monitor and track the movement and behavior of pedestrians or other road users who can be victims, and motor-driven vehicles and non-motor-driven vehicles, and other ground traffic entities.

[0167] The sensors are configured to monitor a crosswalk that crosses the road. Virtual zones (301, 302) can be located on the sidewalk and along the crosswalk. Other sensors are arranged to monitor vehicles and other ground traffic entities traveling on the road following the crosswalk, and virtual zones (303, 304) are strategically arranged to be useful, for example, for detecting incoming vehicles and other ground traffic entities, their distance from the crosswalk, and their speed.

[0168] A system (e.g., an RSE or SRSE related to sensors) collects a data stream from all sensors. When the system is first put into operation, an initial rule-based model can be deployed to assist with device calibration and functionality. Meanwhile, sensor data (e.g., speed and distance from a radar unit, images and videos from a camera) is collected, locally stored in the RSE while being prepared, and in some embodiments, this collected data is transmitted to a remote computer powerful enough to build an AI model of the behavior of different entities at an intersection. In some examples, the RSE itself is an SRSE capable of generating an AI model.

[0169] Accordingly, the data is prepared and a trajectory is constructed for each ground traffic entity passing through the intersection. For example, the trajectory can be derived from radar data by connecting points at different distances belonging to the same entity. The trajectory and behavior of a pedestrian can be derived, for example, from camera and video recordings. By implementing video and image processing techniques, the movement of a pedestrian can be detected in images and videos, and their respective trajectories can be estimated.

[0170] For human behavior, intelligent machine learning-based models typically outperform simple rules based on simple physics. This is because human intentions are difficult to capture and a large data set is required to be able to detect patterns.

[0171] When a machine learning (AI) model is completed on a server, the AI model is downloaded to the RSE, e.g., through the Internet. The RSE then applies the current data captured from the sensors to the AI model, resulting in its prediction of intentions and behaviors, identifying when a dangerous situation is imminent, and triggering corresponding warnings to vehicles and other ground traffic entities, and to road users and drivers who may be affected, as early warnings (e.g., broadcast) so as to enable the road users and drivers who may be affected to execute collision avoidance steps in time.

[0172] This exemplary setting can be combined with any other use case, e.g., traffic at signalized intersections or un-signalized intersections.

[0173] Example 2: Signalized intersection

[0174] In the case of signalized intersections (e.g., those controlled by traffic lights), the overall setting of the system is carried out as in Example 1. One difference can be the type of sensors used to monitor or track the speed, direction of travel, distance, and position of vehicles. The setting for the crosswalk in Example 1 can further be combined with the setting for signalized intersections for a more general solution.

[0175] The concept of the operation for the use case of signalized intersections is to track road users around the intersection using external sensors that collect data about the users or data communicated by the users themselves, predict the behaviors of the users, and broadcast warnings through different communication means regarding an approaching dangerous situation caused generally by violations of the traffic rules at the intersection, e.g., running a red light.

[0176] Data regarding road users can be collected using (a) entity data broadcast by each entity itself regarding the current state of each entity, e.g., through BSM or PSM, and (b) sensors installed externally to the infrastructure or on the vehicle, such as Doppler radar, ultrasonic sensors, vision cameras or thermal cameras, lidar, etc. As described above, the type of sensors selected, and the position and orientation of the sensors selected at an intersection must provide the most extensive coverage of the intersection or the part under investigation, and further, the data collected regarding entities approaching the intersection is the most accurate. Thus, the data collected enables the reconstruction of the current state of road users and the generation of accurate, timely, and useful VBSM (virtual basic safety message) or VPSM (virtual personal safety message). The frequency at which data must be collected depends on the potential hazards of each type of road user and the criticality of potential violations. For example, a motor-driven vehicle moving at high speed at an intersection typically requires 10 data updates per second to achieve real-time collision avoidance, while a pedestrian crossing the intersection at a much lower speed may require as few as 1 data update per second.

[0177] As described so far, FIG. 4 shows an example of a plan view of an intersection with a traffic signal including a detection virtual zone. These zones can segment each approach road to the intersection into separate lanes 410, 411, 412, 413, 405, 406, 407, 408, and further, each lane can be divided into areas corresponding to a general range of distances from the stop bar. The selection of these zones can generally be carried out empirically to suit the characteristics of a particular approach road and intersection. Segmenting the intersection enables a more accurate identification of the relative direction of travel, speed, acceleration, and position for each road user, and thus enables a more appropriate assessment of the potential risks presented by the road user to other surface traffic entities.

[0178] To determine whether the observed traffic situation is a dangerous situation, the system further needs to compare the result of the predicted situation with the state of the traffic lights and take into account local traffic rules (e.g., left-turn lanes, right-turn on red, etc.). Therefore, it is necessary to collect and use intersection signal phase and timing (SPaT) information. SPaT data can generally be collected by directly interfacing with the traffic signal control device at the intersection, by reading data through a wired connection, or by interfacing with the traffic management system to receive the required data, for example, through an API. To ensure that the state of the road users is always synchronized with the state of the traffic signals, it is important to collect SPaT data at a rate as close as possible to the rate at which road user data is collected. An additional complexity in the requirement to know SPaT information is that modern traffic control techniques used to regulate the flow of traffic near intersections do not rely on fixed timings but use algorithms that can adapt dynamically to real-time traffic conditions. Therefore, it is important to incorporate SPaT data prediction algorithms to ensure the highest accuracy in violation prediction. These SPaT data prediction algorithms can be constructed using rule-based methods or machine learning methods.

[0179] For each approach road to an intersection, data is collected by an RSE (or SRSE), and a machine learning (AI) model is configured to explain the behavior of vehicles corresponding to the collected data. Next, current data collected at the intersection is applied to the AI model to generate an early prediction as to whether a vehicle or other ground traffic entity moving on one of the approach roads to the intersection is about to violate, for example, a traffic signal. If a violation is imminent, a message is relayed (e.g., broadcast) from the RSE to ground traffic entities in the vicinity. Vehicles (including the violating vehicle), and pedestrians or other road users who may be affected receive the message and have time to take appropriate measures to avoid a collision. The message can be delivered to the ground traffic entities by the following means, among others, which may also be possible: one or more of a flashing light, a sign, or a radio signal.

[0180] If a vehicle or other entity approaching the intersection is equipped with an OBE or OPE, it can receive a message broadcast from the RSE that a potential hazard has been predicted at the intersection. This enables the user to be warned and to take appropriate measures to avoid a collision. If a violating road user at the intersection is also equipped with an OBE or OPE, the user also receives the broadcast warning. Next, an algorithm in the OBE or OPE can match the user's violating behavior with the message and can appropriately warn the user.

[0181] The determination to send an alert does not depend solely on the behavior of the vehicle represented by the data collected by sensors at the intersection. Although sensors play a major role in the determination, other inputs are also considered. These inputs may include, but are not limited to, information from nearby intersections (if a vehicle ran a signal at a nearby intersection, the probability that the vehicle will do the same at this intersection is higher), information from other connected vehicles, or even information from the vehicle itself, for example, if the vehicle reports that it has a problem.

[0182] Example 3: Intersection without traffic lights

[0183] Controlled intersections without traffic lights, for example, intersections controlled by a stop sign or a yield sign, can also be monitored in the same way. Sensors are used to monitor the access roads controlled by traffic signs, and predictions can be made about the incoming vehicles in the same way as for the incoming vehicles on the access roads to intersections with traffic lights. The road rules at controlled intersections without traffic lights are typically clearly defined. Ground traffic entities on access roads controlled by a stop sign must come to a complete stop. At multi-way stop intersections, the right of way is determined by the order in which ground traffic entities reach the intersection. Special cases associated with one-way stops can be considered. The set of sensors can also monitor access roads without a stop sign. Such a setting can assist in gap negotiation for stop signs. For intersections controlled by a yield sign, ground traffic entities on the access roads controlled by the yield sign must reduce their speed in order to give other ground traffic entities the right of way at the intersection.

[0184] The main problem is that due to internal factors (such as driver distraction) or external factors (such as lack of visibility), ground traffic entities violate the road rules and pose risks to other ground traffic entities.

[0185] In a general example of an intersection controlled by a stop sign (i.e., each approach is controlled by a stop sign), the overall configuration of the system is carried out as in the case of Example 1. One difference can be the type of sensors used to monitor or track the speed, direction of travel, distance, and position of vehicles. Another difference is that the road rules are indicated by road signs without including a traffic signal control device. The settings for the crosswalk in Example 1 can also be combined with the settings of a controlled intersection without signals for a more general solution.

[0186] FIG. 4 can further be understood to show an example of a plan view of a four-way stop intersection including detection virtual zones. These zones can segment each approach to the intersection into separate lanes 410, 411, 412, 413, 405, 406, 407, 408, and can further divide each lane into areas corresponding to a general range of distances from the stop bar. The selection of these zones can generally be done empirically to fit the characteristics of a particular approach and intersection.

[0187] Using a method similar to that described so far for FIG. 4, the current data collected at the intersection is applied to the AI model to generate an early prediction of whether a vehicle or other ground traffic entity moving on one of the approaches to the intersection is about to violate the stop sign. If a violation is imminent, the message can be handled in the same way as in the previous example involving a traffic signal violation.

[0188] Furthermore, similar to the description so far, the determination to send an alert can be based on the aforementioned factors and on other information, such as whether a vehicle has run through a stop sign at a nearby intersection (suggesting a higher probability that the vehicle will do the same at this intersection).

[0189] Figure 18 shows an example of use for an intersection without a controlled traffic signal. It illustrates how an SRSE, including strategically placed sensors, can warn connected entities of an impending dangerous situation arising from an unconnected entity.

[0190] Connected entity 9106 is moving along path 9109. Entity 9106 has the right of way. Unconnected entity 9107 is moving along path 9110. Entity 9107 is the subject of a yield sign 9104 and is attempting to merge onto path 9109 without giving the right of way to entity 9106 and is about to emerge directly into the path of entity 9106. Since entity 9106 does not recognize entity 9107, a dangerous situation is imminent. Since entity 9107 is an unconnected entity, the position and direction of travel of entity 9107 cannot be advertised (broadcast) to other entities sharing the intersection. Further, entity 9106 may not be able to "see" entity 9107 which is not within the direct field of view of entity 9106. If entity 9106 proceeds along its path, entity 9106 may ultimately collide with entity 9107.

[0191] Since the intersection is a smart intersection, radar 9111 mounted on beam 9102 above the road detects entity 9107. Radar 9111 further detects the speed and distance of entity 9107. This information can be relayed as a warning to connected entity 9106 through SRSE 9101. SRSE 9101 includes a machine learning model for entities moving along approach path 9110. Entity 9107 is classified by the model as a potential traffic rule violator and a warning is broadcast to connected entity 9106. This warning is sent in advance, giving entity 9106 sufficient time to react to and prevent the dangerous situation.

[0192] Example 4: Level crossing

[0193] Level crossings are dangerous because they allow motor - driven vehicles, pedestrians, and railway vehicles to pass through. In many cases, the roads following a level crossing enter the blind spot of the driver of a train or other railway vehicle (e.g., a conductor). Since railway vehicle drivers mainly drive based on line - of - sight information, this increases the likelihood of an accident when a road user violates the right - of - way of a railway vehicle and when a road user crosses a level crossing when not permitted to cross.

[0194] The operation of the use case of a level crossing, in the sense that a level crossing is often the collision point between road and railway traffic regulated by traffic rules and signals, is similar to the use case of an intersection with traffic lights. Therefore, this use case further requires collision avoidance warnings to enhance safety near level crossings. Railway traffic may have a planned separate railway right - of - way (e.g., high - speed railway) or may not have a separate railway right - of - way (e.g., light urban railway or tram). When using light railways and trams, these railway vehicles operate on more active roads and need to follow the same traffic rules as road users, so this use case becomes even more important.

[0195] FIG. 6 shows a general use case of a level crossing where a road and a crosswalk cross a railway. Similar to the use case of a crosswalk, sensors are placed to collect data on the movement and intention of pedestrians. Other sensors are used to monitor and predict the movement of road vehicles approaching the intersection. Data on road users can also be collected from road user broadcasts (e.g., BSM or PSM). Data from nearby intersections, vehicles, and the remote command and control center can be used for the decision to trigger an alarm.

[0196] In order to appropriately evaluate the likelihood of violations, it is also necessary to collect data on SPaT for the access roads of the road and the railway.

[0197] Similar to the usage example of the intersection with a traffic signal, the collected data enables the generation of a prediction model using rule-based algorithms and machine learning algorithms.

[0198] In this usage example, the railway vehicle is equipped with an OBE or OPE to receive a collision avoidance warning. If a violation of the railway vehicle's right of way is predicted, the RSE broadcasts an alarm message to warn the driver of the railway vehicle that a road user is on its intended route and enables the driver of the railway vehicle to take preemptive action with sufficient time to avoid a collision.

[0199] If the violating road user is further equipped with an OBE or OPE, the message broadcast by the RSE is further received by the violating road user. The algorithm in the OBE or OPE can then match the user's violation behavior with the received message and can appropriately warn the user.

[0200] Virtual connected ground traffic environment (bridging the gap)

[0201] As explained so far, a useful application of the system is to generate a virtual connected environment instead of unconnected ground traffic entities. The obstacles to the adoption of connected technologies are not only the lack of infrastructure installations, but also the almost non-existence of connected vehicles, road users who can be connected and damaged, and other connected ground traffic entities.

[0202] In connection with connected vehicles, in some regulatory regimes, such vehicles are constantly transmitting what is called a basic safety message (BSM). The BSM includes, among other information, the vehicle's position, direction of travel, speed, and future route. Other connected vehicles can pay attention to these messages and use them to generate a map of the vehicles present in their environment. By knowing where the surrounding vehicles are, vehicles have useful information for maintaining a high level of safety, whether the vehicle is autonomous or not. For example, an autonomous vehicle can maneuver to avoid if there are connected vehicles on its route. Similarly, a driver can receive an alert if there are any other vehicles on the route the driver is planning to follow, such as in the case of a sudden lane change.

[0203] Until all ground traffic entities are equipped to send and receive traffic safety messages and information, some road entities are "dark" or invisible to the rest of the road entities. Dark road entities pose a risk of dangerous situations.

[0204] Dark road entities do not advertise (e.g., broadcast) their positions, so they are invisible to connected entities that can expect all road entities to broadcast information about the road entity (i.e., to be connected entities). In-vehicle sensors can detect obstacles and other road entities, but the range of these sensors tends to be too short to be effective in preventing dangerous situations and collisions. Thus, there is a gap between the connectivity of connected vehicles and the lack of connectivity of unconnected vehicles. The techniques described below aim to bridge this gap by using intelligence regarding infrastructure that can detect all vehicles at intersections or other components of the ground transportation network and can send messages on behalf of unconnected vehicles.

[0205] The system can establish a virtual connected ground traffic environment, for example at intersections, that can bridge the gap between the future where most vehicles (and other ground traffic entities) are expected to be connected entities and the current time when most vehicles and other ground traffic entities have no connectivity. In the virtual connected ground traffic environment, smart traffic lights and other infrastructure installations can use sensors to track all vehicles and other ground traffic entities (connected, unconnected, semi-autonomous, autonomous, non-autonomous), and (in the case of vehicles) can generate virtual BSM messages (VBSMs) instead of them.

[0206] VBSM messages can be considered a subset of BSMs. It may not contain all the fields required to generate a BSM, but it can contain all the location-specific information including location, direction of travel, speed, and trajectory. Since V2X communication is standardized and anonymized, VBSMs and BSMs cannot be easily distinguished and follow the same message structure. The main difference between the two messages is the availability of the source of the information contained in these messages. VBSMs may not contain data and information that are not easily generated by external sensors, such as steering wheel angle, brake status, tire pressure, or wiper activation.

[0207] By installing appropriate sensors, intersections with smart RSEs can detect all road entities moving through the intersection. The SRSE can further convert all data from multiple sensors into a global integrated coordinate system. This global integrated system is represented by the geographical location, speed, and direction of travel of each road entity. Each road entity is detected by intersection equipment regardless of whether each road entity is connected, and a global integrated position is generated in place of each road entity. Therefore, standard safety messages can be broadcast in place of road entities. However, for all entities detected by the RSE, if the RSE broadcasts a safety message, the RSE can send the message in place of the connected road entity. To resolve conflicts, the RSE can filter the connected road entities from its list of dark entities. This can be achieved because the RSE continuously receives safety messages from connected vehicles and because the RSE sensors continuously detect road entities passing through the intersection. If the position of the detected road entity is at a position where a safety message from that position is received by the RSE receiver, the road entity is presumed to be connected and the RSE does not broadcast a safety message in place of that road entity. This is shown in Figure 15.

[0208] By creating a bridge between connected and unconnected vehicles, connected entities (including autonomous vehicles) can safely navigate through intersections with a complete awareness of all nearby road entities.

[0209] This aspect of the technology is shown in FIG. 17. Intersection 9001 includes a plurality of road entities at a given point in time. Some of these entities are unconnected ones 9004, 9006, and others are connected ones 9005, 9007. Road users 9004, 9007 who may be damaged are detected by camera 9002. Motor-driven road entities 9005, 9006 are detected by radar 9003. The position of each road entity is calculated. Broadcasts from connected road entities are further received by RSE 9008. The position of the entity from which a message has been received is compared with the position where the entity was detected. If two entities match within a predetermined tolerance, the entity at that position is considered to be connected, and a safety message is not transmitted in place of that entity. The remaining road entities without a matching received position are considered dark. Instead, a safety message is broadcast.

[0210] For collision warnings and intersection violation warnings, which are integral parts of the V2X protocol, each entity needs to be connected for the system to be effective. That requirement poses a hurdle in the deployment of V2X devices and systems. An intersection equipped with a smart RSE addresses that concern by providing a virtual bridge between connected and unconnected vehicles.

[0211] The US Department of Transportation (DOT) and the National Highway Traffic Safety Administration (NHTSA) have identified many connected vehicle applications that use BSM and help substantially reduce non-impaired crashes and fatalities. These applications include, but are not limited to, Forward Collision Warning (FCW), Intersection Movement Assist (IMA), Left Turn Assist (LTA), Do Not Pass Warning (DNPW), and Blind Spot / Lane Change Warning (BS / LCW). The US DOT and NHTSA define these applications as follows.

[0212] FCW resolves rear-end collisions and warns the driver about stopped, slowing, or slower vehicles ahead. IMA is designed to avoid intersection broadside collisions, warns the driver about vehicles approaching from the side at an intersection, and covers two main scenarios: a path change route in the same or opposite direction and a straight-through crossing route. LTA resolves collisions when a participating vehicle makes a left turn at an intersection and another vehicle moves straight from the opposite direction, and warns the driver of the presence of oncoming traffic when attempting to make a left turn. DNPW assists the driver in avoiding oncoming collisions caused by passing maneuvers and warns the driver of oncoming vehicles when attempting to pass a slower vehicle on an undivided two-lane roadway. BS / LCW resolves collisions when a vehicle makes a lane change / merging maneuver prior to a collision and alerts the driver about approaching vehicles or the presence of those vehicles in their blind spots in adjacent lanes.

[0213] The V2X protocol defines that these applications must be achieved using vehicle-to-vehicle (V2V) communication, where a connected remote vehicle broadcasts basic safety messages to a connected host vehicle. Next, the OBE of the host vehicle attempts to match its own vehicle parameters, such as speed, direction of travel, and trajectory, with these BSMs, and determine whether there is a potential danger or threat caused by the remote vehicle as described hereinbefore in this specification. Further, autonomous vehicles particularly benefit from such applications because the surrounding vehicles can communicate the intention, which is an important information part not included in the data collected from its in-vehicle sensors.

[0214] However, current vehicles are not connected, and as described above, it takes a very long time until the proportion of connected vehicles is high enough for the BSM to function properly as described above. Therefore, if the proportion of connected vehicles is high enough to enable the above applications and fully benefit from V2X communication, a connected vehicle will receive and analyze a large number of BSMs that a connected vehicle would receive in an environment where the proportion of connected vehicles is small, which is not required.

[0215] VBSM can help bridge the gap between the current environment, which includes entities that are mostly not connected, and the future environment, which includes entities that are mostly connected, and can enable the above applications during the interim period. In the technology described herein, a connected vehicle that receives a VBSM processes the VBSM as a regular BSM in that application. Since the VBSM and the BSM follow the same message structure, and since the VBSM contains substantially the same basic information as the BSM, such as speed, acceleration, direction of travel, past trajectory, and predicted trajectory, the result of applying the message to a given application is substantially the same.

[0216] For example, consider an intersection with an unprotected left turn where a connected host vehicle is attempting a left turn at the instant when a distant, unconnected vehicle is moving linearly from the opposite direction with right of way. This is a situation where the execution of the maneuver depends on the judgment of the driver of the host vehicle in this situation. An inappropriate assessment of this situation can lead to conflicts and potentially dangerous approaches or collisions. External sensors installed in the surrounding infrastructure can detect and track the distant vehicle or even both vehicles, collect basic information such as speed, acceleration, direction of travel, and past trajectory, send them to the RSE, and subsequently the RSE can construct a predicted trajectory for the distant vehicle using a rule-based algorithm or a machine learning algorithm or both, fill in the fields required for VBSM, and broadcast it instead of the unconnected distant vehicle. The OBE of the host vehicle receives the VBSM containing information about the distant vehicle and processes the VBSM in its LTA application to determine whether the driver's maneuver poses a potential danger and whether the OBE must display a warning to the driver of the host vehicle to take a forward-looking or corrective action to avoid a collision. Similar results can also be achieved when data is received from the RSE and sensors that a oncoming vehicle with a predicted collision is attempting a left turn while the distant vehicle is connected.

[0217] VBSM can also be used in the operation of lane changes. Such an operation can be dangerous if a vehicle changing lanes is not performing the necessary steps to check the safety of the operation, for example, using the rearview mirror and side mirrors, and checking for blind spots. New advanced driver assistance systems have been developed to help prevent a vehicle from performing a dangerous lane change, such as blind spot warnings using in-vehicle ultrasonic sensors. However, these systems can have drawbacks in cases where the sensors are dirty or the view is blocked. Furthermore, existing systems do not attempt to warn a vehicle at risk about another vehicle attempting to change lanes. V2X communication can help solve this problem through applications such as BS / LCW using BSM, but a vehicle attempting to change lanes may correspond to an unconnected vehicle and as a result may not be able to communicate its intention. VBSM can help achieve its goal. Similar to the usage example of LTA, external sensors installed in the surrounding infrastructure can detect and track an unconnected vehicle attempting to change lanes, collect basic information such as speed, acceleration, direction of travel, and past trajectory, and transmit them to the RSE. Next, the RSE constructs a predicted trajectory for the vehicle changing lanes using rule-based algorithms and machine learning algorithms, fills in the fields required for VBSM, and broadcasts it instead of the unconnected remote vehicle. Next, the OBE of the vehicle at risk receives a VBSM containing information about the vehicle attempting to merge into the same lane, processes the VBSM, and determines whether the operation poses a potential risk and whether it should display a lane change warning to the driver of the vehicle. If the vehicle changing lanes is a connected vehicle, its OBE can similarly receive a VBSM from the RSE regarding the vehicles in its blind spot and determine whether the lane change operation poses a potential risk to the surrounding traffic and whether it should display a blind spot warning to the driver of the vehicle.When both vehicles are connected, both vehicles can broadcast BSM to each other, enabling BS / LCW applications. However, these applications still benefit from applying the same rule-based algorithm or machine learning algorithm (or both) to BSM data, as described above, to predict early the intention of the vehicle changing lanes, which involves determining whether the OBE displays a warning.

[0218] Autonomous vehicle

[0219] The lack of connectivity to unconnected road entities affects autonomous vehicles. The sensors in autonomous vehicles are either short-range or have a narrow field of view. They cannot detect, for example, a vehicle coming near a building at a street corner. They also cannot detect a vehicle that may be hidden behind a delivery truck. If these hidden vehicles are unconnected entities, they are invisible to autonomous vehicles. These situations affect the technical capabilities of autonomous vehicle technology to achieve the safety levels required for large-scale adoption of the technology. Smart intersections can help mitigate this gap and assist in the public's acceptance of autonomous vehicles. Autonomous vehicles only have the same goodness as the sensors they possess. Intersections equipped with smart RSEs can extend the range of in-vehicle sensors around blind corners or beyond large trucks. Such extensions enable autonomous entities and other connected entities to coexist with traditional unconnected vehicles. Such coexistence can accelerate the adoption of autonomous vehicles and the benefits they bring.

[0220] The virtual connected ground traffic environment includes VBSM messages that enable the implementation of vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), and vehicle-to-device (V2D) applications that were difficult to implement by other means.

[0221] The system uses machine learning to quickly and accurately generate the data fields required for various safety messages, pack them into a VBSM message structure, and transmit the messages to ground traffic entities in the vicinity using various media such as, but not limited to, DSRC, Wi-Fi, cellular, or conventional road signs.

[0222] Virtual personal safety messages (VPMS)

[0223] The ground traffic environment can include not only unconnected vehicles but also unconnected people and other road users who can be victims.

[0224] In some regulatory regimes, ground traffic entities that can be victims of connectivity continuously transmit personal safety messages (PSM). PSM can include, among other information, the location, direction of travel, speed, and future route of the ground traffic entity that can be a victim. Connected vehicles and infrastructure can receive these messages, use them to generate a map that includes the entities that can be victims, and enhance the level of safety in the ground traffic network.

[0225] Therefore, a virtual connected ground traffic environment can bridge the gap between a future where most ground traffic entities that can be victims are connected and the current time when most ground traffic entities that can be victims have no connectivity. In a virtual connected ground traffic environment, smart traffic lights and other infrastructure installations can use sensors to track all (connected and unconnected) ground traffic entities that can be victims and generate VPSM on their behalf.

[0226] The VPSM message can be considered a subset of the PSM. The VPSM does not need to include all the fields required to generate the PSM, but may include data necessary for safety assessment and prevention of dangerous situations, and may include location-specific information including location, direction of travel, speed, and trajectory. In some examples, non-standard PSM fields such as, for example, the driver's intent, posture, or direction of view may be further included in the VPSM.

[0227] The system can use machine learning to quickly and accurately generate these fields, pack them into the VPSM message structure, and transmit it to ground traffic entities in the vicinity using various media such as, but not limited to, DSRC, Wi-Fi, cellular, or conventional road signs.

[0228] The VPSM message enables the implementation of pedestrian-to-vehicle (P2V), pedestrian-to-infrastructure (P2I), pedestrian-to-devices (P2D), vehicle-to-pedestrian (V2P), infrastructure-to-pedestrians (I2P), and device-to-pedestrians (D2P) applications that were difficult to implement by other means.

[0229] Figure 16 shows a pedestrian 8102 crossing a crosswalk 8103. The crosswalk 8103 can be a crosswalk at an intersection or a mid-block crosswalk spanning a continuous stretch of road between intersections. A camera 8101 is used to monitor a sidewalk 8104. The global position of the boundary of the camera 8101's field of view 8105 can be determined at the time of installation. The field of view 8105 is covered by a predetermined number of pixels reflected by the specifications of the camera 8101. A road entity 8102 can be detected within the camera's field of view, and its global position can be calculated. The speed and direction of travel of the road entity 8102 can further be identified from its displacement at time s. The path of the road entity 8102 can be represented by a breadcrumb 8106 that is a sequence of positions where the entity 8102 crossed. This data can be used to construct a virtual PSM message. Next, the PSM message can be broadcast to all entities in the vicinity of the intersection.

[0230] Traffic enforcement and behavior enforcement at intersections without traffic lights

[0231] Another useful application of the system is traffic enforcement (e.g., stop signs, yield signs) at intersections without traffic lights, and enforcement of good driving behavior anywhere in the surface transportation network.

[0232] As a byproduct of generating VBSM and VPSM, the system can track and detect road users who do not comply with traffic regulations and who increase the probability of dangerous situations and collisions. Prediction of dangerous situations can be extended to include enforcement. A dangerous situation does not necessarily result in a collision. Near misses are common, increase the stress level of drivers, and can lead to subsequent accidents. The frequency of near misses is positively correlated with the lack of enforcement.

[0233] Furthermore, using VBSM, the system can detect inappropriate driving behaviors, such as sudden lane changes and other forms of reckless driving. The data collected by the sensors can be used for training and can enable a machine learning model to flag ground traffic entities that are engaging in dangerous driving behaviors.

[0234] Authorities typically enforce road rules against ground traffic entities that can include road users who can be victims, but the authorities need to be present near intersections to monitor, detect, and report violations. By using VBSM and VPSM to track unconnected ground traffic entities that can include road users who can be victims, the smart RSE can perform the role of the authorities and enforce road rules at intersections. For example, an unconnected vehicle being tracked by the smart RSE can be detected, identified, and reported to the authorities for violating a stop sign or a yield sign. Similarly, a road user who can be a victim near an intersection being tracked by the smart RSE can be detected, identified, and reported to the authorities for illegally crossing the intersection.

[0235] For enforcement and other purposes, ground traffic entities can be identified using unique identification information including but not limited to license plate recognition. Road users who can be victims can be identified using biometric recognition including but not limited to face recognition, retina recognition, and voice waveform recognition. In special cases including civil incident investigations or criminal investigations, social media networks (e.g., Facebook, Instagram, Twitter) can be further used to support the identification of a violating ground traffic entity or a road user who can be a victim. Examples of social networks that can be utilized include uploading a captured photo of the violator to the social network and requesting social network users who recognize the violator to provide information to the authorities that can help identify the violator.

[0236] Other embodiments also fall within the scope of the claims described below. (Additional Clause 1) An apparatus comprising a device located at an intersection of a transportation network, wherein the device has an input for receiving data from a sensor oriented to monitor a surface transportation entity at the intersection or a surface transportation entity near the intersection, a wireless communication device for transmitting a warning regarding a dangerous situation at the intersection or a dangerous situation near the intersection to a device of one of the surface transportation entities, a processor, a storage, and wherein the storage stores a machine learning model capable of predicting the behavior of a surface transportation entity at the intersection or a surface transportation entity near the intersection at the current time, the machine learning model being based on training data regarding the behavior related to the previous operations of the surface transportation entity at the intersection or a surface transportation entity near the intersection, applies current operation data received from the sensor regarding a surface transportation entity at the intersection or a surface transportation entity near the intersection to the machine learning model to predict the imminent behavior of the surface transportation entity, infers an imminent dangerous situation for one or more of the surface transportation entities at the intersection or a surface transportation entity near the intersection from the predicted imminent behavior, causes the wireless communication device to transmit the warning regarding the dangerous situation to a device of one of the surface transportation entities, and is the storage for instructions executable by the processor for doing so. Apparatus. (Additional Clause 2) The device includes a roadside device The device according to appended claim 1 (Appended claim 3) The device includes a housing for the device The sensor is mounted on the housing The device according to appended claim 1 (Appended claim 4) The warning is transmitted by broadcasting the warning for reception by any of the ground traffic entities at the intersection or the ground traffic entities near the intersection The device according to appended claim 1 (Appended claim 5) The machine learning model includes an artificial intelligence model The device according to appended claim 1 (Appended claim 6) The training data and the operation data include at least one of speed, position, direction of travel, intention, attitude, or direction of view The device according to appended claim 1 (Appended claim 7) The processor is configured to enable generation of the machine learning model in the device The device according to appended claim 1 (Appended claim 8) The machine learning model is generated in the device The device according to appended claim 1 (Appended claim 9) The instructions are executable by the processor to store the training data in the device The device according to appended claim 1 (Appended claim 10) The intersection includes an intersection without traffic lights The device according to appended claim 1 (Appended claim 11) The transportation network includes a road network The device according to appended claim 1 (Appended claim 12) wherein the ground transportation entity includes road users who may be affected by damage The device according to appended claim 1 (Appended claim 13) wherein the ground transportation entity includes vehicles The device according to appended claim 1 (Appended claim 14) wherein the imminent dangerous situation includes a collision or a near miss The device according to appended claim 1 (Appended claim 15) wherein the ground transportation entity includes pedestrians crossing the road at a crosswalk The device according to appended claim 1 (Appended claim 16) comprising another communication device that communicates with a central server The device according to appended claim 1 (Appended claim 17) wherein one of the devices of the ground transportation entity includes a mobile communication device The device according to appended claim 1 (Appended claim 18) An apparatus comprising a device located within a ground transportation entity or the device located on the ground transportation entity, wherein the device is a sensor within the ground transportation entity or the sensor on the ground transportation entity, an input that receives data from the sensor oriented to monitor features near the ground transportation network and other information regarding the context in which the ground transportation entity crosses the ground transportation network, a wireless communication device that receives information regarding the context, a signal processor that applies signal processing to the data from the sensor and the other information regarding the context, a processor, a storage, and comprising wherein the storage Storing a machine learning model capable of predicting the behavior of the operator of the ground traffic entity and the intentions and movements of other ground traffic entities in the vicinity; Applying the currently received data from the sensor and other information regarding the context to predict the behavior of the operator and the intentions and movements of other ground traffic entities in the vicinity; A storage for instructions executable by the processor for doing so; Device. (Appended Claim 19) The instructions are executable by the processor to monitor a user or a person riding in the ground traffic entity; The device according to appended claim 18. (Appended Claim 20) The other information regarding the context includes emergency broadcasts, traffic and safety messages of roadside devices, and messages regarding safety, location, and other operation information from other ground traffic entities; The device according to appended claim 18. (Appended Claim 21) The sensor includes a camera, a distance sensor, a vibration sensor, a microphone, a seat sensor, a hydrocarbon sensor, a sensor for volatile organic compounds and other toxic substances, and a kinematic sensor, or a combination thereof; The device according to appended claim 18. (Appended Claim 22) Instructions are executable by the processor to filter the alarms received in the vehicle by applying the alarms to the machine learning model to predict which alarms are important in terms of current location, environmental conditions, driver behavior, vehicle condition and status, and kinematics; The device according to appended claim 18. (Appended Claim 23) The wireless communication device transmits a warning regarding a dangerous situation to a sign or other infrastructure presentation device; The device according to appended claim 18. (Appended Item 24) A warning includes an instruction or command capable of controlling a specific ground traffic entity, the device according to Appended Item 18. (Appended Item 25) Storing a machine learning model capable of predicting the behavior of a ground traffic entity at an intersection of a traffic network at a current time or a ground traffic entity near the intersection of the traffic network, wherein the machine learning model is based on training data related to the behavior associated with the previous actions of the ground traffic entity at the intersection or the ground traffic entity near the intersection, storing; Applying current operation data received from a sensor regarding the ground traffic entity at the intersection or the ground traffic entity near the intersection to the machine learning model to predict an imminent behavior of the ground traffic entity; Inferring an imminent dangerous situation for one or more of the ground traffic entities at the intersection or the ground traffic entity near the intersection from the predicted imminent behavior; Causing a wireless communication device to transmit a warning regarding the dangerous situation to a device of one of the ground traffic entities; A method including the above. (Appended Item 26) The warning is transmitted by broadcasting the warning for reception by any of the ground traffic entities at the intersection or the ground traffic entity near the intersection. The method according to Appended Item 25. (Appended Item 27) The machine learning model includes an artificial intelligence model. The method according to Appended Item 25. (Appended Item 28) The training data and the operation data include at least one of speed, position, traveling direction, intention, attitude, or viewing direction. The method according to Appended Item 25. (Additional item 29) A method according to claim 25, comprising generating the machine learning model in a device. A method according to claim 25, comprising storing the training data in a device. (Additional item 30) A method according to claim 25, comprising storing the training data in a device. A method according to claim 25, comprising storing the training data in a device. (Additional item 31) Receiving data from a sensor within or on a surface transportation entity, the sensor being oriented to monitor features near a surface transportation network, and other information regarding the context in which the surface transportation entity crosses the surface transportation network; Receiving information regarding the context; Storing a machine learning model capable of predicting the behavior of an operator of the surface transportation entity and the intentions and movements of other surface transportation entities in the vicinity; Applying the currently received data from the sensor and other information regarding the context to predict the behavior of the operator and the intentions and movements of other surface transportation entities in the vicinity; A method comprising the above. (Additional item 32) A method according to claim 31, comprising monitoring a user or a person riding on the surface transportation entity. A method according to claim 31, comprising monitoring a user or a person riding on the surface transportation entity. (Additional item 33) The other information regarding the context includes emergency broadcasts, traffic and safety messages of roadside devices, and messages regarding safety, location, and other operation information from other surface transportation entities. A method according to claim 31, comprising the above. (Additional item 34) The sensor includes a camera, a distance sensor, a vibration sensor, a microphone, a seat sensor, a hydrocarbon sensor, a sensor for volatile organic compounds and other toxic substances, and a kinematic sensor, or a combination thereof. A method according to claim 31, comprising the above. (Supplementary Item 35) Filtering the warnings received in the vehicle by applying the warnings to a machine learning model to predict which warnings are important in terms of the current position, environmental conditions, driver behavior, vehicle condition and status, and kinematics, The method according to Supplementary Item 31. (Supplementary Item 36) Including transmitting a warning regarding a dangerous situation to a sign or other infrastructure presentation device, The method according to Supplementary Item 31. (Supplementary Item 37) The warning includes an instruction or command capable of controlling a specific ground traffic entity, The device according to Supplementary Item 31. (Supplementary Item 38) In a road vehicle moving within a ground traffic network, Receiving messages from an external source regarding the position, movement, and status of other ground traffic entities, Receiving data from in-vehicle sensors regarding the road and driving conditions, and regarding the positions of static objects and moving ground traffic entities around the vehicle, Receiving data regarding the quality of driving by the driver of the road vehicle, Receiving a basic safety message from another ground traffic entity and a personal safety message from a road user who may be damaged, Fusing the received data and the messages, Applying the fused data and the messages to an artificial intelligence model to predict the behavior of the driver of the road vehicle or the behavior of a road user who may be damaged, or the collision risk to the road vehicle, or both the behavior of the driver of the road vehicle or the behavior of a road user who may be damaged and the collision risk to the road vehicle, A method including. (Supplementary Item 39) In the road vehicle, generating a map of the static object and the moving ground traffic entity in the vicinity of the road vehicle. The method according to appended claim 38. (Appended claim 40) Including warning the driver of the road vehicle about the collision risk. The method according to appended claim 38. (Appended claim 41) Including identifying the collision risk based on the probability of the predicted trajectory of other nearby moving ground traffic entities. The method according to appended claim 38. (Appended claim 42) Including filtering the received basic safety message and the personal safety message to reduce the number of warnings provided to the driver of the road vehicle. The method according to appended claim 38. (Appended claim 43) Obtaining operation data for unconnected ground traffic entities moving in the transportation network, Sending a virtual safety message incorporating information about the operation data for the unconnected ground traffic entity to the connected ground traffic entity in the vicinity of the unconnected ground traffic entity. A method including the above. (Appended claim 44) The virtual safety message is a substitute for the safety message that would be sent by the unconnected ground traffic entity if it were assumed to be connected. The method according to appended claim 43. (Appended claim 45) The unconnected ground traffic entity includes a vehicle. The virtual safety message is a substitute for the basic safety message. The method according to appended claim 43. (Appended claim 46) The unconnected ground traffic entity includes road users who may be victims. The virtual safety message is a substitute for the personal safety message, The method according to appended claim 43. (Appended claim 47) The operation data is acquired by an infrastructure sensor, The method according to appended claim 43. (Appended claim 48) An apparatus comprising a device located at an intersection of a transportation network, The device being An input for receiving data from a sensor oriented to monitor surface traffic entities at the intersection or surface traffic entities in the vicinity of the intersection, A wireless communication device for transmitting a warning regarding a dangerous situation at the intersection or a dangerous situation in the vicinity of the intersection to a device of one of the surface traffic entities, A processor, A storage, Comprising, The storage being To store a machine learning model capable of predicting the behavior of surface traffic entities at the intersection or surface traffic entities in the vicinity of the intersection at the current time, the machine learning model being based on training data regarding the behavior related to the previous operations of the surface traffic entities at the intersection or surface traffic entities in the vicinity of the intersection, Applying the currently received operation data from the sensor regarding the surface traffic entities at the intersection or surface traffic entities in the vicinity of the intersection to the machine learning model to predict the imminent behavior of the surface traffic entities including surface traffic entities for which a device involved cannot receive a warning from the wireless communication device, To infer an imminent dangerous situation for a surface traffic entity for which a device involved can receive a warning from the wireless communication device, the imminent dangerous situation being a result of the predicted imminent behavior of the surface traffic entity for which the warning cannot be received, Transmitting the warning regarding the dangerous situation to the device of the ground traffic entity that can receive the warning from the wireless communication device; A storage for instructions executable by the processor for doing so; Device. (Appended Claim 49) The device includes devices along the road; The device according to appended claim 48. (Appended Claim 50) The device includes a housing for the device; The sensor is mounted on the housing; The device according to appended claim 48. (Appended Claim 51) The warning is transmitted by broadcasting the warning for reception by any one of the ground traffic entities at the intersection where the warning can be received or the ground traffic entities near the intersection; The device according to appended claim 48. (Appended Claim 52) The machine learning model includes an artificial intelligence model; The device according to appended claim 48. (Appended Claim 53) The intersection includes an intersection without traffic lights; The device according to appended claim 48. (Appended Claim 54) The intersection includes an intersection with traffic lights; The device according to appended claim 48. (Appended Claim 55) The transportation network includes a road network; The device according to appended claim 48. (Appended Claim 56) The ground traffic entity includes road users who may be damaged; The device according to appended claim 48. (Appended Claim 57) The ground traffic entity includes vehicles; The device according to appended claim 48. (Appended Claim 58) the imminent dangerous situation, including a collision, the device according to appended claim 48. (appended claim 59) the surface traffic entity for which the device involved cannot receive the warning from the wireless communication device, including a vehicle, the surface traffic entity for which the device involved can receive the warning from the wireless communication device, including pedestrians crossing the road at a crosswalk, the device according to appended claim 48. (appended claim 60) comprising another communication device communicating with a central server, the device according to appended claim 48. (appended claim 61) one of the devices of the surface traffic entity comprises a mobile communication device, the device according to appended claim 48. (appended claim 62) using an electronic sensor located near the intersection to monitor the intersection of the surface traffic network and the access road to the intersection, the electronic sensor generating operation data regarding a surface traffic entity moving on the access road or a surface traffic entity moving at the intersection, and one or more of the surface traffic entities being unable to send a safety message to other surface traffic entities near the intersection; sending a virtual safety message to one or more of the surface traffic entities capable of receiving a message based on the operation data generated by the electronic sensor; incorporating information regarding one or more of the surface traffic entities unable to send a safety message into the virtual safety message, each of the incorporated information in the virtual safety message including at least one of the position, traveling direction, speed, and predicted future trajectory of one of the surface traffic entities unable to send a safety message; A method including (Appended claim 63) The information incorporated includes a subset of the information incorporated into a basic safety message or a personal safety message generated by the surface traffic entity, assuming that the incorporated information enables the surface traffic entity to send a basic safety message or a personal safety message. The method according to appended claim 62. (Appended claim 64) Including applying the generated motion data to a machine learning model operating in a device located near the intersection to predict the trajectory of the surface traffic entity that cannot send a safety message. The method according to appended claim 62. (Appended claim 65) At least one of the surface traffic entities includes a motor-driven vehicle. The method according to appended claim 62. (Appended claim 66) The machine learning model is provided to a device located near the intersection by a remote server through the Internet. The method according to appended claim 62. (Appended claim 67) The machine learning model is generated in a device located near the intersection. The method according to appended claim 62. (Appended claim 68) Including training a machine learning model using the motion data generated by the sensor located near the intersection. The method according to appended claim 62. (Appended claim 69) Including sending, to a server, the motion data generated by the sensor located near the intersection for use in training a machine learning model. The method according to appended claim 62. (Appended claim 70) Using an electronic sensor located near the crosswalk to monitor the area within the crosswalk or an area near the crosswalk, wherein the electronic sensor generates motion data regarding a lane user who may be at risk within the crosswalk or a lane user who may be at risk near the crosswalk. Applying the generated motion data to a machine learning model operating on a device located near the crosswalk to predict that one of the lane users who may be at risk is attempting to enter the crosswalk. Wirelessly transmitting a warning to at least one of a device associated with the lane user who may be at risk or a device associated with another ground traffic entity approaching the crosswalk on the road before the lane user who may be at risk enters the crosswalk. A method comprising the above. (Additional item 71) The lane user who may be at risk includes a pedestrian, an animal, or a cyclist. The method according to additional item 70. (Additional item 72) The device associated with the lane user who may be at risk includes a smartwatch or other wearable device, a smartphone, or another mobile device. The method according to additional item 70. (Additional item 73) The other ground traffic entity includes a motor-driven vehicle. The method according to additional item 70. (Additional item 74) The device associated with the other ground traffic entity includes a smartphone or another mobile device. The method according to additional item 70. (Additional item 75) The machine learning model is provided to the device located near the crosswalk by a remote server through the Internet. The method according to additional item 70. (Supplementary Note 76) The method according to claim 70, wherein the machine learning model is generated by the device located near the crosswalk. (Supplementary Note 77) (Supplementary Note 77) The method according to claim 70, comprising training the machine learning model using the motion data generated by the sensor located near the crosswalk. (Supplementary Note 78) (Supplementary Note 78) The method according to claim 70, comprising transmitting, to a server, the motion data generated by the sensor located near the crosswalk for use in training the machine learning model. (Supplementary Note 79) (Supplementary Note 79) The method according to claim 70, comprising segmenting the motion data generated by the sensor located near the crosswalk based on corresponding zones near the crosswalk. (Supplementary Note 80) (Supplementary Note 80) The method according to claim 70, comprising using the electronic sensor to generate motion-related data representing the physical properties of road users who may be victims. (Supplementary Note 81) (Supplementary Note 81) The method according to claim 70, comprising deriving trajectory information regarding road users who may be victims from the motion data generated by the sensor. (Supplementary Note 82) (Supplementary Note 82) An apparatus comprising a device located at a planar intersection of a transportation network, wherein the planar intersection includes a road intersection, a crosswalk, and a railway line, and the device is configured to receive data from sensors oriented to monitor road vehicles and pedestrians at the planar intersection or road vehicles and pedestrians near the planar intersection, and receive phase and timing data for signals on the road and signals on the railway line, A wireless communication device that transmits a warning regarding a dangerous situation at the plane intersection or a dangerous situation near the plane intersection to one of a ground traffic entity, a pedestrian, or a railway vehicle on the railway line, a processor, a storage, and is provided with, wherein the storage stores a machine learning model capable of predicting the behavior of a ground traffic entity at the plane intersection at the current time or a ground traffic entity near the plane intersection, and the machine learning model is based on training data regarding the previous actions and related behaviors of road vehicles and pedestrians at the intersection or road vehicles and pedestrians near the intersection, applies the current motion data received from the sensor regarding road vehicles and pedestrians at the plane intersection or road vehicles and pedestrians near the plane intersection to the machine learning model in order to predict the imminent behavior of the road vehicles and the pedestrians, infers an imminent dangerous situation for a railway vehicle on the railway line at the intersection or a railway vehicle on the railway line near the intersection from the predicted imminent behavior, causes the wireless communication device to transmit the warning regarding the dangerous situation to at least one device among the road vehicle, the pedestrian, and the railway vehicle, and is the storage for instructions executable by the processor for doing so, a device. (Supplementary Claim 83) The warning is transmitted to on-vehicle equipment of the railway vehicle, The device according to Supplementary Claim 82. (Supplementary Claim 84) The railway line is in a separated railway land, The device according to Supplementary Claim 82. (Supplementary Claim 85) The railway line is not in a separated railway land, The device according to appended item 82. (Appended item 86) wherein the device includes a roadside device The device according to appended item 82. (Appended item 87) wherein the warning is transmitted by broadcasting the warning for reception by any one of the ground traffic entities, the pedestrians, or the railway vehicles at the plane intersection, or the ground traffic entities, the pedestrians, or the railway vehicles near the plane intersection The device according to appended item 82. (Appended item 88) wherein the imminent dangerous situation includes a collision or a near miss The device according to appended item 82. (Appended item 89) receiving data from an infrastructure sensor representing the position and movement of a road vehicle being driven or a pedestrian walking in a ground traffic network receiving data in a virtual basic safety message and a virtual personal safety message regarding the states of the road vehicle and the pedestrian applying the received data to a trained machine learning model to identify a dangerous driving or walking behavior of one of the road vehicle or the pedestrian reporting the dangerous driving or walking behavior to an authority A method comprising: (Appended item 90) identifying the road vehicle based on license plate recognition The method according to appended item 89. (Appended item 91) identifying the pedestrian based on biometric recognition The method according to appended item 89. (Appended item 92) identifying the road vehicle or the pedestrian based on social networking The method according to appended item 89. (Supplementary Note 93) Using an electronic sensor located near the intersection to monitor the intersection of the ground transportation network and the access roads to the intersection, wherein the electronic sensor generates motion data regarding a ground transportation entity moving on the access road or a ground transportation entity moving at the intersection Defining separate virtual zones at the intersection and on the access roads to the intersection Segmenting the motion data according to the corresponding virtual zone to which the generated motion data relates Applying the generated motion data for each segment to a machine learning model operating on a device located near the intersection to predict an imminent dangerous situation at the intersection or on one of the access roads, involving one or more of the ground transportation entities Wirelessly transmitting a warning to a device associated with at least one of the ground transportation entities involved before the imminent dangerous situation becomes an actual dangerous situation A method comprising the above steps (Supplementary Note 94) The device associated with each of the ground transportation entities includes a wearable device, a smartphone, or another mobile device The method according to Supplementary Note 93 (Supplementary Note 95) At least one of the ground transportation entities includes a motor-driven vehicle The method according to Supplementary Note 93 (Supplementary Note 96) The machine learning model is provided to the device located near the intersection by a remote server through the Internet The method according to Supplementary Note 93 (Supplementary Note 97) The machine learning model is generated in the device located near the intersection The method according to Supplementary Note 93 (Supplementary Note 98) including training the machine learning model using the motion data generated by the sensor located near the intersection The method according to Supplementary Note 93 (Supplementary Note 99) including transmitting the motion data generated by the sensor located near the intersection to a server for use in training the machine learning model The method according to Supplementary Note 93 (Supplementary Note 100) including using the electronic sensor to monitor an area within a crosswalk or an area near the crosswalk that crosses one of the access roads to the intersection The method according to Supplementary Note 93 (Supplementary Note 101) including using the electronic sensor to generate motion-related data representing the physical properties of road users who may be damaged near the crosswalk The method according to Supplementary Note 93 (Supplementary Note 102) including deriving trajectory information regarding road users who may be damaged from the motion data generated by the sensor The method according to Supplementary Note 93 (Supplementary Note 103) wherein a machine learning model exists for each of the access roads to the intersection The method according to Supplementary Note 93 (Supplementary Note 104) including determining whether to send the warning further based on motion data generated by sensors related to another nearby intersection The method according to Supplementary Note 93 (Supplementary Note 105) including determining whether to send the warning further based on information received from a ground traffic entity moving on the access road or a ground traffic entity moving at the intersection The method according to Supplementary Note 93 (Supplementary Note 106) The intersection has traffic lights, and information regarding the state of the signal is received, The method according to appended claim 93. (Appended claim 107) The intersection does not have traffic lights and is controlled by one or more signs, The method according to appended claim 93. (Appended claim 108) The specified virtual zone includes one or more of the access roads controlled by the signs, The method according to appended claim 107. (Appended claim 109) The signs include a stop sign or a yield sign, The method according to appended claim 107. (Appended claim 110) One of the surface traffic entities includes a railway vehicle, The method according to appended claim 107. (Appended claim 111) An apparatus comprising a device located at an intersection of a transportation network, wherein the device is an input for receiving data from sensors oriented to monitor surface traffic entities at the intersection or surface traffic entities near the intersection, wherein the data from each of the sensors represents at least one position or motion parameter of at least one of the surface traffic entities, wherein the data from each of the sensors is represented in a native format, and wherein the data received from at least two of the sensors do not match with respect to the position or motion parameter, or with respect to the native format, or with respect to both the position or motion parameter and the native format, the input and a processor, a storage, and comprising, wherein the storage Converting the data from each of the sensors into data having a common format independent of the native format of the data of the sensors; Incorporating the data having the common format into a global integrated representation of the ground traffic entities being monitored at the intersection or of the ground traffic entities being monitored in the vicinity of the intersection, the global integrated representation including the position, speed, and direction of travel of each of the ground traffic entities; Identifying the relevance of the position and movement of two of the ground traffic entities using the global integrated representation; Predicting a dangerous situation involving the two ground traffic entities; Sending, to at least one of the two ground traffic entities, a message alerting the at least one of the two ground traffic entities about the dangerous situation; A storage for instructions executable by the processor for performing the above; Device. (Appended Claim 112) The sensor includes at least two of a radar, a lidar, and a camera; The device according to appended claim 111. (Appended Claim 113) The data received from one of the sensors includes image data of a field of view at consecutive instants; The device according to appended claim 111. (Appended Claim 114) The data received from one of the sensors includes reflection points in a 3D space; The device according to appended claim 111. (Appended Claim 115) The data received from one of the sensors includes the distance and speed from the sensor; The device according to appended claim 111. (Supplementary Note 116) wherein the global integrated representation represents the position of the ground traffic entity in a common reference coordinate system; the apparatus according to Supplementary Note 111. (Supplementary Note 117) wherein the apparatus includes at least two of the sensors; wherein the data is received from the at least two sensors; wherein two of the sensors are provided at a fixed position at or near the intersection and have non-overlapping fields of view; the apparatus according to Supplementary Note 111. (Supplementary Note 118) wherein one of the sensors includes a radar; wherein converting the data includes identifying the position of the ground traffic entity from a known position of the radar and the distance from the radar to the ground traffic entity; the apparatus according to Supplementary Note 117. (Supplementary Note 119) wherein one of the sensors includes a camera; wherein converting the data includes identifying the position of the ground traffic entity from a known position, the viewing direction, the tilt of the camera, and the position of the ground traffic entity within the image frame of the camera; the apparatus according to Supplementary Note 117. (Supplementary Note 120) Receiving data from sensors oriented to monitor surface traffic entities at intersections of a surface traffic network or surface traffic entities near intersections of a surface traffic network, wherein the data from each of the sensors represents at least one position or motion parameter of at least one of the surface traffic entities, the data from each of the sensors is represented in a native format, and the data received from at least two of the sensors does not match with respect to the position or motion parameter, or with respect to the native format, or with respect to both the position or motion parameter and the native format, Converting the data from each of the sensors into data having a common format independent of the native format of the data of the sensors, Incorporating the data having the common format into a global integrated representation of the surface traffic entities being monitored at the intersection or the surface traffic entities being monitored near the intersection, wherein the global integrated representation includes the position, speed, and direction of travel of each of the surface traffic entities, Identifying the relevance of the positions and motions of two of the surface traffic entities using the global integrated representation, Predicting a dangerous situation involving the two surface traffic entities, Sending a message alerting at least one of the two surface traffic entities about the dangerous situation to at least one of the two surface traffic entities, A method comprising. (Appended Claim 121) The sensor includes at least two of radar, lidar, and camera, The method according to appended claim 120. (Appended Claim 122) The data received from one of the sensors includes image data of the field of view at consecutive instants. The method according to appended claim 120. (Appended claim 123) The data received from one of the sensors includes reflection points in 3D space. The method according to appended claim 120. (Appended claim 124) The data received from one of the sensors includes the distance and speed from the sensor. The method according to appended claim 120. (Appended claim 125) The global integrated representation represents the position of the surface traffic entity in a common reference coordinate system. The method according to appended claim 120. (Appended claim 126) The data is received from at least two of the two sensors provided at a fixed position at the intersection or in the vicinity of the intersection and having at least partially non-overlapping fields of view. The method according to appended claim 120. (Appended claim 127) One of the sensors includes a radar. The method includes converting the data, including identifying the position of the surface traffic entity from the known position of the radar and the distance from the radar to the surface traffic entity. The method according to appended claim 126. (Appended claim 128) One of the sensors includes a camera. The method includes converting the data, including identifying the position of the surface traffic entity from a known position, the viewing direction, the tilt of the camera, and the position of the surface traffic entity within the image frame of the camera. The method according to appended claim 126. (Aspect 1) To monitor areas within the crosswalk and areas near the crosswalk, use an electronic sensor located near the crosswalk where the road is to be crossed, the electronic sensor generating motion data regarding a roadway user who may be at risk of harm within the crosswalk or a roadway user who may be at risk of harm near the crosswalk, the motion data including the position and direction of the roadway user who may be at risk of harm, and use. To predict that one of the roadway users who may be at risk of harm is attempting to enter the road within the crosswalk or the road near the crosswalk, apply the generated motion data to a machine learning model operating on a device located near the crosswalk, the prediction being made before the roadway user who may be at risk of harm enters the road within the crosswalk or the road near the crosswalk, the machine learning model being trained using motion data generated near the crosswalk and indicative of the intent or behavior of a roadway user who may have been within the crosswalk or near the crosswalk before, the motion data including position, speed, acceleration, and orientation, and apply. Before the roadway user who may be at risk of harm enters the road within the crosswalk or the road near the crosswalk, send a warning to at least one of a device associated with the roadway user who may be at risk of harm, a device associated with another ground traffic entity approaching the crosswalk on the road, and a road sign configured to issue a warning to the roadway user who may be at risk of harm or the driver. A method including. (Aspect 2) The roadway user who may be at risk of harm includes a pedestrian, an animal, or a cyclist. The method according to aspect 1. (Aspect 3) The device associated with the roadway user who may be at risk of harm includes a smartwatch or other wearable device, a smartphone, or another mobile device. The method according to aspect 1. (Aspect 4) wherein the other ground transportation entity includes a motor-driven vehicle, the method according to aspect 1. (Aspect 5) wherein the device related to the other ground transportation entity includes a smartphone or another mobile device, the method according to aspect 1. (Aspect 6) wherein the machine learning model is provided to the device located near the crosswalk by a remote server through the Internet, the method according to aspect 1. (Aspect 7) wherein the machine learning model is generated in the device located near the crosswalk, the method according to aspect 1. (Aspect 8) including training the machine learning model using the motion data generated by the sensor located near the crosswalk, the method according to aspect 1. (Aspect 9) including transmitting, to a server, the motion data generated by the sensor located near the crosswalk for use in training the machine learning model, the method according to aspect 1. (Aspect 10) including segmenting the motion data generated by the sensor located near the crosswalk based on corresponding zones in the vicinity of the crosswalk, the method according to aspect 1. (Aspect 11) including using the electronic sensor to generate motion-related data representing the physical properties of the lane users who may be victimized, the method according to aspect 1. (Aspect 12) including deriving trajectory information regarding the lane users who may be victimized from the motion data generated by the sensor, the method according to aspect 1. (Aspect 13) An apparatus comprising a device located at a planar intersection of a transportation network, wherein the planar intersection includes a road intersection, a crosswalk, and a railway line, and the device receives data from sensors oriented to monitor road vehicles and pedestrians at the planar intersection or road vehicles and pedestrians in the vicinity of the planar intersection, and receives phase and timing data for signals on the road and signals on the railway line, and a wireless communication device that transmits a warning regarding a dangerous situation at the planar intersection or a dangerous situation in the vicinity of the planar intersection to one of a ground transportation entity, a pedestrian, or a railway vehicle on the railway line, a processor, a storage, and comprises, wherein the storage stores a machine learning model capable of predicting the behavior of a ground transportation entity at the planar intersection at the current time or a ground transportation entity in the vicinity of the planar intersection, wherein the machine learning model is based on training data regarding previous actions and related behaviors of road vehicles and pedestrians at the intersection or road vehicles and pedestrians in the vicinity of the intersection, and previous phase and timing data for signals on the road and signals on the railway line, applies the currently received motion data from the sensors regarding road vehicles and pedestrians at the planar intersection or road vehicles and pedestrians in the vicinity of the planar intersection, and the current phase and timing data for signals on the road and signals on the railway line, to the machine learning model to predict the imminent behavior of the road vehicles and the pedestrians, infers an imminent dangerous situation for a railway vehicle on the railway line at the intersection or a railway vehicle on the railway line in the vicinity of the intersection from the predicted imminent behavior, Causing the wireless communication device to transmit the warning regarding the dangerous situation to at least one device among the road vehicle, the pedestrian, and the railway vehicle. The storage for instructions executable by the processor for doing so. Device. (Aspect 14) The warning is transmitted to the on-vehicle equipment of the railway vehicle. The device according to aspect 13. (Aspect 15) The railway line is in a separated railway land. The device according to aspect 13. (Aspect 16) The railway line is not in a separated railway land. The device according to aspect 13. (Aspect 17) The equipment includes roadside equipment. The device according to aspect 13. (Aspect 18) The warning is transmitted by broadcasting the warning for reception by any one of the surface traffic entities, the pedestrian, or the railway vehicle at the grade crossing, or the surface traffic entities, the pedestrian, or the railway vehicle near the grade crossing. The device according to aspect 13. (Aspect 19) The imminent dangerous situation includes a collision or a near miss. The device according to aspect 13. (Aspect 20) Receiving data from an infrastructure sensor representing the position and movement of a road vehicle being driven or a pedestrian walking in a surface traffic network. Receiving data in basic safety messages or virtual basic safety messages, personal safety messages, and virtual personal safety messages regarding the state of the road vehicle and the pedestrian, wherein the basic safety message, the virtual basic safety message, the personal safety message, and the virtual personal safety message include position, traveling direction, and speed information regarding the road vehicle or the pedestrian, the basic safety message and the personal safety message are received from the road vehicle or the pedestrian, and the virtual basic safety message and the virtual personal safety message include position, traveling direction, and speed information regarding the road vehicle and the pedestrian reconstructed from the data received from the infrastructure sensors. Applying the data received from the infrastructure sensors and data from the basic safety message, the virtual basic safety message, the personal safety message, and the virtual personal safety message to a trained machine learning model to identify a dangerous driving or walking behavior of one of the road vehicle or the pedestrian. Automatically reporting the dangerous driving or walking behavior to the authorities. A method comprising the above. (Aspect 21) The method according to aspect 20, including identifying the road vehicle based on license plate recognition. The method according to aspect 20. (Aspect 22) The method according to aspect 20, including identifying the pedestrian based on biometric recognition. The method according to aspect 20. (Aspect 23) The method according to aspect 20, including identifying the road vehicle or the pedestrian based on social networking. The method according to aspect 20.

Claims

1. An apparatus comprising a device located at a planar intersection of a transportation network, wherein the planar intersection includes intersections of roads, crosswalks, and railway lines, and the device is configured to receive data from sensors oriented to monitor road vehicles and pedestrians at the planar intersection or road vehicles and pedestrians in the vicinity of the planar intersection, and receive phase and timing data for signals on the roads and signals on the railway lines, a wireless communication device that transmits a warning regarding a dangerous situation at the planar intersection or a dangerous situation in the vicinity of the planar intersection to one of a ground transportation entity, a pedestrian, or a railway vehicle on the railway line, a processor, a storage, and is configured to wherein the storage is configured to store a machine learning model capable of predicting the behavior of ground transportation entities at the planar intersection at the current time or ground transportation entities in the vicinity of the planar intersection, wherein the machine learning model is based on training data regarding previous actions and related behaviors of road vehicles and pedestrians at the intersection or road vehicles and pedestrians in the vicinity of the intersection, and previous phase and timing data for signals on the roads and signals on the railway lines, apply the current motion data received from the sensors regarding road vehicles and pedestrians at the planar intersection or road vehicles and pedestrians in the vicinity of the planar intersection, and the current phase and timing data for signals on the roads and signals on the railway lines, to the machine learning model to predict the imminent behavior of the road vehicles and the pedestrians, infer an imminent dangerous situation for a railway vehicle on the railway line at the intersection or a railway vehicle on the railway line in the vicinity of the intersection from the predicted imminent behavior, cause the wireless communication device to transmit the warning regarding the dangerous situation to at least one of the road vehicle, the pedestrian, and the railway vehicle, and is the storage for executable instructions by the processor for doing so, an apparatus.

2. The warning is transmitted to an on-vehicle device of the railway vehicle, The apparatus according to claim 1.

3. wherein the railway line is on a separate railway land The apparatus according to claim 1.

4. wherein the railway line is not on a separate railway land The apparatus according to claim 1.

5. wherein the equipment includes equipment along the road The apparatus according to claim 1.

6. wherein the warning is transmitted by broadcasting the warning for reception by any of the ground traffic entities, the pedestrians, or the railway vehicles at the grade crossing, or the ground traffic entities, the pedestrians, or the railway vehicles near the grade crossing The apparatus according to claim 1.

7. wherein the imminent dangerous situation includes a collision or a near miss The apparatus according to claim 1.

Images