Sophisticated onboard apparatus

Abstract

To provide a system and method for giving an early-stage alarm against a dangerous situation.SOLUTION: An apparatus for use in a condition where it is mounted in a first terrestrial traffic entity includes (a) a receiver for information generated by a sensor in an ambient environment of the first terrestrial traffic entity, (b) a processor, and (c) a memory that stores an instruction with which safety message information for a second terrestrial traffic entity is generated based on information generated by the sensor, and then transmitted and which can be executed by the processor.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 62 / 893,616, filed Aug. 29, 2019, the entire content of which is incorporated herein by reference.

[0002] U.S. Patent No. 10,235,882 is incorporated herein by reference.

[0003] This description relates to advanced in-vehicle devices.

Background Art

[0004] Collision avoidance systems are on the rise. King et al. (U.S. Patent Application Publication No. 2007 / 0276600 (A1), 2007), for example, described placing sensors in front of intersections and applying physics-based decision rules to predict whether two vehicles are about to collide at an intersection based on their direction of travel and speed.

[0005] Aoude et al. (U.S. Patent No. 9,129,519 (B2), 2015, the entire content of which is incorporated herein by reference) monitor and model a driver's behavior to enable prediction and prevention of violations in traffic situations at intersections.

Summary of the Invention

Problems to be Solved by the Invention

[0006] Collision avoidance is a primary defense against injury, death, and property damage in surface transportation. Providing early warning of dangerous situations aids collision avoidance.

Means for Solving the Problems

[0007] Generally, in one aspect, a device for use while mounted on a first ground transportation entity includes (a) a receiver for information generated by sensors of the surrounding environment of the first ground transportation entity, (b) a processor, and (c) a memory storing instructions executable by the processor for generating and transmitting safety message information for a second ground transportation entity based on the information generated by the sensors.

[0008] An embodiment may include one or a combination of two or more of the following features. The instructions are executable by a processor to generate a prediction for use in generating safety message information. The prediction is generated by a prediction model. The prediction model is configured to predict a dangerous situation involving a first ground traffic entity, a second ground traffic entity, or another ground traffic entity. The dangerous situation involves crossing a lane of a road by a second ground traffic entity. The second ground traffic entity includes a vehicle, and the dangerous situation includes a skid across a lane by the vehicle. The second ground traffic entity includes a pedestrian or other road user who may be victimized crossing the road. The road user who may be victimized crosses the road at an intersection. The road user who may be victimized crosses the road other than at an intersection. The predicted dangerous situation includes a predicted collision between a third ground traffic entity and the second ground traffic entity. The first ground traffic entity includes a vehicle, and the second ground traffic entity includes a pedestrian or other road user who may be victimized. The third ground traffic entity follows the first ground traffic entity, and the view of the third ground traffic entity is blocked by the first ground traffic entity. The third ground traffic entity is in a lane adjacent to the lane in which the first ground traffic entity is moving. The instructions are executable by a processor to identify an operation parameter of the third ground traffic entity. The second ground traffic entity comes into view of the third ground traffic only in an obstructed state. The second ground traffic entity includes a pedestrian or other road user who may be victimized. The safety message information transmitted by the processor includes a basic safety message. The safety message information transmitted by the processor includes a virtual basic safety message. The safety message information transmitted by the processor includes a personal safety message. The safety message information transmitted by the processor includes a virtual personal safety message. The safety message information transmitted by the processor includes a virtual basic safety message transmitted on behalf of the third ground traffic entity.The third ground transportation entity includes unconnected ground transportation entities. The safety message information transmitted by the processor includes a virtual personal safety message transmitted in place of the third ground transportation entity. The device includes a receiver (d) for information wirelessly transmitted from a source external to the first ground transportation entity. The apparatus of the claim including the first ground transportation entity. The safety message information includes a virtual intersection collision avoidance message (VICA). The safety message information includes an intersection collision avoidance message (ICA). The safety message information includes a virtual combined safety message (VCSM). The safety message information includes a combined safety message (CSM).

[0009] Generally, in one aspect, a device for use while mounted on a first ground transportation entity includes a receiver (a) for first position correction information transmitted from a source external to the first ground transportation entity, a receiver (b) for information representing position parameters or operation parameters of the first ground transportation entity, a processor (c), and a memory (d) storing instructions executable by the processor for generating updated position correction information based on the first position correction information and the information representing operation parameters, and for transmitting a position correction message to another ground transportation entity based on the updated position correction information.

[0010] An embodiment may include one or a combination of two or more of the following features. Position correction information transmitted from a source external to the first ground transportation entity includes a position correction message. Position correction information transmitted from a source external to the first ground transportation entity includes a Radio Technical Commission for Maritime (RTCM) correction message. The position correction information includes GNSS position correction. The position parameters or operation parameters include the current position of the first ground transportation entity. The source external to the first ground transportation entity includes an RSE or an external service configured to transmit an RTCM correction message via the Internet. The instruction is executable by a processor to confirm a reliability level in the updated position correction information.

[0011] Generally, in one aspect, information generated by sensors mounted on a first ground transportation entity regarding the surrounding environment of the first ground transportation entity is received. Safety message information is generated based on the information generated by the sensors and transmitted to a second ground transportation entity.

[0012] An embodiment may include one or a combination of two or more of the following features. The prediction is generated for use in generating safety message information. The prediction is generated by a prediction model. The prediction model is configured to predict a dangerous situation involving a first ground traffic entity, a second ground traffic entity, or another ground traffic entity. The dangerous situation involves crossing a lane of a road by a second ground traffic entity. The second ground traffic entity includes a vehicle, and the dangerous situation includes a skid across a lane by the vehicle. The second ground traffic entity includes a pedestrian or other road user who may be victimized crossing the road. The road user who may be victimized crosses the road at an intersection. The road user who may be victimized crosses a road other than an intersection. The dangerous situation includes a collision between a third ground traffic entity and the second ground traffic entity. The first ground traffic entity includes a vehicle, and the second ground traffic entity includes a pedestrian or other road user who may be victimized. The third ground traffic entity follows the first ground traffic entity, and the vision of the third ground traffic entity is blocked by the first ground traffic entity. The third ground traffic entity is in a lane close to the lane in which the first ground traffic entity is moving. The operating parameters of the third ground traffic entity are identified. The third ground traffic entity comes into view only in an obscured state for the second ground traffic entity. The second ground traffic entity includes a pedestrian or other road user who may be victimized. The safety message information includes a basic safety message. The safety message information includes a virtual basic safety message. The safety message information includes a personal safety message. The safety message information includes a virtual personal safety message. The safety message information includes a virtual basic safety message transmitted instead of the third ground traffic entity. The safety message information transmitted by the processor includes a virtual personal safety message transmitted instead of the third ground traffic. The third ground traffic entity includes a non-connected ground traffic entity.Information wirelessly transmitted from a source external to the first ground traffic entity is received. The safety message information includes a Virtual Intersection Collision Avoidance message (VICA). The safety message information includes a Virtual Intersection Collision Avoidance message (ICA). The safety message information includes a Virtual Combined Safety Message (VCSM). The safety message information includes a Combined Safety Message (CSM).

[0013] Generally, in one aspect, first position correction information transmitted from a source external to the first ground traffic entity is received. Information representing the operating parameters of the first ground traffic entity is received. The updated position correction information is generated based on the first position correction information and the information representing the operating parameters. A position correction message is transmitted, and the position correction message is transmitted to another ground traffic entity based on the updated position correction information.

[0014] Embodiments may include one or a combination of two or more of the following features. The position correction information transmitted from a source external to the first ground traffic entity includes a position correction message. The position correction information transmitted from a source external to the first ground traffic entity includes a Radio Technical Commission for Maritime Services (RTCM) correction message. The position correction information includes GNSS position correction. The operating parameters include the current position of the first ground traffic entity. The source external to the first ground traffic entity includes an RSE or an external service configured to transmit a position correction message via the Internet. The reliability level in the updated position correction information (it) is confirmed.

[0015] These and other aspects, features, and embodiments may be represented as a method, apparatus, system, component, program product, method of doing business, means or steps for performing functions, and other approaches.

[0016] These and other aspects, features, and embodiments will become apparent from the following description including the claims.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Best Mode for Carrying Out the Invention

[0018] With the progress of sensor technology and computers, it has become possible to predict dangerous situations (and provide early warnings of dangerous situations), and by that means, prevent collisions and near misses of ground traffic entities in the implementation of ground traffic (i.e., enable collision avoidance).

[0019] As used herein, the term "ground traffic" is used broadly to include any mode or medium that moves from place to place with contact with land or water on the surface of the earth, for example, walking or running (or performing other walking actions), non-motorized vehicles, motorized vehicles (autonomous, semi-autonomous, and non-autonomous), and rail vehicles.

[0020] As used herein, the term "ground traffic entity" (or, in some cases, simply "entity") is used broadly to include, for example, a person or an individual motorized or non-motorized vehicle involved in a mode of ground traffic, such as, in particular, a pedestrian, a cyclist, a boat, a car, a truck, a tram, a streetcar, or a train. In some cases, the terms "vehicle" or "road user" are used herein as a concise reference to a ground traffic entity.

[0021] As used herein, the term "hazardous situation" is used broadly to include any event, occurrence, sequence, context, or other circumstance that can cause, or can be reduced or avoided, property damage, personal injury, or death that is imminent. In some cases herein, the term "hazard" is used interchangeably with "hazardous situation". In some cases herein, the terms "violation" or "violate" are used with respect to the behavior of an entity that can cause, can lead to, or will lead to a hazardous situation.

[0022] In some embodiments of the technology discussed herein, the surface transportation network is used by a mixture of surface transportation entities that do not have, or do not use, transportation connectivity and surface transportation entities that do have and use transportation connectivity.

[0023] As used herein, the term "connectivity" is used broadly to include any ability of a surface transportation entity for (a) recognizing knowledge information about its surrounding environment, other surface transportation entities in its vicinity, and the traffic situation associated therewith and acting on them, (b) broadcasting or otherwise transmitting data regarding its state, or (c) both (a) and (b). The data transmitted can include its location, direction of travel, speed, or the internal state of its components related to the traffic situation. In some examples, the recognition of a surface transportation entity is based on wirelessly received data regarding other surface transportation entities or the traffic situation related to the operation of the surface transportation entity. The received data can be derived from other surface transportation entities, or infrastructure devices, or both. Typically, connectivity involves transmitting or receiving data in real time, or substantially in real time, or when one or more of the surface transportation entities act based on data in the traffic situation.

[0024] As used herein, the term "traffic situation" is used broadly to include any situation in which two or more ground traffic entities operate in the vicinity of each other and in which the operation or status of each entity can affect or be related to the operation or status of the others.

[0025] As used herein, a ground traffic entity that, in some cases, does not have or does not use connectivity or an aspect of connectivity is referred to as an "unconnected ground traffic entity" or simply an "unconnected entity". As used herein, a ground traffic entity that, in some cases, includes and uses connectivity or an aspect of connectivity is referred to as a "connected ground traffic entity" or simply a "connected entity".

[0026] As used herein, in some cases, the term "cooperating entity" is used to represent a ground traffic entity that broadcasts data, such as data including its position, direction of travel, speed, or the status of an in-vehicle safety system (e.g., brakes, lights, and wipers), to its surrounding environment.

[0027] As used herein, in some cases, the term "non - cooperating entity" is used to represent a ground traffic entity that does not broadcast one or more types of data, such as its position, speed, direction of travel, or status, to its surrounding environment.

[0028] In this specification, in some cases, the term "vicinity" of a ground transportation entity is used broadly, for example, to include an area in which a broadcast by the entity can be received by other ground transportation entities or infrastructure devices. In some examples, the vicinity varies with the location of the entity and the number and characteristics of obstacles in the vicinity of the entity. An entity moving on an open road in a desert has a very wide vicinity because there are no obstacles that would prevent a broadcast signal from the entity from reaching long distances. Conversely, the vicinity in an urban valley is reduced by buildings in the vicinity of the entity. Further, there may be sources of electromagnetic noise that degrade the quality of the broadcast signal and thus reduce the reception distance (vicinity).

[0029] As shown in FIG. 14, the vicinity of an entity 7001 moving along a road 7005 can be represented by concentric circles including an outermost circle 7002 that represents the outermost extent of the vicinity. Any other entity located within the circle 7002 is in the vicinity of the entity 7001. Any other entity located outside the circle 7002 is outside the vicinity of the entity 7001 and cannot receive a broadcast by the entity 7001. The entity 7001 is invisible (unrecognizable) to all entities and infrastructure devices outside its vicinity.

[0030] Typically, the collaborating entities continuously broadcast their state data. Connected entities in the vicinity of the broadcasting entity can receive these broadcasts, process the received data, and act based on the received data. For example, in the case where a potentially vulnerable road user has a wearable device that can receive broadcasts from an entity such as an approaching truck, the wearable device can process the received data and inform the potentially vulnerable user of when it is safe to cross the road. This operation occurs regardless of the location of the collaborating entity or the potentially vulnerable user relative to the "smart" intersection, as long as the user's device can receive the broadcast, i.e., as long as it is within the vicinity of the collaborating entity.

[0031] As used herein, the term "potentially vulnerable road user" or "potentially vulnerable road users" is used broadly to include any user of a roadway, or other feature of a road network, who is not using, for example, a motor-driven vehicle. Potentially vulnerable road users are generally not protected from injury or death or property damage in the event that they collide with a motor-driven vehicle. In some examples, a potentially vulnerable road user can be a person who is walking, running, cycling, or performing any kind of movement that exposes that person to the risk of direct physical contact by a vehicle or other ground-based traffic entity in the event of a collision.

[0032] In some embodiments, the collision avoidance techniques and systems described herein (which may be referred to herein as simply "systems" in some cases) use sensors mounted on infrastructure facilities to monitor, track, detect, and predict the actions (e.g., speed, direction of travel, and position), behaviors (e.g., high speed), and intentions (e.g., violating a stop sign) of ground transportation entities and drivers and their operators. The information provided by the sensors ("sensor data") enables the system to predict dangerous situations in order to increase the chances of collision avoidance and to provide early warnings to the entities.

[0033] As used herein, the term "collision avoidance" is used broadly to include any situation in which, for example, a collision or near miss between two or more ground transportation entities, or between a ground transportation entity and another object in the surrounding environment, that could be caused by a dangerous situation is prevented, or the opportunity for such an interaction is reduced.

[0034] As used herein, the term "early warning" is used broadly to include any notification, alert, instruction, command, broadcast, transmission, or other transmission or receipt of information that, for example, identifies, proposes, or otherwise indicates a dangerous situation and is useful for collision avoidance.

[0035] Road intersections are prime locations where dangerous situations can occur. The technology described in this specification can equip intersections with infrastructure devices including sensors, computing hardware, and intelligence to enable simultaneous monitoring, detection, and prediction of dangerous situations. Data from these sensors is processed after being normalized to one reference coordinate system. Artificial intelligence models of traffic flow along different approaches to the intersection are constructed. These models assist, for example, entities that are likely to violate traffic rules. The models are configured to detect dangerous situations before an actual violation and can thus be considered predictive. Based on the prediction of a dangerous situation, an alert is sent from the infrastructure devices at the intersection to all connected entities in the vicinity of the intersection. Each entity that receives the alert processes the data in the alert and performs alert filtering. Alert filtering is the process of discarding or ignoring alerts that are not beneficial to the entity. If an alert is considered beneficial (i.e., not ignored as a result of filtering), such as an alert of an imminent collision, the entity automatically responds to the alert (e.g., by applying the brakes), or a notification is presented to the driver, or both are done.

[0036] The system can be used for roadways, waterways, and railways, but is not limited to these. In this specification, these and other similar traffic contexts are sometimes referred to as the "surface transportation network."

[0037] In this specification, much of the discussion is about the system in the context of intersections, but it can also apply to other contexts.

[0038] In this specification, the term "intersection" is used broadly to include any actual arrangement of roads, rails, water bodies, or other paths of movement where two or more surface transportation entities moving along paths of the surface transportation network can occupy the same location at a given point in time and location and cause a collision.

[0039] Ground transportation entities using the ground transportation network move at various speeds and can reach a given intersection at different speeds and times. If the speed and distance of an entity from an intersection are known, dividing the distance by the speed (both described in the same unit system) gives the time to reach the intersection. However, the intended speed changes, for example, due to traffic conditions, speed limits on the route, traffic signals, and other factors, so the assumed time to reach the intersection changes continuously. This dynamic change in the assumed time to reach makes it impossible to predict the actual time to reach with 100% reliability.

[0040] Taking into account the factors that affect the operation of an entity requires applying a large number of correlations between the speed of the entity and various influencing factors. The absolute value of the state of the operation of the entity can be observed by sensors that track the entity from the entity or from an external location. The data captured by these sensors can be used to model the motion, behavior, and intention patterns of the entity. Machine learning can be used to generate complex models from large amounts of data. Patterns that cannot be directly modeled using the kinematics of the entity can be captured using machine learning. A trained model can predict whether an entity is about to move or stop at a particular point by using the tracking data of the entity from sensors that track the entity.

[0041] In other words, in addition to directly detecting information regarding surface traffic entities from sensor data, the system uses artificial intelligence and machine learning to process vast amounts of sensor data in order to learn patterns of operation, behavior, and intent of surface traffic entities, for example, at intersections of the surface traffic network, on approaches to such intersections, and at crosswalks of the surface traffic network. Based on the direct use of current sensor data and the results of applying artificial intelligence and machine learning to current sensor data, the system generates early warnings, such as warnings of dangerous situations, and thereby assists in collision avoidance. In connection with early warnings in the form of instructions or commands, the commands or instructions may be directed at specific autonomous or human-driven entities in order to directly control the vehicle. For example, the command or instruction can cause an entity driven by a malicious person determined to be running a red light with the intent to harm a person to slow down or stop.

[0042] The system can be adjusted to predict for that particular intersection and to send warnings to entities in the vicinity of the device broadcasting the warning. For this purpose, the system uses sensors to derive data regarding dangerous entities and to pass current readings from the sensors through the trained model. Thus, the output of the model can predict dangerous situations and broadcast corresponding warnings. Warnings received by connected entities in the vicinity contain information regarding the dangerous entity, and thus the receiving entity can analyze that information to assess the threat posed to it by the dangerous entity. If a threat exists, the receiving entity can take action itself (e.g., slow down) or notify the driver of the receiving entity using a human-machine interface based on visual, audio, tactile, or any type of sensory stimulus. An autonomous entity can take action by itself to avoid a dangerous situation.

[0043] The alert can further be directly transmitted through a cellular network or other network to a mobile phone or other device that is equipped to receive the alert and is owned by a pedestrian. The system identifies potential dangerous entities at an intersection and broadcasts (or directly transmits) the alert to the pedestrian's personal device, which includes a communication unit. The alert can, for example, prevent the pedestrian from entering a crosswalk and thus avoid a potential accident.

[0044] The system can further track the pedestrian and broadcast information related to their status (position, speed, and other parameters) to other entities so that the other entities can take actions to avoid dangerous situations.

[0045] As shown in FIG. 1, the system includes at least the following types of components.

[0046] 1. Roadside Equipment (RSE) 10 that includes or uses sensors 12 for monitoring, tracking, detecting, and predicting the operation (e.g., speed, direction of travel, and position), behavior (e.g., high speed), and intent (e.g., violating a stop sign) of surface traffic entities 14. The RSE can further include or use a data processing unit 11 and a data storage 18. Surface traffic entities exhibit a wide range of behaviors depending on the infrastructure of the surface traffic network, as well as the state of the entity itself, the state of the driver, and the state of other surface traffic entities. To capture the behavior of the entities, the RSE collects information from sensors, other RSEs, OBEs, OPEs, local servers or central servers, and other data processing units. The RSE can further store the data received thereby and can store the processed data in some or all of the steps in the pipeline.

[0047] The RSE can store data in a local storage device or remote storage. The collected data is processed in real time using default logic or logic based on dynamically collected data, which means that the RSE can automatically update its own logic. The data can be processed in one processing unit or a cluster of processing units to obtain results faster. The data can be processed in local or remote processing units, or local or remote clusters of processing units. The RSE can use simple logic or advanced models trained on the collected data. The models can be trained locally or remotely.

[0048] The RSE can preprocess the data before using the trained model to filter out outliers. Outliers can appear due to noise in the sensor, reflections, or some other artifact. The resulting outliers can lead to false alarms that can affect the overall performance of the RSE. The filtering method can be based on data collected by the RSE, OBE, OPE, or online resources. The RSE can interface with other control devices, such as traffic signal control devices at intersections or other locations, to extract information for use in the data processing pipeline.

[0049] The RSE can further include or use communication device 20 to communicate wired or wirelessly with other RSEs, and with OBEs, OPEs, local servers or central servers, and other data processing units. The RSE can use any available standard for communication with other devices. The RSE can use a wired or wireless Internet connection to download and upload data with other devices, a cellular network to send and receive messages from other cellular devices, and a special wireless device to communicate with infrastructure devices and other RSEs at intersections or other locations.

[0050] The RSE can be installed in the vicinity of different types of intersections. For example, at an intersection with traffic lights (e.g., an intersection where traffic is controlled by traffic lights), the RSE 10 is installed near the traffic light control device 26, either within the same enclosure or within a nearby enclosure. Data (e.g., the phases and timings of traffic lights) means the flow 28 between the traffic light control device and the RSE. At an intersection without traffic lights, the RSE 10 is typically located to facilitate connecting it to sensors 12 that are used to monitor the roads or other features of the surface transportation network in the vicinity of the intersection. The proximity of the RSE to the intersection helps maintain a low-latency system that is important for providing the receiving-side surface unit with maximum time to respond to an alert.

[0051] 2. Onboard Equipment (OBE) 36 mounted on, supported by, or within the surface transportation entity 14, including sensors 38 that identify the position and kinematics (operation data) of the entity in addition to safety-related data regarding the entity. The OBE further includes a data processing unit 40, a data storage 42, and a communication device 44 that can communicate wirelessly with other OBEs, OPEs, RSEs, and, in some cases, servers and computing units.

[0052] On-person equipment (OPE) 46 that can be worn, held, attached to, or otherwise connected to a person or animal, which can be, but is not limited to, a mobile phone, a wearable device, or any other device. The OPE includes, if necessary, a data processing unit 48, a data storage 50, and a communication device 52, or can be coupled to the data processing unit 48, the data storage 50, and the communication device 52. In some embodiments, the OPE functions as a special communication unit for road users who can be victimized and are not in a vehicle. In some examples, the OPE can also be used for other purposes. The OPE can include components that provide visual, audible, or tactile alerts to road users who can be victimized.

[0053] Road users who can be victimized can include pedestrians, cyclists, road workers, wheelchair users, scooters, self-balancing devices, battery-powered personal mobility devices, animal-driven carriages, guide or police animals, livestock, herds of animals, and pets.

[0054] Typically, the OPE is carried by a road user who can be victimized and is capable of sending and receiving messages. The OPE can be worn or integrated into a mobile phone, a tablet, a personal mobility device, a bicycle, a wearable device (e.g., a wristwatch, a bracelet, an anklet), or can be attached to a pet's collar.

[0055] Messages sent by the OPE may include kinematic information related to road users who may be victims, including but not limited to time, 3D position, direction of travel, speed, and acceleration. The messages sent may further convey data representing the alert level, current behavior, and future intentions of road users who may be victims, such as, for example, that a road user who may be a victim is currently crossing a crosswalk, listening to music, or attempting to cross a crosswalk. In particular, the message may convey the blob size or data size of the road user who may be a victim, whether an external device (e.g., a stroller, cart, or other device) is present with the road user who may be a victim, whether the road user who may be a victim has a physical disability, or whether any personal assistance is being used. If the road user who may be a victim is a worker, the message may convey the category of the worker and may further describe the type of action being performed by the worker. If clusters of similar road users who may be victims (e.g., groups of pedestrians) have similar characteristics, one message may be sent to avoid multiple message broadcasts.

[0056] Typically, messages received by the OPE are alert messages from roadside devices or entities. The OPE may operate based on the received messages by alerting road users who may be victims. The alert messages convey data useful for providing custom alerts for road users who may be victims. For example, the alert for a road user who may be a victim may present the type of dangerous situation and suggest possible actions. The OPE may apply alert filtering to all received messages and present only relevant messages to road users who may be victims.

[0057] Alarm filtering is based on the results of applying a learning algorithm to historical data related to OPE, which enables customizing the alarm filtering for each of the road users who may be affected. The OPE learning algorithm tracks the responses of road users who may be affected to received alarms and adjusts future alarms to elicit the best response times and best attention from the road users who may be affected. The learning algorithm can also be applied to the data transmitted in the messages sent.

[0058] 4. A data storage server 54, which may be, without limitation, cloud storage, local storage, or any other storage facility enabling the storage and access of data. The data storage server is accessible by the RSE, a computing unit, and, in some cases, by the OBE, the OPE, and the data server, for the purpose of storing data related, for example, to early warning and collision avoidance. The data storage server is accessible from the RSE, and, in some cases, from the OBE, the OPE, and the data server, for the purpose of fetching the stored data. The data may be raw sensor data, data processed by a processing unit, or any other information generated by the RSE, the OBE, and the OPE.

[0059] Sensors at intersections that continuously monitor ground traffic entities can generate a large amount of data every day. This data volume depends on the number and type of sensors. The data is processed locally and in real time at the intersection, for example, and stored for future analysis that requires a data storage unit (e.g., hard disk drive, solid state drive, and other large-capacity storage devices). Local storage devices will become full after a period of time, depending on their storage capacity, the volume of data generated, and the rate at which it is generated. To store the data for future use, the data is uploaded to a remote server with a larger capacity. The remote server can upgrade the storage capacity on demand as needed. The remote server may use a data storage device similar to local storage (e.g., hard disk drive, solid state drive, or other large-capacity storage device) that is accessible through a network connection.

[0060] Data stored locally and on the server for future analysis may include data broadcast by ground traffic entities, received by the RSE, and stored for future analysis. The stored data can be downloaded from the server or other remote sources for processing at the RSE. For example, a machine learning model of an intersection where the RSE is located may be stored on the server or other remote storage and downloaded by the RSE for use in analyzing current data received at the RSE from a local source.

[0061] Computing unit 56, which is a powerful computing machine located in the cloud, or locally (e.g., as part of the RSE), or a combination thereof. Among other functions, the computing unit processes data that can be used to predict the behavior, actions, and intentions of vehicles, pedestrians, or other ground traffic entities using the transportation network, and to generate machine learning-based models. Each of the computing units may include special hardware for processing corresponding types of data (e.g., a graphics processing unit for processing images). In the case of heavy processing loads, the computing units in the RSE can become overloaded. For example, this can occur when additional data generation units (e.g., sensors) are added to the system, causing a computational overload. Overloading can also occur when the logic operating in the computing unit is replaced with more computationally intensive logic. Overloading can occur due to an increase in the number of ground traffic entities being tracked. When a local computational overload occurs, the RSE can offload some of the tasks to another computing unit. The other computing unit can be near the RSE or remote (e.g., a server). Computational tasks can be prioritized, and tasks that are not time-critical can be performed in another computing unit, and the results can be obtained by the local computing unit. For example, the computing unit in the RSE can request another computing unit to perform jobs for analyzing the stored data and training a model using the data. The trained model is then downloaded by the computing unit in the RSE for storage and use there.

[0062] The computing unit in the RSE may use other small computing units to more efficiently perform computationally intensive jobs and reduce time. The available computing units are wisely used to perform most tasks in the shortest time, for example, by splitting tasks between the RSE computing unit and other available computing units. The computing unit may also be attached as an external device to the RSE to add more computing power to the computing unit in the RSE. The externally attached computing unit may include the same or a different architecture compared to the computing unit in the RSE. The externally attached computing unit may communicate with the existing computing unit using any available communication port. The RSE computing unit may request more computing power from the external computing unit as needed.

[0063] In the remaining part of this document, among other things, the roles and functions of the above-mentioned components in the system will be described in detail.

[0064] Roadside Equipment (RSE)

[0065] As shown in FIG. 2, the RSE may include, but is not limited to, the following components.

[0066] 1. One or more communication units 103, 104 that enable the reception or transmission or both of motion data and other data related to a ground transportation entity, and traffic safety data, from and to nearby vehicles or other ground transportation entities, infrastructure, and remote servers, and data storage system 130. In some examples, this type of communication is known as infrastructure - to - everything (I2X), which includes but is not limited to infrastructure - to - vehicles (I2V), infrastructure - to - pedestrians (I2P), infrastructure - to - infrastructure (I2I), and infrastructure - to - devices (I2D), and combinations thereof. The communication may be wireless or wired and may conform to a wide variety of communication protocols.

[0067] 2. Communication unit 103 is used for communication with a ground transportation entity, and unit 104 is used for communication through the Internet with remote servers and data storage system 130.

[0068] 3. A local storage 106 for storing programs, intersection models, and behavior and traffic models. Local storage 106 may be used for temporary storage of data collected from sensor 101.

[0069] 4. Sensors 101 and sensor control device 107 that enable monitoring (e.g., generation of data related thereto) of moving objects such as typically ground traffic entities near the RSE. The sensors may include, but are not limited to, cameras, radars, lidars, ultrasonic detectors, or any other hardware that can detect, or infer from the detected data, for example, the distance to a ground traffic entity, or the speed, direction of travel, or position of a ground traffic entity, or a combination thereof. Sensor fusion is implemented using an aggregate or combination of data from two or more sensors 101.

[0070] 5. A position receiver (102) (e.g., a GPS receiver) that provides positioning data (e.g., coordinates of the position of the RSE) and helps correct positioning errors in the positioning of ground traffic entities.

[0071] 6. A processing unit 105 that acquires and uses data generated by sensors and incoming data from communication units 103, 104. The processing unit processes and stores data locally and, in some embodiments, transmits data for remote storage and further processing. The processing unit further generates messages and alerts that are broadcast or otherwise transmitted via wireless communication facilities to nearby pedestrians, motor vehicles, or other ground traffic entities and, in some examples, to signage or other infrastructure presentation devices. The processing unit also periodically reports the status and status of all RSE systems to a remote server for monitoring.

[0072] 7. An expansion connector 108 that enables control and communication between the RSE and other hardware or other components, such as temperature and humidity sensors, traffic signal control devices, other computing units described above, and other electronic components that may become available in the future.

[0073] On-board equipment (OBE)

[0074] An in-vehicle device is typically original equipment for a ground transportation entity or can be added to the entity by a third-party supplier. As shown in FIG. 3, the OBE may include, but is not limited to, the following components.

[0075] 1. A communication unit 203 that enables the transmission and reception, or both, of data to and from nearby vehicles, pedestrians, cyclists, or other ground transportation entities, and infrastructure, and combinations thereof. The communication unit further enables the transmission or reception (or both) of data between the vehicle or other ground transportation entity and a local or remote server 212 for the purpose of machine learning and for the remote monitoring of ground transportation entities by the server. In some examples, this type of communication is known as vehicle-to-everything (V2X), which includes, but is not limited to, vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), vehicle-to-infrastructure (V2I), vehicle-to-device (V2D), and combinations thereof. The communication may be wireless or wired and may conform to a wide variety of communication protocols.

[0076] Communication unit 204 enables the OBE to communicate with a remote server over the Internet for program updates, data storage, and data processing.

[0077] 2. A local storage 206 for storing programs, intersection models, and traffic models. Local storage 206 may be used for the temporary storage of data collected from sensor 201.

[0078] The sensor 201 and the sensor control device 207 may include, but are not limited to, an external camera, a lidar, a radar, an ultrasonic sensor, or any device that can be used to detect nearby objects, people, or other ground transportation entities. The sensor 201 may further include additional kinematic sensors, a global positioning receiver, and internal and local microphones and cameras.

[0079] 4. A position receiver 202 (e.g., a GPS receiver) that provides positioning data (e.g., the coordinates of the position of a ground transportation entity).

[0080] 5. A processing unit 205 that acquires, uses, generates, and transmits data, including consuming data from a communication unit, transmitting data to a communication unit, and consuming data from sensors within or on a ground transportation entity.

[0081] 6. An expansion connector 208 that enables control and communication between the OBE and other hardware.

[0082] 7. An interface unit that can be additionally introduced or integrated into the head unit, the handle, or the driver's mobile device by one or more methods, for example, using visual, audible, or tactile feedback.

[0083] Smart OBE (SOBE)

[0084] In a world where all vehicles and other surface transportation entities are connected entities, each vehicle or other surface transportation entity can be an entity that collaborates with others and can report its current location, safety status, intentions, and other information to others. Currently, almost all vehicles are not connected entities and cannot report such information to other surface transportation entities, and are operated by people with different levels of skill, happiness, stress, and behavior. Without such connectivity and communication, it becomes difficult to predict the next movement of a vehicle or surface transportation entity, and furthermore, it results in lower capabilities for implementing collision avoidance and providing early warnings.

[0085] The Smart OBE monitors the surrounding environment and the users or occupants of surface transportation entities. It further monitors the condition and status of different systems and subsystems of the entity. The SOBE monitors the external world, for example, by listening for radio wave transmissions from emergency broadcasts, traffic and safety messages from nearby RSEs, and messages regarding safety, location, and other operational information from other connected vehicles or other surface transportation entities. The SOBE further interfaces with in-vehicle sensors such as cameras, distance sensors, vibration sensors, microphones, or any other sensors that can see the road and driving conditions to enable such monitoring. The SOBE further monitors the immediate surrounding environment and generates a map of all stationary and moving objects.

[0086] The SOBE can further monitor the behavior of the users or occupants of a vehicle or other surface transportation entity. The SOBE uses a microphone to monitor the quality of conversations. It can further use other sensors, for example, seat sensors, cameras, hydrocarbon sensors, and sensors for volatile organic compounds and other toxic substances. It can further use kinematic sensors to measure the driver's reactions and behavior and infer the quality of driving therefrom.

[0087] SOBE further receives vehicle-to-vehicle messages (e.g., basic safety messages (BSMs)) from other ground transportation entities and vehicle-to-pedestrian messages (e.g., personal safety messages (PSMs)) from vulnerable road users.

[0088] SOBE then fuses data from this array of sensors, sources, and messages. SOBE can then not only predict the next actions or reactions of the driver or user of the vehicle or other ground transportation entity or vulnerable road user, but also applies the data fused into an artificial intelligence model that can predict intentions and future trajectories and associated near misses or collision risks caused by other vehicles, ground transportation entities, and nearby vulnerable road users. For example, SOBE can use the BSMs received from nearby vehicles to predict that a nearby vehicle is about to enter a lane change maneuver that would create a risk to its own host vehicle and can alert the driver about the impending risk. The risk is calculated by SOBE based on the probabilities of the various predicted future trajectories of the nearby vehicle (e.g., going straight, changing lanes to the right, changing lanes to the left) and the associated collision risks with the host vehicle for each of those trajectories. If the collision risk is higher than a certain threshold, a warning is displayed to the driver of the host vehicle.

[0089] Machine learning is typically required to predict intentions and future trajectories due to the complexity of modeling the behavior of human drivers, which is also affected by external factors (e.g., changing surrounding environment and weather conditions).

[0090] SOBE is characterized by having powerful computing capabilities that can process a number of data feeds, some of which provide data in the order of several megabytes per second. The amount of data available is further proportional to the level of detail required from each sensor.

[0091] The SOBE further includes a powerful signal processing device that can extract useful information from the surrounding environment known to have a high (signal) noise level and a low signal-to-noise ratio. The SOBE further protects the driver from the vast number of alerts received by the vehicle by providing smart alert filtering. The alert filtering is the result of a machine learning model that can distinguish which alerts are important in terms of the current location, surrounding environmental conditions, driver behavior, vehicle condition and status, and kinematics.

[0092] The smart OBE is important for collision avoidance and early warning and for providing a safer transportation network not only for the people or users riding in the vehicle including the SOBE, but also for all users. The SOBE can detect and predict the movement of different entities on the road and thus assist in collision avoidance.

[0093] Over-Person Equipment (OPE)

[0094] As described above, the over-person equipment (OPE) includes any device that is a ground transportation entity, or alternatively exists on or uses a ground transportation network, and can be held by, worn by, or otherwise directly connected to pedestrians, joggers, or other people. Such people can be road users who may be at risk of being collided with by a vehicle and can be victims. The OPE may include, but is not limited to, mobile devices (e.g., smartphones, tablets, digital assistants), wearable devices (e.g., eyewear, watches, bracelets, anklets), and implants. Existing components and features of the OPE can be used to track and report position, speed, and direction of travel. The OPE may be used to receive data, process it, and display alerts to the user in various modes (e.g., visual, audio, tactile).

[0095] Honda has developed a communication system and method for V2P applications that focuses on direct communication between vehicles using OPE and pedestrians. In one example, the vehicle is equipped with an OBE for broadcasting messages to the OPE of pedestrians in the surrounding environment. The messages convey the current status of the vehicle, including, for example, vehicle parameters, speed, and direction of travel. For example, the message can be a basic safety message (BSM). If necessary, the OPE presents an alert to the pedestrian adjusted according to the pedestrian's level of inattentiveness regarding a predicted dangerous situation to avoid a collision. In another example, the pedestrian's OPE broadcasts a message (e.g., a personal safety message (PSM)) to the OBE of vehicles in the surrounding environment where the pedestrian may cross the intended path of the vehicle. If necessary, the vehicle's OBE displays an alert to the vehicle user regarding a predicted danger to avoid a collision. See "Vehicle to pedestrian communication system and method." by Strickland, Richard Dean et al. (U.S. Patent No. 9,421,909).

[0096] The systems described herein use an I2P or I2V approach that uses sensors (primarily in infrastructure) external to vehicles and pedestrians to track and collect data regarding pedestrians and other vulnerable road users. For example, the sensors can track pedestrians crossing a street and vehicles operating at or near the crossing location. The data collected is subsequently used to build predictive models of the intentions and behaviors of pedestrians and vehicle drivers on the road using rule-based and machine learning methods. These models help analyze the data collected and predict the paths and intentions of pedestrians and vehicles. If a danger is predicted, a message is broadcast from the RSE to the OBE or OPE or both to alert each entity of the other's intended path and enable them to take action such as getting a head start with sufficient time to avoid a collision.

[0097] Remote computing (cloud computing and storage)

[0098] Data collected from sensors connected to or embedded in the RSE, OBE, and OPE needs to be processed such that an effective mathematical machine learning model can be generated. This processing requires a lot of data processing power to reduce the time required to generate each model. The processing power required is much more than what is typically locally available in the RSE. To address this, the data can be sent to remote computing facilities that provide the required power and can scale on demand. In this document, the remote computing facilities are referred to as "remote servers" following the terminology used in the literature regarding computing. In some instances, it may be possible to perform some or all of the processing in the RCE by equipping them with high-powered computing capabilities.

[0099] Rule-based processing

[0100] Unlike artificial intelligence and machine learning technologies, rule-based processing can be applied at any time without the need for data collection, training, and model building. Rule-based processing can be deployed from the very beginning of the system's operation and is typically carried out until sufficient training data is obtained to generate a machine learning model. After a new installation, rules are set to process incoming sensor data. This is not only useful for improving road safety but also serves as an appropriate test case to ensure that all components of the system operate as expected. Rule-based processing can be further added and used later as an additional layer to capture rare cases where machine learning may not be able to make accurate predictions. The rule-based approach is based on simple correlations between the collected data parameters (such as speed, range, etc.). The rule-based approach can further provide a baseline for evaluating the performance of machine learning algorithms.

[0101] In rule-based processing, vehicles or other ground traffic entities crossing a part of the ground transportation network are monitored by sensors. If their current speed and acceleration become higher than a threshold that prevents them from stopping before a stop bar (line) on the road, for example, an alarm is generated. A variable area is assigned to each vehicle or other ground traffic entity. That area is labeled as a dilemma zone where the vehicle has not yet been labeled as a violating vehicle. If a vehicle enters the danger zone beyond the dilemma zone because its speed or acceleration, or both, have become higher than a predefined threshold, the vehicle is labeled as a violating entity and an alarm is generated. The thresholds for speed and acceleration are based on physics and kinematics and vary, for example, according to each ground traffic entity approaching an intersection.

[0102] Two conventional rule-based approaches are: 1) static TTI (Time-To-Intersection), and 2) static RDP (Required Deceleration Parameter). See Aoude, Georges S., et al., "Driver behavior classification at intersections and validation on large naturalistic data set.", IEEE Transactions on Intelligent Transportation Systems 13.2 (2012): 724-736.

[0103] Static TTI (Time-To-Intersection) uses the estimated time until reaching an intersection as a classification criterion. In its simplest form, TTI is

Number

[0104] Static RDP (Required Deceleration Parameter) is given the current speed of the vehicle and its position on the road and calculates the deceleration required for the vehicle to stop safely. RDP is

Number

Number

[0105] Similar to the static TTI algorithm, the RDP alert parameter reflects the conservativeness level of the rule-based algorithm.

[0106] In this specification, a rule-based approach is used as a baseline for evaluating the performance of the machine learning algorithms of the present disclosure. In some examples, in this specification, to capture rare cases, i.e., those rare cases that machine learning may not be able to predict, they are operated in parallel with the machine learning algorithms.

[0107] Machine learning

[0108] Modeling a driver's behavior has been shown to be a complex task considering the complexity of human behavior. See H.M. Mandalia, and D.D. Dalvucci, "Using Support Vector Machines for Lane-Change Detection", Human Factors and Ergonomics Society Annual Meeting Proceedings, vol. 49, pp. 1965{1969, 2005. Machine learning techniques are well-suited to modeling human behavior but require "learning" using training data to operate properly. In order to provide excellent detection and prediction results, herein, machine learning is used to model traffic or other features of the surface transportation network detected at intersections during a training period before alert processing is applied to current traffic during the deployment phase. Machine learning can be further used to model a driver's response using in-vehicle data from an on-board equipment (OBE) and can be further based on the history and preferences of in-vehicle sensors and driving records. Herein, further, machine learning models are used to detect and predict the trajectories, behaviors, and intentions of road users (e.g., pedestrians) who may be victims. Machine learning can be further used to model the responses of road users who may be victims from an on-board equipment (OPE). These models may include interactions between entities, between road users who may be victims, and between one or more entities and one or more road users who may be victims.

[0109] Machine learning techniques can be further used to model the behavior of non-autonomous surface transportation entities. By observing a non-autonomous surface transportation entity, or communicating with a non-autonomous surface transportation entity, or both, machine learning can be used to predict its intention, communicate with it, and communicate with other involved entities when a near miss or accident or other dangerous situation is predicted.

[0110] The machine learning mechanism operates in two phases, namely, 1) training and 2) deployment.

[0111] Training phase

[0112] After installation, the RSE begins to collect data from sensors it can access. Since AI model training requires powerful computing capabilities, it is usually carried out on a powerful server that includes multiple parallel processing modules to speed up the training phase. For this reason, the data obtained at the location of the RSE in the ground transportation network can be packaged and sent to a remote powerful server soon after acquisition. This is done using an Internet connection. Next, the data is prepared automatically or with the help of a data scientist. Next, an AI model is built to capture important characteristics of the traffic flow of vehicles and other ground transportation entities for that intersection or other aspects of the ground transportation network. The captured data features may include the position, direction, and movement of vehicles or other ground transportation entities, which can then be converted into intentions and behaviors. By knowing the intentions, the behavior and future behavior of vehicles or other ground transportation entities approaching the traffic location can be predicted with high accuracy using the AI model. The trained AI model is tested on a subset of data that was not included in the training phase. If the performance of the AI model meets expectations, the training is considered complete. This phase is repeated iteratively using different model parameters until a satisfactory performance of the model is achieved.

[0113] Deployment phase

[0114] In some embodiments, the completed and tested AI model is then transmitted over the Internet to the RSE at the traffic location in the ground transportation network. The RSE is then prepared to process new sensor data and to perform predictions and detections of dangerous situations such as traffic signal violations. If a dangerous situation is predicted, the RSE generates an appropriate warning message. If a dangerous situation can be predicted and a warning message is generated, before the predicted dangerous situation occurs, the warning message is broadcast to vehicles and other ground traffic entities in the vicinity of the RSE and received by the vehicles and other ground traffic entities in the vicinity of the RSE. This gives the operator of the vehicle or other ground traffic entity sufficient time to react and to perform collision avoidance. The outputs of the AI model from the various intersections where the corresponding RSEs are located can be recorded and made available online in a dashboard that incorporates all the data generated and displayed in an intuitive and user-friendly manner. Such a dashboard can be used as an interface to the customers of the system, such as city traffic engineers or planners. An example of a dashboard is a map that includes markers indicating the locations of the intersections being monitored, violation events that have occurred, and statistical data and analysis results based on AI predictions and actual results.

[0115] Smart RSE (SRSE: Smart RSE) and Bridging of Connected / Unconnected Entities

[0116] As already suggested, there is a gap between the capabilities and behaviors of connected entities and those of unconnected entities. For example, connected entities are typically collaborative entities that continuously advertise their location and safety system status, such as speed, direction of travel, brake status, and headlight status, to the world. Unconnected entities cannot collaborate and communicate by these means. Thus, even connected entities do not recognize unconnected entities that are out of detection range due to not being in the vicinity of a connected entity or due to interference, distance, or lack of a good vantage point.

[0117] Including appropriate equipment and configuration, an RSE can be enabled to detect all entities using the ground transportation network in their vicinity, including unconnected entities. Special sensors can be used to detect different types of entities. For example, radar is suitable for detecting moving metallic objects such as cars, buses, and trucks. Such road entities are most likely to move in one direction towards an intersection. Cameras are suitable for detecting road users who may be vulnerable to being hit while walking around near an intersection while gauging a safe timing to cross.

[0118] Placing sensors on components of the ground transportation network has at least the following advantages.

[0119] - Good viewpoints: Infrastructure poles, beams, and support cables typically include high viewpoints. High viewpoints enable a more comprehensive view of intersections. This is similar to an air traffic control tower at an airport where the controller can overlook the majority of important and vulnerable users on the ground. In contrast, for ground traffic entities, the view from a good viewpoint of a sensor (camera, lidar, radar, etc., or others) can be obstructed or disrupted by trucks in adjacent lanes, direct sunlight, or other interferences. Sensors at intersections can be selected to be resistant to or less susceptible to such interferences. Radar, for example, is not affected by sunlight and remains effective during evening commuting. Thermal cameras are more likely to detect pedestrians in bright light situations where the view of an optical camera is obstructed.

[0120] - Fixed position: Sensors located at intersections can be adjusted and fixed to detect in specific directions that may be optimal for detecting important targets. This helps the processing software to detect objects more appropriately. As an example, when a camera has a fixed view, information about the background (stationary objects and structures) in the fixed view can be easily detected and used to improve the identification and classification of relatively important moving entities.

[0121] A fixed sensor position further enables easier placement of each entity in a unified global view of the intersection. Since the sensor's view is fixed, the measurements from the sensor can be easily mapped to an integrated global position map of the intersection. Such an integrated map is useful when performing a global analysis of traffic movement from all directions to investigate interactions and dependencies of one traffic flow on another. An example is detecting a near miss (hazardous situation) before it occurs. When two entities are moving along a path where they will cross, the global and integrated view of the intersection enables calculation of the arrival time of each entity at the intersection of their respective paths. If the time is within a certain limit or tolerance, the near miss can be flagged (e.g., targeted with an alert message) before it occurs.

[0122] With the help of sensors installed on infrastructure components, Smart RSE (SRSE) can bridge this gap and enable connected entities to recognize "dark" or unconnected entities.

[0123] Figure 8 shows a scenario illustrating how strategically placed sensors can help connected entities identify the speed and position of unconnected entities.

[0124] The connected entity 1001 moves along path 1007. The entity 1001 is presenting a green signal 1010. The unconnected entity 1002 moves along path 1006. The entity 1002 is presenting a red signal 1009 and is trying to turn right on a red signal along path 1006. This places the entity 1002 directly in the path of the entity 1001. Since the entity 1001 does not recognize the entity 1002, a dangerous situation is imminent. Since the entity 1002 is an unconnected entity, it cannot broadcast (e.g., advertise) its position and direction of travel to other entities sharing the intersection. Further, the entity 1001 cannot "see" the entity 1002 that is being made invisible by the building 1008 even if it is connected. There is a risk that the entity 1001 will go straight through the intersection and collide with the entity 1002.

[0125] If the intersection is configured as a smart intersection, the radar 1004 mounted on the beam 1005 above the road at the intersection detects the entity 1002 and its speed and distance. This information can be relayed to the connected entity 1001 through the SRSE 1011 that functions as a bridge between the unconnected entity 1002 and the connected entity 1001.

[0126] Artificial Intelligence and Machine Learning

[0127] Smart RSE further relies on learning traffic patterns and entity behavior to better predict, prevent, and avoid collisions in dangerous situations. As shown in FIG. 8, radar 1004 constantly detects and provides data for each entity moving along approach path 1012. This data is collected directly or through the RSE, for example, for analysis, to build and train a model that accurately represents traffic along approach path 1012, and is transmitted to the cloud. When the model is complete, it is downloaded to SRSE 1011. This model can then be applied to each entity moving along approach path 1012. If an entity is classified by the model as violating (or attempting to violate) traffic rules, a warning (alert) can be broadcast by the SRSE to all connected entities in the vicinity. This warning, known as an intersection collision avoidance warning, is received by the connected entities and can be used as a basis for considering and avoiding dangerous situations. By using an appropriate traffic model, violating entities can be detected in advance, giving connected entities using the intersection sufficient time to react to and avoid dangerous situations.

[0128] With the help of multiple sensors (some mounted high on components of the ground transportation network infrastructure), an AI model, and an accurate traffic model, the SRSE can obtain a virtual overview of the ground transportation network and recognize each entity within its field of view, including unconnected entities within the field of view that are not "visible" to connected entities within the field of view. The SRSE uses this data to feed the AI model and can provide warnings to connected entities instead of unconnected entities. Connected entities would otherwise be unaware of the presence of unconnected entities sharing the road.

[0129] The SRSE is equipped with high - power computing that is available within the same housing at the location of the SRSE, or to nearby units, or to a server via the Internet. The SRSE can process data directly received from sensors, or data received by broadcast from nearby SRSEs, emergency and weather information, and other data. The SRSE further comprises a large - capacity storage to help store and process data. High - bandwidth connectivity is further required to assist in transmitting raw data and AI models between the SRSE and a more powerful remote server. The SRSE uses AI to extend other traffic hazard detection technologies in order to achieve high accuracy, and to provide additional time for reacting and avoiding collisions.

[0130] The SRSE can still conform to current and new standardized communication protocols, and thus they can be seamlessly interfaced with equipment already deployed on - site.

[0131] The SRSE can further reduce network congestion by sending messages only when necessary.

[0132] Global and integrated intersection topology

[0133] Effective traffic monitoring and control of intersections benefits from an aerial view of the intersection that is not obstructed by obstacles, lighting, or any other interference.

[0134] As described above, different types of sensors can be used to detect different types of entities. Information from these sensors can be different, for example, not matching the position or motion parameters that the data represents, or the native format of the data, or both. For example, radar data typically includes speed, distance, and perhaps additional information such as the number of moving and stationary entities within the radar's field of view. Camera data, in contrast, can represent an image of the field of view at any given point in time. Lidar data can provide the positions of points in 3D space corresponding to the reflection points of laser beams emitted from the lidar at a specific time and direction of travel. Generally, each sensor provides data in a native format that accurately represents the physical quantity they measure.

[0135] To obtain an integrated view (representation) of an intersection, the fusion of data from different types of sensors is useful. For the purpose of fusion, data from various sensors is converted into a common (integrated) format that is independent of the sensors used. The data included in the integrated format from all of the sensors includes the global position, speed, and direction of travel of each entity using the intersection, regardless of how it was detected.

[0136] By using this integrated global data, the smart RSE can not only detect and predict the movement of entities, but also identify the relative positions and directions of travel of different entities with respect to each other. Therefore, the SRSE can achieve improved detection and prediction of dangerous situations.

[0137] For example, in the scenario shown in FIG. 9, the motor-driven entity 2001 and the potentially affected road user 2002 share the same crosswalk. The entity 2001 moves along the road 2007 and is detected by the radar 2003. The potentially affected road user 2002 walking along the sidewalk 2006 is detected by the camera 2004. The potentially affected road user 2002 may decide to cross the road 2007 using the crosswalk 2005. Doing so places the road user 2002 in the path of the entity 2001, creating a potentially dangerous situation. Assuming that the data 2003 and 2004 from each of the sensors are considered independently and no other information is considered, each of the sensors may only detect the entity within its respective field of view, so the dangerous situation is not identified. Furthermore, each of the sensors may be unable to detect objects that they are not designed to detect. However, when an integrated view is considered by the SRSE, the positions and dynamics of the entity 2001 and the potentially affected road user 2002 can be placed in the same reference coordinate system, i.e., a geographic coordinate system, such as a map projection, or another coordinate system. When considered within a common reference system, the fused data from the sensors can be used to detect and predict potentially dangerous situations that may occur between the two entities 2001 and 2002. The following paragraphs explain the conversion between the sensor space and the integrated space.

[0138] Conversion from Radar Data to an Integrated Reference

[0139] As shown in FIG. 10, radar 3001 is used to monitor road entities moving along a road including two lanes 3005 and 3008 each containing a center line 3006 and 3007. Stop bar 3003 indicates the ends of lanes 3005 and 3008. T3006 can be defined by a set of markers 3003 and 3004. Although FIG. 10 shows only two markers, generally, the center line is a piecewise linear function. The global positions of markers 3003 and 3004 (and other markers not shown) are predefined by the lane design and known to the system. The exact global position of radar 3001 can be further specified. Thus, distances 3009 and 3010 from radar 3001 to markers 3003 and 3004 can be calculated. Distance 3011 from radar 3001 to entity 3002 can be measured by radar 3001. Using simple geometry, the system can identify the position of entity 3002 using the measured distance 3011. Since it is derived from the global positions of markers 3003, 3004, and radar 3001, the result is a global position. Since each lane can be approximated by a generalized piecewise linear function, the above method can be applied to any lane that can be monitored by radar.

[0140] FIG. 11 shows a similar scenario on a curved road. Radar 4001 monitors entities moving along road 4008. Markers 4003 and 4004 represent linear segments 4009 of the center line 4007 (piecewise linear function). Distances 4005 and 4006 represent the normal distances between the plane 4010 of radar 4001 and markers 4003 and 4004, respectively. Distance 4007 is the measured distance of entity 4002 from radar plane 4010. Continuing with the previous explanation, given the global positions of radar 4001 and markers 4003 and 4004, the global position of entity 4002 can be calculated using simple arithmetic of ratios.

[0141] Conversion from Camera Data to Integrated Criteria

[0142] By knowing the height, global position, orientation, tilt, and field of view of the camera, calculating the global position of each pixel in the camera image becomes straightforward using existing 3D geometric laws and transformations. As a result, when an object is identified in the image, its global position can be easily estimated by knowing the pixels it occupies. It is beneficial to note that the type of camera is irrelevant if its specifications, such as sensor size, focal length, or field of view, or combinations thereof, are known.

[0143] FIG. 12 shows a side view of a camera 5001 looking at an entity 5002. The height 5008 and tilt angle 5006 of the camera 5001 can be determined at installation. The field of view 5007 can be obtained from the specifications of the camera 5001. The global position of the camera 5001 can be further determined at installation. From the known information, the system can identify the global positions of points 5003 and 5004. The distance between point 5003 and point 5004 is further divided into pixels in the image generated by the camera 5001. The number of these pixels is known from the specifications of the camera 5001. The pixels occupied by the entity 5002 can be identified. Thus, the distance 5005 can be calculated. The global position of the entity 5002 can be further calculated.

[0144] The global integrated view of any intersection can be put together by fusing information from various sensors. Figure 13 shows a plan view of a four-way intersection. Each leg of the intersection is separated by a median 6003. The intersection in the figure is monitored by two different types of sensors, radar and camera, and the principles described herein can be generalized to other types of sensors. In this example, the radar monitoring area 6001 overlaps the camera monitoring area 6002. By having a unified global view, each entity moving between areas continues to be tracked within the unified global view. This enables, for example, the identification of the relevance between the actions of different entities by SRSE. Such information enables a truly universal bird's-eye view of intersections and roadways. The integrated data from the sensors can then be fed into an artificial intelligence program as described in the following paragraphs.

[0145] Figure 2 above showed the components of the RSE. Additionally, in SRSE, the processing unit may further include one or more special processing units that can process data in parallel. An example of such a unit is a graphics processing unit or GPU. With the assistance of a GPU or similar hardware, machine learning algorithms can operate much more efficiently in SRSE and provide results in real time. Such a processing architecture enables real-time prediction of dangerous situations and thus enables early warnings to be sent to provide entities with sufficient time to react and avoid collisions. Additionally, since SRSE can perform processing using data from different sensors and different types of sensors, SRSE can construct an integrated view of intersections useful for traffic flow analysis and detection and prediction of dangerous situations.

[0146] Usage examples

[0147] A wide variety of examples can yield benefits in terms of the system and the early warnings it can provide for collision avoidance. Examples are provided here.

[0148] Example 1: Ground transportation entities that can be victimized

[0149] As shown in FIG. 4, the lanes crossing a typical intersection 409 may include a crosswalk that includes specific crossing areas 401, 402, 403, 404 that can be used by pedestrians and other road users who can be victimized (road users who can be victimized) to walk across the lanes. Sensors suitable for detecting such crossing or other users who can be victimized are located at one or more vantage points that enable monitoring of the crosswalk and the surrounding environment of the crosswalk. During the training phase, the data collected can be used to train an artificial intelligence model to learn about the behavior of road users who can be victimized at the intersection. During the deployment phase, the AI model can then use current data regarding road users who can be victimized, for example, to predict that a road user who can be victimized is about to cross a lane and to make that prediction before the road user who can be victimized begins to cross. If the behavior and intentions of pedestrians and other road users who can be victimized, drivers, vehicles, and other people, and ground transportation entities can be predicted in advance, an early warning (e.g., an alert) can be sent to any or all of them. The early warning can enable a vehicle to stop, slow down, change route, or a combination thereof, and can enable a road user who can be victimized to refrain from crossing the road if a dangerous situation is predicted to be imminent.

[0150] Generally, sensors are used to monitor all areas of possible movement of road users and vehicles that may be affected in the vicinity of an intersection. The type of sensor used depends on the type of object being monitored and tracked. Some sensors are better at tracking people and bicycles or other non-motorized vehicles. Some sensors are better at monitoring and tracking motorized vehicles. If the sensor provides appropriate data at a sufficient data rate that may depend on the type of object being monitored and tracked, the type of sensor is irrelevant, and the solutions described herein are sensor- and hardware-agnostic. For example, Doppler radar is a suitable sensor for monitoring and tracking the speed and distance of vehicles. The data rate or sampling rate is the rate at which the radar can provide successive new data values. The data rate must be fast enough to capture the dynamics of the movement of the object being monitored and tracked. The higher the sampling rate, the more detailed the capture, the more robust and accurate the representation of the movement by the data. If the sampling rate is too low and the vehicle moves a significant distance between two sample instances, it becomes difficult to model the behavior due to missed details during the intervals when no data is generated.

[0151] For a crosswalk, sensors monitor pedestrians and other road users (e.g., cyclists) who cross in the area of intersections and in the vicinity of intersections and who may be subject to other hazards. Data from these sensors can be segmented to represent a state with respective different virtual zones to aid in detection and location. The zones can be selected to correspond to respective critical areas where hazardous situations can be assumed, such as, for example, sidewalks, sidewalk entrances, and the approach roads 405, 406, 407, 408 to intersections. Activities and other states in each zone are recorded. The records can include, but are not limited to, kinematics (e.g., position, direction of travel, speed, and acceleration), and facial and body features (e.g., eyes, posture).

[0152] The number of sensors, the number of zones, and the shape of the zones are specific to each intersection and each approach road to the intersection.

[0153] FIG. 5 shows a plan view of a typical exemplary scene showing different zones used to monitor and track the movement and behavior of pedestrians or other road users who may be subject to hazards, and motor-driven and non-motor-driven vehicles and other surface traffic entities.

[0154] The sensors are configured to monitor a crosswalk that spans a road. Virtual zones (301, 302) can be located on the sidewalk and along the crosswalk. Other sensors are arranged to monitor vehicles and other surface traffic entities traveling on the road following the crosswalk, and virtual zones (303, 304) are strategically arranged to aid in detecting, for example, incoming vehicles and other surface traffic entities, their distances from the crosswalk, and their speeds.

[0155] A system (e.g., an RSE or SRSE related to sensors) collects a data stream from all sensors. When the system is first put into operation, an initial rule-based model can be deployed to assist with device calibration and functionality. During that time, sensor data (e.g., speed and distance from a radar unit, images and videos from a camera) is collected and stored locally in the RSE for preparation and, in some embodiments, to be transmitted to a remote computer powerful enough to use this collected data to build an AI model of the behavior of different entities at an intersection. In some examples, the RSE is an SRSE capable of generating the AI model itself.

[0156] Next, the data is prepared and trajectories are constructed for each ground traffic entity passing through the intersection. For example, the trajectories can be derived from radar data by connecting points at different distances belonging to the same entity together. Pedestrian trajectories and behavior can be derived, for example, from camera and video recordings. By implementing video and image processing techniques, pedestrian movements can be detected in the images and videos and their respective trajectories can be estimated.

[0157] For human behavior, intelligent machine learning-based models typically outperform simple rules based on simple physics. This is because human intentions are difficult to capture and a large data set is required to be able to detect patterns.

[0158] When a machine learning (AI) model is completed on a server, it is downloaded to the RSE, for example, through the Internet. The RSE then applies the current data captured from the sensors to the AI model to have it predict intentions and behaviors, identify when a dangerous situation is imminent, and trigger corresponding warnings that are disseminated (e.g., broadcast) to vehicles and other ground traffic entities, as well as to road users and drivers who may be affected, as early warnings in time for the road users and drivers who may be affected to perform collision avoidance steps.

[0159] This exemplary setting can be combined with any other use case, such as traffic at a signalized intersection or an un-signalized intersection.

[0160] Example 2: Signalized intersection

[0161] In the case of a signalized intersection (e.g., one controlled by traffic lights), the overall setting of the system is done as in Example 1. One difference can be the types of sensors used to monitor or track the speed, direction of travel, distance, and position of vehicles. The setting for the crosswalk in Example 1 can also be combined with the setting for a signalized intersection for a more general solution.

[0162] The concept of the operation for the use case of a signalized intersection is to track road users around the intersection using external sensors that collect data about the users or data communicated by the users themselves, predict their behaviors, and generally broadcast warnings through different communication means regarding an impending dangerous situation resulting from a violation of the intersection's traffic rules, such as running a red light.

[0163] Data regarding road users can be collected using (a) entity data broadcast by each entity itself regarding its current state, e.g., via BSM or PSM, and (b) sensors installed externally on infrastructure or vehicles, such as Doppler radars, ultrasonic sensors, vision cameras or thermal cameras, lidars, etc. As described above, the type of sensors selected, and their position and orientation at the intersection must provide the broadest coverage of the intersection, or the part thereof under investigation, and furthermore the data collected regarding entities approaching the intersection is most accurate. Thus, the collected data enables the reconstruction of the current state of road users and the generation of accurate, timely, and useful VBSM (Virtual Basic Safety Message) or VPSM (Virtual Personal Safety Message). The frequency at which data must be collected depends on the potential risks and the criticality of potential violations for each type of road user. For example, a motor-driven vehicle moving at high speed at an intersection typically requires 10 data updates per second to achieve real-time collision avoidance, while a pedestrian crossing the intersection at a much lower speed may require as few as 1 data update per second.

[0164] As described above, FIG. 4 shows an example of a plan view of a signalized intersection including detection virtual zones. These zones may segment each approach to the intersection into separate lanes 410, 411, 412, 413, 405, 406, 407, 408, and may further divide each lane into areas corresponding to a general distance range from the stop bar. The selection of these zones can generally be performed empirically to fit the characteristics of a particular approach and intersection. Segmenting the intersection enables a more accurate identification of the relative direction of travel, speed, acceleration, and position for each road user, and thus a more appropriate assessment of the potential risks posed by the road user to other surface traffic entities.

[0165] To determine whether the observed traffic situation is a dangerous situation, the system further needs to compare the result of the predicted situation with the state of the traffic lights and to take into account local traffic rules (e.g., left-turn lanes, right-turn on red, etc.). Therefore, it is necessary to collect and use signal phase and timing (SPaT) information at intersections. SPaT data can generally be collected by directly interfacing with the traffic signal control device at the intersection through a wired connection for reading the data, or by interfacing with the traffic management system to receive the required data, for example, through an API. To ensure that the state of the road users is always synchronized with the state of the traffic signals, it is important to collect SPaT data at a rate as close as possible to the rate at which road user data is collected. A further complication in the requirement to know SPaT information is that modern traffic control techniques used to regulate the traffic flow near intersections do not rely on fixed timings but use algorithms that can adapt dynamically to the real-time traffic situation. Therefore, it is important to incorporate SPaT data prediction algorithms to ensure the highest accuracy in violation prediction. These SPaT data prediction algorithms can be constructed using rule-based methods or machine learning methods.

[0166] For each approach road to an intersection, data is collected by RSE (or SRSE), and a machine learning (AI) model is configured to explain the behavior of vehicles corresponding to the collected data. Next, the current data collected at the intersection is applied to the AI model to generate an early prediction as to whether a vehicle or other ground traffic entity moving on one of the approach roads to the intersection is about to violate, for example, a traffic signal. If a violation is imminent, a message is relayed (e.g., broadcast) from the RSE to ground traffic entities in the vicinity. Vehicles (including the violating vehicle) and pedestrians or other road users who may be affected receive the message and have time to take appropriate measures to avoid a collision. The message can be delivered to the ground traffic entity by one or more of the following techniques, namely, a flashing light, a sign, or a wireless signal, among others.

[0167] If a vehicle or other entity approaching the intersection is equipped with an OBE or OPE, it can receive a message broadcast from the RSE about a potential hazard being predicted at the intersection. This enables the user to be warned and to take appropriate measures to avoid a collision. If a violating road user at the intersection is further equipped with an OBE or OPE, the user receives the broadcast warning as well. The algorithm in the OBE or OPE can then adjust the message to include the user's violating behavior and can appropriately warn the user.

[0168] The determination to send an alert does not solely depend on the behavior of the vehicle represented by the data collected by sensors at an intersection. Although sensors play a significant role in the determination, other inputs are also considered. These inputs may include, but are not limited to, information from nearby intersections (if a vehicle ran through a signal at a nearby intersection, the probability that the vehicle will do the same at this intersection is higher), information from other connected vehicles, or even information from the vehicle itself, such as when the vehicle reports that it has a problem.

[0169] Example 3: Intersection without traffic lights

[0170] Controlled intersections without traffic lights, such as intersections controlled by stop signs or yield signs, can be similarly monitored. Sensors are used to monitor the access roads controlled by traffic signs, and predictions regarding approaching vehicles can be made, similar to those for approaching vehicles on the access roads to intersections with traffic lights. The road rules at controlled intersections without traffic lights are typically clearly defined. Ground traffic entities on access roads controlled by stop signs must come to a complete stop. At multi-way stop intersections, the right of way is determined by the order in which ground traffic entities reach the intersection. In special cases, one-way stops can be considered. The set of sensors can similarly monitor access roads without stop signs. Such a setting can assist in gap negotiation for stop signs. For intersections controlled by yield signs, ground traffic entities on access roads controlled by yield signs must reduce their speed to give the right of way to other ground traffic entities at the intersection.

[0171] The main issue is that due to internal factors (such as driver distraction) or external factors (such as lack of visibility), ground traffic entities violate road rules and pose risks to other ground traffic entities.

[0172] In a general example of an intersection controlled by a stop sign (i.e., each approach is controlled by a stop sign), the overall setup of the system is done as in Example 1. One difference could be the type of sensors used to monitor or track the speed, direction, distance, and position of vehicles. Another difference is that there is no traffic signal control device and the road rules are indicated by road signs. The setup for the crosswalk in Example 1 can be further combined with the setup of a controlled intersection without signals for a more general solution.

[0173] It can be further understood that FIG. 4 shows an example of a plan view of a four-way stop intersection including detection virtual zones. These zones can segment each approach to the intersection into separate lanes 410, 411, 412, 413, 405, 406, 407, 408, and can further divide each lane into areas corresponding to a general distance range from the stop bar. The selection of these zones can generally be done empirically to fit the characteristics of a particular approach and intersection.

[0174] Using a method similar to what has been described so far for FIG. 4, the current data collected at the intersection is applied to an AI model to generate an early prediction of whether a vehicle or other ground traffic entity moving on one of the approaches to the intersection is about to violate the stop sign. If a violation is imminent, the message can be handled in the same way as in the previous example involving a traffic signal violation.

[0175] Similar to the explanation so far, the determination to send an alert can be based on the aforementioned factors and on other information, such as whether the vehicle has run through a stop sign at a nearby intersection, which suggests that the vehicle is more likely to do the same at this intersection.

[0176] FIG. 18 shows an example of use for an intersection without a controlled traffic signal. FIG. 18 illustrates how an SRSE including strategically placed sensors can warn a connected entity of an impending dangerous situation arising from an unconnected entity.

[0177] Connected entity 9106 is moving along path 9109. Entity 9106 has the right of way. Unconnected entity 9107 is moving along path 9110. Entity 9107 is presented with a yield sign 9104 and merges onto path 9109 without giving the right of way to entity 9106, placing entity 9107 directly in the path of entity 9106. Since entity 9106 does not recognize entity 9107, a dangerous situation is imminent. Since entity 9107 is an unconnected entity, it cannot advertise (broadcast) its position and direction of travel to other entities sharing the intersection. Further, entity 9106 may not be able to "see" entity 9107 which is not within its direct line of sight. If entity 9106 continues along its path, entity 9106 may ultimately collide with entity 9107.

[0178] Since the intersection is a smart intersection, radar 9111 mounted on beam 9102 above the road detects entity 9107. Radar 9111 further detects the speed and distance of entity 9107. This information can be relayed as a warning to connected entity 9106 through SRSE 9101. SRSE 9101 includes a machine learning model for entities moving along approach path 9110. Entity 9107 is classified by the model as a potential traffic rule violator and a warning (alert) is broadcast to connected entity 9106. This warning is sent in advance to give entity 9106 sufficient time to react to and prevent the dangerous situation.

[0179] Example 4: Level crossing

[0180] Level crossings are dangerous because they allow motor - driven vehicles, pedestrians, and railway vehicles to pass through. In many cases, the roads following a level crossing enter the blind spot of the driver of a train or other railway vehicle (e.g., the conductor). Since railway vehicle drivers mainly drive based on line - of - sight information, this increases the likelihood of an accident when a road user violates the right - of - way of a railway vehicle and crosses the level crossing when not permitted to cross.

[0181] In the sense that level crossings are often the collision points between road and railway traffic regulated by traffic rules and signals, the operation of the use cases of level crossings is similar to that of intersections with traffic lights. Therefore, this use case further requires collision avoidance warnings to enhance safety near level crossings. Railway traffic can have a planned separate railway right - of - way, such as a high - speed railway, or may not have a separate railway right - of - way, such as a light urban railway or a tram. In the case of light railways and trams, these railway vehicles also operate on active roads and need to follow the same traffic rules as road users, so this use case becomes even more important.

[0182] FIG. 6 shows a general use case of a level crossing where a road and a crosswalk cross a railway. Similar to the use case of the crosswalk, sensors are placed to collect data on the movement and intention of pedestrians. Other sensors are used to monitor and predict the movement of road vehicles approaching the intersection. Data on road users can also be collected from road user broadcasts (e.g., BSM or PSM). Data from nearby intersections, vehicles, and remote command and control centers can be used in the decision to trigger an alarm.

[0183] In order to appropriately evaluate the likelihood of violations, it is necessary to further collect data on SPaT for the access roads of roads and railways.

[0184] Similar to the usage example of the intersection with a traffic signal, the collected data enables the generation of prediction models using rule-based algorithms and machine learning algorithms.

[0185] In this usage example, the railway vehicle is equipped with an OBE or OPE to receive a collision avoidance warning. If a violation of the railway vehicle's right of way is predicted, the RSE broadcasts an alarm message to warn the driver of the railway vehicle that a road user is present on its intended route, enabling the driver of the railway vehicle to take action with sufficient time to avoid a collision.

[0186] If the violating road user is further equipped with an OBE or OPE, the message broadcast by the RSE is further received by the violating road user. The algorithm in the OBE or OPE can then adjust the received message including the user's violation behavior and appropriately warn the user.

[0187] (Bridging the gap) Virtual connected ground traffic environment

[0188] As described above, a useful application of the system is to generate a virtual connected surrounding environment instead of unconnected ground traffic entities. The obstacles to the adoption of connected technologies are not only the lack of infrastructure installations, but also the almost non-existence of connected vehicles, road users who can be connected and damaged, and other connected ground traffic entities.

[0189] In connection with connected vehicles, in some regulatory regimes, such vehicles always transmit what is called a basic safety message (BSM). The BSM includes, among other information, the vehicle's position, direction of travel, speed, and future route. Other connected vehicles can pay attention to these messages and use them to generate a map of the vehicles present in their surrounding environment. By knowing where the vehicles in the surrounding environment are, a vehicle can maintain a high level of safety, whether it is autonomous or not. For example, an autonomous vehicle can avoid maneuvers if there are connected vehicles on its route. Similarly, a driver can receive an alert if there are any other vehicles on the route that the driver is planning to take, for example, if there is a sudden lane change.

[0190] Until all ground transportation entities are equipped to send and receive traffic safety messages and information, some road entities are "dark" or invisible to the rest of the road entities. Dark road entities pose a risk of dangerous situations.

[0191] Dark road entities do not advertise (e.g., broadcast) their positions and are thus invisible to connected entities that may assume that all road entities broadcast their information (i.e., are connected entities). In-vehicle sensors can detect obstacles and other road entities, but the range of these sensors tends to be too short to be effective in preventing dangerous situations and collisions. Thus, there is a gap between the connectivity of connected vehicles and the lack of connectivity of non-connected vehicles. The techniques described below are aimed at bridging this gap by using intelligence regarding infrastructure that can detect all vehicles at intersections or other components of the ground transportation network and send messages on behalf of non-connected vehicles.

[0192] The system can bridge the gap between the future, where most vehicles (and other ground transportation entities) are expected to be connected entities, and the current time when most vehicles and other ground transportation entities have no connectivity, and establish a virtual connected ground transportation environment, for example, at intersections. In the virtual connected ground transportation environment, smart traffic lights and other infrastructure installations can use sensors to track all vehicles and other ground transportation entities (connected, unconnected, semi-autonomous, autonomous, non-autonomous), and (in the case of vehicles) can generate virtual BSM messages (VBSM) instead of them.

[0193] The VBSM message can be considered a subset of the BSM. It may not contain all the fields required to generate a BSM, but it can contain all the location-specific information including position, direction of travel, speed, and trajectory. Since V2X communication is standardized and anonymized, VBSM and BSM cannot be easily distinguished and follow the same message structure. The main difference between the two messages is the availability of the source of the information contained in these messages. The VBSM may not contain data and information that are not easily generated by external sensors, such as steering wheel angle, brake status, tire pressure, or wiper activation.

[0194] By installing appropriate sensors, intersections with Smart RSEs can detect all road entities moving through the intersection. The SRSE can further convert all data from multiple sensors into a global integrated coordinate system. This global integrated system is represented by the geographical location, speed, and direction of travel of each road entity. Each road entity is detected by intersection equipment regardless of whether it is connected, and a global integrated position is generated in its place. Standard safety messages can thus be broadcast instead of the road entity. However, if the RSE broadcasts safety messages for all entities it detects, the RSE can send messages instead of the connected road entity. To resolve collisions, the RSE can filter the connected road entities from its list of dark entities. This can be achieved since the RSE continuously receives safety messages from connected vehicles and the RSE sensors continuously detect road entities passing through the intersection. If at a location where a safety message is received by the RSE receiver, the position of the detected road entity matches that location, the road entity is presumed to be connected and the safety message is not broadcast by the RSE in its place. This is shown in Figure 15.

[0195] By bridging between connected and unconnected vehicles, connected entities (including autonomous vehicles) can safely maneuver through the intersection with a complete awareness of all nearby road entities.

[0196] This aspect of the technology is shown in FIG. 17. Intersection 9001 involves a plurality of road entities at a given point in time. Some of these entities 9004, 9006 are not connected, while others 9005, 9007 are connected. Road users 9004, 9007 who may be affected are detected by camera 9002. Motor-driven road entities 9005, 9006 are detected by radar 9003. The position of each road entity is calculated. Broadcasts from connected road entities are further received by RSE 9008. The position of the entity from which a message was received is compared with the position at which the entity was detected. If two entities match within a predetermined tolerance, the entities at that position are considered connected and no safety message is sent instead. The remaining road entities with no matching received positions are considered dark. Instead, safety messages are broadcast for them.

[0197] For collision warnings and intersection violation warnings, which are integrated parts of the V2X protocol, each entity needs to be connected for the system to be effective. That requirement is an obstacle in the deployment of V2X devices and systems. Intersections equipped with smart RSEs address that concern by providing a virtual bridge between connected and unconnected vehicles.

[0198] The U.S. DOT (Department of Transportation) and NHTSA (National Highway Traffic Safety Administration) have identified a number of connected vehicle applications that use BSM and that can help substantially reduce crashes and fatalities not caused by malfunction. These applications include, but are not limited to, Forward Collision Warning (FCW), Intersection Movement Assist (IMA), Left Turn Assist (LTA), Do Not Pass Warning (DNPW), and Blind Spot / Lane Change Warning (BS / LCW). The U.S. DOT and NHTSA define these applications as follows.

[0199] FCW resolves rear-end collisions and warns the driver about stopped, slowing, or slower vehicles ahead. IMA is designed to avoid intersection broadside collisions, warns the driver about vehicles approaching from the side at intersections, and covers two main scenarios, namely, the path change route in the same or opposite direction and the straight intersection route. LTA resolves collisions when a turning vehicle makes a left turn at an intersection and other vehicles move straight from the opposite direction, and warns the driver of the presence of oncoming traffic approaching when attempting to make a left turn. DNPW assists the driver in avoiding oncoming collisions caused by passing maneuvers and warns the driver of an approaching oncoming vehicle when attempting to pass a slower vehicle on an undivided two-lane roadway. BS / LCW resolves collisions when a vehicle makes a lane change / merging maneuver prior to a collision and alerts the driver of the presence of vehicles in their blind spots in adjacent or nearby lanes.

[0200] The V2X protocol stipulates that these applications must be achieved using vehicle-to-vehicle (V2V) communication, where one connected remote vehicle broadcasts basic safety messages to the connected host vehicle. The OBE of the host vehicle then attempts to adjust its own vehicle parameters such as speed, direction of travel, and trajectory with these BSMs, and determine whether there are potential hazards or threats caused by the remote vehicle as described hereinbefore in this specification. Furthermore, autonomous vehicles particularly benefit from such applications because it enables vehicles in the surrounding environment to communicate intentions that are an important part of information not included in the data collected from their on-board sensors.

[0201] However, current vehicles are not connected, and as described above, it takes a very long time until the proportion of connected vehicles is high enough for the BSM to function properly as described above. Therefore, in an environment where the proportion of connected vehicles is small, connected vehicles do not need to receive and analyze a large number of BSMs as they would in an environment where the proportion of connected vehicles is high enough to enable the above applications and receive sufficient benefits from V2X communication.

[0202] VBSM can help bridge the gap between the current environment, which includes entities mostly not connected, and the future environment, which includes entities mostly connected, and enable the above applications during the intervening period. In the technology described herein, a connected vehicle that receives a VBSM processes it as a regular BSM in the application. VBSM follows the same message structure as BSM, and since VBSM includes substantially the same basic information as BSM, such as speed, acceleration, direction of travel, past trajectory, and predicted trajectory, the result of applying the message to a given application is substantially the same.

[0203] For example, considering an intersection with an unprotected left turn, where a connected host vehicle is attempting a left turn while a distant unconnected vehicle is moving straight from the opposite direction with right of way. This is a situation where the execution of the maneuver depends on the judgment of the driver of the host vehicle regarding the situation. An inappropriate assessment of the situation can lead to conflicts and potential collision-prone approaches or collisions. External sensors installed in the surrounding infrastructure can detect and track the distant vehicle or both vehicles, collect basic information such as speed, acceleration, direction of travel, and past trajectories, and send them to the RSE, which can then use a rule-based algorithm or a machine learning algorithm or both to construct a predicted trajectory for the distant vehicle, fill in the fields required for VBSM, and broadcast it instead of the distant unconnected vehicle. The OBE of the host vehicle receives the VBSM containing information about the distant vehicle and processes it in its LTA application to determine whether the driver's maneuver poses a potential danger and whether the OBE must display a warning to the driver of the host vehicle to take anticipatory or corrective actions to avoid a collision. Similar results can be further achieved when the distant vehicle is connected and the RSE and sensors receive data regarding a left turn by an oncoming vehicle where a collision is predicted.

[0204] VBSM can also be used in the operation of lane changes. When a vehicle changing lanes fails to perform necessary steps for ensuring the safety of the operation, such as using rearview mirrors and side mirrors and checking blind spots, such an operation can be dangerous. New advanced driver assistance systems, such as blind spot warnings using in-vehicle ultrasonic sensors, have been developed to help prevent vehicles from performing dangerous lane changes. However, these systems can have drawbacks when the sensors are dirty or the view is blocked. Furthermore, existing systems do not attempt to warn a vehicle at risk about another vehicle attempting to change lanes. V2X communication helps solve this problem through applications such as BS / LCW using BSM, but the vehicle attempting to change lanes can be an unconnected vehicle and thus may not be able to communicate its intention. VBSM can help achieve its goal. Similar to the usage example of LTA, external sensors installed in the infrastructure of the surrounding environment can detect and track an unconnected vehicle attempting a lane change operation, collect basic information such as speed, acceleration, direction of travel, and past trajectory, and send them to the RSE. The RSE can then use rule-based algorithms and machine learning algorithms to construct a predicted trajectory for the vehicle changing lanes, fill in the fields required for VBSM, and broadcast it instead of the unconnected distant vehicle. The OBE of the vehicle at risk then receives a VBSM containing information about the vehicle attempting to merge into the same lane, processes it, and determines whether the operation poses a potential risk and whether it should display a lane change warning to the driver of the vehicle. If the vehicle changing lanes is a connected vehicle, its OBE can similarly receive a VBSM from the RSE regarding the vehicle in its blind spot and determine whether the lane change operation poses a potential risk to the surrounding traffic and whether it should display a blind spot warning to the driver of the vehicle.When both vehicles are connected, both vehicles can broadcast BSMs to each other, enabling BS / LCW applications. However, these applications still benefit from applying the same rule-based algorithm or machine learning algorithm (or both) as described above to the BSM data to predict early the intention of the vehicle changing lanes, and to determine whether the OBE displays a warning.

[0205] Autonomous vehicle

[0206] The lack of connectivity to unconnected road entities affects autonomous vehicles. The sensors in autonomous vehicles are short-range or have a narrow field of view. They cannot detect, for example, a vehicle coming near a building at a street corner. They also cannot detect a vehicle that may be hidden behind a delivery truck. These hidden vehicles are invisible to autonomous vehicles if they are unconnected entities. These situations affect the technological capabilities of autonomous vehicles to achieve the safety levels required for large-scale adoption of the technology. Smart intersections can help mitigate this gap and assist in the public's acceptance of autonomous vehicles. Autonomous vehicles are only as good as the sensors they have. Intersections equipped with smart RSEs can extend the coverage range of in-vehicle sensors around blind corners or beyond large trucks. Such an extension enables autonomous entities and other connected entities to coexist with conventional unconnected vehicles. Such coexistence can accelerate the adoption of autonomous vehicles and the benefits they bring.

[0207] The virtual connected ground traffic environment includes VBSM messages that enable the implementation of vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), and vehicle-to-device (V2D) applications that were difficult to implement by other means.

[0208] The system may use machine learning to quickly and accurately generate the data fields required for various safety messages, and to pack them into a VBSM message structure, and to send messages to ground traffic entities in the vicinity using various media such as, but not limited to, DSRC, Wi-Fi, cellular, or conventional road signs.

[0209] Virtual personal safety message (VPMS)

[0210] The ground traffic environment may include not only unconnected vehicles, but also unconnected people and other road users who may be victims.

[0211] In some regulatory regimes, ground traffic entities that may be connected victims continuously send personal safety messages (PSMs). PSMs may include, among other information, the location, direction of travel, speed, and future route of the ground traffic entity that may be a victim. Connected vehicles and infrastructure may receive these messages and use them to generate a map that includes the entities that may be victims and increases the level of safety in the ground traffic network.

[0212] Therefore, a virtual connected ground traffic environment can bridge the gap between the future, where most ground traffic entities that may be victims are assumed to be connected, and the current time, where most ground traffic entities that may be victims have no connectivity. In a virtual connected ground traffic environment, smart traffic lights and other infrastructure installations may use sensors (connected and unconnected) to track all ground traffic entities that may be victims and generate VPSMs in their place.

[0213] The VPSM message can be considered a subset of the PSM. The VPSM does not need to include all the fields required to generate the PSM, but may include data necessary for safety assessment and prevention of dangerous situations, and may include location-specific information including position, direction of travel, speed, and trajectory. In some examples, non-standard PSM fields such as, for example, the driver's intent, posture, or direction of view may be further included in the VPSM.

[0214] The system may use machine learning to generate these fields quickly and accurately, to pack them into the VPSM message structure, and to transmit it to ground traffic entities in the vicinity using various media such as, but not limited to, DSRC, Wi-Fi, cellular, or conventional road signs.

[0215] The VPSM message enables the implementation of pedestrian-to-vehicle (P2V), pedestrian-to-infrastructure (P2I), pedestrian-to-device (P2D), vehicle-to-pedestrian (V2P), infrastructure-to-pedestrian (I2P), and device-to-pedestrian (D2P) applications that were difficult to implement by other means.

[0216] Figure 16 shows a pedestrian 8102 crossing a crosswalk 8103. The crosswalk 8103 can be at an intersection or in the middle of a block spanning a continuous road between intersections. A camera 8101 is used to monitor a sidewalk 8104. The global position of the boundary of the camera 8101's field of view 8105 can be determined at the time of installation. The field of view 8105 is covered by a predetermined number of pixels reflected by the specifications of the camera 8101. A road entity 8102 can be detected within the camera's field of view, and its global position can be calculated. The speed and direction of travel of the road entity 8102 can be further identified from its movement at s time points. The path of the road entity 8102 can be represented by a breadcrumb 8106 which is a sequence of positions where the entity 8102 crossed. This data can be used to construct a virtual PSM message. Next, the PSM message can be broadcast to all entities in the vicinity of the intersection.

[0217] Traffic enforcement and behavior enforcement at intersections without traffic lights

[0218] Another useful application of the system is traffic enforcement (e.g., stop signs, yield signs) at intersections without traffic lights, and enforcement of good driving behavior anywhere in the surface transportation network.

[0219] As a byproduct of generating VBSM and VPSM, the system can track and detect road users who do not comply with traffic regulations and who increase the probability of dangerous situations and collisions. Prediction of dangerous situations can be extended to include enforcement. A dangerous situation does not need to result in a collision. Near misses are common, increase the stress level of drivers, and can lead to later accidents. The frequency of near misses is positively correlated with the lack of enforcement.

[0220] Furthermore, using VBSM, the system can detect inappropriate driving behaviors such as sudden lane changes and other forms of reckless driving. Data collected by sensors can be used for training and to enable a machine learning model to flag ground traffic entities that are engaging in dangerous driving behaviors.

[0221] Enforcement authorities typically enforce road rules against ground traffic entities, including road users who may be victims, but the authorities need to be present in the vicinity of intersections to monitor, detect, and report violations. By using VBSM and VPSM to track unconnected ground traffic entities, including road users who may be victims, the smart RSE can act in the role of an enforcement authority and enforce road rules at intersections. For example, an unconnected vehicle tracked by the smart RSE can be detected, identified, and reported to the authorities for violating a stop sign or a yield sign. Similarly, a road user who may be a victim near an intersection tracked by the smart RSE can be detected, identified, and reported to the authorities for illegally crossing the intersection.

[0222] For enforcement and other purposes, ground traffic entities can be identified using unique identification information including but not limited to license plate recognition. Road users who may be victims can be identified using biometric recognition including but not limited to face recognition, retina recognition, and voice waveform recognition. In special cases including civil incident investigations or criminal investigations, social media networks (e.g., Facebook, Instagram, Twitter) can be further used to support the identification of violating ground traffic entities or road users who may be victims. Examples of social networks that can be leveraged include uploading a captured photo of the violator to the social network and asking social network users who recognize the violator to provide intelligence to the enforcement authorities that can help identify the violator.

[0223] High SOBE

[0224] The Smart RSE uses sensors and prediction models to predict dangerous situations and then can send virtual safety messages (such as ICA, VBSM, VPSM, VICA, and VCSM (virtual combined safety messages)) on behalf of unconnected road users, including those who may be victimized. The Smart OBE uses incoming virtual safety messages or standard safety messages (such as BSM, PSM, VBSM, VPSM, VICA, and VCSM), vehicle sensors, and a prediction model to predict dangerous situations and can alert the driver of the host vehicle. The High SOBE does all of that and can (a) send virtual BSM, virtual PSM, virtual ICA, and VCSM, and standard messages, whenever applicable, on behalf of other road users, even those that are unconnected (functioning as a Smart RSE), and especially those who may be victimized, (b) send a high-level BSM to its own ground traffic entity that includes information based on the prediction of its own behavioral intent, and (c) send messages that function as an RSE, such as GPS corrections.

[0225] This specification refers to BSM, PSM, ICA, VBSM, VPSM, and VICA. There are other types of safety messages that exist or may be developed, including the Cooperative Perception Message (CPM) under development as reported at https: / / www.sae.org / standards / content / j2945 / 8 / . References to BSM, PSM, ICA, VBSM, VPSM, and VICA are intended to further refer to existing and future other safety messages including CPM. As proposed, for example, CPM may include data regarding multiple objects (and, in that sense, containers for these objects) such as VBSM, VPSM, and VICA. In this specification, references are made in some cases to containers that may be for VBSM, VPSM, VICA, and other types of virtual safety messages, such as messages for VCSM. ESOBE may generate VCSM based on its detection of ground traffic entities and other objects using sensors of its host vehicle. VCSM may be periodically broadcast by ESOBE to make unconnected or occluded ground traffic entities or other objects recognizable to other ground traffic entities.

[0226] In this example, an on-board equipment (OBE) is described that can have advanced and further performance that surpasses the already described OBE and SOBE in several aspects. In some embodiments described herein, such an advanced SOBE (ESOBE: enhanced SOBE) has the ability to actually function as a smart RSE, for example, by enhancing sensors already present in a vehicle (the "host vehicle") in which the ESOBE exists together with a prediction model, which is superior to the existing capabilities. In some examples, the ESOBE can (1) generate and broadcast virtual safety messages instead of one or more other vehicles, and (2) use the intention prediction of its own behavior to generate advanced standard safety messages for host ground traffic entities, and (3) transmit them, together with the virtual safety messages they transmit in their roles as in-vehicle RSEs (e.g., GPS corrections), to, for example, other vehicles or road users who may be affected. In this specification, the in-vehicle RSE is sometimes referred to as an advanced RSE ("ERSE": enhanced RSE).

[0227] The following describes scenarios and applications for an advanced SOBE ("ESOBE") technology that includes the following.

[0228] 1. The ESOBE can, for example, generate and broadcast a virtual BSM instead of a skidding and unconnected other vehicle. Next, a third connected vehicle that cannot visually recognize the skidding vehicle from its position because its view is obstructed receives the virtual BSM and can use it, for example, to generate an early warning of a dangerous situation for the driver of the third vehicle.

[0229] 2. ESOBE generates and broadcasts a virtual PSM on behalf of pedestrians or other road users who may be at risk of being hit, for example, at a marked or designated crosswalk at an intersection. Next, a third connected vehicle that has a road user who may be at risk of being hit from its position obscured from view (including moving or stationary vehicles such as public transport vehicles) receives the virtual PSM and may use it to generate, for example, an early warning of a dangerous situation for the driver of the third vehicle.

[0230] 3. ESOBE generates and broadcasts a virtual PSM on behalf of pedestrians or other road users who may be at risk of being hit at a crosswalk in the middle of a block rather than at an intersection. Next, a third connected vehicle that has a road user who may be at risk of being hit from its position obscured from view receives the virtual PSM and may use it to generate, for example, an early warning of a dangerous situation for the driver of the third vehicle.

[0231] 4. ESOBE generates and broadcasts an enhanced standard BSM from its host vehicle by including in the BSM a prediction of a forward collision identified based on information received by ESOBE from, for example, an in-vehicle camera or sensor or both of the host vehicle. A vehicle following the host vehicle may use the advanced BSM to generate an early warning for the driver of the following vehicle that the host vehicle may brake hard, for example, within 1 to 2 seconds.

[0232] 5. ESOBE generates and broadcasts GPS corrections to other surface transportation entities. In this mode, ESOBE operates substantially as an RSE, which helps to expand the coverage of RSE corrections by enhancing the GPS of the OBE itself. The OBE can further store and obtain the most recent GPS correction received from the nearest RSE it approaches.

[0233] In all of these scenarios described above and below, and in other scenarios, the messages broadcast by the ESOBE may include virtual messages bundled in the VCSM.

[0234] Skidding vehicle obscured and angled collision

[0235] FIG. 19 shows a vehicle or another ground traffic entity 1907 that is moving within lane 1914 and skidding into an adjacent lane 1916 across a multi-lane road 1912 (e.g., not equipped with an OBE or otherwise not connected). The skidding of the vehicle may occur, for example, due to strong braking, a slippery road surface, or another reason. In this scenario, vehicle 1906 is moving within lane 1916 and is equipped with an ESOBE 1908 and a sensor 1918. The sensor 1918 can be, but is not limited to, a camera, radar, lidar, ultrasonic distance sensor, etc., and combinations thereof. The ESOBE 1908 processes the data feed from the sensor in real time and generates periodic basic safety messages (BSMs) for vehicle 1906.

[0236] A third vehicle 1902 is moving within lane 1920. The skidding vehicle 1907, or anything in front of vehicle 1906, or anything partially to the side of vehicle 1906, is obscured or partially obscured so as not to be visible from vehicle 1902.

[0237] When the sensor 1918 of vehicle 1906 detects a skidding vehicle 1907, the ESOBE 1908 of vehicle 1906 identifies the measured parameters of vehicle 1907 (assuming in this scenario that vehicle 1907 does not have an OBE). The measured parameters can include one or more of speed, direction of travel, route history, route prediction, brake status, or others, or combinations thereof. Based on these measured parameters, the ESOBE generates and broadcasts a virtual BSM in place of vehicle 1907 which does not have an OBE.

[0238] Based on the measured parameters identified by the ESOBE of vehicle 1906 and the virtual BSM obtained as a result of being broadcast by the ESOBE of vehicle 1906, even if the driver of vehicle 1902 and the in-vehicle sensors in vehicle 1902 cannot visually recognize vehicle 1907 because vehicle 1907 is blocked by vehicle 106, vehicle 1902 will recognize the skidding vehicle 1907. Using this information available to the SOBE of vehicle 1902 together with the measured parameters of its own motion including speed, direction of travel, etc., the SOBE of vehicle 1902 can predict the threat of a possible collision with vehicle 1907 and accordingly alert its driver.

[0239] In addition to making it easier for a vehicle in the surrounding environment (e.g., vehicle 1902 in this scenario) to recognize an unconnected vehicle that is not visible, if the ESOBE of vehicle 1906 detects the possibility of an angled collision between vehicle 1907 and vehicle 1902 within lane 1920 by predicting the skidding path of vehicle 1907 with respect to vehicle 1902, the ESOBE of vehicle 1906 can generate and transmit an intersection collision avoidance message (ICA). If vehicle 1902 is capable of receiving and handling ICA messages, vehicle 1902 can receive them, process them, and alert its driver that an angled (intersection) collision may occur.

[0240] In contrast, in typical DSRC (Dedicated Short Range Communication), ICA is generated and transmitted only at road intersections to alert drivers regarding angled collisions (e.g., when one vehicle is predicted to cross the path of another vehicle). In the technology described herein, since the ESOBE comprises a prediction algorithm, the ICA can be triggered, generated, and transmitted from the ESOBE of a moving vehicle even at locations other than road intersections. Such ICA can be used to alert drivers and to enable drivers to avoid angled collisions that may occur in a straight road segment as described above.

[0241] Obstructed Pedestrian: Crossing at Crosswalk

[0242] In an embodiment according to this scenario, the ESOBE can function as a broadcaster of virtual PSM, for example, on behalf of a pedestrian or other road user who may be at risk.

[0243] As shown in FIG. 20, in this scenario, pedestrian 2000 is crossing two-lane road 2012 at crosswalk 2010. Vehicle 2006 moving within lane 2014 obstructs (blocks the view of) pedestrian 2000 from vehicle 2002 moving within lane 2016. Zone 2004 indicates the recognition range (visible range) of the driver and the sensors of vehicle 2002. Vehicle 2006 is restricting zone 2004, and what is behind vehicle 2006 (relative to vehicle 2002) is blocked from being visible to vehicle 2002. If vehicle 2002 continues to move without recognizing the position, speed, and direction of movement of pedestrian 2000, a serious accident may occur.

[0244] The bus 2006 is equipped with an ESOBE2008 and a sensor 2018. The sensor 2018 can be, but is not limited to, a camera, a radar, a lidar, an ultrasonic distance sensor, etc., and combinations thereof. The ESOBE2008 processes the simultaneous data feeds of the sensor 2018 in real time. When the ESOBE2008 detects a pedestrian 2000 using the sensor data, the ESOBE2008 automatically starts to broadcast a virtual PSM message instead of the pedestrian 2000, and the virtual PSM message can be received by the vehicle 2002. As a result, even if the driver and the in-vehicle sensors of the vehicle 2002 cannot visually recognize the pedestrian 2000 on the other side of the vehicle 2006, the vehicle 2002 will be able to recognize the pedestrian 2000 crossing the lane 2014.

[0245] Obstructed Pedestrian: Crosswalk in the middle of a block

[0246] In this example, except that the pedestrian crosses a road in the middle of a block away from a crosswalk or other formal road intersection, this scenario is the same as the previous scenario.

[0247] As shown in FIG. 21, a pedestrian 2100 is crossing a road 2112 in front of a vehicle 2106. The pedestrian 2100 is blocked from the view of the driver and the sensors of the vehicle 2102. The recognition zone of the vehicle 2102 is indicated by the dotted line area 2104.

[0248] The vehicle 2106 is equipped with an ESOBE2108 that can transmit and receive V2X messages such as PSM and BSM, for example. The ESOBE2108 is further capable of processing simultaneous data feeds from the sensor 2118.

[0249] After being detected by ESOBE2108, information including, but not limited to, the global position, velocity, and direction of travel of pedestrian 2100 is encoded in a virtual PSM message, which is then broadcast. Vehicle 2102 receives the virtual PSM message and thus recognizes pedestrian 2100. The in-vehicle algorithm of vehicle 2102 can predict whether an imminent dangerous situation exists and take appropriate actions including, but not limited to, alerting the driver, automatically decelerating the vehicle, or stopping the vehicle.

[0250] Occluded Pedestrian: Forward Collision

[0251] Figure 22 shows pedestrian 2400 crossing lane road 2412 at crosswalk 24102. A larger vehicle (e.g., a truck) 2406 is moving within lane 2414 and is equipped with ESOBE2408 and sensor 2418. Sensor 2418 can be, but is not limited to, a camera, radar, lidar, ultrasonic distance sensor, etc., and combinations thereof. ESOBE2408 processes the data feed from the sensors in real time.

[0252] Another vehicle 2402 is moving behind the larger vehicle 2406 within the same lane 2414. Pedestrian 2400 or something in front of vehicle 2406 is blocked from the view of vehicle 2402.

[0253] If vehicle 2406 has to suddenly brake due to the presence of pedestrian 2400 (which vehicle 2402 does not recognize), vehicle 2402 can collide with the rear end of vehicle 2406.

[0254] When the pedestrian 2400 is detected using the sensor 2418, the ESOBE 2408 of the vehicle 2406 immediately starts to broadcast a virtual PSM on behalf of the pedestrian 2400 in addition to the normal basic safety message (BSM) it transmits for itself. As a result, together with the knowledge information of the presence of the vehicle 2406 (from the BSM), the vehicle 2402 further recognizes the pedestrian 2400 crossing the lane 2414 (from the virtual PSM), even if the driver of the vehicle 2402 and the in-vehicle sensors can neither visually recognize nor detect the vehicle 2406 beyond.

[0255] The ESOBE in the vehicle 2406 performs artificial intelligence processing that learns from its driver's behavior in various situations. For example, when the pedestrian 2400 is detected, the ESOBE 2408 applies an AI algorithm to predict whether the driver 2406 of the vehicle is about to apply the brakes and to predict the future point in time when that might occur. Based on these predictions, the ESOBE may decide to add braking information to the BSM message it broadcasts to other vehicles earlier than would occur if it were a typical system. This earlier delivery of the braking message can give other vehicles, such as the vehicle 2402 for example, more time to predict a collision in advance. In other words, other vehicles can benefit from the AI capabilities in the ESOBE of the first vehicle.

[0256] ESOBE - Position Correction Service

[0257] The availability of high-precision GPS position data is important for DSRC-based V2X safety applications or other applications that assume or require higher accuracy than typical GPS accuracy. ESOBE may provide GPS correction information to enhance the accuracy of GPS position data for nearby ground traffic entities. As shown in FIG. 23, four vehicles (2502, 2503, 2504, and 2505) moving in a given direction in different lanes of a road may be able to use GPS correction data in scenarios such as those described below.

[0258] In this scenario, vehicle (e.g., truck or bus) 2506 is moving within one lane 2516 and is equipped with an ESOBE 2508 capable of transmitting and receiving V2X messages. Vehicle 2506 is shown as a large vehicle, but vehicle 2506 can be any ground traffic entity of any size.

[0259] Assume that (a) there is no RSE in the vicinity of this area, (b) the vehicle 2506 (including its ESOBE) has recently passed through an area where there is an RSE with differential GNSS (global navigation satellite system) broadcast capabilities via DSRC (using technologies such as RTK, DGPS, or Wide Area RTK), (c) vehicle 2506 is no longer in the vicinity of the RSE, and (d) vehicle 2506 (and its ESOBE) has passed through an area where a specific RSE exists and the specific RSE has been transmitting periodic RTCM (Radio Technical Commission for Maritime Services) correction messages.

[0260] When these RTCM correction messages were received, the ESOBE 2508 in vehicle 2506 corrected its own position and stored the RTCM correction data for later use. While a particular RSE was within range, the ESOBE used the received correction messages to correct its position and continued to update these messages for later use.

[0261] After the ESOBE exited the coverage area of a particular RSE, the ESOBE used the stored correction data to construct a newer correction message based on its current position and provided a correction service to other vehicles (2502, 2503, 2504, 2505 in FIG. 23) (based on the newer correction message) in the same manner as the correction service provided by the RSE assuming the RSE was within range. The distance of the ESOBE from a particular RSE for which correction data was collected from that particular RSE, and the current position information of the ESOBE can be used to update the correction message (e.g., generate a newer correction message) before re-broadcasting them as part of the correction service. By this approach, the ESOBE effectively operates as a base station to provide accurate RTCM correction data to other road users in areas where the RSE is not within range. Thus, the ESOBE not only transfers correction data from the RSE or other external sources but also updates the correction data based on the current position of the host vehicle and other factors mentioned in this example.

[0262] In addition, the ESOBE executes an algorithm to ascertain the accuracy of the reconstructed correction data before broadcasting the reconstructed correction data. The correction data is broadcast only if the algorithm confirms a very high level of confidence in the reconstructed correction data.

[0263] In some embodiments, the algorithm for retransmission of the correction data may include other information, namely, The period elapsed since the ESOBE received the correction data from the RSE, the distance moved from the RSE, the direction of movement (travel direction) for determining whether the correction still applies at the current vehicle position, the reliability level of the original correction data received from the RSE, and combinations thereof are considered.

[0264] In some examples, another feature of the ESOBE is that if the ESOBE can directly access an external service that transmits an RTCM correction feed via the Internet, the ESOBE can use this feed to generate DSRC RTCM correction messages alone. The ESOBE can decide to switch to this mode in a scenario where there is no RSE or when the reliability level of the data received from the RSE is not appropriate. In this example, the ESOBE has the intelligence to select a better source of correction information.

[0265] Among the advantages of the ESOBE, being able to build or reconstruct on its own what is received from the RSE and being able to transmit correction messages helps ground traffic entities, including low-performance and inexpensive GPS devices, correct their positions, helps ground traffic entities execute safety algorithms more reliably, expands the effective coverage area of the RSE, and broadcasts RTCM corrections via the DSRC network or other short-range vehicle networks using V2X standard correction messages.

[0266] Other embodiments also fall within the scope of the claims described below. (Additional Item 1) An apparatus comprising equipment for use in a state mounted on a first ground transportation entity, said equipment comprising: (a) a receiver for information generated by sensors of the surrounding environment of the first ground transportation entity; (b) a processor; (c) a memory storing instructions executable by the processor to generate and transmit safety message information for a second ground transportation entity based on the information generated by one or more of said sensors; comprising: the apparatus. (Additional Item 2) The apparatus according to Additional Item 1, wherein the instructions are executable by the processor to generate a prediction for use in generating the safety message information. The apparatus according to Additional Item 1. (Additional Item 3) The apparatus according to Additional Item 2, wherein the prediction is generated by a prediction model. The apparatus according to Additional Item 2. (Additional Item 4) The apparatus according to Additional Item 3, wherein the prediction model is configured to predict a dangerous situation involving one or more of the first ground transportation entities, one or more of the second ground transportation entities, or one or more other ground transportation entities. The apparatus according to Additional Item 3. (Additional Item 5) The apparatus according to Additional Item 4, wherein the dangerous situation involves crossing a lane of a road by one or more of the second ground transportation entities. The apparatus according to Additional Item 4. (Additional Item 6) The apparatus according to Additional Item 5, wherein the one or more of the second ground transportation entities include one or more vehicles, and the dangerous situation includes a skid across the lane by the one or more vehicles. The apparatus according to Additional Item 5. (Additional Item 7) The apparatus according to Additional Item 5, wherein the one or more of the second ground transportation entities include one or more pedestrians crossing the road or one or more other road users who may be harmed. The apparatus according to Additional Item 5. (Additional Item 8) The apparatus according to Additional Item 7, wherein the road user who may be harmed crosses the road at an intersection. The apparatus according to Additional Item 7. (Additional Item 9) The apparatus according to Additional Item 7, wherein the road user who may be harmed crosses the road other than at an intersection. The apparatus according to Additional Item 7. (Additional Item 10) The predicted dangerous situation includes a predicted collision between a third ground traffic entity and the second ground traffic entity. The apparatus according to appended claim 4. (Appended claim 11) The first ground traffic entity includes a vehicle, and the second ground traffic entity includes a pedestrian or other road user who may be victimized. The apparatus according to appended claim 3. (Appended claim 12) The third ground traffic entity follows the first ground traffic entity, and the view of the third ground traffic entity is blocked by the first ground traffic entity. The apparatus according to appended claim 10. (Appended claim 13) The third ground traffic entity is in a lane adjacent to the lane in which the first ground traffic entity is moving. The apparatus according to appended claim 10. (Appended claim 14) The instruction is executable by the processor to specify the operating parameters of the third ground traffic entity. The apparatus according to appended claim 1. (Appended claim 15) The third ground traffic entity can only enter the field of view of the second ground traffic entity when in a blocked state. The apparatus according to appended claim 3. (Appended claim 16) The second ground traffic entity includes a pedestrian or other road user who may be victimized. The apparatus according to appended claim 3. (Appended claim 17) The safety message information transmitted by the processor includes a basic safety message. The apparatus according to appended claim 1. (Appended claim 18) The safety message information transmitted by the processor includes a virtual basic safety message. The apparatus according to appended claim 1. (Appended claim 19) The safety message information transmitted by the processor includes a personal safety message. The apparatus according to appended claim 1. (Appended claim 20) The safety message information transmitted by the processor includes a virtual personal safety message. The apparatus according to appended claim 1. (Appended claim 21) The safety message information transmitted by the processor includes a virtual basic safety message transmitted instead of the third ground traffic entity. The apparatus according to appended claim 1. (Appended claim 22) The third ground traffic entity includes an unconnected ground traffic entity. The apparatus according to appended claim 21. (Appended claim 23) The safety message information transmitted by the processor includes a virtual personal safety message transmitted instead of the third ground traffic entity. The apparatus according to supplementary note item 1. (Supplementary note item 24) The device includes the receiver for information wirelessly transmitted from a source outside the first ground traffic entity. The apparatus according to supplementary note item 1. (Supplementary note item 25) Including the first ground traffic entity. The apparatus according to supplementary note item 1. (Supplementary note item 26) The safety message information includes a virtual intersection collision avoidance message (VICA). The apparatus according to supplementary note item 1. (Supplementary note item 27) The safety message information includes an intersection collision avoidance message (ICA). The apparatus according to supplementary note item 1. (Supplementary note item 28) The safety message information includes a virtual combined safety message (VCSM). The apparatus according to supplementary note item 1. (Supplementary note item 29) The safety message information includes a combined safety message (CSM). The apparatus according to supplementary note item 1. (Supplementary note item 30) An apparatus comprising equipment for use in a state mounted on a first ground traffic entity, the equipment being: (a) a receiver for first position correction information transmitted from a source outside the first ground traffic entity; (b) the receiver for information representing the position parameters or operation parameters of the first ground traffic entity; (c) a processor; (d) a memory storing instructions executable by the processor to generate updated position correction information based on the first position correction information and the information representing the operation parameters, and to transmit a position correction message to another ground traffic entity based on the updated position correction information; Including Apparatus. (Supplementary note item 31) The position correction information transmitted from the source outside the first ground traffic entity includes the position correction message. The apparatus according to supplementary note item 30. (Supplementary note item 32) The position correction information transmitted from the source outside the first ground traffic entity includes a Radio Technical Commission for Maritime Services (RTCM) correction message. The apparatus according to supplementary note item 30. (Supplementary note item 33) The position correction information includes GNSS position correction. The apparatus according to supplementary note item 30. (Supplementary note item 34) The position parameters or the operation parameters include the current position of the first ground traffic entity. The apparatus according to supplementary note item 30. (Supplementary note item 35) The source, which is external to the first ground transportation entity, includes an RSE or an external service configured to transmit the position correction message via the Internet. The apparatus according to claim 30. (Claim 36) The processor is capable of executing the command to check the reliability level in the updated position correction information. The apparatus according to claim 30. (Claim 37) Receiving information generated by a sensor mounted on the first ground transportation entity regarding the surrounding environment of the first ground transportation entity, generating and transmitting safety message information for a second ground transportation entity based on the information generated by the sensor. A method including the above. (Claim 38) Including generating a prediction for use in generating the safety message information. The method according to claim 37. (Claim 39) The prediction is generated by a prediction model. The method according to claim 38. (Claim 40) The prediction model is configured to predict a dangerous situation involving the first ground transportation entity, the second ground transportation entity, or another ground transportation entity. The method according to claim 39. (Claim 41) The dangerous situation involves crossing a lane of the road by the second ground transportation entity. The method according to claim 40. (Claim 42) The second ground transportation entity includes a vehicle, and the dangerous situation includes a skid across a lane by the vehicle. The method according to claim 41. (Claim 43) The second ground transportation entity includes a pedestrian crossing the road or another road user who may be harmed. The method according to claim 41. (Claim 44) The road user who may be harmed crosses the road at an intersection. The method according to claim 43. (Claim 45) The road user who may be harmed crosses the road other than at an intersection. The method according to claim 43. (Claim 46) The predicted dangerous situation includes a predicted collision between a third ground transportation entity and the second ground transportation entity. The method according to claim 40. (Claim 47) The first ground transportation entity includes a vehicle, and the second ground transportation entity includes a pedestrian or another road user who may be harmed. The method according to claim 40. (Claim 48) The third ground traffic entity follows the first ground traffic entity, and the view of the third ground traffic entity is blocked by the first ground traffic entity. The method according to appended claim 46. (Appended claim 49) The third ground traffic entity is in a lane adjacent to the lane in which the first ground traffic entity is moving. The method according to appended claim 46. (Appended claim 50) Including identifying the operating parameters of the third ground traffic entity. The method according to appended claim 37. (Appended claim 51) The third ground traffic entity only enters the field of view of the second ground traffic entity in a blocked state. The method according to appended claim 46. (Appended claim 52) The second ground traffic entity includes pedestrians or other road users who may be victimized. The method according to appended claim 37. (Appended claim 53) The safety message information includes a basic safety message. The method according to appended claim 37. (Appended claim 54) The safety message information includes a virtual basic safety message. The method according to appended claim 37. (Appended claim 55) The safety message information includes a personal safety message. The method according to appended claim 37. (Appended claim 56) The safety message information includes a virtual personal safety message. The method according to appended claim 37. (Appended claim 57) The safety message information includes a virtual basic safety message transmitted instead of the third ground traffic entity. The method according to appended claim 37. (Appended claim 58) The third ground traffic entity includes unconnected ground traffic entities. The method according to appended claim 53. (Appended claim 59) The safety message information transmitted by the processor includes a virtual personal safety message transmitted instead of the third ground traffic entity. The method according to appended claim 53. (Appended claim 60) Including receiving information wirelessly transmitted from a source outside the first ground traffic entity. The method according to appended claim 37. (Appended claim 61) The safety message information includes a virtual intersection collision avoidance message (VICA). The method according to appended claim 37. (Appended claim 62) The safety message information includes an intersection collision avoidance message (ICA). The method according to appended claim 37. (Appended claim 63) The safety message information includes a virtual combined safety message (VCSM). The device according to appended claim 37. (Supplementary Note 64) The safety message information includes a combined safety message (CSM), the apparatus according to paragraph 37. (Supplementary Note 65) Receiving first position correction information transmitted from a source external to a first surface transportation entity; Receiving information representing operating parameters of the first surface transportation entity; Generating updated position correction information based on the first position correction information and the information representing the operating parameters; Transmitting a position correction message to another surface transportation entity based on the updated position correction information; A method comprising the steps. (Supplementary Note 66) The position correction information transmitted from the source external to the first surface transportation entity includes the position correction message, the method according to paragraph 65. (Supplementary Note 67) The position correction information transmitted from the source external to the first surface transportation entity includes a Radio Technical Commission for Maritime Services (RTCM) correction message, the method according to paragraph 65. (Supplementary Note 68) The position correction information includes GNSS position correction, the method according to paragraph 65. (Supplementary Note 69) The operating parameters include the current position of the first surface transportation entity, the method according to paragraph 65. (Supplementary Note 70) The source external to the first surface transportation entity includes an RSE or an external service configured to transmit a position correction message via the Internet, the method according to paragraph 65. (Supplementary Note 71) Including verifying a reliability level in the updated position correction information, the method according to paragraph 65.

Claims

1. 1. An apparatus comprising equipment for use on board a first ground transportation entity, the equipment comprising: (a) a receiver for first position correction information transmitted from a source external to the first ground traffic entity; (b) the receiver for information representative of a position or operational parameter of the first ground transportation entity; and (c) a processor; and (d) a memory storing instructions executable by the processor to generate updated position correction information based on the first position correction information and the information representing the operating parameter, and to transmit a position correction message to another ground transportation entity based on the updated position correction information; and Including, Device.

2. the position fix information transmitted from the source external to the first ground transportation entity includes the position fix message; 2. The apparatus of claim 1.

3. the position correction information transmitted from the source external to the first ground traffic entity comprises a Radio Technical Commission for Maritime (RTCM) correction message; 2. The apparatus of claim 1.

4. the position fix information includes a GNSS position fix; 2. The apparatus of claim 1.

5. the location parameters or the operational parameters include a current location of the first ground transportation entity; 2. The apparatus of claim 1.

6. the source external to the first ground transportation entity includes an RSE or an external service configured to transmit the position fix message over the Internet; 2. The apparatus of claim 1.

7. the instructions are executable by the processor to verify a level of confidence in the updated position fix information.

2. The apparatus of claim 1.

8. receiving first position correction information transmitted from a source external to the first ground traffic entity; receiving information representative of an operational parameter of the first ground transportation entity; generating updated position correction information based on the first position correction information and the information representative of the operational parameter; transmitting a position correction message to another ground transportation entity based on the updated position correction information; and A method comprising:

9. the position fix information transmitted from the source external to the first ground transportation entity includes the position fix message; The method according to claim 8.

10. the position correction information transmitted from the source external to the first ground traffic entity comprises a Radio Technical Commission for Maritime (RTCM) correction message; The method according to claim 8.

11. the position fix information includes a GNSS position fix; The method according to claim 8.

12. the operational parameters include a current location of the first ground transportation entity; The method according to claim 8.

13. the source external to the first ground transportation entity includes an RSE or an external service configured to transmit a position fix message over the Internet; The method according to claim 8.

14. ascertaining a level of confidence in the updated position fix information; The method according to claim 8.

Images