Enhanced vehicle equipment

Abstract

Among other things, equipment for use onboard a first ground transportation entity has (a) a receiver for information generated by a sensor of the first ground transportation entity's environment, (b) a processor, and (c) a memory storing instructions executable by the processor to generate safety message information based on the information generated by the sensor and send it to a second ground transportation entity.

Description

[0001] This application enjoys the benefit of the filing date of U.S. Provisional Patent Application No. 62 / 893,616, filed on August 29, 2019, the entire contents of which are incorporated herein by reference. Background Art

[0002] The contents of US Patent 10,235,882 are incorporated herein by reference.

[0003] The present description relates to enhanced vehicle equipment.

[0004] Collision avoidance systems are becoming more and more sophisticated. For example, King et al. (U.S. Patent Publication No. 2007 / 0276600A1, 2007) describe placing sensors in front of an intersection and applying physics-based decision rules to predict whether two vehicles are about to collide at the intersection based on travel direction and speed.

[0005] In Aoude et al. (U.S. Pat. No. 9,129,519 B2, 2015, the entire contents of which are incorporated herein by reference), driver behavior is monitored and modeled to allow prediction and prevention of violations of traffic conditions at intersections.

[0006] Collision avoidance is the primary defense against loss of life and property in ground transportation. Providing early warning of hazardous situations helps avoid collisions. Summary of the invention

[0007] Generally speaking, in one aspect, equipment for use onboard a first ground transportation entity has (a) a receiver for information generated by a sensor of the first ground transportation entity's environment, (b) a processor, and (c) a memory storing instructions executable by the processor to generate safety message information based on the sensor-generated information and transmit it to a second ground transportation entity.

[0008] Implementations may include one of the following features or a combination of two or more of the following features. The instructions are executable by a processor to generate a prediction for generating safety message information. The prediction is generated by a prediction model. The prediction model is configured to predict a dangerous situation involving a first ground transportation entity, a second ground transportation entity, or another ground transportation entity. The dangerous situation involves the second ground transportation entity crossing a road lane. The second ground transportation entity includes a vehicle, and the dangerous situation includes the vehicle skidding on the lane. The second ground transportation entity includes a pedestrian or other vulnerable road user crossing the road. The vulnerable road user is crossing the road at an intersection. The vulnerable road user is crossing the road outside the intersection. The predicted dangerous situation includes a predicted collision between a third ground transportation entity and a second ground transportation entity. The first ground transportation entity includes a vehicle, and the second ground transportation entity includes a pedestrian or other vulnerable road user. The third ground transportation entity follows the first ground transportation entity, and the third ground transportation entity's view from the first ground transportation entity is blocked. The third ground transportation entity is located in a lane adjacent to the lane in which the first ground transportation entity is traveling. The instructions are executable by the processor to determine motion parameters of the third ground transportation entity. The second ground transportation entity has only an obstructed view of the third ground transportation. The second ground transportation entity includes pedestrians or other vulnerable road users. The safety message information sent by the processor includes a basic safety message. The safety message information sent by the processor includes a virtual basic safety message. The safety message information sent by the processor includes a personal safety message. The safety message information sent by the processor includes a virtual personal safety message. The safety message information sent by the processor includes a virtual basic safety message sent on behalf of the third ground transportation entity. The third ground transportation entity includes an unconnected ground transportation entity. The safety message information sent by the processor includes a virtual personal safety message sent on behalf of the third ground transportation entity. The equipment has (d) a receiver for information wirelessly sent from a source external to the first ground transportation entity. The requested device includes the first ground transportation entity. The safety message information includes a virtual intersection collision avoidance message (VICA). The safety message information includes an intersection collision avoidance message (ICA). The safety message information includes a virtual combined safety message (VCSM). The safety message information includes a combined safety message (CSM).

[0009] In general, in one aspect, equipment for onboard use on a first ground transportation entity has (a) a receiver for first position correction information sent from a source external to the first ground transportation entity, (b) a receiver for information representing a parameter of a position or motion of the first ground transportation entity, (c) a processor, and (d) a memory storing instructions executable by the processor to generate updated position correction information based on the first position correction information and the information representing the motion parameter, and to send a position correction message to another ground transportation entity based on the updated position correction information.

[0010] An implementation may include one of the following features or a combination of two or more of the following features. The position correction information sent by a source external to the first ground transportation entity includes a position correction message. The position correction information sent from a source external to the first ground transportation entity includes a Maritime Radio Technical Commission (RTCM) correction message. The position correction information includes a GNSS position correction. The position or motion parameter includes a current position of the first ground transportation entity. The source external to the first ground transportation entity includes an RSE or an external service configured to transmit RTCM correction messages over the Internet. The instructions are executable by a processor to confirm a confidence level in the updated position correction information.

[0011] In general, in one aspect, information about an environment of a first ground transportation entity that has been generated by a sensor installed on the first ground transportation entity is received. Safety message information is generated based on the information generated by the sensor and sent to a second ground transportation entity.

[0012] Implementations may include one of the following features or a combination of two or more of the following features. Generate a prediction for generating safety message information. The prediction is generated by a prediction model. The prediction model is configured to predict a dangerous situation involving a first ground transportation entity, a second ground transportation entity, or another ground transportation entity. The dangerous situation involves the second ground transportation entity crossing a road lane. The second ground transportation entity includes a vehicle, and the dangerous situation includes the vehicle skidding on the lane. The second ground transportation entity includes a pedestrian or other vulnerable road user crossing the road. The vulnerable road user is crossing the road at an intersection. The vulnerable road user is crossing the road at a location other than an intersection. The dangerous situation includes a collision between a third ground transportation entity and the second ground transportation entity. The first ground transportation entity includes a vehicle, and the second ground transportation entity includes a pedestrian or other vulnerable road user. The third ground transportation entity follows the first ground transportation entity, and the third ground transportation entity's view from the first ground transportation entity is obstructed. The third ground transportation entity is located in a lane adjacent to the lane in which the first ground transportation entity is traveling. Determine motion parameters of the third ground transportation entity. The second ground transportation entity only has an obstructed view of the third ground transportation entity. The second ground transportation entity includes a pedestrian or other vulnerable road user. The safety message information includes a basic safety message. The safety message information includes a virtual basic safety message. The safety message information includes a personal safety message. The safety message information includes a virtual personal safety message. The safety message information includes a virtual basic safety message sent on behalf of a third ground transportation entity. The safety message information sent by the processor includes a virtual personal safety message sent on behalf of the third ground transportation. The third ground transportation entity includes an unconnected ground transportation entity.

[0013] Information wirelessly transmitted from a source external to the first ground transportation entity is received.

[0014] The safety message information includes a virtual intersection collision avoidance message (VICA). The safety message information includes a virtual intersection collision avoidance message (ICA). The safety message information includes a virtual combined safety message (VCSM). The safety message information includes a combined safety message (CSM).

[0015] In general, in one aspect, first position correction information that has been sent from a source external to a first ground transportation entity is received. Information representing a motion parameter of the first ground transportation entity is received. Updated position correction information is generated based on the first position correction information and the information representing the motion parameter. A position correction message is sent, and a position correction message is sent to another ground transportation entity based on the updated position correction information.

[0016] Implementations may include one of the following features or a combination of two or more of the following features. The position correction information sent by a source external to the first ground transportation entity includes a position correction message. The position correction information sent from a source external to the first ground transportation entity includes a Radio Technical Commission for Maritime (RTCM) correction message. The position correction information includes a GNSS position correction. The motion parameters include a current position of the first ground transportation entity. The source external to the first ground transportation entity includes an RSE or an external service configured to transmit the position correction message over the Internet. A confidence level in the updated position correction information is confirmed.

[0017] These and other aspects, features, and implementations may be represented as methods, apparatus, systems, components, program products, methods of doing business, means or steps for performing the functions, and in other ways.

[0018] These and other aspects, features and implementations will be apparent from the following description including the claims.

[0019] describe

[0020] Figure 1 , 2 , 3 and 15 are block diagrams.

[0021] Figure 4 , 5 , 8 to 11, 13, 14 and 17 to 23 are schematic diagrams of the road network viewed from above.

[0022] Figure 6 and Figure 7 is an annotated perspective view of the intersection.

[0023] Fig.12 and 16 is a schematic side and perspective view of the road network.

[0024] With advances in sensor technology and computers, it has become feasible to predict hazardous situations (and provide early warning) while conducting ground transportation and in this way prevent collisions and near misses for ground transportation entities (ie, collision avoidance).

[0025] We use the term “ground transportation” broadly to include, for example, any means or medium of moving from one place to another that requires contact with land or water on the Earth’s surface, such as walking or running (or engaging in other pedestrian activities), nonmotorized vehicles, motorized vehicles (motorized, semi-motorized, and non-motorized), and rail vehicles.

[0026] We use the term "surface transportation entity" (or sometimes simply "entity") broadly to include, for example, a person or discrete motorized or non-motorized vehicle participating in a surface transportation mode, such as a pedestrian, bicyclist, boat, car, truck, tram, streetcar, or train, etc. We sometimes use the term "vehicle" or "road user" as a shorthand reference to a surface transportation entity.

[0027] We use the term "hazardous situation" broadly to include, for example, any event, incident, sequence, situation, or other circumstance that could result in imminent property damage or personal injury or death and that could be reduced or avoided. We sometimes use the term "hazard" interchangeably with "hazardous situation." We sometimes use the words "violation" or "breach" for an entity's actions that have, could, or will result in a hazardous situation.

[0028] In some implementations of the techniques we discuss herein, the ground transportation network is being used by a mix of ground transportation entities that do not have or are not using transportation connectivity and ground transportation entities that have and are using transportation connectivity.

[0029] We use the term "connectivity" broadly to include, for example, any ability of a ground transportation entity to (a) be aware of and act upon its surroundings, other ground transportation entities in its vicinity, and traffic conditions relevant thereto, (b) broadcast or otherwise transmit data about its status, or (c) both (a) and (b). The transmitted data may include its location, direction of travel, speed, or the internal states of its components as they relate to traffic conditions. In some cases, the ground transportation entity's awareness is based on wirelessly received data about other ground transportation entities or traffic conditions relevant to the operation of the ground transportation entity. The received data may come from other ground transportation entities or infrastructure equipment, or both. Typically, connectivity involves sending or receiving data in real time or substantially real time or in a timely manner to enable one or more ground transportation entities to take action based on the data in a traffic situation.

[0030] We use the term "traffic condition" broadly to include any situation in which two or more ground transportation entities are operating in proximity to each other and in which the operation or status of each entity may affect or be related to the operation or status of the others.

[0031] We sometimes refer to ground transportation entities that do not have or are not using connectivity or aspects of connectivity as “non-connected ground transportation entities” or simply “non-connected entities.” We sometimes refer to ground transportation entities that have and are using connectivity or aspects of connectivity as “connected ground transportation entities” or simply “connected entities.”

[0032] We sometimes use the term "cooperating entity" to refer to ground transportation entities that broadcast data to their surroundings, including, for example, location, direction of travel, speed, or the status of onboard safety systems (such as brakes, lights, and wipers).

[0033] We sometimes use the term "non-cooperative entity" to refer to a ground transportation entity that does not broadcast one or more types of data to its surroundings (e.g., its location, speed, direction of travel, or status).

[0034] We sometimes use the term "nearby area" of a ground transportation entity broadly to include, for example, the area within which broadcasts from that entity can be received by other ground transportation entities or infrastructure equipment. In some cases, the nearby area will vary with the location of the entity and the number and nature of obstacles surrounding the entity. An entity traveling on an open road in the desert will have a very wide nearby area because there are no obstacles to prevent the broadcast signal from the entity from reaching great distances. In contrast, the nearby area of ​​an urban canyon will be reduced due to the buildings surrounding the entity. In addition, there may be sources of electromagnetic noise that degrade the quality of the broadcast signal and therefore reduce the reception distance (nearby area).

[0035] like Fig.14 As shown, the vicinity of entity 7001 traveling along road 7005 can be represented by concentric circles, with the outermost circle 7002 representing the outermost range of the vicinity. Any other entity within circle 7002 is in the vicinity of entity 7001. Any other entity outside circle 7002 is outside the vicinity of entity 7001 and cannot receive the broadcast of entity 7001. Entity 7001 is invisible to all entities and infrastructure equipment outside its vicinity.

[0036] Typically, cooperating entities continuously broadcast their status data. Connected entities in the vicinity of the broadcasting entity are able to receive these broadcasts and can process and act on the received data. For example, if a vulnerable road user has a wearable device that can receive broadcasts from an entity, such as an oncoming truck, the wearable device can process the received data and let the vulnerable user know when it is safe to cross the road. This action occurs regardless of the location of the cooperating entity or the vulnerable user relative to the "smart" intersection as long as the user's device can receive the broadcast, i.e., is in the vicinity of the cooperating entity.

[0037] We use the term "vulnerable road user" or "weak road user" broadly to include, for example, any road user or other characteristic of the road network who does not use a motor vehicle. A vulnerable road user is generally not immune from injury or death or property damage in a collision with a motor vehicle. In some examples, a vulnerable road user may be someone who is walking, running, cycling, or engaging in any type of activity that puts them at risk of direct physical contact with a vehicle or other ground transportation entity in the event of a collision.

[0038] In certain embodiments, the collision avoidance techniques and systems described in this document (which we sometimes simply refer to as the "system") use sensors mounted on infrastructure fixtures to monitor, track, detect, and predict the movements (e.g., speed, direction of travel, and location), behavior (e.g., high speed), and intentions (e.g., about to violate a no-stop sign) of ground transportation entities and their drivers and operators. The information provided by the sensors (the "sensor data") enables the system to predict hazardous situations and provide early warnings to entities to increase the chances of collision avoidance.

[0039] We use the term "collision avoidance" broadly to include, for example, any situation in which a collision or near miss that could result from a hazardous situation between two or more ground transportation entities or between a ground transportation entity and another object in the environment is prevented or the chance of such a collision is reduced.

[0040] We use the term "warning" broadly to include, for example, any notification, alarm, instruction, command, broadcast, transmission, or other information sent or received that identifies, suggests, or in any way indicates a hazardous situation and is useful in avoiding a collision.

[0041] Road intersections are prime locations where dangerous situations can occur. The techniques we describe here enable infrastructure devices at intersections, including sensors, computing hardware, and smart devices, to enable simultaneous monitoring, detection, and prediction of dangerous situations. Data from these sensors is normalized into a single reference frame and then processed. Artificial intelligence models of traffic flow along different roads leading to the intersection are built. For example, these models help entities that are more likely to violate traffic regulations. Models are built to detect dangerous situations before actual violations occur and can therefore be considered predictions. Based on the prediction of dangerous situations, alerts are sent from infrastructure devices at the intersection to all connected entities in the vicinity of the intersection. Each entity that receives an alert processes the data in the alert and performs alert filtering. Alert filtering is the process of discarding or ignoring alerts that are not beneficial to the entity. If the alert is deemed beneficial (i.e., not ignored due to filtering), such as an alert of an impending collision, the entity either automatically reacts to the alert (e.g., by applying the brakes) or provides a notification to the driver or both.

[0042] The system can be used for, but is not limited to, roads, waterways, and railroads. We sometimes refer to these and other similar transportation scenarios as "surface transportation networks."

[0043] Although we often discuss this system in the context of intersections, it can be applied to other situations as well.

[0044] We use the term "intersection" broadly to include, for example, any real-world arrangement of roads, railroads, bodies of water, or other travel paths where two or more ground transportation entities traveling along paths of a ground transportation network may occupy the same position at certain times and locations, thereby creating a collision.

[0045] Ground transportation entities using the ground transportation network move at various speeds and may arrive at a given intersection at different speeds and times of day. If the speed and distance of an entity from an intersection are known, then dividing the distance by the speed (both expressed in the same unit system) will give the time to arrive at the intersection. However, the estimated time to arrive at an intersection is constantly changing because speeds vary due to traffic conditions, route speed limits, traffic signals, and other factors. This dynamic change in the estimated time of arrival makes it impossible to predict the actual arrival time with 100% confidence.

[0046] To account for factors that affect the motion of an entity, a number of relationships need to be applied between the velocity of the entity and various influencing factors. The absolute value of the motion state of an entity can be observed from sensors that track the entity either from the entity or from an external location. The data captured by these sensors can be used to model the motion patterns, behaviors, and intentions of the entity. Machine learning can be used to generate complex models from large amounts of data. Patterns that cannot be directly modeled using entity kinematics can be captured using machine learning. A trained model can predict whether an entity is going to move or stop at a specific point by using entity tracking data from sensors that track the entity.

[0047] In other words, in addition to detecting information about ground transportation entities directly from sensor data, the system uses artificial intelligence and machine learning to process large amounts of sensor data to learn the movement patterns, behaviors, and intentions of ground transportation entities (e.g., at intersections of the ground transportation network, on roads to such intersections, and at crosswalks of the ground transportation network). Based on the direct use of current sensor data and the results of applying artificial intelligence and machine learning to current sensor data, the system generates warnings such as dangerous situation alerts to help avoid collisions. For warnings in the form of instructions or commands, the commands or instructions can be directed to specific motorized or human-driven entities to directly control the vehicle. For example, an instruction or command can slow down or stop an entity driven by a malicious person who has been determined to run a red light in an attempt to harm others.

[0048] The system can be customized to make predictions for that specific intersection and send alerts to entities within the vicinity of the device broadcasting the alert. To do this, the system uses sensors to acquire data about dangerous entities and passes the current readings of the sensors to a trained model. The output of the model can then predict dangerous situations and broadcast corresponding alerts. The alerts received by connected entities in the vicinity contain information about the dangerous entities, so the receiving entity can analyze the information to assess the threat posed by the dangerous entity to it. If a threat exists, the receiving entity can take action on its own (e.g., slow down) or notify the driver of the receiving entity using a human-machine interface based on visual, audio, tactile, or any type of sensory stimulation. The maneuverable entity can take action on its own to avoid the dangerous situation.

[0049] The alert may also be sent directly via a cellular or other network to a mobile phone or other device that is equipped to receive the alert and is owned by the pedestrian. The system identifies potentially dangerous entities at the intersection and broadcasts (or directly sends) an alert to the pedestrian's personal device that has a communication unit. For example, the alert may prevent the pedestrian from entering a crosswalk, thereby avoiding a potential accident.

[0050] The system can also track pedestrians and broadcast information related to their status (position, speed, and other parameters) to other entities so that other entities can take action to avoid dangerous situations.

[0051] like Figure 1 As shown, the system includes at least the following components:

[0052] 1. Roadside Equipment (RSE) 10 that includes or uses sensors 12 to monitor, track, detect, and predict the movement (e.g., speed, direction of travel, and location), behavior (e.g., high speed), and intentions (e.g., will violate a no-stop sign) of ground transportation entities 14. The RSE also includes or may use a data processing unit 11 and data storage 18. Ground transportation entities exhibit a wide range of behaviors, which depend on the infrastructure of the ground transportation network and the state of the entity itself, the state of the driver, and the state of other ground transportation entities. To capture the behavior of the entity, the RSE collects information from sensors, other RSEs, OBEs, OPEs, local or central servers, and other data processing units. The RSE also saves the data it receives and may also save processed data at some or all steps in the pipeline.

[0053] RSE can save data on local storage devices or remote storage. The collected data is processed in real time using predefined logic or logic based on dynamically collected data, which means RSE can automatically update its own logic. Data can be processed on a single processing unit or a cluster of processing units to get results faster. Data can be processed on local or remote processing units or clusters of local or remote processing units. RSE can use simple logic or complex models trained on the collected data. The model can be trained locally or remotely.

[0054] RSE can pre-process the data before filtering outliers using a trained model. Outliers may appear due to noise, reflections, or some other artifacts in the sensor. The resulting outliers may cause false positives, thus affecting the performance of the entire RSE. The filtering method can be based on data collected by the RSE, OBE, OPE, or online resources. RSE can interface with other controllers (such as traffic light controllers at intersections or other locations) to extract information for use in the data processing pipeline.

[0055] The RSE also includes or can utilize communication equipment 20 to communicate with other RSEs, OBEs, OPEs, local or central servers, and other data processing units by wired or wireless means. The RSE can communicate with other equipment using any available standard. The RSE can use a wired or wireless Internet connection to download and upload data to other equipment, use a cellular network to send and receive messages from other cellular devices, and use dedicated radio equipment to communicate with infrastructure equipment and other RSEs at intersections or other locations.

[0056] The RSE may be installed at different types of intersections. For example, at a signalized intersection (e.g., an intersection where traffic is controlled by lights), the RSE 10 is installed near a traffic light controller 26, either in the same housing or in an adjacent housing. Data (e.g., traffic light phase and timing) flows between the traffic light controller and the RSE 28. At an unsignalized intersection, the RSE 10 is typically positioned to make it easy to connect it to a sensor 12 that monitors other features of the road or ground transportation network in the vicinity of the intersection. The proximity of the RSE to the intersection helps maintain a low latency system, which is critical to providing maximum time for receiving ground units to respond to alerts.

[0057] 2. An onboard equipment (OBE) 36 installed on or carried by or in the ground transportation entity 14 comprises sensors 38 which determine the position and kinematics (motion data) of the entity in addition to safety-related data about the entity. The OBE also comprises a data processing unit 40, data storage 42 and communication equipment 44 which can communicate wirelessly with other OBEs, OPEs, RSEs and possibly servers and computing units.

[0058] 3. On personal equipment (OPE) 46, which may be, but is not limited to, a mobile phone, a wearable device, or any other device capable of being worn, held, attached to, or otherwise interacting with a person or animal. If desired, the OPE may include or be coupled to a data processing unit 48, data storage 50, and communication equipment 52. In some implementations, the OPE is used as a dedicated communication unit for non-vehicle vulnerable road users. In some cases, the OPE may also be used for other purposes. The OPE may have components that provide visual, audio, or tactile alerts to vulnerable road users.

[0059] Vulnerable road users can include pedestrians, cyclists, road workers, people in wheelchairs, scooters, self-balancing devices, battery-powered personal transport, animal-powered carriages, guide or police animals, farm animals, herd animals and pets.

[0060] Typically, an OPE is owned by a vulnerable road user and is capable of sending and receiving messages. The OPE can be connected or integrated into a mobile phone, tablet, personal transportation, bicycle, wearable device (e.g., watch, bracelet, anklet), or attached to a pet collar.

[0061] The message sent by the OPE may include kinematic information related to the vulnerable road user, including but not limited to the time of day, 3D position, direction of travel, speed, and acceleration. The message sent may also carry data representing the alert level, current behavior, and future intentions of the vulnerable road user, such as whether the vulnerable road user is currently crossing a crosswalk, listening to music, or about to cross a crosswalk. Among other things, the message may convey the blob size or data size of the vulnerable road user, whether the vulnerable road user has an external device (e.g., a stroller, cart, or other device), whether the vulnerable road user has a disability or is using any personal assistance. If the vulnerable road user is a worker, the message may convey the class of the worker and may also describe the type of activity the worker is performing. When a cluster of similar vulnerable road users (e.g., a group of pedestrians) has similar characteristics, a single message may be sent to avoid multiple message broadcasts.

[0062] Typically, the messages received by the OPE are alert messages from roadside equipment or entities. The OPE may act on the received messages by alerting vulnerable road users. The alert messages will carry data that helps to customize the alerts for vulnerable road users. For example, an alert to a vulnerable road user may present a type of hazardous situation and suggest possible actions. The OPE may apply alert filtering to all received messages and present only relevant messages to vulnerable road users.

[0063] Alert filtering is based on the results of applying a learning algorithm to historical data related to OPE, allowing alert filtering to be customized for each vulnerable road user. The OPE learning algorithm tracks the vulnerable road user's response to received alerts and customizes future alerts to obtain the best response time and the best attention from vulnerable road users. The learning algorithm can also be applied to the data carried in the messages sent.

[0064] 4. Data storage server 54, which may be, but is not limited to, cloud storage, local storage or any other storage facility that allows for storage and retrieval of data. The data storage server may be accessed by the RSE, computing unit and possibly by the OBE, OPE and data server for the purpose of storing data related to warning and collision avoidance. The data storage server may be accessed from the RSE and possibly from the OBE, OPE and data server for the purpose of retrieving the stored data. The data may be raw sensor data, data processed by the processing unit or any other information generated by the RSE, OBE and OPE.

[0065] Sensors at intersections that continuously monitor ground transportation entities can generate a large amount of data every day. The amount of data depends on the number and type of sensors. The data can be processed in real time or saved for future analysis, which requires local (e.g., at the intersection) data storage units (e.g., hard drives, solid-state drives, and other mass storage devices). Local storage devices will be filled up over a period of time, depending on their storage capacity, the amount of data generated, and the rate at which data is generated. In order to retain the data for future use, the data is uploaded to a remote server with more capacity. The remote server can upgrade the storage capacity on demand as needed. The remote server can use a data storage device similar to a local storage device (e.g., a hard drive, a solid-state drive, or other mass storage device) that can be accessed via a network connection.

[0066] Data stored locally and on a server for future analysis may include data broadcast by ground transportation entities and received by the RSE, which is saved for future analysis. The stored data may be downloaded from a server or other remote source for processing at the RSE. For example, a machine learning model for an intersection at which the RSE is located may be stored on a server or other remote storage and downloaded by the RSE for use in analyzing current data received from a local source at the RSE.

[0067] 5. Computing unit 56, which is a powerful computing machine located in the cloud or locally (e.g., as part of an RSE), or a combination of these. Among other functions, the computing unit processes available data to generate predictions, machine learning-based models of movement, behavior, and intentions of vehicles, pedestrians, or other ground transportation entities using the transportation network. Each computing unit may have dedicated hardware to process the corresponding type of data (e.g., a graphics processing unit for processing images). In the case of heavy processing load, the computing unit in the RSE may be overloaded. For example, this may occur when additional data generating units (e.g., sensors) are added to the system, resulting in computational overload. Overload may also occur if the logic running in the computing unit is replaced by more computationally intensive logic. Overload may be caused by an increase in the number of ground transportation entities being tracked. When local computational overload occurs, the RSE can offload some tasks to another computing unit. The other computing unit can be near the RSE or remote, such as a server. Computing tasks can be prioritized, and tasks that are not time-critical can be completed in another computing unit, and the results retrieved by the local computing unit. For example, a computing unit in RSE can request another computing unit to run a job for analyzing saved data and training a model using that data. The trained model will then be downloaded by the computing unit of RSE and stored and used there.

[0068] Computing units at the RSE can use other small computing units to perform computationally intensive jobs more efficiently and save time. Available computing units are used judiciously to perform the most tasks in the shortest time, for example, by dividing tasks between RSE computing units and other available computing units. Computing units can also be connected to the RSE as external devices to add more computing power to computing units in the RSE. Externally attached computing units can have the same or different architecture compared to computing units in the RSE. Externally attached computing units can communicate with existing computing units using any available communication ports. RSE computing units can request more computing power from external computing units as needed.

[0069] The rest of this document will explain in detail the role and function of the above components in the system, etc.

[0070] Roadside Equipment (RSE)

[0071] like Figure 2 As shown, RSE may include, but is not limited to, the following components:

[0072] 1. One or more communication units 103, 104, which are capable of receiving motion data and other data related to ground transportation entities and traffic safety data from nearby vehicles or other ground transportation entities, infrastructure and remote servers and data storage systems 130, or sending motion data and other data related to ground transportation entities and traffic safety data to nearby vehicles or other ground transportation entities, infrastructure and remote servers and data storage systems 130, or both. In some cases, this type of communication is referred to as infrastructure to every (I2X), including but not limited to infrastructure to vehicle (I2V), infrastructure to pedestrian (I2P), infrastructure to infrastructure (I2I) and infrastructure to device (I2D) and combinations thereof. The communication can be wireless or wired and conform to a variety of communication protocols.

[0073] 2. Communication unit 103 is used to communicate with ground transportation entities, and unit 104 is used to communicate with remote servers and data storage systems 130 via the Internet.

[0074] 3. Local memory 106, used to store programs, intersection models, and behavior and traffic models. It can also be used to temporarily store data collected from sensors 101.

[0075] 4. Sensors 101 and sensor controllers 107 that allow monitoring (e.g., generating data about) mobile objects, such as ground transportation entities that are typically in the vicinity of the RSE. Sensors may include, but are not limited to, cameras, radars, lidars, ultrasonic detectors, or any other hardware that can sense or infer from sensed data the distance to a location, speed, direction of travel, location, or a combination thereof, etc. of ground transportation entities. Sensor fusion is performed using aggregation or combination of data from two or more sensors 101.

[0076] 5. A position receiver (102) (eg, a GPS receiver) that provides positioning data (eg, position coordinates of the RSE) and helps correct positioning errors in the positioning of ground transportation entities.

[0077] 6. Processing unit 105, which will acquire and use data generated from sensors and incoming data from communication units 103, 104. The processing unit will process and store data locally, and in some implementations, transmit data for remote storage and further processing. The processing unit will also generate messages and alerts that are broadcast or otherwise sent via wireless communication facilities to nearby pedestrians, motor vehicles or other ground transportation entities, and in some cases to signs or other infrastructure presentation devices. The processing unit will also periodically report the operating conditions and status of all RSE systems to a remote server for monitoring.

[0078] 7. Expansion connector 108, allowing control and communication between the RSE and other hardware or other components (such as temperature and humidity sensors, traffic light controllers, other computing units as described above, and other electronic devices that may become available in the future).

[0079] On-Board Equipment (OBE)

[0080] Vehicle equipment can typically be original to the ground transportation entity or added to the entity by a third-party supplier. Figure 3 As shown, an OBE may include, but is not limited to, the following components:

[0081] 1. A communication unit 203 that is capable of sending and receiving data, or both, to nearby vehicles, pedestrians, cyclists, or other ground transportation entities and infrastructure, and combinations thereof. The communication unit also allows for the transmission or reception (or both) of data between the vehicle or other ground transportation entity and a local or remote server 212 for machine learning purposes and for remote monitoring of ground transportation entities by the server. In some cases, this type of communication is referred to as vehicle-to-everything (V2X), including but not limited to vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), vehicle-to-infrastructure (V2I), vehicle-to-device (V2D), and combinations thereof. Communications may be wireless or wired and conform to a variety of communication protocols.

[0082] The communication unit 204 will allow the OBE to communicate with a remote server via the Internet for program updates, data storage, and data processing.

[0083] 2. Local memory 206, used to store programs, intersection models and traffic models. It can also be used to temporarily store data collected from sensors 201.

[0084] 3. Sensors 201 and sensor controller 207, which may include but are not limited to external cameras, lidar, radar, ultrasonic sensors, or any device that can be used to detect nearby objects or people or other ground traffic entities. Sensors 201 may also include additional motion sensors, global positioning receivers, and internal and local microphones and cameras.

[0085] 4. A location receiver 202 (eg, a GPS receiver) that provides positioning data (eg, coordinates of the location of a ground transportation entity).

[0086] 5. A processing unit 205 that acquires, uses, generates and transmits data, including consuming data from and sending data to a communication unit and consuming data from sensors in or on a ground transportation entity.

[0087] 6. Expansion connector 208, allowing control and communication between the OBE and other hardware.

[0088] 7. An interface unit that can be retrofitted or integrated into a head-mounted device, steering wheel or driver mobile device in one or more ways (e.g. using visual, auditory or tactile feedback).

[0089] Smart OBE(SOBE)

[0090] In a world where all vehicles and other ground transportation entities are connected entities, each vehicle or other ground transportation entity can be an entity that cooperates with other entities and can report its current location, safety status, intentions, and other information to other entities. Currently, almost all vehicles are not connected entities, cannot report such information to other ground transportation entities, and are operated by people with different skills, happiness, stress, and behavior levels. Without this connectivity and communication, it becomes difficult to predict the next move of a vehicle or ground transportation entity, and this leads to a decrease in the ability to implement collision avoidance and provide early warning.

[0091] The smart OBE monitors the environment and the users or passengers of the ground transportation entity. It also keeps an eye on the health and status of the entity's different systems and subsystems. The SOBE monitors the outside world by listening to radio transmissions from, for example, emergency broadcasts, traffic and safety messages from nearby RSEs, and messages from other connected vehicles or other ground transportation entities regarding safety, location, and other motion information. The SOBE also interfaces with onboard sensors that can monitor road and driving conditions, such as cameras, distance sensors, vibration sensors, microphones, or any other sensors that allow such monitoring. The SOBE will also monitor the surrounding environment and create a map of all static and moving objects.

[0092] SOBE can also monitor the behavior of users or passengers of vehicles or other ground transportation entities. SOBE uses microphones to monitor the quality of conversations. It can also use other sensors such as seat sensors, cameras, hydrocarbon sensors, and sensors for volatile organic compounds and other toxic substances. It can also use motion sensors to measure the driver's reactions and behavior and infer the driving quality based on this.

[0093] SOBE also receives vehicle-to-vehicle messages (e.g., Basic Safety Messages (BSMs)) from other ground transportation entities and vehicle-to-pedestrian messages (e.g., Personal Safety Messages (PSMs)) from vulnerable road users.

[0094] The SOBE will then fuse the data from this array of sensors, sources, and messages. It will then apply the fused data to an artificial intelligence model that is not only able to predict the next action or reaction of the driver or user of a vehicle or other ground transportation entity or vulnerable road user, but also the intention and future trajectory and the associated near miss or collision risk due to other vehicles, ground transportation entities, and vulnerable road users in the vicinity. For example, the SOBE can use the BSM received from nearby vehicles to predict that nearby vehicles are about to enter a lane change maneuver that poses a risk to their own vehicle, and can warn the driver of the upcoming risk. For each of these trajectories, the risk is calculated by the SOBE based on the probability of various future predicted trajectories of nearby vehicles (e.g., going straight, changing lanes to the right, changing lanes to the left) and the associated risk of collision with the host vehicle. If the risk of collision is above a certain threshold, a warning is displayed to the driver of the host vehicle.

[0095] Due to the complexity of modeling human driver behavior, machine learning is often required to predict intentions and future trajectories, which are further affected by external factors (e.g., changing environmental and weather conditions).

[0096] SOBE features a large computing capacity capable of processing large data feeds, some of which provide multiple terabytes of data per second. The amount of data available is also proportional to the level of detail required for each sensor.

[0097] SOBE will also have a powerful signal processing setup to be able to extract useful information from environments known to have high (signal) noise levels and low signal-to-noise ratios. SOBE will also protect the driver from the large number of alerts received by the vehicle by providing intelligent alert filtering. Alert filtering is the result of a machine learning model that will be able to determine which alert is important given the current location, environmental conditions, driver behavior, vehicle operation and status, and kinematics.

[0098] Smart OBEs are important for collision avoidance and warning, and are also very important for providing a safer transportation network for all users, not just the occupants or users of vehicles that include SOBEs. SOBEs can detect and predict the movement of different entities on the road, thus helping in collision avoidance.

[0099] Personal Equipment (OPE)

[0100] As previously mentioned, personal devices (OPEs) include any device that may be held, attached to, or otherwise directly interfaced with a pedestrian, jogger, or ground transportation entity or other person who otherwise appears on or uses a ground transportation network. For example, such a person may be a vulnerable road user who is vulnerable to vehicle impact. OPEs may include, but are not limited to, mobile devices (e.g., smartphones, tablets, digital assistants), wearable devices (e.g., glasses, watches, bracelets, anklets), and implants. Existing components and features of OPEs can be used to track and report location, speed, and direction of travel. OPEs can also be used to receive and process data and display alerts to users in various modes (e.g., visual, sound, tactile).

[0101] Honda has developed a communication system and method for V2P applications that focuses on direct communication between vehicles and pedestrians using OPE. In one scenario, a vehicle is equipped with an OBE that broadcasts a message to the OPEs of surrounding pedestrians. The message carries the current state of the vehicle, including, for example, vehicle parameters, speed, and direction of travel. For example, the message can be a basic safety message (BSM). If necessary, the OPE will alert the pedestrian about a predicted dangerous situation based on the pedestrian's level of distraction to avoid a collision. In another scenario, the pedestrian's OPE broadcasts a message (such as a personal safety message (PSM)) to the OBEs of surrounding vehicles that the pedestrian may cross the vehicle's predetermined path. If necessary, the vehicle's OBE will display an alert to the vehicle user about the predicted danger to avoid a collision. See Strickland, Richard Dean, et al., "Vehicle-to-Pedestrian Communication System and Method," U.S. Patent 9,421,909. The system we describe here uses an I2P or I2V approach to track and collect data on pedestrians and other vulnerable road users using sensors external to vehicles and pedestrians (primarily infrastructure). For example, sensors can track pedestrians crossing the street and vehicles traveling at or near an intersection. The collected data will then be used to build predictive models of the intentions and behaviors of pedestrians and vehicle drivers on the road using rule-based and machine learning methods. These models will help analyze the collected data and predict the paths and intentions of pedestrians and vehicles. If a hazard is predicted, a message will be broadcast from the RSE to the OBE or OPE or both, alerting each entity to the other's expected path and allowing each of them enough time to take preemptive action to avoid a collision.

[0102] Remote computing (cloud computing and storage)

[0103] Data collected from sensors connected to or contained in RSEs, OBEs, and OPEs needs to be processed in order to generate effective mathematical machine learning models. This processing requires a large amount of data processing power to reduce the time required to generate each model. The required processing power far exceeds the capacity typically available locally at the RSE. To address this problem, the data can be transferred to a remote computing facility that provides the required power and can be scaled as needed. We refer to the remote computing facility as a "remote server," which is consistent with the terminology used in the computing literature. In some cases, it is possible to perform some or all of the processing at the RCE by equipping it with high-performance computing power.

[0104] Rule-based processing

[0105] Unlike AI and machine learning techniques, rule-based processing can be applied at any time, without the need for data collection, training, and model building. Rule-based processing can be deployed from the beginning of a system’s operation, and is typically done so until enough training data has been acquired to create a machine learning model. After a new installation, rules are set up to process incoming sensor data. Not only does this help improve road safety, but it also serves as a good test case to ensure that all components of the system are working as expected. Rule-based processing can also be added and used later as an additional layer to capture rare situations where machine learning may not be able to make an accurate prediction. Rule-based approaches are based on simple relationships between parameters of the collected data (e.g., speed, range, etc.). Rule-based approaches can also provide a benchmark for evaluating the performance of machine learning algorithms.

[0106] In rule-based processing, a vehicle or other ground transportation entity passing through a portion of a ground transportation network is monitored by sensors. If, for example, its current speed and acceleration exceed a threshold value that would prevent it from stopping before a stop strip (line) on the road, an alert is generated. A variable area is assigned to each vehicle or other ground transportation entity. This area is marked as a dilemma area, where the vehicle has not yet been marked as an offending vehicle. If the vehicle crosses the dilemma area into the danger zone because its speed or acceleration or both exceed predefined thresholds, the vehicle is marked as an offending entity and an alert is generated. For example, the thresholds for speed and acceleration are based on physics and kinematics and vary with each ground transportation entity approaching an intersection.

[0107] Two traditional rule-based methods are 1) static TTI (Time-To-Intersection), and 2) static RDP (Required Deceleration Parameter). See Strickland, Richard Dean, et al., "Driver Behavior Classification at Intersections and Validation on a Large Naturalistic Dataset," IEEE Transactions on Intelligent Transportation Systems 13.2 (2012): 724-736.

[0108] Static TTI (Time-To-Intersection) uses the estimated time to reach the intersection as the classification criterion. In its simplest form, TTI is calculated as Where r is the distance to the intersection line at the intersection, and v is the current speed of the vehicle or other ground traffic entity. <TTI req , then the vehicle is classified as dangerous, where TTI req It is the time required for the vehicle to stop safely after braking. req The parameter reflects how conservative the rule-based algorithm is. TTI is calculated at the start of braking and is identified when the vehicle decelerates beyond a deceleration threshold (e.g., -0.075g). If the vehicle never exceeds this threshold, classification is performed at a specified resort time, which is typically between 1 and 2 seconds of the estimated remaining time to reach the intersection.

[0109] The static RDP (Required Deceleration Parameter) calculates the deceleration required to stop the vehicle safely at its current speed and position on the road. The RDP is calculated as Where r is the distance to the intersection line, v is the current speed of the vehicle or other ground traffic entity, and g is the gravity acceleration constant. If the required deceleration of the vehicle is greater than the selected RDP threshold RDP warn , then the vehicle is classified as dangerous (i.e. the vehicle has or will create a dangerous situation). In practice, if at any time r <r alert ,in, The vehicle is classified as dangerous.

[0110] Similar to the static TTI algorithm, RDP alert The parameter reflects how conservative the rule-based algorithm is.

[0111] We use rule-based methods as a baseline for evaluating the performance of our machine learning algorithms, and in some cases we run them in parallel with machine learning algorithms to capture rare cases that machine learning may not be able to predict.

[0112] Machine Learning

[0113] Modeling driver behavior has proven to be a complex task given the complexity of human behavior. See H. M. Mandalia and D. D. Dalvucci, \Using Support Vector Machines for Lane Change Detection, Proceedings of the Annual Meeting of the Human Factors and Ergonomics Society, Vol. 49, pp. 1965{1969, 2005. Machine learning techniques are well suited to modeling human behavior but need to be "learned" using training data to work properly. To provide superior detection and prediction results, we use machine learning to model traffic detected at intersections or other features of the ground transportation network during training before applying the alert process to current traffic during the deployment phase. Machine learning can also be used to model driver responses using onboard data from onboard equipment (OBEs), and can also be based on onboard sensors and a history of driving records and preferences. We also use machine learning models to detect and predict the trajectory, behavior, and intentions of vulnerable road users, such as pedestrians. Machine learning can also be used to model the responses of vulnerable road users to on-site equipment (OPEs). These models may include interactions between entities, vulnerable road users, and between one or more entities and one or more vulnerable road users.

[0114] Machine learning techniques can also be used to model the behavior of non-motorized ground transportation entities. Machine learning can be used to predict the intentions of non-motorized ground transportation entities by observing or communicating with them, or both, and to predict their intentions and communicate with them and other relevant entities when a near miss or accident or other hazardous situation is predicted.

[0115] Machine learning mechanisms work in two phases: 1) training and 2) deployment.

[0116] Training phase

[0117] Once installed, the RSE begins collecting data from the sensors it has access to. Since AI model training requires intensive computing power, it is usually performed on powerful servers with multiple parallel processing modules to speed up the training phase. For this reason, data acquired at the RSE location on the ground transportation network can be packaged and sent to a remote powerful server soon after acquisition. This is done using an Internet connection. The data is then prepared automatically or with the help of a data scientist. An AI model is then built to capture the important characteristics of the traffic flow of vehicles and other ground transportation entities at that intersection or other aspects of the ground transportation network. The captured data features may include the location, direction, and movement of vehicles or other ground transportation entities, which can then be translated into intent and behavior. Knowing the intent, we can use the AI ​​model to predict the actions and future behavior of vehicles or other ground transportation entities approaching the traffic location with high accuracy. The trained AI model is tested on a subset of the data that was not included in the training phase. If the performance of the AI ​​model meets expectations, the training is considered complete. This phase is iteratively repeated with different model parameters until satisfactory model performance is achieved.

[0118] Deployment Phase

[0119] In some implementations, the complete tested AI model is then transmitted via the Internet to the RSE at the traffic location in the ground transportation network. The RSE is then ready to process new sensor data and predict and detect dangerous situations, such as traffic light violations. When a dangerous situation is predicted, the RSE will generate an appropriate alert message. Dangerous situations can be predicted, alert messages generated, and broadcast and received to vehicles and other ground transportation entities in the vicinity of the RSE before the predicted dangerous situation occurs. This gives the operators of vehicles or other ground transportation entities enough time to react and avoid collisions. The output of the AI ​​model from each intersection where the corresponding RSE is located can be recorded and made available online in a dashboard that contains all the generated data and displays it in an intuitive and user-friendly manner. Such a dashboard can be used as an interface with system customers (e.g., city traffic engineers or planners). An example of a dashboard is a map with markers indicating the location of the monitored intersections, the violations that have occurred, and statistics and analysis based on AI predictions and actual results.

[0120] SmartRSE (SRSE) and connected entity / disconnected entity bridge

[0121] As mentioned earlier, there is a gap between the capabilities and behaviors of connected and non-connected entities. For example, connected entities are often cooperative entities that constantly inform the world of their location and safety system status, such as speed, direction of travel, brake status, and headlight status. Non-connected entities cannot cooperate and communicate in these ways. Therefore, even connected entities are unaware of non-connected entities that are not in the vicinity of connected entities or out of sensor range due to interference, distance, or lack of a good vantage point.

[0122] Properly equipped and configured, an RSE can detect all entities within its vicinity that use the ground transportation network, including non-connected entities. Specialized sensors can be used to detect different types of entities. For example, radar is suitable for detecting moving metal objects such as cars, buses, and trucks. Such road entities are most likely to move in a single direction toward an intersection. Cameras are suitable for detecting vulnerable road users who may be wandering at an intersection looking for a safe time to cross.

[0123] Placing sensors on components of a ground transportation network has at least the following advantages:

[0124] - Good vantage points: Infrastructure poles, beams, and supporting cables often have elevated vantage points. Elevated vantage points allow for a more complete view of the intersection. This is like an observation tower at an airport, where the controller can have a comprehensive view of the most important and vulnerable users on the ground. In contrast, for ground transportation entities, the view from the vantage point of a sensor (camera, lidar, radar, etc... or other) may be blocked or interfered with by trucks in adjacent lanes, direct sunlight, or other interference. Sensors at intersections can be selected to be immune to such interference or less susceptible to such interference. For example, radar is not affected by sunlight and remains effective during the evening commute. Thermal cameras are more likely to detect pedestrians in bright light situations where the optical camera's view is obstructed.

[0125] - Fixed position: Sensors located at intersections can be adjusted and fixed to sense a specific direction that is optimized for detecting important targets. This will help the processing software detect objects better. As an example, if the camera has a fixed view, the background (non-moving objects and structures) information in the fixed view can be easily detected and used to improve the recognition and classification of relatively important moving entities.

[0126] Fixed sensor locations also make it easier to place each entity in a unified global view of the intersection. Because the sensor view is fixed, the measurements from the sensors can be easily mapped to a unified global position map of the intersection. Such a unified map is useful when performing a global analysis of traffic movements from all directions to study the interactions and dependencies of one traffic flow on another. An example is detecting a near miss (a dangerous situation) before it occurs. When two entities travel along intersecting paths, a global and unified view of the intersection will be able to calculate the time it takes for each entity to reach the intersection of the respective paths. If the time is within a certain limit or tolerance, the near miss may be flagged before it occurs (for example, by making it the subject of an alert message).

[0127] With the help of sensors installed on infrastructure components, smart RSE (SRSE) can bridge this gap and allow connected entities to understand “dark” or non-connected entities.

[0128] Figure 8 A scenario is described that explains how strategically placed sensors can help connected entities identify the speed and location of non-connected entities.

[0129] Connected entity 1001 is traveling along path 1007. Entity 1001 has green light 1010. Non-connected entity 1002 is traveling along path 1006. It has red light 1009 but will turn right along path 1006 on red. This places it directly in the path of entity 1001. Since entity 1001 is unaware of entity 1002, a dangerous situation is imminent. Because entity 1002 is a non-connected entity, it cannot broadcast (e.g., announce) its location and orientation to other entities sharing the intersection. In addition, entity 1001, even if connected, cannot "see" entity 1002, which is obscured by building 1008. There is a risk that entity 1001 will cross the intersection directly and hit entity 1002.

[0130] If the intersection is configured as a smart intersection, the radar 1004 mounted on the beam 1005 above the intersection road will detect the entity 1002 and its speed and distance. This information can be relayed to the connected entity 1001 through the SRSE 1011 which acts as a bridge between the non-connected entity 1002 and the connected entity 1001.

[0131] Artificial Intelligence and Machine Learning

[0132] Smart RSEs also rely on learning traffic patterns and entity behaviors to better predict and prevent dangerous situations and avoid collisions. Figure 8As shown, radar 1004 is always sensing and providing data for each entity moving along road 1012. This data is collected and transmitted to the cloud directly or through, for example, an RSE for analysis and for building and training a model that accurately represents the traffic along road 1012. When the model is complete, it is downloaded to SRSE 1011. The model can then be applied to each entity moving along road 1012. If the model classifies the entity as being (or about to be) violating traffic regulations, the SRSE may broadcast a warning (alarm) to all connected entities in the vicinity. This warning (called an intersection collision avoidance warning) will be received by the connected entity, and action can be taken based on considering the dangerous situation and avoiding the collision. Using an appropriate traffic model, the violating entity can be detected in advance, giving the connected entities using the intersection enough time to react and avoid dangerous situations.

[0133] With the help of multiple sensors (some installed high up on components of the ground transportation network's infrastructure), AI models, and accurate traffic models, SRSE can have a virtual overview of the ground transportation network and understand every entity in its field of view (including non-connected entities in its field of view, which are not "visible" to connected entities in its field of view). SRSE can use this data to feed the AI ​​model and provide alerts to connected entities on behalf of non-connected entities. Otherwise, connected entities would not know that there are non-connected entities sharing the road.

[0134] The SRSE may have high performance computing available in the same premises at the location of the SRSE, or by connecting to a nearby unit or to a server over the Internet. The SRSE may process data received directly from sensors, or data received from nearby SRSE broadcasts, emergency and weather information, and other data. The SRSE is also equipped with large capacity storage to help store and process data. High bandwidth connections are also required to help transfer raw data and AI models between the SRSE and a more powerful remote server. The SRSE uses AI to enhance other traffic hazard detection technologies to achieve high accuracy and provide additional time to react and avoid collisions.

[0135] SRSEs are compatible with current and new standardized communication protocols, so they can seamlessly integrate with equipment already deployed in the field.

[0136] SRSE can also reduce network congestion by sending messages only when necessary.

[0137] Global and unified intersection topology

[0138] Effective traffic monitoring and control at intersections benefits from a bird’s-eye view of the intersection that is not hampered by obstacles, lights or any other distractions.

[0139] As mentioned above, different types of sensors can be used to detect different types of entities. The information from these sensors may differ, for example, in terms of inconsistencies related to the position or motion parameters represented by their data or the native format of the data, or both. For example, radar data typically includes speed, distance, and possibly other information such as the number of moving and stationary entities in the radar's field of view. In contrast, camera data can represent an image of the field of view at any moment in time. Lidar data can provide the location of points in 3D space that correspond to reflection points of a laser beam emitted from the lidar at a specific time and direction of travel. Typically, each sensor provides data in a native format that accurately represents the physical quantity they measure.

[0140] In order to obtain a unified view (representation) of the intersection, it is useful to fuse data from different types of sensors. For the purpose of fusion, the data from the various sensors is converted into a common (unified) format independent of the sensors used. The data contained in the unified format from all sensors will include the global position, speed and direction of travel of each entity using the intersection, regardless of how it was detected.

[0141] With this unified global data, intelligent RSE can not only detect and predict the movement of entities, but also determine the relative position and direction of travel of different entities with respect to each other. As a result, SRSE can achieve improved detection and prediction of dangerous situations.

[0142] For example, in Fig. 9In the scenario shown, a motorized entity 2001 and a vulnerable road user 2002 share the same crosswalk. Entity 2001 is traveling along road 2007 and is detected by radar 2003. Vulnerable road user 2002 walking along sidewalk 2006 is detected by camera 2004. Vulnerable road user 2002 may decide to use crosswalk 2005 to cross road 2007. Doing so places road user 2002 in the path of entity 2001, creating a potentially dangerous situation. If the data from each of sensors 2003 and 2004 were considered independently and without considering other information, no dangerous situation would be identified because each sensor can only detect entities in its respective field of view. Furthermore, each sensor may not be able to detect objects that they are not designed to detect. However, when the SRSE considers a unified view, the location and dynamics of entity 2001 and vulnerable road user 2002 can be placed in the same reference system: a geographic coordinate system, such as a map projection or other coordinate system. When considered within a common reference system, the fused data from the sensors can be used to detect and predict possible dangerous situations between the two entities 2001 and 2002. In the following paragraphs, we will discuss the transformation between the sensor space and the unified space.

[0143] Radar data to unified reference conversion

[0144] like Fig.10 As shown, radar 3001 is used to monitor road entities traveling along a road having two lanes 3005 and 3008, which have center lines 3006 and 3007, respectively. Stop strip 3003 indicates the end of lanes 3005 and 3008. T 3006 can be defined by a set of markings 3003 and 3004. Fig.10 Only two markers are shown, but typically, the centerline is a piecewise linear function. The global positions of markers 3003 and 3004 (and other markers, not shown) are predefined by the design of the road and are known to the system. The precise global position of radar 3001 can also be determined. Therefore, the distances 3009 and 3010 of markers 3003 and 3004 from radar 3001 can be calculated. The distance 3011 of entity 3002 from radar 3001 can be measured by radar 3001. Using simple geometry, the system can determine the position of entity 3002 using the measured distance 3011. The result is a global position because it comes from the global positions of markers 3003, 3004, and radar 3001. Since every road can be approximated by a generalized piecewise linear function, the above method can be applied to any road that a radar can monitor.

[0145] Fig.11A similar situation is shown on a curved road. Radar 4001 monitors an entity moving along road 4008. Markers 4003 and 4004 represent linear segments 4009 (of a piecewise linear function) of centerline 4007. Distances 4005 and 4006 represent the normal distances between the plane 4010 of radar 4001 and markers 4003 and 4004, respectively. Distance 4007 is the measured distance of entity 4002 from radar plane 4010. Based on the discussion above, given the global position of radar 4001 and markers 4003 and 4004, the global position of entity 4002 can be calculated using simple ratio arithmetic.

[0146] Camera data to unified reference conversion

[0147] Knowing the camera's height, global position, orientation, tilt, and field of view, it becomes straightforward to compute the global position of each pixel in the camera image using existing 3D geometry rules and transformations. Thus, when an object is identified in an image, its global position can be easily inferred by knowing the pixels it occupies. It is important to note that the type of camera does not matter if its specifications are known (e.g., sensor size, focal length, or field of view, or a combination of these).

[0148] Fig.12 A side view of camera 5001 observing entity 5002 is shown. The height 5008 and tilt angle 5006 of camera 5001 can be determined at installation. The field of view 5007 can be obtained from the specifications of camera 5001. The global position of camera 5001 can also be determined at installation. Based on the known information, the system can determine the global position of points 5003 and 5004. The distance between points 5003 and 5004 is also divided into pixels on the image created by camera 5001. This number of pixels is known from the specifications of camera 5001. The pixels occupied by entity 5002 can be determined. Therefore, the distance 5005 can be calculated. The global position of entity 5002 can also be calculated.

[0149] By fusing information from a variety of sensors, a global unified view of any intersection can be pieced together. Fig.13A top view of a four-way intersection is depicted. Each road in the intersection is divided by a median 6003. The intersection in the figure is monitored by two different types of sensors: radar and camera, and the principles discussed here can be generalized to other types of sensors. In this example, the radar monitoring area 6001 overlaps with the camera monitoring area 6002. With a unified global view, each entity traveling between areas will remain tracked within the unified global view. This makes it possible for the SRSE to easily determine the relationship between the movements of different entities. Such information will allow a truly universal bird's-eye view of intersections and roads. The unified data from the sensors can then be fed into an artificial intelligence program, as described below.

[0150] Discussed above Figure 2 The components of an RSE are described. Furthermore, in an SRSE, the processing unit may also include one or more specialized processing units that can process data in parallel. An example of such a unit is a graphics processing unit or GPU. With the help of a GPU or similar hardware, machine learning algorithms can run more efficiently on the SRSE and can provide results in real time. This processing architecture enables real-time prediction of hazardous situations and, therefore, enables early warnings to be sent, giving entities enough time to react and avoid collisions. Furthermore, since the SRSE can run processes that can use data from different sensors and different types of sensors, the SRSE can build a unified view of the intersection, which will help in analyzing traffic flow and detection and prediction of hazardous situations.

[0151] Use Cases

[0152] A wide variety of use cases can benefit from this system and the early warning it provides for collision avoidance. Examples are provided here.

[0153] Case 1: Vulnerable Ground Transportation Entities

[0154] like Figure 4As shown, a road crossing a typical intersection 409 may have a pedestrian crosswalk, including specific intersection areas 401, 402, 403, 404 that pedestrians and other vulnerable road users (vulnerable road users) can use to cross the road. Sensors suitable for detecting such intersections or other vulnerable users are placed at one or more advantageous locations to allow monitoring of the crosswalk and its surroundings. During the training phase, the collected data can be used to train an artificial intelligence model to understand the behavior of vulnerable road users at intersections. During the deployment phase, the AI ​​model can then use current data about vulnerable road users to predict, for example, that a vulnerable road user is about to cross the road, and make predictions before the vulnerable road user begins to cross. When the behavior and intentions of pedestrians and other vulnerable road users, drivers, vehicles, and other people and ground transportation entities can be predicted in advance, an early warning (e.g., an alarm) can be sent to any or all of them. The early warning can cause the vehicle to stop, slow down, change the path, or a combination thereof, and can cause vulnerable road users to avoid crossing the road when a dangerous situation is expected to occur.

[0155] Typically, sensors are used to monitor all areas of possible movement of vulnerable road users and vehicles in the area near the intersection. The type of sensor used depends on the type of subject being monitored and tracked. Some sensors are better at tracking people and bicycles or other non-motorized vehicles. Some sensors are better at monitoring and tracking motorized vehicles. The solution described here is sensor and hardware agnostic, in that the type of sensor is irrelevant if the sensor provides appropriate data at a sufficient data rate, which may depend on the type of subject being monitored and tracked. For example, a Doppler radar would be a suitable sensor for monitoring and tracking vehicle speed and distance. The data rate or sampling rate is the rate at which the radar is able to provide continuous new data values. The data rate must be fast enough to capture the dynamics of the motion of the subject being monitored and tracked. The higher the sampling rate, the more details are captured, and the more robust and accurate the representation of the motion through the data becomes. If the sampling rate is too low, and the vehicle travels a very long distance between two sample instances, it becomes difficult to model the behavior due to the details missed during the intervals where no data was generated.

[0156] For crosswalks, sensors will monitor pedestrians and other vulnerable road users (such as cyclists) crossing the road at the intersection and in the area near the intersection. The data from these sensors can be segmented to represent conditions with different virtual zones to help with detection and positioning. These zones can be selected to correspond to corresponding key areas where dangerous conditions may occur, such as sidewalks, entrances to sidewalks, and roads to intersections 405, 406, 407, 408. Activity and other conditions in each area are recorded. Recordings can include, but are not limited to, kinematics (such as position, direction of travel, speed, and acceleration) and facial and body features (such as eyes, posture).

[0157] The number of sensors, the number of zones, and the shapes of the zones vary for each intersection and the roads leading to each intersection.

[0158] Figure 5 Depicted is a floor plan of a typical example setup showing different areas for monitoring and tracking the movement and behavior of pedestrians or other vulnerable road users, motorized and non-motorized vehicles, and other ground transportation entities.

[0159] Sensors are arranged to monitor a crosswalk across a road. Virtual areas (301, 302) may be placed on the sidewalk and along the crosswalk. Other sensors are placed to monitor vehicles and other ground transportation entities traveling on the road leading to the crosswalk, and virtual areas (303, 304) are strategically placed to help detect incoming vehicles and other ground transportation entities, their distance from the crosswalk, and, for example, their speed.

[0160] The system (e.g., an RSE or SRSE associated with the sensors) collects data streams from all sensors. When the system is first put into operation, an initial rule-based model can be deployed to help with equipment calibration and functionality. At the same time, sensor data (e.g., speed and distance from radar units, images and video from cameras) is collected and stored locally on the RSE in preparation for transmission to a remote computer powerful enough in certain embodiments to build an AI model of the behavior of different entities at the intersection using the collected data. In some cases, the RSE is an SRSE capable of generating AI models on its own.

[0161] The data is then prepared and trajectories are constructed for each ground traffic entity that passes through the intersection. For example, trajectories can be extracted from radar data by stitching together points of different distances belonging to the same entity. For example, pedestrian trajectories and behaviors can be extracted from camera and video recordings. By performing video and image processing techniques, the movement of pedestrians can be detected in images and videos, and their respective trajectories can be inferred.

[0162] For human behavior, models based on intelligent machine learning often outperform simple rules based on simple physics. This is because human intent is difficult to capture and large datasets are required to detect patterns.

[0163] For example, when a machine learning (AI) model is completed on a server, it is downloaded to the RSE via the Internet. The RSE then applies current data captured from sensors to the AI ​​model, allowing it to predict intentions and behaviors, determine when a dangerous situation is imminent, and trigger corresponding alerts that are distributed (e.g., broadcast) to vehicles and other ground transportation entities and vulnerable road users and drivers as timely warnings to enable vulnerable road users and drivers to take collision avoidance measures.

[0164] This example setup can be combined with any other use case, such as traffic at a signalized intersection or level crossing.

[0165] Case 2: Signalized intersection

[0166] In the case of a signalized intersection (e.g., an intersection controlled by traffic lights), the overall setup of the system is the same as in Case 1. One difference may be the type of sensors used to monitor or track vehicle speed, direction of travel, distance, and location. The pedestrian crossing setup of Case 1 can also be combined with the signalized intersection setup to obtain a more general solution.

[0167] The operational concept of the signalized intersection use case is to track road users around the intersection using external sensors that collect data about the users or that are transmitted by the users themselves, predict their behavior and broadcast warnings via different communication methods about upcoming dangerous situations, which are usually due to violations of intersection traffic rules, such as violating red light signals.

[0168] Data about road users can be collected using: (a) entity data about its current state broadcast by each entity itself, such as through a BSM or PSM, and (b) sensors mounted on the infrastructure or outside the vehicle, such as Doppler radar, ultrasonic sensors, visual or thermal imagers, LiDAR, etc. As mentioned above, the type of sensor selected and its location and orientation at the intersection should provide the most comprehensive coverage of the intersection or the partial coverage of the intersection under study, and the data collected about entities approaching the intersection is the most accurate. In this way, the collected data will allow the reconstruction of the current state of the road users and the creation of accurate, timely, and useful VBSM (Virtual Basic Safety Message) or VPSM (Virtual Personal Safety Message). The frequency of data collection depends on the potential danger of each type of road user and the severity of the potential violation. For example, a motor vehicle traveling at high speed at an intersection typically requires 10 data updates per second to achieve real-time collision avoidance; a pedestrian crossing the intersection at a much lower speed may require as little as 1 data update per second.

[0169] As mentioned earlier, Figure 4 An example of a signalized intersection plan view with detection virtual zones is depicted. The zones may divide each approach to the intersection into separate lanes 410, 411, 412, 413, 405, 406, 407, 408, and may further divide each lane into zones corresponding to a general range of distances from the stop strip. The zones may be selected empirically to generally match the characteristics of a particular road and intersection. Segmenting the intersection allows for a more accurate determination of the relative direction of travel, speed, acceleration, and location of each road user, and thus a better assessment of the potential hazard of road users to other ground transportation entities.

[0170] In order to determine whether the observed traffic situation is a dangerous one, the system also needs to compare the results of the predicted situation with the traffic light status and take into account local traffic rules (such as left turn lanes, right turn on red lights, and other rules). Therefore, it is necessary to collect and use signal phase and timing (SPaT) information at the intersection. SPaT data can be collected by directly interfacing with the traffic light controller at the intersection, usually reading the data through a wired connection, or by interfacing with a traffic management system to receive the required data, such as through an API. It is very important to collect SPaT data at a rate as close as possible to the rate at which road user data is collected to ensure that the road user status is always synchronized with the traffic signal status. Another complexity in understanding the requirement for SPaT information is that modern traffic control strategies used to adjust traffic flow around intersections are not based on fixed timings, but use algorithms that can dynamically adapt to real-time traffic conditions. Therefore, it is important to combine SPaT data prediction algorithms to ensure the highest accuracy of violation prediction. These SPaT data prediction algorithms can be developed using rule-based methods or machine learning methods.

[0171] For each road leading to the intersection, the RSE (or SRSE) collects data and builds a machine learning (AI) model to describe the behavior of the vehicle corresponding to the collected data. The current data collected at the intersection is then applied to the AI ​​model to produce an early prediction of whether a vehicle or other ground transportation entity traveling on one of the roads leading to the intersection is, for example, about to violate a traffic light. If a violation is about to occur, a message is relayed (e.g., broadcast) from the RSE to ground transportation entities in the vicinity. Vehicles (including the violating vehicle) and pedestrians or other vulnerable road users will receive the message and have time to take appropriate preemptive measures to avoid a collision. The message can be delivered to the ground transportation entity by one or more of the following means, including: flashing lights, signs, or radio signals.

[0172] If a vehicle or other entity approaching the intersection is equipped with an OBE or OPE, it will be able to receive a message broadcast from the RSE that a potential hazard is predicted to exist at the intersection. This allows the user to be warned and take appropriate preemptive measures to avoid a collision. If the offending road user at the intersection is also equipped with an OBE or OPE, that user will also receive the broadcast alert. The algorithm on the OBE or OPE can then coordinate the message with the user's violation and warn the user appropriately.

[0173] The decision to send an alert is not based solely on the vehicle's behavior as represented by the data collected by sensors at the intersection. Although sensors play a major role in the decision, other inputs are also considered. These inputs may include, but are not limited to, information from nearby intersections (if a vehicle turns on its lights at a nearby intersection, there is a higher probability that it will do the same at this intersection), information from other cooperating vehicles, or even the vehicle itself, for example if it reports that it has a malfunction.

[0174] Case 3: Unsignalized intersection

[0175] Non-signalized controlled intersections, such as those controlled by a stop sign or yield sign, can also be monitored. Sensors are used to monitor roads controlled by traffic signs and can make predictions about incoming vehicles similar to predictions for incoming vehicles on roads leading to signalized intersections. The rules of the road for non-signalized controlled intersections are usually well defined. Ground traffic on roads controlled by a stop sign must come to a stop. In multi-way stop intersections, the right of way is determined by the order in which ground traffic arrives at the intersection. A special case may be considered for one-way stops. A set of sensors can also monitor roads without stop signs. Such a setup can help with stop sign gap negotiation. For yield sign controlled intersections, ground traffic on roads controlled by a yield sign must reduce their speed to yield the right of way to other ground traffic in the intersection.

[0176] A major challenge is that due to internal (e.g., driver distraction) or external (e.g., lack of visibility) factors, ground traffic entities violate road rules and put other ground traffic entities at risk.

[0177] In the general case of a stop sign controlled intersection (i.e., each road is controlled by a stop sign), the overall system setup is the same as done in Case 1. One difference may be the type of sensors used to monitor or track vehicle speed, direction of travel, distance, and location. Another difference is the lack of traffic light controllers, and the rules of the road are indicated by road signs. The pedestrian crossing setup of Case 1 can also be combined with the non-signalized controlled intersection setup to obtain a more general solution.

[0178] Figure 4 It can also be understood that an example of a four-way stop intersection plan view is depicted with detection virtual zones. These zones can divide each road to the intersection into separate lanes 410, 411, 412, 413, 405, 406, 407, 408, and can also divide each lane into zones corresponding to a general distance range from the stop strip. The selection of these zones can be made empirically to generally match the characteristics of a particular road and intersection.

[0179] With the above Figure 4 In a manner similar to that described in , current data collected at the intersection is applied to an AI model to produce an early prediction of whether a vehicle or other ground traffic entity traveling on one of the roads leading to the intersection is, for example, about to violate a stop sign. If a violation is about to occur, the message can be processed similarly to the case described earlier involving a traffic light violation.

[0180] Also similar to the previous description, the decision to send an alert can be based on the factors described previously and other information, such as whether the vehicle is driving into a stop sign at a nearby intersection, indicating a higher probability that it will do the same thing at this intersection.

[0181] Fig.18 The use case of a controlled non-signalized intersection is illustrated. It explains how a SRSE with strategically placed sensors can warn connected entities of impending hazardous situations caused by non-connected entities.

[0182] Connected entity 9106 is traveling along path 9109. Entity 9106 has the right of way. Non-connected entity 9107 is traveling along path 9110. Entity 9107 has a yield sign 9104 and will merge onto path 9109 without granting the right of way to entity 9106, placing it directly in the path of entity 9106. Since entity 9106 is unaware of entity 9107, a dangerous situation is imminent. Because entity 9107 is a non-connected entity, it cannot announce (broadcast) its location and proceed toward other entities sharing the intersection. Additionally, entity 9106 may not be able to "see" entity 9107, which is not in its direct field of view. If entity 9106 proceeds along its path, it may eventually collide with entity 9107.

[0183] Because the intersection is a smart intersection, the radar 9111 mounted on the beam 9102 above the road will detect the entity 9107. It will also detect the speed and distance of the entity 9107. This information can be relayed to the connected entity 9106 through the SRSE 9101 as an alert. The SRSE 9101 has a machine learning model for entities moving along the road 9110. The entity 9107 will be classified as a potential violator of traffic rules by the model and a warning (alert) will be broadcast to the connected entity 9106. This warning is sent in advance, giving the entity 9106 enough time to react and prevent a dangerous situation.

[0184] Case 4: Level Crossing

[0185] Level crossings are dangerous because they may carry motor vehicles, pedestrians and rail vehicles. In many cases, the road leading to a level crossing falls into the blind spot of the operator of a train or other rail vehicle, such as the conductor. Since rail vehicle drivers operate primarily based on visual information, this increases the likelihood of an accident if a road user violates the rail vehicle's right of way and crosses the level crossing when they are not allowed to pass.

[0186] The Level Crossing use case operates similarly to the Signalized Intersection use case in the sense that a level crossing is a point of conflict between road and rail traffic that is typically subject to traffic rules and signals. Therefore, this use case also requires collision avoidance warnings to increase safety around level crossings. Rail traffic can have systematically segregated rights of way, such as high-speed rail, or no segregated rights of way, such as light rail or trams. For light rail and trams, the use case becomes even more important because these rail vehicles also operate on living roads and must obey the same traffic rules as road users.

[0187] Figure 6 Depicted is a general use of a level crossing where a road and a pedestrian crossing cross a railway. Similar to the pedestrian crossing use case, sensors are placed to collect data about pedestrian movements and intentions. Other sensors are used to monitor and predict the movements of road vehicles set to approach the intersection. Data about road users can also be collected from road user broadcasts (e.g., BSM or PSM). Data from nearby intersections, vehicles, and remote command and control centers can be used in the decision to trigger an alert.

[0188] There is also a need to collect SPaT data on both road and rail routes in order to fully assess the likelihood of violations.

[0189] Similar to the signalized intersection use case, the collected data can create predictive models using rule-based and machine learning algorithms.

[0190] In this use case, rail vehicles are equipped with OBE or OPE to receive collision avoidance warnings. When a violation of the rail vehicle right of way is predicted, the RSE will broadcast an alert message to warn the rail vehicle driver that the road user is on its intended path and allow the rail vehicle driver enough time to take preemptive action to avoid a collision.

[0191] If the offending road user is also equipped with an OBE or OPE, the message broadcast by the RSE will also be received by the offending road user. The algorithm on the OBE or OPE can then coordinate the received message with the user's violation and warn the user appropriately.

[0192] Virtually connected ground transportation environment (bridging the gap)

[0193] As mentioned above, one useful application of the system is to create a virtual connected environment on behalf of non-connected ground transportation entities. One barrier to the adoption of connected technologies is not only the lack of infrastructure installation, but also the near non-existence of connected vehicles, connected vulnerable road users, and connected other ground transportation entities.

[0194] In the case of connected vehicles, in some regulatory regimes, such vehicles always send so-called Basic Safety Messages (BSMs). BSMs contain, among other information, the vehicle's position, direction of travel, speed, and future path. Other connected vehicles can listen to these messages and use them to create a map of the vehicles present in their surroundings. Knowing the position of surrounding vehicles, vehicles, whether motorized or not, will have information that helps maintain a high level of safety. For example, if a motorized vehicle has a connected vehicle in its path, it can avoid making a move. Similarly, the driver can be warned if there are other vehicles in the path they plan to follow, such as a sudden lane change.

[0195] Until all surface transportation entities are equipped with the ability to send and receive traffic safety messages and information, some road entities will be "dark" or invisible to the rest of the road entities. Dark road entities pose a risk of hazardous conditions.

[0196] Dark road entities do not advertise (e.g., broadcast) their locations and, therefore, are invisible to connected entities that may expect all road entities to broadcast their information (i.e., to become connected entities). Although onboard sensors can detect obstacles and other road entities, the range of these sensors is often too short to effectively prevent hazardous situations and collisions. As a result, there is a gap between the connectivity of connected vehicles and the lack of connectivity of non-connected vehicles. The techniques described below aim to bridge this gap by using intelligence on the infrastructure that can detect all vehicles at an intersection or other component of a ground transportation network and send messages on behalf of non-connected vehicles.

[0197] The system can establish a virtually connected ground transportation environment, such as at an intersection, which can bridge the gap between the future in which most vehicles (and other ground transportation entities) are expected to be connected entities and the present in which most vehicles and other ground transportation entities do not have connectivity. In the virtually connected ground transportation environment, smart traffic lights and other infrastructure installations can use sensors to track all vehicles and other ground transportation entities (connected, non-connected, semi-mobile, mobile, non-mobile) and (in the case of vehicles) generate virtual BSM messages (VBSM) on their behalf.

[0198] The VBSM message can be considered a subset of the BSM. It may not contain all the fields required to create a BSM, but it can contain all positioning information, including position, driving direction, speed, and trajectory. Since V2X communications are standardized and anonymized, VBSM and BSM cannot be easily distinguished and follow the same message structure. The main difference between the two messages is the availability of information sources that populate these messages. VBSM may lack data and information that is not easily generated by external sensors, such as steering wheel angle, brake status, tire pressure, or wiper activation.

[0199] With the appropriate sensors installed, an intersection with a smart RSE can detect all road entities traveling through the intersection. The SRSE can also transform all the data from multiple sensors into a global unified coordinate system. This global unified coordinate system is represented by the geographic location, speed, and direction of travel of each road entity. Each road entity, whether connected or not, is detected by the intersection equipment and a global unified location is generated on its behalf. Therefore, standard safety messages can be broadcast on behalf of road entities. However, if the RSE broadcasts safety messages to all entities it detects, it can send messages on behalf of connected road entities. To resolve conflicts, the RSE can filter connected road entities from its list of dark entities. This is possible because the RSE is constantly receiving safety messages from connected vehicles and the RSE sensors are constantly detecting road entities crossing the intersection. If the location of the detected road entity matches the location from which the RSE receiver received a safety message, it is assumed that the road entity is connected and the RSE will not broadcast a safety message on its behalf. This is done in Fig.15 Chinese commentary.

[0200] By building a bridge between connected and non-connected vehicles, connected entities (including motor vehicles) can safely navigate intersections with full awareness of all nearby road entities.

[0201] This aspect of the technology Fig.17. An intersection 9001 has multiple road entities at a given time. Some of these entities are non-connected 9004, 9006, while others are connected 9005, 9007. Vulnerable road users 9004, 9007 are detected by camera 9002. Motorized road entities 9005, 9006 are detected by radar 9003. The position of each road entity is calculated. RSE 9008 also receives broadcasts from connected road entities. The position of the entity receiving the message is compared with the position of the detected entity. If the two entities match within a predetermined tolerance range, the entity at that position is considered connected and no safety message is sent on its behalf. The remaining road entities that do not match the received position are considered dark. Safety messages are broadcast on their behalf.

[0202] For collision warnings and intersection violation warnings that are part of the V2X protocol, each entity needs to be connected for the system to work. This requirement is a barrier to the deployment of V2X devices and systems. Intersections equipped with smart RSE will solve this problem by creating a virtual bridge between connected and non-connected vehicles.

[0203] The U.S. DOT (Department of Transportation) and NHTSA (National Highway Traffic Safety Administration) have identified many connected vehicle applications that will use BSM and help significantly reduce non-impairment accidents and fatalities. These applications include, but are not limited to, Forward Collision Warning (FCW), Intersection Movement Assist (IMA), Left Turn Assist (LTA), Do Not Pass Warning (DNPW), and Blind Spot / Lane Change Warning (BS / LCW). The U.S. DOT and NHTSA define these applications as follows.

[0204] FCW addresses rear-end collisions and warns drivers when the vehicle ahead is stopped, slowed, or decelerated. IMA is designed to avoid intersection collisions and warns drivers of vehicles approaching from the side at an intersection covering two main scenarios: turning paths in the same or opposite directions and straight crossing paths. LTA addresses collisions where one of the involved vehicles is turning left at an intersection while the other vehicle is traveling straight in the opposite direction, and warns drivers of oncoming oncoming vehicles when attempting to turn left. DNPW helps drivers avoid reverse collisions caused by passing maneuvers and warns drivers of oncoming oncoming vehicles when attempting to pass a slower vehicle on an undivided two-lane road. BS / LCW addresses collisions where vehicles have performed lane change / merge maneuvers prior to the collision and warns drivers of vehicles in the adjacent lane that are in or approaching their blind spot.

[0205] The V2X protocol specifies that these applications should be implemented using vehicle-to-vehicle (V2V) communications, where a connected remote vehicle will broadcast basic safety messages to a connected host vehicle. The host vehicle's OBE will then attempt to reconcile these BSMs with its own vehicle parameters (such as speed, direction of travel, and trajectory) and determine if there is a potential hazard or threat as presented by the remote vehicle as described in the above applications. In addition, motor vehicles will particularly benefit from such applications because it allows surrounding vehicles to communicate intent, which is key information not included in the data collected from their onboard sensors.

[0206] However, today's vehicles are not connected, and as mentioned earlier, it will take a long time for BSMs to work as described above until the percentage of connected vehicles becomes high. Therefore, connected vehicles do not need to receive and analyze a large number of BSMs in an environment with a small percentage of connected vehicles, as they would otherwise receive in an environment with a large enough percentage of connected vehicles to enable the described applications and fully benefit from V2X communications.

[0207] VBSM can help bridge the gap between the current environment with a large number of non-connected entities and the future environment with a large number of connected entities, and enable the above applications during the transition period. In the technique we describe here, a connected vehicle receiving a VBSM will process it as a regular BSM in the application. Since VBSM and BSM follow the same message structure, and VBSM contains essentially the same basic information as BSM, such as speed, acceleration, driving direction, past and predicted trajectories, the results of applying the message to a given application will be essentially the same.

[0208] For example, consider an intersection with an unprotected left turn where a connected host vehicle is about to attempt a left turn while a non-connected remote vehicle is traveling straight ahead from the opposite direction and has the right of way. This is a situation where the completion of the maneuver depends on the judgment of the host vehicle's driver about the situation. An incorrect assessment of the situation could result in a conflict and a potential near collision or collision. External sensors installed on the surrounding infrastructure can detect and track the remote vehicle or even both vehicles, collect basic information such as speed, acceleration, direction of travel, and past trajectory, and transmit it to the RSE, which in turn can construct a predicted trajectory for the remote vehicle using rule-based or machine learning algorithms, or both, populate the required fields of the VBSM, and broadcast it on behalf of the non-connected remote vehicle. The host vehicle's OBE will receive the VBSM with information about the remote vehicle and process it in its LTA application to determine if the driver's actions are potentially dangerous and whether the OBE should display a warning to the host vehicle's driver to take preemptive or corrective action to avoid a collision. Similar results can also be achieved if the remote vehicle is connected and receives from the RSE and sensors that the opposing vehicle is attempting a left turn in the event of a predicted collision.

[0209] VBSM can also be used for lane change maneuvers. If a vehicle changes lanes without performing the necessary steps to check the safety of the maneuver, such as checking the rear and side mirrors and blind spots, such a maneuver can be dangerous. New advanced driver assistance systems, such as blind spot warnings using onboard ultrasonic sensors, have been developed to help prevent vehicles from performing dangerous lane changes. However, these systems can be flawed when the sensors are dirty or the field of view is obstructed. Existing systems do not attempt to warn dangerous vehicles that another vehicle is attempting to change lanes. V2X communications help solve this problem through applications such as BS / LCW using BSM, but the vehicle attempting the lane change may be a non-connected vehicle and therefore unable to communicate its intentions. VBSM can help achieve this goal. Similar to the LTA use case, external sensors installed on the surrounding infrastructure can detect and track non-connected vehicles attempting lane change maneuvers, collect basic information such as speed, acceleration, direction of travel, and past trajectory, and transmit it to the RSE. The RSE will in turn use rule-based and machine learning algorithms to build a predicted trajectory for the vehicle's lane change, populate the required fields of the VBSM, and broadcast it on behalf of the non-connected remote vehicle. The OBE of the at-risk vehicle would then receive the VBSM with information about a vehicle about to merge into the same lane, process it and determine if the maneuver is potentially dangerous and if a lane change warning should be displayed to the vehicle driver. If the lane-changing vehicle is a connected vehicle, its OBE can similarly receive a VBSM about vehicles in its blind spot from the RSE and determine if the lane-changing maneuver is potentially dangerous to surrounding traffic and if a blind spot warning should be displayed to the driver of the vehicle. If both vehicles are connected, the two vehicles will be able to broadcast BSMs to each other and enable BS / LCW applications. However, these applications will still benefit from applying the same rule-based or machine learning algorithms (or both) on the BSM data to predict the vehicle's intention to change lanes early on, with the OBE deciding whether to display a warning.

[0210] Motor Vehicles

[0211] The lack of connectivity in non-connected road entities affects motor vehicles. Sensors on motor vehicles are either short-range or have a narrow field of view. They cannot detect, for example, a vehicle driving around a building on a street corner. They also cannot detect a vehicle that may be hidden behind a delivery truck. These hidden vehicles, if they were non-connected entities, would not be visible to the motor vehicle. These situations affect the ability of motor vehicle technology to achieve the level of safety required for mass adoption of the technology. Smart intersections can help close this gap and aid public acceptance of motor vehicles. A motor vehicle is only as good as the sensors it has. An intersection equipped with smart RSE can extend the range of onboard sensors to around blind corners or beyond large trucks. Such an extension would allow motor vehicles and other connected entities to coexist with traditional non-connected vehicles. This coexistence could accelerate the adoption of motor vehicles and the benefits they bring.

[0212] The virtual connected ground transportation environment includes VBSM messaging, which enables vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), and vehicle-to-device (V2D) applications that would otherwise be difficult to implement.

[0213] The system can use machine learning to quickly and accurately generate the data fields required for various safety messages, package them into a VBSM message structure, and send the messages to nearby ground transportation entities using various media, such as but not limited to DSRC, WiFi, cellular, or traditional road signs.

[0214] Virtual Personal Security Message (VPMS)

[0215] The ground transportation environment may include not only non-connected vehicles, but also non-connected people and their vulnerable road users.

[0216] In some regulatory regimes, connected vulnerable ground transportation entities continuously send Personal Safety Messages (PSMs). PSMs contain, among other information, the location, direction of travel, speed, and future path information of the vulnerable ground transportation entity. Connected vehicles and infrastructure can receive this information and use it to create maps that include vulnerable entities and improve the safety level of the ground transportation network.

[0217] Therefore, the virtually connected ground transportation environment can bridge the gap between the future when the most vulnerable ground transportation entities are expected to be connected and the present when the most vulnerable ground transportation entities are not connected. In the virtually connected ground transportation environment, smart traffic lights and other infrastructure installations can use sensors to track all vulnerable ground transportation entities (connected, non-connected) and generate VPSMs on their behalf.

[0218] The VPSM message can be considered a subset of the PSM. A VPSM does not need to contain all the fields required to create a PSM, but can contain data required to safely assess and prevent hazardous situations, and can include positioning information, including position, direction of travel, speed, and trajectory. In some cases, non-standard PSM fields may also be included in the VPSM, such as the driver's intent, posture, or gaze direction.

[0219] The system can use machine learning to quickly and accurately generate these fields, package them into a VPSM message structure, and send the message to nearby ground transportation entities using a variety of mediums, such as but not limited to DSRC, WiFi, cellular, or traditional road signs.

[0220] VPSM messages enable pedestrian-to-vehicle (P2V), pedestrian-to-infrastructure (P2I), pedestrian-to-device (P2D), vehicle-to-pedestrian (V2P), infrastructure-to-pedestrian (I2P), and device-to-pedestrian (D2P) applications that would otherwise be difficult to implement.

[0221] Fig.16 A pedestrian 8102 is depicted crossing a crosswalk 8103. The crosswalk 8103 may be a mid-block crosswalk at an intersection or on a stretch of road between intersections. A camera 8101 is used to monitor a crosswalk 8104. The global position of the boundaries of the field of view 8105 of the camera 8101 may be determined at installation. The field of view 8105 is covered by a predetermined number of pixels reflected by the specifications of the camera 8101. A road entity 8102 may be detected within the field of view of the camera and its global position may be calculated. The speed and direction of travel of the road entity 8102 may also be determined based on its displacement at s times. The path of the road entity 8102 may be represented by breadcrumbs 8106, which is a series of locations that the entity 8102 has passed through. This data may be used to construct a virtual PSM message. The PSM message may then be broadcast to all entities in the vicinity of the intersection.

[0222] Traffic and behavioral enforcement at unsignalized intersections

[0223] Another useful application of the system is in traffic enforcement at unsignalized intersections (e.g., stop signs, yield signs) and enforcement of good driving behavior anywhere on the surface transportation network.

[0224] As a byproduct of generating VBSM and VPSM, the system can track and detect road users who do not comply with traffic regulations and increase the probability of dangerous situations and collisions. The prediction of dangerous situations can be extended to include enforcement. Dangerous situations do not have to end in a collision. Near misses are common and increase driver stress levels, leading to subsequent accidents. The frequency of near misses is positively correlated with the lack of enforcement.

[0225] Additionally, using VBSM, the system can detect improper driving behaviors such as sudden lane changes and other forms of reckless driving. The data collected by the sensors can be used to train and enable machine learning models to flag ground transportation entities that engage in dangerous driving behaviors.

[0226] Law enforcement typically enforces road rules against ground-based entities, including vulnerable road users, but authorities need to be present in the vicinity of intersections to monitor, detect, and report violations. By using VBSM and VPSM to track non-connected ground-based entities, including vulnerable road users, smart RSE can play the role of law enforcement and enforce road rules at intersections. For example, a non-connected vehicle tracked by a smart RSE can be detected violating a stop or yield sign and can be identified and reported to the authorities. Similarly, a vulnerable road user near an intersection tracked by a smart RSE can be detected illegally crossing the intersection, can be identified, and reported to the authorities.

[0227] For law enforcement and other purposes, ground transportation entities may be identified using unique identifiers, including but not limited to license plate recognition. Vulnerable road users may be identified using biometric recognition, including but not limited to facial, retinal, and acoustic recognition. In special circumstances, including civil or criminal investigations, social media networks (e.g., Facebook, Instagram, Twitter) may also be used to support the identification of violating ground transportation entities or vulnerable road users. An example of leveraging social networks is uploading captured images of violators on a social network and requesting social network users who recognize the violators to provide law enforcement agencies with intelligence that may help identify the violators.

[0228] Enhanced SOBE

[0229] An intelligent RSE can use sensors and prediction models to predict dangerous situations and subsequently send virtual safety messages (such as ICA, VBSM, VPSM, VICA and VCSM (virtual combined safety messages)) on behalf of non-connected road users, including vulnerable road users. An intelligent OBE can use incoming virtual or standard safety messages (such as BSM, PSM, VBSM, VPSM, VICA and VCSM), vehicle sensors and prediction models to predict dangerous situations and alert the driver of the host vehicle. An enhanced SOBE can do all of this and send a. (acting as an intelligent RSE) virtual BSM, virtual PSM, virtual ICA and VCSM and standard messages on behalf of other road users (even non-connected road users and especially vulnerable road users) where applicable, b. advanced BSM for its own ground transportation entity, including information based on intentional predictions of its own behavior and c. messages such as GPS corrections (acting as an RSE).

[0230] Throughout this document, we refer to BSM, PSM, ICA, VBSM, VPSM, and VICA. Other types of safety messages may exist or are under development, including the under-development Cooperative Perception Message (CPM), as reported at https: / / www.sae.org / standards / content / j2945 / 8 / . References to BSM, PSM, ICA, VBSM, VPSM, and VICA are also intended to refer to other existing and future safety messages, including CPM. For example, as proposed, a CPM may include data about (and in that sense is a container for) multiple objects (e.g., VBSM, VPSM, and VICA). We sometimes refer to messages that may be containers of VBSM, VPSM, VICA, and other types of virtual safety messages (e.g., VCSM). An ESOBE may generate a VCSM based on its detection of ground transportation entities and other objects using the sensors of its host vehicle. VCSMs may be broadcast periodically by an ESOBE to make other ground transportation entities aware of unconnected or obscured ground transportation entities or other objects.

[0231] Here, we describe onboard equipment (OBEs) that may have enhanced and additional capabilities beyond the previously described OBEs and SOBEs in certain respects. In some implementations described herein, such enhanced SOBEs (ESOBEs) may have capabilities that actually go beyond serving as a smart RSE, for example, by leveraging sensors already present on the vehicle in which the ESOBE is present (the "host vehicle"), as well as predictive models. In some cases, for example, the ESOBE may send to other vehicles or vulnerable road users: (1) virtual safety messages on behalf of one or more other vehicles, and (2) enhanced standard safety messages of the host ground transportation entity using intentional predictions of its own behavior, and (3) virtual safety messages sent by them operating in the role of an onboard RSE (e.g., GPS corrections). We sometimes refer to the onboard RSE as an enhanced RSE ("ERSE").

[0232] Below, we describe the scenarios and applications of the Enhanced SOBE (“ESOBE”) technology, including the following:

[0233] 1. ESOBE, for example on behalf of another vehicle that is skidding and not connected, generates and broadcasts a virtual BSM. A third connected vehicle that cannot see the skidding vehicle due to obstructed vision then receives the virtual BSM and can use it to generate an early warning of a dangerous situation, for example to the driver of the third vehicle.

[0234] 2. ESOBE generates and broadcasts a virtual PSM, for example representing a pedestrian or other vulnerable road user crossing at a marked or designated crosswalk (e.g., an intersection). A third connected vehicle (including a moving or stationary vehicle, such as a public transportation vehicle) that is unable to see the vulnerable road user crossing the road due to an obstructed view then receives the virtual PSM and can use it to generate an early warning of the dangerous situation to, for example, the driver of the third vehicle.

[0235] 3. ESOBE generates and broadcasts a virtual PSM, for example representing a pedestrian or other vulnerable road user crossing at a mid-block crosswalk instead of at an intersection. A third connected vehicle that is unable to see the vulnerable road user crossing the road due to an obstructed view then receives the virtual PSM and can use it to generate an early warning of the hazardous situation to, for example, the driver of the third vehicle.

[0236] 4. ESOBE generates and broadcasts an enhanced standard BSM from its host vehicle, for example, by including in the BSM a forward collision prediction determined based on information received by ESOBE from the host vehicle's onboard camera or sensor or both. Vehicles following the host vehicle can use the enhanced BSM to generate an early warning to the drivers of the following vehicles that the host vehicle may brake suddenly within, for example, 1-2 seconds.

[0237] 5. The ESOBE generates and broadcasts GPS corrections to other ground traffic entities. In this mode, the ESOBE essentially acts as an RSE, helping to extend the coverage of RSE corrections by leveraging the OBE's own GPS. The OBE can also save and retrieve the most recent GPS corrections it received from the nearest RSE it is close to.

[0238] In all of these scenarios described above and below, as well as in other scenarios, the messages broadcast by ESOBE may include virtual messages bundled in the VCSM.

[0239] Occluded skidding vehicles and corner collisions

[0240] Fig.19 A vehicle or another ground transportation entity 1907 (e.g., not equipped with an OBE, or otherwise not connected) is depicted traveling in lane 1914 and sliding across a multi-lane road 1912 into an adjacent lane 1916. The vehicle skid may occur, for example, due to sudden braking, slippery road conditions, or other reasons. In this case, vehicle 1906 is traveling in lane 1916 and carrying ESOBE 1908 and sensor 1918. Sensor 1918 may be, but is not limited to, a camera, radar, lidar, ultrasonic ranging sensor, etc., and combinations thereof. ESOBE 1908 processes the data feed from the sensor in real time and generates periodic basic safety messages (BSMs) for vehicle 1906.

[0241] A third vehicle 1902 is traveling in lane 1920. Any object in front of or partially to the side of the skidding vehicle 1907 or vehicle 1906 is obscured or partially obscured from view from vehicle 1902.

[0242] When a skidding vehicle 1907 is detected using the sensors 1918 of the vehicle 1906, the ESOBE 1908 of the vehicle 1906 determines the measured parameters of the vehicle 1907 (assuming in this case that the OBE is not equipped). The measured parameters may include one or more of speed, driving direction, path history, path prediction, braking state, or other or a combination thereof. Based on these measured parameters, the ESOBE generates and broadcasts a virtual BSM on behalf of the unequipped vehicle 1907.

[0243] Based on the measured parameters determined by the ESOBE of vehicle 1906 and the resulting virtual BSM broadcast by the ESOBE of vehicle 1906, vehicle 1902 has become aware of the skidding vehicle 1907, even though neither the driver of vehicle 1902 nor the onboard sensors of vehicle 1902 can see vehicle 1907 because it is obscured by vehicle 106. Using this information available to the SOBE of vehicle 1902 together with the measured parameters of its own motion, including speed, direction of travel, etc., the SOBE of vehicle 1902 can predict the threat of a possible collision with vehicle 1907 and warn its driver accordingly.

[0244] In addition to making surrounding vehicles (such as vehicle 1902 in this case) more aware of unseen unconnected vehicles, the ESOBE of vehicle 1906 can generate and send an intersection collision avoidance message (ICA) if the ESOBE of vehicle 1906 detects the possibility of a corner collision between vehicle 1907 and vehicle 1902 in lane 1920 by predicting the skid path of vehicle 1907. If vehicle 1902 is capable of receiving and processing ICA messages, it can receive them, process them, and warn its driver of the possible corner (intersection) collision.

[0245] In contrast, in typical DSRC (dedicated short range communication), ICA is generated and sent only at road intersections to warn drivers of corner collisions (e.g., predicting that one vehicle will cross the path of another vehicle). In the technology we describe here, since the ESOBE is equipped with a prediction algorithm, ICA can be triggered, generated, and sent from the ESOBE of a moving vehicle even at locations other than road intersections. Such ICA can be used to warn drivers, enabling them to avoid corner collisions that may occur even on straight road sections as described above.

[0246] Obscured pedestrians: Crosswalk

[0247] In an implementation according to this scenario, the ESOBE may act as a broadcaster of a virtual PSM, for example, on behalf of pedestrians or other vulnerable road users.

[0248] like Fig. 20As shown, in this case, pedestrian 2000 is crossing a two-lane road 2012 at a crosswalk 2010. Vehicle 2006 traveling in lane 2014 obscures pedestrian 2000's view of vehicle 2002 traveling in lane 2016 (obstructs line of sight). Area 2004 depicts the perception range (visual capabilities) of the driver and sensors of vehicle 2002. Vehicle 2006 limits area 2004, and anything behind vehicle 2006 (relative to vehicle 2002) is obscured from view from vehicle 2002. If vehicle 2002 continues to move without knowledge of pedestrian 2000's location, speed, and direction of travel, a serious accident could occur.

[0249] The bus 2006 is equipped with an ESOBE 2008 and sensors 2018. The sensors 2018 may be, but are not limited to, cameras, radars, lidars, ultrasonic ranging sensors, etc., and combinations thereof. The ESOBE 2008 processes the synchronized data feeds of the sensors 2018 in real time. When the ESOBE 2008 detects a pedestrian 2000 using the sensor data, the ESOBE 2008 automatically begins broadcasting virtual PSM messages on behalf of the pedestrian 2000 that can be received by the vehicle 2002. As a result, the vehicle 2002 is aware of the pedestrian 2000 that is crossing the lane 2014, even though the driver of the vehicle 2002 and the onboard sensors cannot see the pedestrian 2000 on the other side of the vehicle 2006.

[0250] Occluded pedestrians: mid-block intersection

[0251] This scenario is similar to the previous one, except here the pedestrian is crossing the street mid-block away from a crosswalk or other formal road intersection.

[0252] like Fig.21 As shown, pedestrian 2100 is crossing road 2112 in front of vehicle 2106. Pedestrian 2100 is obscured from the field of view of the driver and sensors of vehicle 2102. The perception area of ​​vehicle 2102 is depicted by dashed area 2104.

[0253] The vehicle 2106 is equipped with an ESOBE 2108 capable of sending and receiving V2X messages such as PSM and BSM. The ESOBE 2108 is also capable of processing synchronized data feeds from sensors 2118.

[0254] Once detected by ESOBE 2108, information including but not limited to the global position, speed, and direction of travel of pedestrian 2100 is encoded into a virtual PSM message, which is broadcast. Vehicle 2102 receives the virtual PSM message and then becomes aware of pedestrian 2100. Algorithms onboard vehicle 2102 are able to predict whether there is an imminent hazardous situation and take appropriate action, including but not limited to warning the driver or slowing down or automatically stopping the vehicle.

[0255] Forward collision with obscured pedestrian

[0256] Fig. 22 A pedestrian 2400 is depicted crossing a two-lane road 2412 at a crosswalk 2410. A larger vehicle (e.g., a truck) 2406 is traveling in lane 2414 and carries an ESOBE 2408 and a sensor 2418. The sensor 2418 may be, but is not limited to, a camera, radar, lidar, ultrasonic ranging sensor, etc., and combinations thereof. The ESOBE 2408 processes the data feed from the sensor in real time.

[0257] Another vehicle 2402 is traveling behind a larger vehicle 2406 in the same lane 2414. Pedestrian 2400 or anything else in front of vehicle 2406 is obscured from the view of vehicle 2402.

[0258] If vehicle 2406 must brake suddenly due to the presence of pedestrian 2400 (unbeknownst to vehicle 2402 ), vehicle 2402 may collide with the rear end of vehicle 2406 .

[0259] When pedestrian 2400 is detected using sensor 2418, in addition to the regular basic safety messages (BSMs) that vehicle 2406's ESOBE 2408 sends on its own behalf, it immediately begins broadcasting a virtual PSM on behalf of pedestrian 2400. As a result, vehicle 2402 is aware of pedestrian 2400 (from the virtual PSM) crossing lane 2414, along with knowledge of vehicle 2406's presence (from the BSM), even though neither the driver nor the onboard sensors of vehicle 2402 can see or sense the overtaking vehicle 2406.

[0260] The ESOBE in vehicle 2406 performs artificial intelligence processing that learns from the behavior of its driver in various situations. For example, upon detecting pedestrian 2400, ESOBE 2408 applies an AI algorithm to predict whether the driver of vehicle 2406 is about to apply the brakes and predict the future time when this may occur. Based on these predictions, the ESOBE may decide to add braking information in a BSM message that is broadcast to other vehicles more quickly than would otherwise be possible in a typical system. For example, this earlier delivery of a braking message may give other vehicles, such as vehicle 2402, more time to predict a collision ahead. In other words, other vehicles can benefit from the AI ​​capabilities in the first vehicle's ESOBE.

[0261] ESOBE-Position Correction Service

[0262] The availability of high-precision GPS position data is important for DSRC-based V2X safety applications or other applications that expect or require higher than typical GPS accuracy. ESOBE can provide GPS correction information to improve the accuracy of GPS position data for nearby ground transportation entities. Fig.23 As shown, four vehicles (2502, 2503, 2504, and 2505) traveling in different lanes of a road in a given direction may be able to use GPS correction data, for example, in the scenario described below, etc.

[0263] In this case, a vehicle (e.g., a truck or bus) 2506 is traveling in a lane 2516 and carrying an ESOBE 2508 capable of sending and receiving V2X messages. Although vehicle 2506 is depicted as a large vehicle, it can be any ground transportation entity of any size.

[0264] Assume that (a) no RSEs are present in the vicinity of the area, (b) vehicle 2506 (including its ESOBE) has recently passed through an area in which an RSE having differential GNSS (Global Navigation Satellite System) broadcast capability over DSRC (using technology such as RTK, DGPS, or wide-area RTK), (c) vehicle 2506 is no longer in the vicinity of that RSE, and (d) vehicle 2506 (and its ESOBE) has passed through an area in which a particular RSE is present and that particular RSE is transmitting periodic RTCM (Radio Technical Commission for Maritime) correction messages.

[0265] Upon receiving these RTCM correction messages, the ESOBE 2508 on the vehicle 2506 corrects its own position and also stores the RTCM correction data for later use. When a particular RSE is within range, the ESOBE continues to use the received correction messages to correct its position and update these messages for later use.

[0266] Once the ESOBE is outside the coverage area of ​​a particular RSE, the ESOBE uses the stored correction data to construct a more recent correction message based on its current position and sends it to other vehicles ( Fig.23 2502, 2503, 2504, 2505) in the figure provide a correction service similar to the correction service that would be provided by the RSE if the RSE was within range. The distance of the ESOBE to the specific RSE from which the correction data has been collected and the current position information of the ESOBE can be used to update the correction message (e.g., create a newer correction message) before rebroadcasting the correction message as part of the correction service. In this way, the ESOBE can effectively act as a base station to provide accurate RTCM correction data to other road users who do not have an RSE in range. Therefore, the ESOBE not only forwards correction data from the RSE or other external source, but also updates the correction data based on the current position of the host vehicle and other factors mentioned herein.

[0267] In addition, before broadcasting the correction data, ESOBE runs an algorithm to determine the correctness of the reconstructed correction data. Only when the algorithm confirms that the confidence of the regenerated correction data is very high, the correction data is broadcast.

[0268] In some implementations, the algorithm for retransmitting correction data considers, among other information:

[0269] The time elapsed since the ESOBE received the correction data from the RSE, the distance travelled from the RSE, the direction of travel (driving direction) to determine if the correction still applies to the current vehicle position, and the confidence level of the original correction data received from the RSE, and combinations thereof.

[0270] Another feature of ESOBE in some examples is that if it has direct access over the Internet to an external service that transmits an RTCM correction feed, it can use that feed to generate DSRC RTCM correction messages on its own. In situations where there is no RSE present or the confidence level of the data received from the RSE is insufficient, ESOBE can decide to switch to this mode. Here, ESOBE has the intelligence to select a better source of correction information.

[0271] The benefits of ESOBE being able to construct correction messages on its own or reconstruct correction messages from correction messages received from RSE and send them are: helping ground transportation entities with low-end, inexpensive GPS devices correct their positions; helping ground transportation entities to more reliably execute safety algorithms; expanding the effective coverage of RSE; and broadcasting RTCM corrections on DSRC networks or other short-range vehicular networks using V2X standard correction messages.

[0272] Other implementations are within the scope of the following claims.

Claims

1. An enhanced vehicle-mounted device, comprising: Equipment used in the first ground transportation entity vehicle, The equipment has (a) a receiver for information generated by one or more sensors of the environment of the first ground transportation entity, (b) processors, and (c) a memory storing instructions executable by the processor to perform operations comprising: detecting a second ground transportation entity and a third ground transportation entity in proximity to the first ground transportation entity based at least on the information generated by the one or more sensors; generating, based on detecting the second ground transportation entity, (i) a prediction of a future change in movement of the first ground transportation entity, and (ii) a prediction of a future time at which the predicted future change in movement will occur; Based on detecting the third ground transportation entity, determining a motion parameter of the third ground transportation entity and predicting a collision between the third ground transportation entity and the second ground transportation entity; generating a safety message, the safety message comprising a prediction of the future motion change and a prediction of the future time; and The safety message is sent to a third ground transportation entity.

2. The enhanced vehicle-mounted device according to claim 1, characterized in that: Generating the prediction of the future change of motion and the prediction of the future time includes applying an artificial intelligence model trained to predict the behavior of an operator.

3. The enhanced vehicle-mounted device according to claim 2, characterized in that: The artificial intelligence model is trained on operators of the first ground transportation entity.

4. The enhanced vehicle-mounted device according to claim 1, characterized in that: The future change of movement includes braking.

5. The enhanced vehicle-mounted device according to claim 1, characterized in that: The prediction of the future change in movement includes that the future change in movement will be initiated by an operator of the first ground transportation entity.

6. The enhanced vehicle-mounted device according to claim 1, characterized in that: The safety message includes a basic safety message.

7. The enhanced vehicle-mounted device according to claim 1, characterized in that: The third ground transportation entity is following the first ground transportation entity, and the second ground transportation entity's view from the third ground transportation entity is blocked.

8. The enhanced vehicle-mounted device according to claim 1, characterized in that: The safety message includes movement parameters of the second ground transportation entity.

9. The enhanced vehicle-mounted device according to claim 1, characterized in that: The second ground transportation entity is a disconnected ground transportation entity.

Images