RF and optical message association

Abstract

The present disclosure provides "RF and optical message association". Data fusion of radio frequency (RF) data and visual data is provided. A vehicle-to-everything (V2X) message is received from a remote road entity via a transceiver of a vehicle, the V2X message including one or more V2X data elements describing the remote road entity. Modulated light data is captured within a sensing area around a self-aware road entity using a camera of the vehicle, the modulated light data being transmitted from the remote road entity. A modulated light transmission in the modulated light data is decoded to identify one or more light data elements in the modulated light data that describe the road entity. The association of the one or more V2X data elements with the one or more lamp data elements is performed to identify an associated V2X camera object, where the V2X message and the modulated light data are fused into a combined representation of the remote road entity.

Description

Technical Field

[0001] Aspects of the present disclosure generally relate to using optical signaling to associate camera objects with vehicle-to-vehicle (V2V) radio frequency (RF) messages. Background Art

[0002] Vehicle-to-everything (V2X) allows vehicles to exchange information with other vehicles as well as with infrastructure, pedestrians, networks and other devices. Vehicle-to-infrastructure (V2I) communications enable applications to facilitate and accelerate communications or transactions between vehicles and infrastructure. In a vehicle telematics system, the Telematics Control Unit (TCU) can be used for various remote control services such as over-the-air (OTA) software downloads, emergency calls and turn-by-turn navigation. Summary of the invention

[0003] In one or more illustrative embodiments, a self-aware road entity for performing data fusion of RF data and visual data includes: a transceiver; a camera; and one or more controllers of the self-aware road entity. The one or more controllers are configured to: receive a V2X message from a remote road entity via the transceiver, the V2X message including one or more V2X data elements describing the remote road entity; capture modulated light data within a sensing area around the self-aware road entity using the camera, the modulated light data being sent from the remote road entity; decode the modulated light transmission in the modulated light data to identify one or more light data elements in the modulated light data describing the remote road entity; perform association of the one or more V2X data elements with the one or more light data elements to identify an associated V2X camera object, wherein the V2X message and the modulated light data refer to the same remote road entity, wherein the V2X data elements and the modulated light data are fused into a combined representation of the remote road entity; and process the combined representation of the remote road entity based on the association.

[0004] In one or more illustrative examples, a method for data fusion of RF data and visual data is performed. A V2X message is received from a road entity via a transceiver, the V2X message including one or more V2X data elements describing the road entity. Using a camera, modulated light data is captured within a sensing area around a vehicle, the modulated light data being transmitted from the road entity. Modulated light transmissions in the modulated light data are decoded to identify one or more light data elements in the modulated light data describing the road entity. An association of the one or more V2X data elements with the one or more light data elements is performed to identify an associated V2X camera object, wherein the V2X message and the modulated light data refer to the same road entity, wherein the V2X data elements and the modulated light data are fused into a combined representation of the road entity. The combined representation of the road entity is processed based on the association.

[0005] In one or more illustrative examples, a non-transitory computer-readable medium includes instructions for performing data fusion of RF and visual data, which, when executed by one or more controllers of a self-aware road entity, cause the self-aware road entity to perform operations including: receiving a V2X message from a remote road entity via a transceiver of the self-aware road entity, the V2X message including one or more V2X data elements describing the remote road entity; capturing modulated light data within a sensing area around the self-aware road entity using a camera of the self-aware road entity, the modulated light data being sent from the remote road entity; decoding modulated light transmissions in the modulated light data to identify one or more optical data elements in the modulated light data describing the remote road entity; performing an association of the one or more V2X data elements with the one or more optical data elements to identify an associated V2X camera object, wherein the V2X message and the modulated light data refer to the same road entity, wherein the one or more V2X data elements and the modulated light data are fused into a combined representation of the remote road entity; and processing the combined representation of the remote road entity based on the association. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 An example system for performing data fusion using optical signaling to associate objects detected in captured images with received V2X messages is shown;

[0007] Figure 2 An example of a vehicle receiving a V2X message and capturing sensor data is shown;

[0008] Figure 3 An example of details of processing these V2X messages and sensor data by a data fusion application is shown;

[0009] Figure 4 An example process for performing data fusion using optical signaling to associate objects detected in captured images with received V2X messages is shown;

[0010] Figure 5 shows an example process for training a machine learning model given data collected from a vehicle; and

[0011] Figure 6 An example of a computing device for performing collaborative sensor sharing among traffic participants is shown. DETAILED DESCRIPTION

[0012] As required, detailed embodiments of the present disclosure are disclosed herein; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be implemented in various and alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or minimized to show details of particular components. Therefore, the specific structural details and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the present disclosure in different ways.

[0013] Vehicles can receive situational awareness information from multiple sources. As some examples, these sources can include RF messages via V2X communications, as well as sensor information captured using cameras, radio detection and ranging (radar), and light detection and ranging (lidar).

[0014] The captured sensor information may show the vehicle or other object being tracked. Furthermore, the V2X communication messages may include information such as vehicle heading, location, and size. Due to this overlap, the V2X information may be used to associate the vehicle captured in the camera image with the V2X message. The accuracy of this association may depend on factors such as lighting conditions, the correctness of the location information in the V2X message, object detection confidence, and the like. In some examples, multiple camera frames or data from other sensors (such as radar) may be required to calculate higher order derivatives of the position of the target vehicle (e.g., velocity, rate of change of velocity). These calculations may be inaccurate due to errors in the location information in the V2X message and partial visibility of the vehicle in the image, as well as other issues.

[0015] To address these issues, methods can be implemented using modulated optical communications. These optical communications can be performed between a headlight as a transmitter and a camera system of another vehicle as a receiver. As some examples, the headlights can include the headlights and / or taillights of the vehicle. The light-based communication channel can be used to exchange information to assist in the data fusion process. For example, the headlights can be modulated with signal information such as an identifier (ID), object type, location, speed, heading, etc. used in a V2X message. The camera system of an adjacent vehicle can capture the optical communication and use the ID used to associate the vehicle with a V2X message received via the V2X interface. For example, a vehicle camera can scan for light sources and can decode the modulated information from any detected light source. The receiving vehicle can associate the ID and object type in the signaled data with the same field in the V2X message to perform the association. By fusing data using this method, a more accurate association can be made between visual data and RF data.

[0016] Figure 1 An example system 100 for using optical signaling to associate an object detected in a captured image with a received V2X message is shown. The system 100 may include a vehicle 102 having a controller 104 and a camera 106 that communicate via one or more vehicle buses 108. The vehicle 102 may also include a TCU 110 configured to provide communication services via a transceiver 112. The TCU 110 may include a processor 114 and a storage device 116. The vehicle 102 may also include external lights 118 configured to modulate signals through light-based communications. The TCU 110 may be used to send and receive V2X messages 120. The camera 106 may be configured to capture such modulated light data 122. The V2X message 120 and the modulated light data 122 may relate to a road entity 124 in a road 126, a roadside unit (RSU) 130, or other objects in the environment surrounding the vehicle 102 (e.g., road signs, smart and connected infrastructure, etc.). The data fusion application 132 may be installed to the TCU 110 and may be configured to correlate the ID and object type information in the modulated light data 122 with the corresponding information in the V2X message 120 to build an accurate correlation between the visual data and the RF data. It should be noted that the system 100 is merely an example and that a system 100 having more, fewer, and different components may be used.

[0017] Vehicle 102 may include various types of automobiles, crossover utility vehicles (CUVs), sport utility vehicles (SUVs), trucks, recreational vehicles, motorcycles, boats, airplanes, or other mobile machines for transporting people or goods. Such vehicles 102 may be human-driven or autonomous. In many cases, vehicle 102 may be powered by an internal combustion engine. As another possibility, vehicle 102 may be a battery electric vehicle powered by one or more electric motors. As another possibility, vehicle 102 may be a hybrid electric vehicle powered by both an internal combustion engine and one or more electric motors, such as a series hybrid electric vehicle, a parallel hybrid electric vehicle, or a parallel / series hybrid electric vehicle.

[0018] Vehicle 102 may be a vehicle with driver assistance features driven by a driver. In other examples, the vehicle may be a semi-autonomous vehicle (AV). These AV or driver assistance features may be supported via received V2X data and / or optical data. The level of automation may vary between different levels of driver assistance technology and fully automated unmanned vehicles. Since the type and configuration of vehicle 102 may vary, the capabilities of vehicle 102 may vary accordingly. As some other possibilities, vehicle 102 may have different capabilities in terms of passenger capacity, traction capacity and capacity, and storage. For ownership, inventory, and other purposes, vehicle 102 may be associated with a unique identifier, such as a vehicle identification number (VIN). It should be noted that although motor vehicle 102 is being used as an example of a traffic participant, other types of traffic participants, such as bicycles, scooters, and pedestrians, which may be equipped with V2X technology, may be used in addition or alternatively.

[0019] The vehicle 102 may include multiple controllers 104 configured to perform and manage various vehicle 102 functions under the power of the vehicle battery and / or drive train. As shown, the example vehicle controllers 104 are represented as discrete controllers 104 (i.e., controllers 104A to 104G). However, the vehicle controllers 104 may share physical hardware, firmware, and / or software, such that functionality from multiple controllers 104 may be integrated into a single controller 104, and the functionality of various such controllers 104 may be distributed across multiple controllers 104.

[0020] As some non-limiting examples of vehicle controllers 104: the powertrain controller 104A may be configured to provide control of engine operating components (e.g., idle control components, fuel delivery components, emission control components, etc.), and to monitor the status of such engine operating components (e.g., the status of engine codes); the body controller 104B may be configured to manage various power control functions, such as exterior lights 118, interior lighting, keyless entry, remote starting, and access point status verification (e.g., the closed status of the hood, doors, and / or trunk of the vehicle 102); the radio transceiver controller 104C may be configured to communicate with a key fob, mobile device, or other local The vehicle 102 device communicates; the autonomous controller 104D may be configured to provide commands to control the powertrain, steering, or other aspects of the vehicle 102; the climate control management controller 104E may be configured to provide control of heating and cooling system components (e.g., compressor clutch, blower fan, temperature sensors, etc.); the global positioning system (GPS) controller 104F may be configured to provide vehicle location information; and the human-machine interface (HMI) controller 104G may be configured to receive user input via various buttons or other controls, and provide vehicle status information to the driver, such as fuel level information, engine operating temperature information, and the current location of the vehicle 102.

[0021] The controller 104 of the vehicle 102 may utilize various cameras 106 in order to receive information about the surrounding environment of the vehicle 102. In an example, these cameras 106 may include one or more of a visible light camera 106 (e.g., an advanced driver assistance system (ADAS) camera 106), an infrared camera 106, and / or a multi-spectral camera 106. In some examples, the camera 106 may include a capture device that operates using other methods, such as a radar system and / or a lidar system. In some examples, the camera 106 may include a rolling shutter complementary metal oxide semiconductor (CMOS) camera 106 configured to receive light-based transmissions.

[0022] The vehicle bus 108 may include various communication methods available between the vehicle controllers 104 and between the TCU 110 and the vehicle controllers 104. As some non-limiting examples, the vehicle bus 108 may include one or more of a controller area network (CAN), an Ethernet network, and a media oriented systems transport (MOST) network.

[0023] The TCU 110 may include network hardware configured to facilitate communication between the vehicle controllers 104 and with other devices of the system 100. For example, the TCU 110 may include or otherwise have access to a transceiver 112 configured to facilitate communication with other vehicles 102 or with infrastructure. The TCU 110 may be configured to communicate via a broadcast peer-to-peer protocol such as PC5 to facilitate cellular V2X (C-V2X) communications with devices such as other vehicles 102. The TCU 110 may also be configured to communicate via a cellular network to communicate with other devices over the Internet. It should be noted that these protocols are examples only and different peer-to-peer and / or cellular technologies may be used.

[0024] The TCU 110 may include various types of computing devices that support the execution of the functions of the TCU 110 described herein. In an example, the TCU 110 may include one or more processors 114 configured to execute computer instructions, and a storage device 116 medium on which computer executable instructions and / or data may be maintained. Computer-readable storage media (also referred to as processor-readable media or storage devices 116) include any non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor). Typically, the processor 114 receives instructions and / or data, such as from a storage device 116, etc. to a memory, and uses the data to execute instructions, thereby performing one or more processes, including one or more of the processes described herein. Computer executable instructions may be compiled or interpreted according to a computer program created using a variety of programming languages ​​and / or technologies, which include, but are not limited to, JAVA, C, C++, C#, FORTRAN, PASCAL, VISUAL BASIC, PYTHON, JAVASCRIPT, PERL, etc., individually or in combination.

[0025] The TCU 110 may be configured to facilitate the collection of V2X messages 120 and / or other vehicle information from a vehicle controller 104 connected to one or more vehicle buses 108. The TCU 110 may also be configured to facilitate the collection of modulated light data 122 captured by a camera 106 of the vehicle 102. Examples of modulated light data 122 may include information captured in visible light transmissions, infrared light transmissions, ultraviolet light transmissions, multi-frequency modulated light transmissions, and the like.

[0026] The V2X message 120 may include collected information retrieved from the controller 104 via the vehicle bus 108. In many examples, the collected information data may include information useful for autonomous vehicle operation or driver-assisted vehicle operation. As some non-limiting examples, the networked vehicle data information retrieved by the TCU 110 may include the sender's ID and data about the sender, such as latitude, longitude, time, heading angle, speed, lateral speed change, longitudinal speed change, yaw rate, throttle position, brake status, steering angle, headlight status, wiper status, outside temperature, turn signal status, vehicle length, vehicle width, vehicle mass, and bumper height. The networked vehicle data information may also include weather data (such as ambient temperature, ambient air pressure, etc.), traction control status, wiper status, or other vehicle status information (such as the status of external vehicle lights, vehicle type, anti-lock braking system (ABS) system status, etc.).

[0027] The V2X message 120 may also include information about the detected road entity 124. For example, the V2X message 120 may indicate information such as the location, size, and / or profile of the detected road entity 124 along the road 126. The V2X message 120 may also indicate the type of the road entity 124 (e.g., a pedestrian, a car, a truck, debris, etc.).

[0028] In some examples, a machine learning model 128 may be executed to classify the road entity 124 into a plurality of object classifications, and the V2X message 120 may indicate the most likely type that the road entity 124 is determined to be (or in other examples, the road entity 124 is various types of road entities 124). The machine learning model 128 may be executed by various devices, such as one or more controllers 104 of the vehicle 102, the TCU 110, the RSU 130, etc. It should be noted that from the perspective, the vehicle 102 may be a self-aware road entity 124 of other vehicles 102. Therefore, various aspects discussed with respect to the vehicle 102 may be performed by the road entity 124 from its perspective, and vice versa. In addition, operations discussed as being performed by the vehicle 102 as a self-aware road entity 124 may also be performed by other types of road entities 124 (such as by the RSU 130). The road entity 124 may then be equipped with V2X and / or light modulation capabilities. Messages from V2X or light modulation may include details about the road entity 124, which in turn may be used to enhance the classification capabilities of the machine learning model 128 (e.g., the likelihood of classifying the road entity 124 as one of a plurality of object classifications and / or each classification).

[0029] In some examples, the road entity 124 may additionally involve communication via one or more RSUs 130. The RSU 130 may be a device with processing and networking capabilities and may be designed to be placed near the road 126 for communication with the vehicle 102. For example, the RSU 130 may include hardware configured to communicate via a broadcast peer-to-peer protocol (such as PC5) to facilitate C-V2X communication with the vehicle 102. In another example, the machine learning model 128 may be hosted by the RSU 130 to allow the road entity 124 to be classified without using the computing power of the vehicle 102. The RSU 130 may accordingly be able to communicate with multiple vehicles 102 along a specific road 126 or in a specific area. The RSU 130 may also have a wired or wireless backhaul capability to allow communication with other elements of the traffic control system, such as via a Uu interface, such as Ethernet or a cellular connection to a cellular network infrastructure.

[0030] While a single vehicle bus 108 is shown, it should be noted that in many examples, multiple vehicle buses 108 are included, with a subset of the controllers 104 connected to each vehicle bus 108. Thus, to access a given controller 104, the TCU 110 may be configured to maintain a mapping of which vehicle buses 108 are connected to which controllers 104, and access the corresponding vehicle bus 108 for that particular controller 104 when communication with the controller 104 is desired.

[0031] The TCU 110 may also be configured to periodically transmit V2X messages 120 for reception by other vehicles 102. The location, size, and heading of the vehicle 102 may be broadcast by the vehicle 102 in the V2X message 120. In an example, the frequency of sending the V2X message 120 may be approximately once every ten milliseconds. The V2X radio may be a relatively short-range communication unit (e.g., having a range of approximately one kilometer). In one example, the V2X radio may operate on a cellular V2X (e.g., C-V2X 3GPP). In another example, the V2X radio may operate on an Institute of Electrical and Electronics Engineers (IEEE) 802.11p dedicated short-range communication (DSRC). In one example, the V2X message 120 may take the form of a BSM message as described in the Society of Automotive Engineers (SAE) standard document J2735. In another example, the V2X message 120 may take the form of a sensor data sharing message (SDSM) message as described in the SAE standard document J3224.

[0032] The TCU 110 may also be configured to receive V2X messages 120 from other vehicles 102 or other road entities 124. In an example, the vehicle 102 may be referred to as a host vehicle (HV), and remote vehicles (RVs) around the HV may communicate with the vehicle 102 as road entities 124. In another example, the HV may communicate with other road entities 124, such as bicyclists, pedestrians, etc.

[0033] The body controller 104B may be configured to modulate information using external lights 118. Similar to the reception of the V2X message 120, the information may be collected as modulated light data 122 sent by other road entities 124. Visible light communication (VLC) such as LiFi may be used to send the information. LiFi is a wireless communication technology that uses light to transmit data and location between devices. In an example, the body controller 104B may use LiFi or other types of VLC to send information using external lights 118 to be sensed by the camera 106 of the other vehicle 102. In some examples, the modulated light data 122 may include the same data elements as those included in the V2X message 120. In other examples, the modulated light data 122 may include more, fewer, or different data elements included in the V2X message 120. However, at least some overlap of data elements (e.g., location, ID, etc.) may be provided between the V2X message 120 and the modulated light data 122 to allow the recipient to associate the information.

[0034] Management of the transmission, reception, and filtering of the V2X messages 120 may be handled by a data fusion application 132 executed by the TCU 110. The data fusion application 132 may also be configured to receive the modulated light data 122 from the camera 106 and fuse the modulated light data 122 with the V2X messages 120 to construct an overall view of the road 126. The fused representation may include, for example, data elements from the V2X messages 120 and / or the modulated light data 122, as well as image data showing specific road entities 124 to which the data relates. Figures 2 to 5 These aspects are discussed in detail.

[0035] Figure 2 An example 200 is shown of a vehicle 102 receiving a V2X message 120 and capturing modulated light data 122. The sensing area 202 may be defined by the sensing capabilities of a camera 106 mounted to the vehicle 102. As shown, the modulated light data 122 is captured within the sensing area 202 of the forward-facing camera 106 of the vehicle 102. It should be noted that this is merely an example, and the vehicle 102 may have additional sensing areas 202 around other portions of the vehicle 102.

[0036] Also shown is that the vehicle 102 is receiving V2X messages 120 from the first vehicle road entity 124A and from the second vehicle road entity 124B. These V2X messages 120 may be sent by the first vehicle road entity 124A and the second vehicle road entity 124B, respectively. In addition, the vehicle 102 receives modulated light data 122 from the first vehicle road entity 124A and the second vehicle road entity 124B. This information may be modulated into the operation of the exterior lights 118 of the first vehicle road entity 124A and the second vehicle road entity 124B.

[0037] Vehicles 102 equipped with C-V2X functionality may periodically broadcast V2X messages 120. The transmission frequency of the V2X messages 120 may depend on factors such as the density of vehicles 102, and may be in the range of 10 Hz to about 2 Hz. The V2X messages 120 may be received even when the vehicles 102 or road entities 124 are non-line-of-sight (NLOS). Thus, the V2X messages 120 make available information about objects that are obscured from the sensing area 202 of the camera 106 (or not visible due to low-light conditions).

[0038] The V2X message 120 may contain an ID that changes periodically to ensure that the road entity 124 is not tracked by other vehicles 102 or other C-V2X capable devices. The V2X message 120 may include additional information about the sender, such as universal time (UTC), latitude, longitude, heading, absolute speed, length, width, path history, and speed.

[0039] The modulated light data 122 may include information modulated on one or more optical frequencies. Compared to the V2X message 120, the modulated light data 122 is by definition line of sight (LOS). This modulation may be imperceptible to the human eye, but can be deciphered by the camera 106. The camera 106 may also be configured to monitor the environment in conjunction with the controller 104, detect road entities 124, estimate speed, heading and / or other information about the detected road entities 124, and provide timely alerts about potential object interactions. The modulated light data 122 may also include the same ID that changes periodically. This may allow the recipient to confirm that the modulated light data 122 is related to the V2X message 120. The modulated light data 122 may include additional information about the sender, such as UTC, latitude, longitude, heading, absolute speed, length, width, path history, and speed.

[0040] Figure 3An example 300 of details of processing these V2X messages 120 and modulated light data 122 by the data fusion application 132 is shown. As shown, the data fusion application 132 receives the V2X message 120 from the transceiver 112 and also receives the modulated light data 122 from the camera 106. The data fusion application 132 may include a coordinate transformer 302, a data demodulator 304, a data synchronizer 306, a buffer 308, and a V2X / camera object associator 310.

[0041] The coordinate transformer 302 may receive the V2X message 120. Based on the V2X message 120, the coordinate transformer 302 may map the V2X message 120 and the modulated light data 122 to a common coordinate system. This may include, for example, placing the V2X message 120 as an overlay on the captured modulated light data 122 based on the location information included in the V2X message 120.

[0042] The data demodulator 304 may be configured to decode any modulated information included in the modulated light data 122. As described above, the time-varying modulated light data 122 may include modulated VLC transmitted using the exterior lights 118 of the road entity 124. This information may be identified using LiFi and / or other VLC technologies to retrieve the modulated data stream. The modulated data may be decoded to augment the modulated light data 122 (and / or the identified road entity 124 in the modulated light data 122) with decoded information such as ID, object type, heading, speed, location, and / or any information decoded from the modulated stream of the modulated light data 122. Notably, the V2X message 120 and the modulated light data 122 may utilize the same ID to improve the accuracy of data fusion.

[0043] Similar to how the coordinate transformer 302 aligns the V2X message 120 and the modulated light data 122, the data synchronizer 306 may be configured to align the V2X message 120 and the modulated light data 122 in time. This may be performed by ordering the V2X message 120 based on the included timestamp and / or reception time in conjunction with the capture time of the modulated light data 122. The results of the synchronization may be provided to the buffer 308 for processing by the V2X / camera object associator 310.

[0044] The V2X / camera object associator 310 may be configured to associate the V2X message 120 in the buffer 308 with the modulated light data 122 in the buffer 308. For example, this may be performed by matching the positions of the V2X message 120 and the modulated light data 122 determined by the coordinate transformer 302, such as by mapping the position information from the V2X message 120 and the position of the road entity 124 collected from the modulated light data 122. The mapping may include, for example, associating the V2X message 120 and the modulated light data 122 based on a common ID for the two data streams. Alternatively, in other examples, the association may also include associating based on other information fields, such as position, speed, heading, object type, etc.

[0045] In some examples, the V2X / camera object associator 310 may also be configured to utilize the machine learning model 128 to perform object detection in the modulated light data 122, or in other examples, receiving object detection information may be performed by the machine learning model 128 from an upstream object pre-processor. In one example, the machine learning model 128 may be performed by a device external to the vehicle 102 through communication with the TCU 110. In an example, the TCU 110 may send data to the RSU 130 or a mobile edge computing (MEC) device or server in communication with the vehicle 102 for processing.

[0046] For example, object detection may be performed to identify the location of road entities 124 in the image captured by camera 106. In some examples, this may allow data demodulator 304 to focus demodulation on areas of modulated light data 122 that contain vehicle road entities 124. Additionally, this may allow V2X messages 120 to be associated with specific objects in the image captured by camera 106. In some examples, the combined representation may be provided to machine learning model 128 to help associate the image captured by camera 106 with remote road entities 124.

[0047] The result of this association may result in one of three possibilities: an associated V2X camera object 312, an unassociated V2X object 314, and an unassociated camera object 316. An associated V2X camera object 312 refers to an object in which a V2X message 120 and modulated light data 122 are fused into a combined representation. The combined representation may include, for example, data elements from a V2X message 120 and / or modulated light data 122, as well as image data showing a specific road entity 124 to which the data is associated. An unassociated V2X object 314 refers to an object for which a V2X message 120 is received without associated modulated light data 122. An unassociated camera object 316 refers to an object for which a road entity 124 in the modulated light data 122 is identified without an associated V2X message 120. In some examples, it may be inferred that the unassociated V2X object 314 is NLoS, and it may be inferred that the associated V2X camera object 312 and the unassociated camera object 316 are LoS.

[0048] The associated V2X camera object 312, the unassociated V2X object 314, and the unassociated camera object 316 may be provided for various downstream tasks. In an example, the associated V2X camera object 312, the unassociated V2X object 314, and the unassociated camera object 316 may be provided to a road object processing 318 feature, such as driver alerts, autonomous or semi-autonomous driving, and the like.

[0049] In another aspect, the unassociated camera objects 316 may be provided to the SDSM generator 320. The SDSM generator 320 may be configured to generate the V2X messages 120 based on the vehicle 102's observation of the corresponding unassociated camera objects 316. By sending the V2X messages 120, the vehicle 102 may be able to inform other road entities 124 of aspects of the detected unassociated camera objects 316.

[0050] In some examples, this may be done only for unassociated camera objects 316 to alert roadway entities 124 of detections that were not received by vehicle 102 in V2X messages 120 , as these objects are likely unknown to other roadway entities 124 .

[0051] In other examples, the SDSM generator 320 may generate the V2X message 120 for all associated V2X camera objects 312, unassociated V2X objects 314, and unassociated camera objects 316. In yet another example, the SDSM generator 320 may generate the V2X message 120 for all associated V2X camera objects 312, unassociated V2X objects 314, and unassociated camera objects 316 if the V2X RF congestion is below a threshold, but only generate the message for the unassociated camera object 316 if the V2X RF congestion exceeds the threshold.

[0052] Figure 4 An example process 400 is shown for performing data fusion using optical signaling to associate objects detected in captured images with received V2X messages 120. In an example, in the context of the system 100, the process 400 may be performed by a data fusion application 132 executed by a TCU 110 of a vehicle 102. It should be noted that this is only one possibility, and other types of self-aware road entities 124 (such as RSU 130, MEC, remote server, etc.) may perform one or more aspects of the process 400.

[0053] At operation 402, the TCU 110 receives a V2X message 120. In an example, the TCU 110 may utilize the transceiver 112 to receive the V2X message 120 from other vehicles 102 or other road entities 124 that are configured to periodically transmit the V2X message 120. The transmission frequency of the V2X message 120 may depend on factors such as the density of the vehicles 102, and may be in the range of 10 Hz to about 2 Hz. The V2X message 120 may be received by the TCU 110 even when the transmitting vehicle 102 or the road entity 124 is NLOS.

[0054] At operation 404, the TCU 110 receives the modulated light data 122. In an example, the TCU 110 may receive the modulated light data 122 captured by the camera 106 of the vehicle 102. For example, the camera 106 may include a rolling shutter CMOS camera 106 configured to receive light-based transmissions. These light-based transmissions may be sent as LiFi or other forms of VLC from other vehicles 102 using their exterior lights 118 and / or from other exterior lights 118 of various road entities 124.

[0055] At operation 406, the TCU 110 optionally identifies the road entity 124 in the modulated light data 122. In an example, the V2X / camera object correlator 310 may perform object detection in the modulated light data 122. In other examples, the V2X / camera object correlator 310 may receive object detection information from another device, such as the RSU 130. Object detection may be performed, for example, to identify the location of the road entity 124 in the modulated light data 122.

[0056] At operation 408, the TCU 110 decodes the light transmission received in the modulated light data 122. In an example, the data demodulator 304 may decode the modulation information included in the modulated light data 122 received by the camera 106. In some examples, if operation 406 is performed, the data demodulator 304 may focus demodulation on an area of ​​the modulated light data 122 that includes the vehicle road entity 124.

[0057] At operation 410, the TCU 110 performs object association between the V2X message 120 and the modulated light data 122. In an example, the V2X / camera object associator 310 may associate the V2X message 120 by matching information from the V2X message 120 with information determined based on the optical transmission of the modulated light data 122. This may include, for example, associating the V2X message 120 and the modulated light data 122 based on a common ID for the two data streams. Alternatively, in other examples, the association may also include association based on other information fields such as position, speed, heading, object type, etc. The result of this association may result in one of the following three possibilities: an associated V2X camera object 312, an unassociated V2X object 314, and an unassociated camera object 316.

[0058] At operation 412, the TCU 110 processes the objects based on the association. For example, the associated V2X camera object 312, the unassociated V2X object 314, and the unassociated camera object 316 may be provided to road object processing 318 features, such as driver alerts, autonomous or semi-autonomous driving, etc. Importantly, by fusing the data using the information modulated in the modulated light data 122, a more accurate and less processor intensive association may be made between the visual data and the RF data than with purely image-based recognition techniques.

[0059] At operation 414 , the TCU 110 optionally transmits the V2X messages 120 using the SDSM generator 320 . The SDSM generator 320 may be configured to generate the V2X messages 120 based on the vehicle 102 's observations of the respective unassociated camera objects 316 . These generated V2X messages 120 may be broadcast by the transceiver 112 .

[0060] In other examples, the SDSM generator 320 may generate the V2X message 120 for all associated V2X camera objects 312, unassociated V2X objects 314, and unassociated camera objects 316 (or may generate the message for all associated V2X camera objects 312, unassociated V2X objects 314, and unassociated camera objects 316 if V2X RF congestion is below a threshold, and only generate the message for the unassociated camera objects 316 if V2X RF congestion exceeds the threshold).

[0061] After operation 414, process 400 ends. It should be noted that in many examples, the operations of process 400 can be performed continuously in a loop. It should also be noted that one or more of the operations of process 400 can be performed simultaneously and / or out of order as shown in process 400.

[0062] Figure 5 An example process 500 is shown for training a machine learning model 128 in view of data collected from a vehicle 102. In an example, the process 500 may be performed by a training device, which may include one or more of the computing devices discussed in detail herein, such as a TCU 110, an RSU 130, a MEC, a remote server, etc.

[0063] At operation 502, the training device receives the V2X message 120 and the modulated light data 122 captured by the vehicle 102. The data may be captured as discussed in detail above as the vehicle 102 traverses the road 126. In another example, data may be captured as one or more vehicles 102 traverse a test road 126 having a set of test road entities 124 placed at known measurement locations.

[0064] At operation 504, the training device constructs training data. For example, one or more frames of the modulated light data 122 may be used as input images for object detection. In addition, the locations included in the V2X message 120 and / or the modulated light data 122 may be used as center points or bounding boxes as ground truth for the locations of the identified road entities 124 in the image data. These locations may be mapped into the image data using techniques consistent with the operation of the coordinate transformer 302 discussed above.

[0065] At operation 506, the training device performs machine learning using the received data. The training may be performed using the location of the V2X message 120 and / or the modulated light data 122 as ground truth to allow the machine learning model 128 to learn to recognize the location of the road entity 124 in the image data. In an example, the training may include retraining the machine learning model 128, which includes at least a portion of the data received at operation 502. In another example, the training may include fine-tuning the machine learning model 128, for example, to improve the recognition of a pre-trained model in specific environmental factors (such as weather, season, location, etc.) without a complete retraining.

[0066] At operation 508, the training device provides the updated machine learning model 128 for use. In an example, this may include deploying the updated machine learning model 128 to the TCU 110, RSU 130, etc. for use. For example, the machine learning model 128 may be deployed for performing data fusion, such as discussed with respect to process 400. In another example, the machine learning model 128 may be deployed for a task without data fusion, such as recognizing a road entity 124 without additional use of the modulated light data 122. After operation 508, process 500 ends.

[0067] Thus, the data fusion application 132 can be used to collaboratively combine sensing from multiple data sources based on the light modulation information without duplication of information. In addition, although the road entities 124 detected in the modulated light data 122 can be shared in the V2X message 120, the data fusion application 132 can prevent excessive increase in wireless channel congestion, thereby enabling multiple vehicles 102 to collaboratively share data in an efficient manner.

[0068] This approach results in a more accurate association as the receiver receives ground truth rather than inferred / computed information from the camera 106. This approach may be particularly useful in urban areas where the Global Navigation Satellite System (GNSS) accuracy of the location information populated in the V2X message 120 may not be precise enough to help match the vehicle road entity 124 seen by the camera 106 during the association process.

[0069] Additionally, RF signals may be received from the front, rear, and adjacent vehicle-road entities 124. The optical channel allows some of these to be ignored, thereby reducing the processing burden. For example, if the vehicle 102 receives RFC-V2X and an optical object from the taillights of the vehicle-road entity 124, it may be possible to ignore the message because the vehicle 102 is behind the vehicle-road entity 124. Moreover, the method improves the accuracy of SDSM generation by increasing the confidence of the association between the detected camera object and the V2X message 120.

[0070] Figure 6 An example 600 of a computing device 602 for performing collaborative camera 106 sharing between road entities 124 is shown. Figure 6 , and refer to Figures 1 to 5 , the vehicle 102, the controller 104, the TCU 110, the transceiver 112, the processor 114, the road entity 124, and the RSU 130 may be examples of such a computing device 602. As shown, the computing device 602 includes a processor 604, which is operatively connected to a storage device 606, a network device 608, an output device 610, and an input device 612. It should be noted that this is merely an example, and a computing device 602 having more, fewer, or different components may be used.

[0071] Processor 604 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) and / or a graphics processing unit (GPU). In some examples, processor 604 is a system on chip (SoC) that integrates the functionality of a CPU and a GPU. SoC may optionally include other components (such as storage device 606 and network device 608) into a single integrated device. In other examples, the CPU and GPU are connected to each other via a peripheral connection device (such as a peripheral component interconnect (PCI) express) or another suitable peripheral data connection. In one example, the CPU is a commercially available central processing device that implements an instruction set, such as one of the microprocessor (MIPS) instruction set families of x86, ARM, Power, or non-interlocked pipeline stages.

[0072] Regardless of the details, during operation, the processor 604 executes stored program instructions, such as those of the data fusion application 132, retrieved from the storage device 606. The stored program instructions accordingly include software that controls the operation of the processor 604 to perform the operations described herein. The storage device 606 may include both non-volatile memory devices and volatile memory devices. Non-volatile memory includes solid-state memory, such as NAND flash memory, magnetic storage media, and optical storage media, or any other suitable data storage device that retains data when the system is disabled or loses power. Volatile memory includes static and dynamic random access memory (RAM) that stores program instructions and data during operation of the system 100. Examples of data stored to the storage device 606 may include V2X messages 120, modulated light data 122, data stored in the buffer 308, and associated V2X camera objects 312, unassociated V2X objects 314, and unassociated camera objects 316.

[0073] The GPU may include hardware and software for displaying at least two-dimensional (2D) and optionally three-dimensional (3D) graphics to an output device 610. The output device 610 may include a graphics or visual display device, such as an electronic display screen, a projector, a printer, or any other suitable device for reproducing a graphical display. As another example, the output device 610 may include an audio device, such as a speaker or headphones. As another example, the output device 610 may include a tactile device, such as a mechanically elevable device, which in one example may be configured to display Braille or another physical output that may be touched to provide information to a user.

[0074] Input device 612 may include any of a variety of devices that enable computing device 602 to receive control input from a user. Examples of suitable input devices that receive human-computer interface input may include a keyboard, mouse, trackball, touch screen, voice input device, graphics tablet, etc.

[0075] The network devices 608 may each include any of a variety of devices that enable the vehicle 102 and the road entity 124 to send and / or receive data from an external device over a network. Examples of suitable network devices 608 include an Ethernet interface, a Wi-Fi transceiver, a cellular transceiver, or a Bluetooth or Bluetooth Low Energy (BLE) transceiver, an ultra-wideband (UWB) transceiver, or other network adapters or peripheral interconnect devices that receive data from another computer or external data storage device, which may be used to receive large data sets in an efficient manner.

[0076] The process, method or algorithm disclosed herein may be delivered to / implemented by a processing device, a controller or a computer, which may include any existing programmable electronic control unit or a dedicated electronic control unit. Similarly, the process, method or algorithm may be stored as data and instructions that can be executed by a controller or a computer in many forms, including but not limited to information permanently stored on a non-writable storage medium such as a read-only memory (ROM) device and information that can be modified and stored on a writable storage medium such as a floppy disk, a tape, a compact disk (CD), a RAM device and other magnetic and optical media. The process, method or algorithm may also be implemented as a software executable object. Alternatively, the process, method or algorithm may be embodied in whole or in part using a suitable hardware component or a combination of hardware, software and firmware components, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a state machine, a controller or other hardware component or device.

[0077] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms covered by the claims. The words used in the specification are descriptive words rather than restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of various embodiments can be combined to form other embodiments of the present invention that may not be explicitly described or shown. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations in terms of one or more desired characteristics, it should be recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, strength, durability, life cycle, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc. For this reason, in terms of one or more characteristics, to the extent that any embodiment is described as less than other embodiments or prior art implementations are ideal, these embodiments are not outside the scope of the present disclosure and may be desired for specific applications.

[0078] With respect to the processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring according to a certain ordered sequence, such processes may be practiced by performing the described steps in an order different from that described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the processes herein is provided for the purpose of illustrating certain embodiments and should in no way be construed as limiting the claims.

[0079] Therefore, it should be understood that the above description is intended to be illustrative and not restrictive. Upon reading the above description, many embodiments and applications other than the examples provided will be apparent. The scope should not be determined with reference to the above description, but should be determined with reference to the entire scope of the appended claims and equivalents to which such claims are entitled. It is expected and anticipated that the technology discussed herein will develop in the future, and the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that the present application is capable of modification and variation.

[0080] All terms used in the claims are intended to be given their broadest reasonable constructions and their ordinary meanings as understood by persons skilled in the art described herein unless an explicit indication to the contrary is given herein. Specifically, use of singular articles such as "a," "an," "the," and "said" should be construed to recite one or more of the indicated elements unless a claim recites an explicit limitation to the contrary.

[0081] An abstract of the present disclosure is provided to allow the reader to quickly determine the nature of the technical disclosure. It should be understood that the abstract will not be used to interpret or limit the scope or meaning of the claims. In addition, in the aforementioned detailed description, it can be seen that various features are grouped together in various embodiments for the purpose of making the text of the present disclosure flow smoothly. This method of the present disclosure should not be interpreted as reflecting the intention that the claimed embodiments require more features than the features explicitly recited in each claim. Instead, as reflected in the attached claims, the inventive subject matter lies in less than all the features of a single disclosed embodiment. Therefore, the attached claims are hereby incorporated into the detailed description, with each claim itself as a separately claimed subject matter.

[0082] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms of the present invention. Rather, the words used in the specification are descriptive rather than restrictive terms, and it should be understood that various changes may be made without departing from the spirit and scope of the present invention. In addition, the features of the various embodiments of implementation may be combined to form other embodiments of the present invention.

[0083] According to the present invention, a self-aware road entity for performing data fusion of radio frequency (RF) data and visual data is provided, which has: a transceiver; a camera; and one or more controllers of the self-aware road entity, the one or more controllers being configured to: receive a vehicle-to-the-world (V2X) message from a remote road entity via the transceiver, the V2X message including one or more V2X data elements describing the remote road entity; use the camera to capture modulated light data within a sensing area around the self-aware road entity, the modulated light data being sent from the remote road entity; decode the modulated light transmission in the modulated light data to identify one or more optical data elements in the modulated light data describing the remote road entity; perform association of the one or more V2X data elements with the one or more optical data elements to identify an associated V2X camera object, wherein the V2X message and the modulated light data refer to the same remote road entity, wherein the V2X data elements and the modulated light data are fused into a combined representation of the remote road entity; and process the combined representation of the remote road entity based on the association.

[0084] According to an embodiment, the modulated light data includes one or more of an identifier (ID), an object type, a heading, a speed or a position of the remote road entity, and the association is performed by matching one or more of the ID, the object type, the heading, the speed or the position with corresponding information in the V2X message.

[0085] According to an embodiment, the self-aware road entity includes external lights, and the one or more controllers of the self-aware road entity are configured to: use the transceiver to broadcast a V2X message including vehicle information for reception by the road entity, the vehicle information including one or more of the ID, object type, heading, speed or position of the self-aware road entity; and use the external lights to send a modulated light transmission describing the self-aware road entity for reception by the road entity, the modulated light transmission also including the vehicle information.

[0086] According to an embodiment, the one or more controllers are also programmed to: identify remote road entities in the modulated light data by performing object detection on the image captured by the camera; and focus the decoding of the modulated light transmission on the area captured by the camera that contains the remote road entity.

[0087] According to an embodiment, the one or more controllers are further programmed to: generate a V2X message based on the association for any modulated light data received from the remote road entity without a corresponding V2X message; and broadcast the generated V2X message using the transceiver.

[0088] According to an embodiment, the one or more controllers are further programmed to provide a driver alert using the combined representation.

[0089] According to an embodiment, the one or more controllers are further programmed to provide the combined representation to an autonomous or semi-autonomous driving function of the self-aware road entity.

[0090] According to the present invention, a method for performing data fusion of radio frequency (RF) data and visual data includes: receiving a vehicle-to-the-world (V2X) message from a road entity via a transceiver, the V2X message including one or more V2X data elements describing the road entity; using a camera to capture modulated light data within a sensing area around a vehicle, the modulated light data being sent from the road entity; decoding modulated light transmissions in the modulated light data to identify one or more optical data elements in the modulated light data describing the road entity; performing association of the one or more V2X data elements with the one or more optical data elements to identify an associated V2X camera object, wherein the V2X message and the modulated light data refer to the same road entity, wherein the V2X data elements and the modulated light data are fused into a combined representation of the road entity; and processing the combined representation of the road entity based on the association.

[0091] In one aspect of the present invention, the modulated light data includes one or more of an identifier (ID), an object type, a heading, a speed, or a position of the road entity, and the association is performed by matching one or more of the identifier (ID), the object type, the heading, the speed, or the position with corresponding information in the V2X message.

[0092] In one aspect of the present invention, the method includes: using the transceiver to broadcast a V2X message including vehicle information for reception by a road entity, the vehicle information including one or more of the vehicle's ID, object type, heading, speed or position; and using the vehicle's external lights to send a modulated light transmission describing the vehicle for reception by a road entity, the modulated light transmission also including the vehicle information.

[0093] In one aspect of the invention, the method comprises: identifying road entities in the modulated light data by performing object detection on images captured by the camera; and focusing the decoding of the modulated light transmission on an area captured by the camera that contains the road entity.

[0094] In one aspect of the present invention, the method includes: decoding a second modulated light transmission in the modulated light data; determining based on the association that a V2X message has not been received from any road entity corresponding to the second modulated light transmission; generating a sensor data sharing V2X message including information from the second modulated light transmission; and broadcasting the sensor data sharing V2X message using the transceiver for reception by other road entities.

[0095] In one aspect of the invention, the method includes providing a driver alert using the combined representation.

[0096] In one aspect of the invention, the method comprises providing the combined representation to an autonomous or semi-autonomous driving function of the vehicle.

[0097] In one aspect of the invention, the method includes utilizing a machine learning model to recognize remote road entities in one or more frames captured using the camera.

[0098] In one aspect of the invention, the method comprises: creating a training set using at least the V2X message and the modulated light data, wherein the locations included in the V2X message and / or the modulated light data are used as ground truth for the locations of road entities identified in image data; and retraining the machine learning model using the training set to enhance detection of road entities.

[0099] In one aspect of the invention, the method includes deploying the retrained machine learning model to recognize road entities in image data.

[0100] According to the present invention, a non-transitory computer-readable medium is provided, which has instructions for executing radio frequency (RF) data and visual data. When the instructions are executed by one or more controllers of a self-aware road entity, the self-aware road entity performs the following operations, including: receiving a vehicle-to-the-world (V2X) message from a remote road entity via a transceiver of the self-aware road entity, the V2X message including one or more V2X data elements describing the remote road entity; using a camera of the self-aware road entity to capture modulated light data within a sensing area around the self-aware road entity, the modulated light data being sent from the remote road entity; decoding the modulated light transmission in the modulated light data to identify one or more optical data elements describing the remote road entity in the modulated light data; performing an association of the one or more V2X data elements with the one or more optical data elements to identify an associated V2X camera object, wherein the V2X message and the modulated light data refer to the same road entity, wherein the one or more V2X data elements and the modulated light data are fused into a combined representation of the remote road entity; and processing the combined representation of the remote road entity based on the association.

[0101] According to an embodiment, the modulated light data includes one or more of an identifier (ID), an object type, a heading, a speed or a position of the remote road entity, and the association is performed by matching one or more of the identifier (ID), the object type, the heading, the speed or the position with corresponding information in the V2X message.

[0102] According to an embodiment, the present invention is also characterized by instructions, which, when executed by one or more controllers of the self-aware road entity, cause the self-aware road entity to perform the following operations, including: identifying remote road entities in the modulated light data by performing object detection on the image captured by the camera; and focusing the decoding of the modulated light transmission on the area captured by the camera that contains the remote road entity.

[0103] According to an embodiment, the present invention is also characterized by instructions, which, when executed by one or more controllers of the self-aware road entity, cause the self-aware road entity to perform the following operations, including: decoding a second modulated light transmission in the modulated light data; determining based on the association that no V2X message has been received from any remote road entity corresponding to the second modulated light transmission; generating a sensor data sharing V2X message including information from the second modulated light transmission; and broadcasting the sensor data sharing V2X message using the transceiver for reception by other road entities.

[0104] According to an embodiment, the invention is further characterized by instructions that, when executed by one or more controllers of the self-aware road entity, cause the self-aware road entity to perform operations including: providing a driver alert using the combined representation.

[0105] According to an embodiment, the present invention is also characterized by instructions, which, when executed by one or more controllers of the self-aware road entity, cause the self-aware road entity to perform the following operations, including: providing the combined representation to the autonomous or semi-autonomous driving function of the self-aware road entity.

[0106] According to an embodiment, the present invention is also characterized by instructions, which, when executed by one or more controllers of the self-aware road entity, cause the self-aware road entity to perform the following operations, including: using the transceiver to broadcast a V2X message including vehicle information for reception by a road entity, the vehicle information including one or more of the ID, object type, heading, speed or position of the self-aware road entity; and using the external lights of the self-aware road entity to send a modulated light transmission describing the self-aware road entity for reception by a remote road entity, the modulated light transmission also including the vehicle information.

[0107] According to an embodiment, the present invention is also characterized by instructions, which, when executed by one or more controllers of the self-aware road entity, cause the self-aware road entity to perform the following operations, including: providing the combined representation to a machine learning model to help associate the image with the remote road entity.

Claims

1. A self-aware road entity for performing data fusion of radio frequency (RF) data and visual data, comprising: Transceiver; camera; as well as One or more controllers of the self-aware road entity, the one or more controllers configured to receiving, via the transceiver, a vehicle-to-everything (V2X) message from a remote roadway entity, the V2X message comprising one or more V2X data elements describing the remote roadway entity, capturing, using the camera, modulated light data within a sensing area around the self-aware road entity, the modulated light data being transmitted from the remote road entity, decoding the modulated light transmission in the modulated light data to identify one or more optical data elements in the modulated light data describing the remote road entity, performing an association of the one or more V2X data elements with the one or more light data elements to identify an associated V2X camera object, wherein the V2X message and the modulated light data refer to the same remote road entity, wherein the V2X data elements and the modulated light data are fused into a combined representation of the remote road entity, and The combined representation of the remote road entity is processed based on the association.

2. The self-aware road entity of claim 1 , wherein the modulated light data comprises one or more of an identifier (ID), an object type, a heading, a speed, or a position of the remote road entity, and the associating is performed by matching one or more of the ID, the object type, the heading, the speed, or the position with corresponding information in the V2X message.

3. The self-aware road entity of claim 1 , wherein the self-aware road entity includes exterior lights, and the one or more controllers of the self-aware road entity are configured to: broadcasting, using the transceiver, a V2X message including vehicle information for receipt by a road entity, the vehicle information including one or more of an ID, object type, heading, speed, or location of the self-aware road entity; and A modulated light transmission describing the self-aware road entity is transmitted using the exterior light for receipt by the road entity, the modulated light transmission also including the vehicle information.

4. The self-aware road entity of claim 1, wherein the one or more controllers are further programmed to: identifying remote road entities in the modulated light data by performing object detection on images captured by the camera; and The decoding of the modulated light transmission is focused on an area captured by the camera that contains a remote road entity.

5. The self-aware road entity of claim 1, wherein the one or more controllers are further programmed to: generating a V2X message based on the association for any modulated light data received from the remote roadway entity without a corresponding V2X message; and The generated V2X message is broadcasted using the transceiver.

6. The self-aware road entity of claim 1, wherein the one or more controllers are further programmed to provide a driver alert using the combined representation.

7. The self-aware road entity of claim 1, wherein the one or more controllers are further programmed to provide the combined representation to an autonomous or semi-autonomous driving function of the self-aware road entity.

8. A method for performing data fusion of radio frequency (RF) data and visual data, comprising: receiving, via a transceiver, a vehicle-to-everything (V2X) message from a roadway entity, the V2X message comprising one or more V2X data elements describing the roadway entity; capturing modulated light data within a sensing area around the vehicle using a camera, the modulated light data being transmitted from the road entity; decoding the modulated light transmission in the modulated light data to identify one or more light data elements in the modulated light data describing the road entity; performing an association of the one or more V2X data elements with the one or more light data elements to identify an associated V2X camera object, wherein the V2X message and the modulated light data refer to a same road entity, wherein the V2X data elements and the modulated light data are fused into a combined representation of the road entity; and The combined representation of the road entity is processed based on the association.

9. The method of claim 8, wherein the modulated light data comprises one or more of an identifier (ID), an object type, a heading, a speed, or a position of the road entity, and the associating is performed by matching one or more of the identifier (ID), the object type, the heading, the speed, or the position with corresponding information in the V2X message.

10. The method of claim 8, further comprising: broadcasting, using the transceiver, a V2X message including vehicle information for receipt by a road entity, the vehicle information including one or more of an ID, an object type, a heading, a speed, or a location of the vehicle; and A modulated light transmission describing the vehicle is transmitted using exterior lights of the vehicle for receipt by a roadway entity, the modulated light transmission also including the vehicle information.

11. The method of claim 8, further comprising: identifying road entities in the modulated light data by performing object detection on images captured by the camera; as well as The decoding of the modulated light transmission is focused on an area captured by the camera that contains a road entity.

12. The method of claim 8, further comprising: decoding a second modulated light transmission in the modulated light data; determining, based on the association, that a V2X message is not received from any roadway entity corresponding to the second modulated optical transmission; generating a sensor data sharing V2X message including information from the second modulated light transmission; as well as The sensor data sharing V2X message is broadcasted using the transceiver for reception by other road entities.

13. The method of claim 8, further comprising one or more of the following: providing a driver warning using the combined representation; and The combined representation is provided to an autonomous or semi-autonomous driving function of the vehicle.

14. The method of claim 8, further comprising: utilizing a machine learning model to recognize remote road entities in one or more frames captured using the camera; creating a training set using at least the V2X messages and the modulated light data, wherein locations included in the V2X messages and / or the modulated light data are used as ground truth for the locations of road entities identified in image data; retraining the machine learning model using the training set to enhance detection of road entities; as well as The retrained machine learning model is deployed to recognize road entities in image data.

15. A non-transitory computer readable medium comprising instructions for performing data fusion of radio frequency (RF) data and visual data, the instructions, when executed by one or more controllers of a self-aware road entity, causing the self-aware road entity to perform operations comprising: receiving, via a transceiver of the self-aware roadway entity, a vehicle-to-everything (V2X) message from a remote roadway entity, the V2X message comprising one or more V2X data elements describing the remote roadway entity; capturing modulated light data within a sensing area around the self-aware road entity using a camera of the self-aware road entity, the modulated light data being sent from the remote road entity; decoding the modulated light transmission in the modulated light data to identify one or more optical data elements in the modulated light data describing the remote road entity; performing an association of the one or more V2X data elements with the one or more light data elements to identify an associated V2X camera object, wherein the V2X message and the modulated light data refer to a same road entity, wherein the one or more V2X data elements and the modulated light data are fused into a combined representation of the remote road entity; and The combined representation of the remote road entity is processed based on the association.

Images