Transmission of optical display metadata

Abstract

Disclosed is a transmitter vehicular control system (110) of a transmitting vehicle (104). The transmitter vehicular control system comprises: a transmitter memory (114) storing transmitter machine executable instructions (130); a transmitter computational system (112). Execution of the transmitter machine executable instructions causes the transmitter computational system to: control (200) an optical display system (120) of the transmitting vehicle to generate an optical display (122) using optical display commands (132); receive (202) transmitter location data (134) from a location system (118) of the transmitting vehicle; construct (204) optical display metadata (136) from the optical display commands (132) and the transmitter location data (134); broadcast (206) the optical display metadata via a wireless inter-vehicle communication system (108).

Description

TRANSMISSION OF OPTICAL DISPLAY METADATAFIELD OF THE INVENTION

[0001] The invention relates to the field of transportation, in particular to optical displays provided by vehicles.BACKGROUND

[0002] Vehicles currently use a variety of fixed lights, such as blinkers or warning lights, to communicate or signal other drivers. Recently more complicated systems have been developed where optical displays can be projected onto the roadway to assist the driver of a vehicle. For example, a predicted vehicle path can be projected on the roadway. This may also be used to communicate with pedestrians and other drivers. A car could project a walk symbol in front of the car to communicate that the car has stopped for the pedestrian to cross the street. In another example, a projector could project turn signals or other warning signs onto the street to warn other drivers.SUMMARY OF THE INVENTION

[0003] The invention provides for a system comprising a transmitter vehicular control system of a transmitting vehicle and a receiver vehicular control system of a receiving vehicle, a transmitter vehicular control system, a receiver vehicular control system, a method of operating a transmitter vehicular control system, a method of operating a receiver vehicular control system, and a computer program in the independent claims. Embodiment are given in the dependent claims.

[0004] In one aspect a transmitter vehicular control system of a transmitting vehicle is disclosed. The transmitter vehicular control system comprises a transmitter memory storing transmitter machine-executable instructions. The transmitter vehicular control system further comprises a transmitter computational system. Execution of the transmitter machine-executable instructions causes the transmitter computational system to control an optical display system of thetransmitting vehicle to generate an optical display using optical display commands. Execution of the transmitter machine-executable instructions further causes the transmitter computational system to receive transmitter location data from a location system of the transmitting vehicle. Execution of the transmitter machine-executable instructions further causes the transmitter computational system to construct optical display metadata from the optical display commands and the transmitter location data. Execution of the transmitter machine-executable instructions further causes the transmitter computational system to broadcast the optical display metadata via a wireless inter-vehicle communication system.

[0005] In another aspect a receiver vehicular control system of a receiving vehicle is disclosed. The receiver vehicular control system comprises a receiver memory storing receiver machineexecutable instructions and an artificial intelligence module. The artificial intelligence module is configured to classify road objects in a digital image. The receiver vehicular control system further comprises a receiver computational system. Execution of the receiver machineexecutable instructions causes the receiver computational system to receive optical display metadata descriptive of an optical display generated by a transmitting vehicle via a wireless intervehicle communication system. The display metadata is descriptive of a transmitter location data of the transmitting vehicle. The execution of the receiver machine-executable instructions further causes the receiver computational system to receive a digital image that comprises the optical display. Execution of the receiver machine-executable instructions further causes the receiver computational system to receive a classification of the optical display in response to inputting the digital image and the optical display metadata into the artificial intelligence module. The artificial intelligence module is configured to modify the classification of the optical display using the optical display metadata.

[0006] In another aspect a method of operating a transmitter vehicular control system of a transmitting vehicle is disclosed. The method comprises controlling an optical display system of the transmitting vehicle to generate an optical display using optical display commands. The method further comprises receiving transmitter location data from a location system of the transmitting vehicle. The method further comprises constructing optical display metadata from the optical display commands and the transmitter location data. The method further comprises broadcasting the optical display metadata via a wireless inter-vehicle communication system.

[0007] In another aspect a method of operating a receiver vehicular control system of a receiving vehicle is disclosed. The method comprises receiving optical display metadata descriptive of an optical display generated by a transmitting vehicle via a wireless inter-vehicle communication system. The display metadata is descriptive of a transmitter location data of thetransmitting vehicle. The method further comprises receiving a digital image that comprises the optical display. The method further comprises receiving a classification of the optical display in response to inputting the digital image and the optical display metadata into an artificial intelligence module. The artificial intelligence module is configured to classify road objects in the digital image. The artificial intelligence module is configured to modify the classification of the optical display using the optical display metadata.

[0008] In another aspect a computer program comprising machine-executable instructions for implementing the method of operating a transmitter vehicular control system of a transmitting vehicle is disclosed.

[0009] In another aspect a computer program comprising machine-executable instructions for implementing the method of operating a receiver vehicular control system of a receiving vehicle is disclosed.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the following, examples are described in greater detail making reference to the drawings in which:

[0011] Fig. 1 is a schematic of a transportation system that comprises both a transmitting vehicle and a receiving vehicle.

[0012] Fig. 2 is a block diagram that illustrates a method of using the transportation system.

[0013] Fig. 3 illustrates a use case of the transportation system of Fig. 1.DETAILED DESCRIPTION

[0014] In the following, similar elements are denoted by the same reference numerals.

[0015] In one example a transmitter vehicular control system of a transmitting vehicle is disclosed. A transmitting vehicle as disclosed encompasses a car, truck, motorcycle, moped, bicycle, or other vehicle designed or configured for travelling on a roadway. The transmitter vehicular control system comprises a transmitter memory storing transmitter machineexecutable instructions. The transmitter vehicular control system further comprises a transmitter computational system. Execution of the transmitter machine-executable instructions causes the transmitter computational system to control an optical display system of the transmitting vehicle to generate an optical display using optical display commands.

[0016] The optical display system may for example be used to generate optical signals or projections onto the vehicle or the roadway depending upon the different examples. The optical display commands encompass commands which are used to directly control the optical display system to provide the optical display. Execution of the transmitter machine-executable instructions further causes the transmitter computational system to receive transmitter location data from a location system of the transmitting vehicle. The location system may for example encompass any system which is able to provide a location of the transmitting vehicle. This may for example be a GPS or GLONASS system in some examples. In other examples other radio location methods may also be used. The location system may also use other methods of localization based on a correlation between: exteroceptive sensors measurements e.g. data such as point clouds, or features (corners / edges / etc.) or objects (e.g. pole, crosswalk, etc.) measured by Lidar, Radar, Ultrasonics sensors; and / or an a priori map (with GNSS coordinates), priorly stored in the vehicle (containing data & landmarks to enable the correlation). The transmitter location data encompasses a location of the transmitting vehicle.

[0017] Execution of the transmitter machine-executable instructions further causes the transmitter computational system to construct optical display metadata from the optical display commands and the transmitter location data. The optical display metadata may for example encompass a classification or detailed structure of the optical display. The transmitter location data may in some examples be used to provide precise details on the location of the optical display.

[0018] The optical display may, for example, be located on the roadway, or on particular regions of the vehicle (e.g. "vehicle left side") if the optical display is a color change or a panel. Execution of the transmitter machine-executable instructions further causes the transmitter computational system to broadcast the optical display metadata via a wireless inter-vehicle communication system.

[0019] The inter-vehicle communication system encompasses a wireless communication system which is able to form a data connection between two or more vehicles. This example may be beneficial because the optical display generated by the optical display system may cause confusion or cause an artificial intelligence unit to improperly classify the optical display. For example, the optical display may fool an autonomous or semi-autonomous vehicle into believing it is at a false location or that the optical display is in fact a road hazard, such as an obstacle, and may cause an evasive or reactive action from this other vehicle. In some cases, the evasive or reactive action may be inappropriate considering the context, and may lead to an accident.Broadcasting the optical display metadata may provide a means for other vehicles to automatically recognize and react appropriately to the optical display.

[0020] In another example the optical display system is a projector configured for projecting an optical display onto a surface adjacent to the vehicle. For example, the optical display could be projected onto a roadway. This example is particularly beneficial because projections onto surfaces adjacent to the transmitting vehicle may in particular confuse the artificial intelligence modules of other vehicles. This example may have the benefit that it may provide for added safety when a projector is used to project optical displays adjacent to the transmitting vehicle.

[0021] In another example the optical display metadata comprises a geometrical description of the optical display. This geometrical description may for example be a descriptor or description of a particular shape or type of geometrical figures or figures which are used to form the optical display. In other examples this geometrical description may actually provide coordinates for one or more portions of the optical display. This may be beneficial because it may provide for extremely detailed information which can be transmitted in the optical display metadata. This may better help an artificial intelligence of a receiving vehicle to properly interpret the optical display.

[0022] The optical display metadata could also provide the type of optical display, which could describe the projection of symbols onto the roadway, a description of a vehicle body that is changing color, a description of a panel on the vehicle displaying symbols, and also an indication of automated driving mode. The type of symbol could include in some examples: "crosswalk", "snow flake", "warning sign", "arrow", "complex pattern", etc.

[0023] The optical display metadata could also comprise other information depicting the physical aspect of the optical display e.g. color, brightness. The optical display metadata in additional examples may also include the message conveyed by the optical display (and its target audience). For example, the optical display metadata could classify the optical display, for example, as a "welcome message " for the driver, "warning about icy road" for the driver & other road users, "intended maneuver=turn right" for other road users, "you can cross" for the pedestrian, and etc.

[0024] In further examples, the optical display metadata could comprise a time sequence of the current & planned optical displays (e.g., symbol 1 during 100ms, symbol 2 during Is, etc.). In further examples, the optical display metadata could comprise a confidence level in each previously provided information. The confidence level could for example be used as an input to an artificial intelligence model to aid in either classifying the optical display or determining the appropriate response to the optical display.

[0025] In another example the geometrical description comprises one or more physical locations of the one or more portions of the optical display. For example, one or more locations in the optical display may have GPS or other detailed coordinates which indicate their position. This may better help an artificial intelligence module interpreting the optical display to properly interpret it. In some examples, the other location / position information may refer to the vehicle region on which the optical display is occurring. For example, if the optical display is a color change according to a particular pattern, or a panel displaying a pedestrian that is crossing the road, the position may indicate which vehicle region / side is implied).

[0026] In another example the optical display is an exterior display panel on the vehicle. For example, the vehicle may have panels which are used to provide information or displays on the vehicle. However, these symbols or renderings may inadvertently confuse an artificial intelligence module which is used to classify images. By providing the optical display metadata for these displays it may provide for greater safety because the artificial intelligence modules of receiving vehicles are not confused.

[0027] In another example the optical display is any one of the following: a turn signal, a yield signal, a cross walk projection, a predicted vehicle path, a welcome display for the driver, a vehicle malfunction warning, a road hazard indicator, a warning and a warning of an occupant to exit the vehicle. This example may be beneficial because any or all of these symbols may be improperly classified by an artificial intelligence module of a receiving vehicle.

[0028] In another example the optical display is a lighting system configured to indicate a manual driving mode or an automated driving mode of the transmitting vehicle. This example may be beneficial because a change in the lighting system from the manual driving mode to the automated driving mode or vice versa may provide a situation where the artificial intelligence module of a receiving vehicle is confused or misclassifies the transmitting vehicle. Providing the optical display metadata may enable the artificial intelligence module to better classify or interpret the transmitting vehicle.

[0029] In another example the optical display is an optical system configured for dynamically changing vehicle color. For example, portions of the surface of the vehicle may provide for a means of changing the light emitted or may actively project an image which is used to provide a change in the vehicle color. When the vehicle color is changing this may for example cause an artificial intelligence module of a receiving vehicle to misclassify or falsely classify the transmitting vehicle. The broadcasting of the optical display metadata may provide for a means of the artificial intelligence module to better classify images of the transmitting vehicle.

[0030] In another example, the wireless transmission system is a radio-frequency data exchange system. For example, there may be radio transceivers in the transmitting and receiving vehicles that use a protocol to directly exchange data.

[0031] In another example, the wireless transmission system is a Wi-Fi data exchange system that uses the typical Wi-Fi protocol used by computers. This may provide for an effective means of exchanging data between vehicles that is also cost effective.

[0032] In another example the wireless transmission system is a V2X system which may be configured for exchanging data between different vehicles. The V2X system may be beneficial because some vehicles already incorporate this protocol as well as providing for a safe and effective means of transferring data. Various V2X systems provide or operate on different operating principles and some use Wi-Fi and some use cellular data exchange systems.

[0033] The optical display metadata which comprises information describing the optical display (i.e. the symbol projected on the road) could be provided in existing, upcoming or totally new standardized vehicle-to-infrastructure messages, vehicle-to-vehicle messages and / or infrastructure-to-vehicle messages, such as for example CAM (ETSI EN 302 637-2), DENM (ETSI EN 302 637-3), MAP (ISO TS 19091 and SAE J2735), IVI (ISO TS 19321), CPM (ETSI TS 103 324), VAM (ETSI TS 103 300-3).

[0034] The wireless communication technology employed to send and / or receive this information may be vehicle-to-vehicle communication, in particular, Wi-Fi / ITS-G5 (IEEE 802. lip or 802.11bd) or 4G / 5G (3GPP Release 14 and more, meaning "LTE-V2X" or "5G-V2X"). In case this V2X information is exchanged between the vehicle and an infrastructure, such as an external control unit, vehicle-to-infrastructure communication based on 4G, 5G, 6G, or 7G (or other generations of cellular communication) may be used.

[0035] In another example the wireless transmission system is a cellular data exchange system. The use of a cellular data exchange system may also be used to exchange data between various vehicles. This may have the advantage that the SIM card used for the cellular system may provide for positive identification of vehicles. The use of a cellular system may also provide assistance in providing the location of the transmitting and receiving vehicles. This may for example be useful in situations where satellite location systems such as GPS are not functioning properly.

[0036] In another example a receiver vehicular control system of a receiving vehicle is disclosed. The receiver vehicular control system comprises a receiver memory storing receiver machineexecutable instructions and an artificial intelligence module. The artificial intelligence module is configured to classify road objects in a digital image. A digital image as used herein encompasses single digital images, a video feed, and also a temporal sequence of digital images. The artificialintelligence module may refer to a single machine learning module or component such as a neural network. In other examples the artificial intelligence module may contain multiple artificial intelligence modules and / or procedural code.

[0037] The road objects which are classified, may for example include: road users such as cars, trucks, pedestrians, and etc.; road furniture & infrastructure such as traffic lights, traffic signs, guardrails, fences; street furniture such as poles, electricity boxes, post boxes, road background such as trees, vegetations, buildings, and lane marking, Stop line, Symbols on the roadway (e.g. painted arrows in turning lanes, etc.).

[0038] In one example, the artificial intelligence module is configured to classify road objects in a digital image. In further examples, it may be used afterwards the classification of these road objects for various purposes such as controlling or driving the vehicle.

[0039] In another example the Al module may be fully or partially inside or incorporated into an ADAS or Automated driving system. The receiver vehicular control system comprises a receiver computational system. Execution of the receiver machine-executable instructions causes the receiver computational system to receive optical display metadata descriptive of an optical display generated by a transmitting vehicle in a wireless inter-vehicle communication system. The display metadata is descriptive of a transmitter location data of the transmitting vehicle. The optical display metadata that is provided provides both details on the optical display as well as the location of the optical display. Execution of the receiver machine-executable instructions further causes the receiver computational system to receive a digital image that comprises the optical display. This may for example be received from a camera or video system that is incorporated into the receiving vehicle.

[0040] Execution of the receiver machine-executable instructions further causes the receiver computational system to receive a classification of the optical display in response to inputting the digital image and the optical display metadata into the artificial intelligence module. The artificial intelligence module is configured to modify the classification of the optical display using the optical display metadata. This modification may for example be performed in several different ways. In some instances, the artificial intelligence module may have a machine learning component that is trained to receive the optical display metadata and then to modify the output of the artificial intelligence module in response. In other examples, since detailed information on the location and the details of the optical display are provided, the optical display metadata may be used to simply remove or delete a classification corresponding to the optical display. This may for example be useful in totally preventing the optical display from modifying the behavior of the receiving vehicle.

[0041] In another example, the receiver does not comprise a LiDAR or RADAR. In some autonomous or semi-autonomous vehicles both optical systems and LiDAR / RADAR systems are used. As these operate in different wavelengths and contain different information the combination of the optical data and LiDAR / RADAR may be useful in rejecting artifacts or objects which are actually not present and may cause an accident when the receiving vehicle reacts. LiDAR and RADAR systems work on the principle of a reflected RADAR or LiDAR signal. The broadcasting of the optical display metadata may provide for a way of discriminating and eliminating false objects caused by an optical display. As this system is able to help reject these false objects the receiver vehicle does not need to comprise a LiDAR or a RADAR. LIDAR / RADAR data may comprise a raw point cloud, or features such as contours, shapes, or even classified objects such as a car or pedestrian.

[0042] In another example the process of modifying the classification of the optical display comprises ignoring or deleting the classification of the optical display. This example may be beneficial because it removes the possibility of an autonomous or assisted driving system from being affected by the optical display altogether because the classification has been ignored or deleted.

[0043] The artificial intelligence module may be implemented as an image processing pipeline for the autonomous or assisted driving system in some examples. In one example, the image processing pipeline may contain at least some of the following steps: image acquisition, image pre-processing, feature extraction, object / scene recognition, object / scene perception fusion, and driving behavior decision. The image acquisition is the acquisition of the digital image from the physical world by using sensors such as cameras. In the image pre-processing step, various steps may be performed. This may include image enhancement to improve image interpretability via e.g. adjustment of brightness, contrast, and / or color balance. It may also include image restoration to remove artifacts due to noise, blurring, and / or compression. Image denoising may also be performed to remove or reduce noise, which for example was introduced during image acquisition. The image preprocessing may also include image segmentation to divide the digital image into meaningful distinct regions, for example according to color criteria. Other various preprocessing steps may also be performed such as image resizing.

[0044] After image pre-processing the pre-processed digital image may better reflect reality and have better interpretability. Feature extraction may then be performed on this pre-processed image. In feature extraction, the features such as information in the image enabling objects to be distinguished from each other are retrieved. This may include particular shapes such as lines, polygons, rectangles, and etc.; contours such as edges of objects, the location of textures in theimage, and the extractions of sub-parts of the image such as segmented regions with information about the segmentation's shape and color. This list of features is then used for object recognition.

[0045] In object recognition, the list of features is used to identify and generate a list of specific objects which may correspond to specific segmentation regions. This classification may contain information about the class of objects such as labeling the object as a pedestrian, car, truck, curb, lane boundary, and etc.; object dimensions; object kinematics such as speed and acceleration; and object position and orientation.

[0046] This list of specific objects is then used for the object / scene perception fusion. In this step, the data from other sensors such as other cameras, LIDAR, RADAR, and ultrasonic sensors is appended or combined with the list of specific objects. The list of objects / scene elements and their particular characteristics is then provided to the driver assistance module or the receiver vehicle.

[0047] Within this chain of steps, the artificial intelligence module (or image pipeline) may be configured to modify the classification of the optical display using the optical display metadata in several different ways. In the image pre-processing step, the location in the digital image where the optical display is located is pre-processed differently. For example, different brightness and / or contrast adjustments may be applied. This may for example prevent or reduce the chance that during feature extraction false features are generated. In this example, the brightness and / or contrast of the optical display could be artificially reduced to avoid feature extraction.

[0048] In addition to or as an alternative to the modified image pre-processing, the optical display metadata may be used to delete or remove features corresponding to the optical display. As the optical display metadata contains a location of the optical display this location may be identified and features corresponding to the optical display can be removed or deleted.

[0049] In addition to these steps or in combination with them, the object I scene recognition may also be modified. In this example, the object recognition processing step considers the extracted features of the optical display differently. For example, the located contours may be excluded from object recognition or may be explicitly labeled as being an optical display. In either case, this may prevent the object / scene element perception fusion from being performed incorrectly.

[0050] In addition to these steps or in combination the optical display metadata may be used during the object fusion processing step to exclude or properly label the optical display. The optical display metadata may in addition to or in combination also be passed directly to thedriver assistance module or the autonomous driving system. In this case the optical display metadata may be used to directly affect or modify the behavior of these modules.

[0051] In another example execution of the receiver machine-executable instructions further causes the receiver computational system to modify operation of the driver assistance module of the receiver vehicle or modify operation of an autonomous driving system of the receiver vehicle using the modified classification of the optical display. This example may be beneficial because it may provide for improved safety because the driver assistance module or the autonomous driving system will not be improperly affected by the optical display. For example, an ADAS (assisted driving system) may decide to send a driver warning and switch off, in order to give back immediately the vehicle control to the driver. In another example, an ADS (automated driving system) may decide to slow down, or to stop in a safe area (e.g. out of traffic) or to request the driver to take back the vehicle control.

[0052] Other software modules or modules which may incorporate the driver assistance module may also adjust their behavior by using the modified optical display classification. For example, different subsystems from a driver assistance system or an autonomous driving system such as driving behavior decision subsystem, localization on a digital map subsystem, or digital map update management subsystem may use the modified classification.

[0053] The artificial intelligence module or the driver assistance module may also be incorporated into or work with other systems such as an intelligent lighting system, or an augmented reality system, or other similar or subsequent system.

[0054] These software systems may for example, also include a driving behavior decision subsystem, a localization on a digital map subsystem, or a digital map update management subsystem, and intelligent lighting system, or an augmented reality system. In various example, one could also add that the artificial intelligence module may additionally provide a confidence indicator in response to receiving the optical display metadata as input. This confidence indicator may reflect in particular:

[0055] 1) how much the module has avoided or mitigated the optical display (based on the optical display metadata)

[0056] 2) the quality / trustworthiness of the optical display metadata)

[0057] This confidence indicator may be used to adjust driving behavior, or to estimate the confidence indicator that is outputted by other software modules that are using the modified optical display metadata. This may be illustrated in several examples below:

[0058] In a first example, the objects recognized / classified (after these steps) are then processed by a localization system. This localization system may be inside or outside the Al module, and inside or outside the ADAS / ADS.

[0059] This localization system may use the optical display metadata to adjust its behavior. For example, this localization system may use crosswalks detected by vehicle sensors (e.g. cameras) to determine the vehicle position / location, by correlating this detected crosswalk with an a priori map (priorly stored in the vehicle) containing crosswalks with their spatial coordinates.

[0060] If the optical display is a crosswalk projected on the roadway, optical display metada describing this false crosswalk will be used by the localization system so that this false crosswalk will be ignored, which will improve the reliability of the position determined by the localization system.

[0061] In a second example, the objects recognized / classified (after these steps) are then processed by a system different from or incorporated into an assisted or automated driving module. For example, this system may be :

[0062] - an Augmented Reality system (displaying on the windshield information for the driver, such as important objects the driver should be aware of)

[0063] - or a Map update manager (in charge of updating a digital map describing the road : for example, if a new crosswalk is detected by vehicle sensors, a new crosswalk will be added in the digital map, then this updated digital map will be shared to millions of other vehicles)

[0064] If the optical display is a crosswalk projected on the roadway, optical display metada describing this false crosswalk will be used by these 2 systems so that (respectively)

[0065] - the Augmented Reality system ignores the false crosswalk (does not display it to the driver), or on the contrary, warns the driver that it is a false crosswalk he should not consider

[0066] - the Map update manager ignores the false crosswalk and does not add it in the digital map.

[0067] It is understood that one or more of the aforementioned examples may be combined as long as the combined examples are not mutually exclusive.

[0068] Fig. 1 illustrates an example of a transportation system 100 that is formed by the combination of a roadway 102, a transmitting vehicle 104, and a receiving vehicle 106. There is a wireless inter-vehicle communication system 108 that enables the transmitting vehicle 104 and the receiving vehicle 106 to exchange data. The transmitting vehicle 104 comprises a transmitter vehicular control system 110. This comprises a transmitter computational system 112 that is in communication with a transmitter memory 114. In this example there is a transceiver 116 and a global positioning system 118 that are also in communication with the transmitter computationalsystem 112. The transmitter computational system 112 is further in communication with an optical display system 120 that is able to project an optical display 122 onto the roadway 102.

[0069] The transmitter 114 is shown as storing transmitter machine-executable instructions 130. The transmitter machine-executable instructions 130 enable the transmitter computational system 112 to perform various computational tasks and control the other components of the transmitter vehicular control system 110. The transmitter memory 114 is further shown as storing optical display commands 132 that comprise instructions which enable the transmitter computational system 112 to control the optical display system 120 to project the optical display 122. The transmitter memory 114 is further shown as containing transmitter location data 134 that has been received by the global positioning system 118. The transmitter memory 114 is further shown as containing optical display metadata 136 that has been constructed from the optical display commands 132 and the transmitter location data 134. For example, the optical display metadata 136 may contain various properties such as the color, brightness and location of the optical display 122. In some examples the optical display metadata 136 may contain identifiers which identify the optical display 122 or even detailed GPS coordinates of various markers or locations within the optical display 122. For example, the GPS coordinates of the vertices of a polygon. The wireless inter-vehicle communication system 108 is used to transfer the optical display metadata 136 to the receiving vehicle 106.

[0070] The receiving vehicle 106 is further shown as comprising a receiver vehicular control system 140. The receiver vehicular control system 140 is shown as comprising a receiver computational system 142 that is in communication with a receiver memory 144, a camera system 146, a transceiver 148, and a global positioning system 150.

[0071] The receiver memory 144 is shown as storing receiver machine-executable instructions 154 which enable the receiver computational system 142 to control the other components of the receiver vehicular control system 140. The receiver memory 144 is further shown as containing the optical display metadata 136 that has been received via the wireless inter-vehicle communication system 108. The receiver memory 144 is further shown as containing a digital image 155 that was acquired by controlling the camera system 146. The camera system 146 may for example be a camera or a video system. The digital image 155 contains an image of the optical display 122. The receiver memory 144 is further shown as containing an artificial intelligence module 158 that may be incorporated into an autonomous driving system 162. The artificial intelligence module 158 may receive the digital image 155 and the optical display metadata 136 to provide a modified classification 160 of the optical display 122.

[0072] A difficulty is that the artificial intelligence module 158 may mistake the optical display 122 for a road hazard that causes the autonomous driving system 162 to erroneously react to. The use of the optical display metadata 136 may be used to properly classify the optical display 122 so that it does not cause a malfunction of the autonomous driving system 162. An advantage of this system is that the optical display metadata 136 can be used to ensure that the autonomous driving system 162 is not fooled and a LiDAR system is not needed to provide secondary confirmation that the optical display 122 is in fact real or is just a projection.

[0073] Fig. 2 shows a flowchart of a method which illustrates how to use the transportation system 100 of Fig. 1. In step 200 the optical display system 120 is controlled by the transmitter computational system 122 to generate the optical display 122 using the optical display commands 132. In step 202 the transmitter computational system 112 receives the transmitter location data 134 from the location system, which is in this case the global positioning system 118, of the transmitting vehicle 104.

[0074] In step 204 the transmitter computational system 112 constructs the optical display metadata 136 from the optical display commands 132 and the transmitter location data 134. In step 206 the transmitter computational system 112 controls the transceiver 116 to transmit the optical display metadata 136 across the wireless inter-vehicle communication system 108 to the receiver vehicular control system 140.

[0075] In step 208 the receiver computational system 142 receives the optical display metadata 136 via the transceiver 148. In step 210 the digital image 155 is received from the camera 146 or imaging system. In step 212 the modified classification 160 is received from the artificial intelligence module 158 in response to inputting the digital image 155 and the optical display metadata 136.

[0076] Fig. 3 illustrates a use case of the transportation system 100. In this case the transmitting vehicle 104 is parked to the side of the roadway 102 and is projecting an optical display 122 onto the roadway 102. In this example the optical display 122 is a so-called welcome symbol projected onto the roadway 102. A difficulty with this is that the receiver vehicle 106 may experience errors in its autonomous driving system 162 if the artificial intelligence module 158 accidentally classifies the welcome symbol 122 as an actual object on the roadway 102. The transmitter vehicle 104 sends the optical display metadata 136 via a wireless inter-vehicle communication system 108 which in this example is a V2X communication system. The camera system 146 of the receiver vehicle 106 receives the digital image 155 which contains the unknown obstacle which is the optical display 122. This image 155 as well as the optical display metadata 136 is input intothe artificial intelligence module 158. In this case the artificial intelligence module is an object perception.

[0077] Various types of information 136 may be provided to the artificial intelligence module 158. It could for example provide the type of symbol, which might include a cross shape, two parallel lines, a snowflake, a warning sign, a crosswalk, an arrow or a more complex pattern. This may also include various geometric and physical description of the symbol such as the type of shape or symbol and its properties, which might include the composition using lines, triangles, polygons, circles and other geometric shapes. It may include details of the dimensions, orientation and the position or spatial coordinates. The geometric and physical description may also include various physical aspects such as the color, brightness or other information. Also other information that could be provided such as when the type of optical display is: a projection of symbols onto the roadway, a vehicle body that is changing color, a panel on the vehicle displaying symbols, an indication of automated driving mode. This other information may also include a time sequence of the current & planned optical displays. For example: symbol 1 during 100ms, symbol 2 during Is, and etc. A confidence level in each previously provided information may also be included.

[0078] There may also be a message which is conveyed by the symbol which is included in the optical display metadata 136 such as the message which is intended or provided by the symbol. The message may for example include a "welcome message " for the driver, "warning about icy road" for the driver & other road users, "intended maneuver=turn right" for other road users, "you can cross" for the pedestrian, etc.

[0079] In any case, the artificial intelligence module 158 provides the modified classification 160 of the optical display 122 and provides it to the autonomous driving system 162 or autonomous driving subsystem. The autonomous driving system 162 may perform such things as making driving decisions, whether or not to perform emergency braking, which is one aspect of vehicle control.

[0080] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed examples.

[0081] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as an apparatus, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a"circuit," "module" or "system." Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer executable code embodied thereon.

[0082] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A 'computer-readable storage medium' as used herein encompasses any tangible storage medium which may store instructions which are executable by a processor or computational system of a computing device. The computer-readable storage medium may be referred to as a computer-readable non-transitory storage medium. The computer-readable storage medium may also be referred to as a tangible computer readable medium. In some embodiments, a computer-readable storage medium may also be able to store data which is able to be accessed by the computational system of the computing device. Examples of computer- readable storage media include, but are not limited to: a floppy disk, a magnetic hard disk drive, a solid state hard disk, flash memory, a USB thumb drive, Random Access Memory (RAM), Read Only Memory (ROM), an optical disk, a magneto-optical disk, and the register file of the computational system. Examples of optical disks include Compact Disks (CD) and Digital Versatile Disks (DVD), for example CD-ROM, CD-RW, CD-R, DVD-ROM, DVD-RW, or DVD-R disks. The term computer readable-storage medium also refers to various types of recording media capable of being accessed by the computer device via a network or communication link. For example, data may be retrieved over a modem, over the internet, or over a local area network. Computer executable code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0083] A computer readable signal medium may include a propagated data signal with computer executable code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0084] 'Computer memory' or 'memory' is an example of a computer-readable storage medium. Computer memory is any memory which is directly accessible to a computational system. 'Computer storage' or 'storage' is a further example of a computer-readable storage medium.Computer storage is any non-volatile computer-readable storage medium. In some embodiments computer storage may also be computer memory or vice versa.

[0085] A 'computational system' as used herein encompasses an electronic component which is able to execute a program or machine executable instruction or computer executable code. References to the computational system comprising the example of "a computational system" should be interpreted as possibly containing more than one computational system or processing core. The computational system may for instance be a multi-core processor. A computational system may also refer to a collection of computational systems within a single computer system or distributed amongst multiple computer systems. The term computational system should also be interpreted to possibly refer to a collection or network of computing devices each comprising a processor or computational systems. The machine executable code or instructions may be executed by multiple computational systems or processors that may be within the same computing device or which may even be distributed across multiple computing devices.

[0086] Machine executable instructions or computer executable code may comprise instructions or a program which causes a processor or other computational system to perform an aspect of the present invention. Computer executable code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages and compiled into machine executable instructions. In some instances, the computer executable code may be in the form of a high-level language or in a precompiled form and be used in conjunction with an interpreter which generates the machine executable instructions on the fly. In other instances, the machine executable instructions or computer executable code may be in the form of programming for programmable logic gate arrays.

[0087] The computer executable code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0088] Aspects of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program productsaccording to embodiments of the invention. It is understood that each block or a portion of the blocks of the flowchart, illustrations, and / or block diagrams, can be implemented by computer program instructions in form of computer executable code when applicable. It is further under stood that, when not mutually exclusive, combinations of blocks in different flowcharts, illustrations, and / or block diagrams may be combined. These computer program instructions may be provided to a computational system of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the computational system of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0089] These machine executable instructions or computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0090] The machine executable instructions or computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.REFERENCE SIGNS LIST100 transportation system102 roadway104 transmitting vehicle106 receiving vehicle108 wireless inter-vehicle communication system110 transmitter vehicular control system112 transmitter computational system114 transmitter memory116 transceiver118 global positioning system120 optical display system122 optical display130 transmitter machine executable instructions132 optical display commands134 transmitter location data136 optical display metadata140 receiver vehicular control system142 receiver computational system144 receiver memory146 camera system148 transceiver150 global positioning system154 receiver machine executable instructions155 digital image158 artificial intelligence module160 modified classification of optical display162 autonomous driving system200 control an optical display system of the transmitting vehicle to generate an optical display using optical display commands202 receive transmitter location data from a location system of the transmitting vehicleconstruct optical display metadata from the optical display commands and the transmitter location data broadcast the optical display metadata via a wireless inter-vehicle communication system receive optical display metadata descriptive of an optical display generated by a transmitting vehicle via a wireless inter-vehicle communication system, wherein the display metadata is descriptive of transmitter location data of the transmitting vehicle receive a digital image that comprises the optical display receive a classification of the optical display in response to inputting the digital image and the optical display metadata into the artificial intelligence module

Claims

CLAIMS1. A system comprising a transmitter vehicular control system (110) of a transmitting vehicle (104) and a receiver vehicular control system (140) of a receiving vehicle (106), wherein the transmitter vehicular control system comprises: a transmitter memory (114) storing transmitter machine executable instructions (130); a transmitter computational system (112), wherein execution of the transmitter machine executable instructions causes the transmitter computational system to: o control (200) an optical display system (120) of the transmitting vehicle to generate an optical display (122) using optical display commands (132); o receive (202) transmitter location data (134) from a location system (118) of the transmitting vehicle; o construct (204) optical display metadata (136) from the optical display commands (132) and the transmitter location data (134); o broadcast (206) the optical display metadata via a wireless inter-vehicle communication system (108), wherein the receiver vehicular control system comprises: a receiver memory (144) storing receiver machine executable instructions (154) and an artificial intelligence module (158), wherein the artificial intelligence module is configured to classify road objects in a digital image (155); a receiver computational system (142), wherein execution of the receiver machine executable instructions causes the receiver computational system to: o receive (208) optical display metadata (136) descriptive of an optical display (122) generated by a transmitting vehicle (104) via a wireless inter-vehicle communication system (108), wherein the display metadata is descriptive of transmitter location data (134) of the transmitting vehicle;o receive (210) the digital image that comprises the optical display; and o receive (212) a classification (160) of the optical display in response to inputting the digital image and the optical display metadata into the artificial intelligence module, wherein the artificial intelligence module is configured to modify the classification of the optical display using the optical display metadata.

2. The system of claim 1, wherein the optical display system comprises a projector configured for projecting an optical display onto a surface adjacent to the transmitting vehicle.

3. The system of claim 1, wherein the optical display metadata comprises a geometrical description of the optical display, wherein optionally the geometrical description comprises one or more physical locations of one or more portions of the optical display.

4. The system of claim 1, 2, or 3, wherein anyone of the following:- the optical display comprises an exterior display panel on the vehicle and / or- the optical display comprises a lighting system configured to indicate a manual driving mode or an automated driving mode and / or- wherein the optical display comprises an optical system configured for dynamically changing vehicle color5. The system of any one of the preceding claims, wherein the optical display is any one of the following: a turn signal, a yield signal, a cross walk projection, a predicted vehicle path, a welcome display for the driver, a vehicle malfunction warning, a road hazard indicator, and a warning of an occupant to exit the vehicle.

6. The system of any one of the preceding claims, wherein the wireless transmission system is any one of the following: a radio-frequency data exchange system, a wi-fi data exchange system; a V2X system, and a cellular data exchange system.

7. The system of claim 1, wherein modifying the classification of the optical display comprises ignoring or deleting the classification of the optical display.

8. The system of claim 1, wherein execution of the receiver machine executable instructions further causes the receiver computational system to modify operation of a driver assistance module of the receiver vehicle or modify operation of an autonomous driving system of the receiver vehicle using the modified classification of the optical display.

9. A method of operating a transmitter vehicular control system (110) of a transmitting vehicle (104), wherein the method comprises: o controlling (200) an optical display system (120) of the transmitting vehicle to generate an optical display (120) using optical display commands; o receiving (202) transmitter location data (134) from a location system (118) of the transmitting vehicle; o constructing (204) optical display metadata (136) from the optical display commands and the transmitter location data; and o broadcasting (206) the optical display metadata via a wireless inter-vehicle communication system (108).

10. A method of operating a receiver vehicular control system (140) of a receiving vehicle (106), wherein the method comprises: o receiving (208) optical display metadata (136) descriptive of an optical display (122) generated by a transmitting vehicle (104) via a wireless inter-vehicle communication system (108), wherein the display metadata is descriptive of transmitter location data of the transmitting vehicle; o receiving (210) a digital image (155) that comprises the optical display; and o receiving (212) a classification (160) of the optical display in response to inputting the digital image and the optical display metadata into an artificial intelligence module (158), wherein the artificial intelligence module is configured to classify road objects in the digital image, wherein the artificial intelligence module is configured to modify the classification of the optical display using the optical display metadata.

11. A computer program comprising machine executable instructions (130, 154) for implementing the method of claim 11 or 12.

Images