Driving assistance for non-autonomous vehicles in autonomous environments

Abstract

Disclosed are techniques for providing driving assistance to a non-autonomous vehicle when operating in an autonomous vehicle environment. In one embodiment, a method is disclosed that includes: establishing a secure connection with an object selected from a group consisting of a road and lanes of the road; receiving a packet from the object, the packet describing a condition of the object; verifying the packet; generating an enhanced display using data within the packet; and displaying the enhanced display in a vehicle.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 363,033, filed on Mar. 25, 2019, and entitled "Driver Assistance for Non-Autonomous Vehicle in an Autonomous Environment", the entire disclosure of which is hereby incorporated herein by reference.

[0003] Copyright Notice

[0004] This application contains copyrighted material. The copyright owner does not object to the facsimile reproduction by anyone of the patent disclosure as it appears in the Patent and Trademark Office file or records, but reserves all copyrights in any event in all other respects. Background Art

[0005] The disclosed embodiments are directed to autonomous vehicles, and more particularly, to a system for allowing non-autonomous vehicles to navigate in an autonomous vehicle environment.

[0006] As more human-operated vehicles are replaced by autonomous vehicles, various features of roads and other environments will necessarily change. Specifically, current roads are designed for human operators. Road signs and warnings are primarily visual (e.g., road signs, traffic lights, construction warnings, etc.). With fewer human-operated vehicles, these visual indicators will cease to be useful and will thus be able to be removed.

[0007] However, the removal of the above-mentioned indicators cannot be achieved until all vehicles on the road are autonomous vehicles. This scenario delays the removal of the visual indicators or may prevent the removal of these indicators. For example, while autonomous vehicles are being rapidly adopted, the adoption of 100% autonomous vehicles will obviously take much longer. In addition, there are still some scenarios where vehicles (e.g., emergency or repair vehicles) may still retain human operators. Therefore, under the current system, visual indicators cannot be removed and may never be removed.

[0008] Current systems have made a great deal of effort to improve the performance of autonomous vehicles. "Intelligent" roads (also known as intelligent transportation systems) have been proposed or designed to allow continuous communication with autonomous vehicles. However, these proposals or systems do not have a way to notify non-autonomous vehicles of road conditions or road signs. In addition, many of these systems (or particularly the proposals) fail to include sufficient security measures to prevent man-in-the-middle attacks and other malicious behaviors that could endanger autonomous vehicles or passengers.

[0009] Accordingly, there is a need in the art for displays utilizing invisible environmental indicators to enhance non-autonomous vehicles. SUMMARY OF THE INVENTION

[0010] Disclosed herein are methods, computer-readable media, and apparatuses for providing driving assistance to a non-autonomous vehicle when operating in an autonomous vehicle environment.

[0011] As will be described herein, the disclosed embodiments describe systems that can enhance non-autonomous vehicles using indicator data received from the environment. In some embodiments, the non-autonomous vehicle is equipped with an augmented reality (AR) or head-up display that presents real-time indicators (e.g., traffic lights, stop signs, construction notices, speed limits, weather, etc.) transmitted by roads or devices near the road. In some embodiments, these signals can be combined with real-time data regarding vehicle operation to provide an accurate visualization of the operations the vehicle is performing and the operations that may be required in the future. In some embodiments, visual indicators are used in place of text indicators (e.g., visualization of a worker instead of the text "workers present") to provide clear indicators that can be used regardless of whether the driver's language is synchronized with the local language. The disclosed embodiments utilize secure identification and communication protocols (e.g., DICE-RIoT) to ensure communication between the vehicle and the road / markers.

[0012] In one embodiment, a method is disclosed that includes: establishing a secure connection with an object selected from a group consisting of a road and lanes of the road; receiving a packet from the object, the packet describing a condition of the object; validating the packet; generating an enhanced display using data within the packet; and displaying the enhanced display in the vehicle.

[0013] In one embodiment, a non-transitory computer-readable storage medium is disclosed for tangibly storing computer program instructions executable by a computer processor, the computer program instructions defining the steps of: establishing a secure connection with an object selected from a group consisting of a road and lanes of the road; receiving a packet from the object, the packet describing a condition of the object; validating the packet; generating an enhanced display using data within the packet; and displaying the enhanced display in the vehicle.

[0014] In one embodiment, a device is disclosed that includes: a processor; and a storage medium for tangibly storing program logic thereon for execution by the processor, the stored program logic including: logic executed by the processor for establishing a secure connection with an object selected from a group consisting of a road and lanes of the road; logic executed by the processor for receiving a packet from the object, the packet describing a condition of the object; logic executed by the processor for verifying the packet; logic executed by the processor for generating an enhanced display using data within the packet; and logic executed by the processor for displaying the enhanced display in a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The foregoing and other objects, features, and advantages of the present disclosure will become apparent from the following description of embodiments as illustrated in the accompanying drawings, in which like reference numerals refer to the same parts throughout the various views. The drawings are not necessarily to scale; in fact, emphasis is placed on illustrating the principles of the present disclosure.

[0016] Figure 1 A flowchart illustrating a method for receiving an indicator packet according to some embodiments of the present disclosure.

[0017] Figure 2 A flowchart illustrating a method for generating a display from an indicator packet according to some embodiments of the present disclosure.

[0018] Figure 3 A flowchart illustrating a method for responding to an indicator packet according to some embodiments of the present disclosure.

[0019] Figure 4 A flowchart illustrating a method for processing vehicle data generated in response to an enhanced display.

[0020] Figure 5 A packet structure diagram according to some embodiments of the present disclosure.

[0021] Figure 6 A diagram of a blockchain and its respective blocks according to some embodiments of the present disclosure.

[0022] Figure 7 A block diagram of an autonomous vehicle according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Figure 1 A flowchart illustrating a method for receiving an indicator packet according to some embodiments of the present disclosure.

[0024] In steps 102 and 104, the vehicle travels along the road and maintains / updates an RFID (or similar) connection with the road and / or with the lanes of the road. Although RFID is used as an example, other protocols such as Wi-Fi, CDMA, W-CDMA, DSRC, or 5G LTE can be used instead of RFID.

[0025] In the illustrated embodiment, the DICE-RIoT protocol is used to authenticate and secure the communication between the vehicle and the road (or lane). Details of how to utilize the DICE-RIoT protocol are described in more detail in co-pending application 16 / 034,763, filed on July 13, 2018, and entitled "Secure Vehicular Communication" (Dkt. No. 2018-0274), and the details of that application are incorporated herein by reference in their entirety.

[0026] Briefly, the vehicle and the road / lane are equipped with transmitters that include a RIoT core processing module. This processing module is configured to generate a secure "triple" that includes an encrypted identifier, a public key, and a security certificate. These values are generated based on the Fusion Device Secret (FDS), which is only available to the immutable bootloader (level 0) upon startup or restart. The bootloader uses the FDS to generate a composite device identifier that is based on the hash of the FDS and the "level 1" code.

[0027] The RIoT core additionally creates an asymmetric device ID key pair using a deterministic key generation function seeded with the CDI. Additionally, the RIoT core generates an encrypted device identifier and a certificate. The device identifier, the certificate, and the public device ID key are exported to a layer 1 (L1) software executed in an operating system (OS) of, for example, the vehicle or road transmitter. Additionally, the RIoT core generates a certificate that contains the device ID public key. The certificate allows other devices to confirm that the device ID public key is associated with the L0 and L1 layers of the RIoT core. In one embodiment, the RIoT core signs the certificate using the device ID private key. In another embodiment, the RIoT core generates a certificate signing request ("CSR") structure (e.g., a PKCS10 request) for the device ID public key signed by the device ID private key.

[0028] The RIoT core exports the device ID public key, the certificate, and the vehicle identifier to the L1 software. The L0 and L1 layers refer to software or firmware executed on a device (e.g., the road), while L2 refers to a layer on an external device (e.g., the vehicle). This triple is typically represented as {ID Lk public, ID Lk certificate, K Lk{Disclosure}. The L1 software may include system-level or user-level code. There is no limitation on the specific nature of the code. However, as will be described, one of the main functions of the L1 software is to securely transmit road or environmental information.

[0029] In addition, the L1 on a given device may employ a dual encryption technique for generating the ID certificate portion of a triple of L2 code that is exported to another device. Specifically, the L1 code first uses the ID L1public key as the encryption key to encrypt K L1public key using a first encryption module, thereby generating an encrypted ciphertext K'. This ciphertext is then used as the data supplied to a second encryption module, while the K L1private generated by a second generator is used as the encryption key. The result of the second encryption module is a double-encrypted ciphertext K'' that is output as the ID L1 certificate. The specific choice of encryption algorithm (and corresponding decryption algorithm) is not limited in this disclosure.

[0030] Each device also includes certificate checking circuitry and / or software.

[0031] Each device is equipped with certificate checking (in hardware and / or software), which decrypts the ID L1 certificate using the K L1 public key via a first decryption module. The resulting key M' is then used as the key for a second decryption module, which decrypts the ID L1 public key using the key M'. Finally, the resulting plaintext M'' is compared with the original public key (K L1public ) via a comparator. If the result of the comparator is affirmative, the triple is confirmed to be valid. If not, the triple is marked as invalid. The certificate checker can be executed independently of the triple generation circuitry / software. In this way, the triples of the respective entities can be verified by the L2 code.

[0032] In the illustrated embodiment, the vehicle and the road exchange and verify triples to ensure the integrity of the recipient. After this verification, the vehicle and the road establish a secure connection (such as a TLS connection) to communicate securely. As illustrated, this connection can be switched to other roads when the vehicle is in motion.

[0033] In step 106, the method receives an indicator packet from the road.

[0034] In the illustrated embodiment, the indicator packet refers to data to be displayed to the vehicle operator. Generally, the indicator packet is associated with an indication. The indications include, for example, objects such as road signs, traffic lights, road conditions, construction notices, and any object or event that is generally visible on the road or otherwise represented by signs or other visual indicators.

[0035] Each indication is associated with a location. For example, a traffic light indicator is associated with a point (or area) near an intersection. As described above, the physical traffic light can be removed from the intersection. Thus, the indicator is a virtual representation of these former physical objects. In some embodiments, the indicator includes coordinates (or a geofence) and a computationally processable description of the indicator. For example, a traffic light is associated with a point on a road and describes, for example, the timing of the traffic light. This type of indicator can be managed by a central (e.g., government) agency and can include fixed or semi-permanent indicators. As another example, a speed limit sign can include an indicator. In this context, the indicator can be permanent throughout the road and is not limited to a specific point. Thus, the indicator can be transmitted repeatedly as the vehicle travels along a given road. As another example, the indicator can include a temporary indicator. For example, a construction crew can record a construction event occurring on a given section of a road, which has a fixed start and end time. In another embodiment, a physical transmitter can be inserted into the road, allowing for the installation of special-purpose indicators within the road.

[0036] The indicator data is grouped and transmitted from the road to the vehicle via a secure channel established between the road and the vehicle. Figure 5 is a block diagram of grouping according to some embodiments of the present disclosure, and the disclosure of the figure is incorporated herein by reference.

[0037] In step 108, the method verifies the group signature.

[0038] As Figure 5 described in more detail, the sender signs each group using a private key. In step 108, the receiver (e.g., the vehicle) can use the sender's public key included in the message itself to verify the identity and authenticity of the sender. Thus, in step 108, the method ensures that the sender is indeed who they claim to be.

[0039] In step 112, if the signature verification fails, then the method discards the message and records the failed message. In one embodiment, the recording of the message includes presenting a new response block to a blockchain as Figure 6 described.

[0040] In step 114, if the signature verification passes, then the method determines whether the message is directed to the appropriate lane. In steps 102 and 104, the vehicle maintains the identifier of the current lane and coordinates with the road via a connection establishment process. The group received in step 106 can include a group broadcast by the road to all vehicles but identifying a specific lane. In step 114, the method ensures that the lane included in the group matches the lane in which the vehicle is located at a given moment.

[0041] In step 116, if two validations pass, then the method updates the Advanced Driver Assistance System (ADAS).

[0042] As is known in the art, ADAS is a system that can automate, adjust, and enhance vehicle systems for safe and better driving. ADAS includes electronic stability control, anti-lock braking, lane departure warning, adaptive cruise control, and traction control subsystems (either individually or in combination), as well as any additional systems.

[0043] In the illustrated embodiment, indicator data can be used to further refine the data used by these ADAS systems. For example, road sign indicators can be used to improve the GPS navigation system, the driver's reaction to speed warnings can be used to improve the hill descent control system, speed limit indicators can be used to optimize the speed limit control system, and other uses. This specification places no limitation on the types of ADAS systems that can be used in conjunction with indicator data, and this specification contemplates any such use.

[0044] In some embodiments, the packet received in step 106 may include a temporary indicator. For example, the packet may indicate that a natural disaster or a human accident (e.g., flood, earthquake, motor vehicle accident, etc.) is occurring. This information can be transferred to the ADAS to appropriately update those systems.

[0045] Additionally, in some embodiments, the method may update the database of speed limits in the ADAS system or update the traffic light status of the traffic lights stored in the ADAS.

[0046] In step 118, the method uses the packet data to generate a display. This process is described in more detail in Figure 2 In.

[0047] Figure 2 A flowchart illustrating a method for generating a display from an indicator packet according to some embodiments of the present disclosure.

[0048] In step 202, the method receives a packet. In the illustrated embodiment, the packet includes a packet received from a road or lane as described in Figure 1 As described in. As illustrated in Figure 5 The packet contains a payload that describes the indicator and may include, for example, a location (e.g., coordinates), an indicator type (e.g., speed limit sign), data about the indicator (e.g., the legal speed limit), and any other data related to the indicator. The foregoing example of a speed limit sign is used herein.

[0049] In step 204, the method receives an indicator sub-screen, and in step 206, the method fills the sub-screen with the packet payload data.

[0050] In one embodiment, the sub - screen includes one (or more) graphical element(s) associated with a given indicator type. In some embodiments, the vehicle stores these sub - screens locally. In other embodiments, the vehicle may retrieve the sub - screens from a remote central location. In alternative embodiments, the sub - screens may be included within a payload.

[0051] In some embodiments, the sub - screen includes two fixed elements, an animated element, and a placeholder element. The fixed elements include shapes that do not move relative to other shapes within the sub - screen. The animated element contains elements that move along a fixed path relative to the fixed elements. The placeholder element includes replaceable components of the sub - screen, such as text content, contextual graphics, etc. In some embodiments, the sub - screen may be location - specific and may be customized based on the vehicle's jurisdiction.

[0052] Continuing with the speed - limit example, the sub - screen may contain a fixed element, including a U.S. speed - limit sign (e.g., a white rectangular shape) with the text "Speed Limit". The sub - screen may additionally contain a placeholder for inserting the speed limit into the sub - screen. As another example, a construction sub - screen may contain an animated portion of a construction crew at work. In some embodiments, the fixed elements may also be internationalized to accommodate the vehicle's operator. Thus, the aforementioned sub - screen may say "Maximum Speed (VELOCIDAD MAXIMA)" instead of "Speed Limit". Additionally, in some embodiments, the data inserted into the placeholder may be translated based on jurisdiction. For example, the speed reported in a road report may be in kilometers per hour, while the U.S. sign may be converted to miles per hour, and vice versa (less likely).

[0053] Note that in some embodiments, the sub - screen does not contain text elements and relies primarily on graphical or numerical elements. For example, the speed limit may be represented as the vehicle's current speed and maximum speed (e.g., "45 / 65 MPH"). Additionally, when the user approaches 65, the speed - limit sub - screen may change color from green to orange to red. Further, the sub - screen may be equipped with an audio portion that plays a warning when the user reaches the speed limit.

[0054] In step 208, the method optionally superimposes the vehicle graphics over the sub - screen.

[0055] As indicated, step 208 may be optional. For example, the vehicle overlay may not be displayed on the parking sign. However, as an additional example, the indicator may include a dangerous curve indicator. In existing roads, this sign includes a general n-shaped curve or a similar shape. Alternatively, the sign is simply called "DANGEROUS CURVE AHEAD". Compared with this system, the indicator may include an identification of the type of dangerous curve, but may also specify the speed specific to the dangerous curve and the path including the curve. Using this data, the sub-image associated with the dangerous curve may be more complex than a graphical element. For example, the sub-image may include a rendering of the curve itself dynamically generated using the path data, and may be updated dynamically when the vehicle approaches the curve and when the user is within the curve.

[0056] In step 210, the method generates a distortion function based on the position of the indicator.

[0057] As described above, the display may include an augmented reality display or a head-up display. Therefore, the generated display must be appropriately distorted to simulate its position in the real-world space. In this way, the method uses the geographical coordinates (fixed or well-known) of the indicator and the geographical location of the vehicle to generate a distortion function for the sub-image. Therefore, the distortion function distorts or warps the perspective of the sub-image when displayed.

[0058] Since the vehicle is often in a state of continuous motion, the method generates a function that distorts the sub-image based on the real-time vehicle position and the display positioning within the operator's field of view. In this way, the method generates a function with inputs of the current position, speed, and acceleration, and manipulates the outline of the sub-image such that the sub-image simulates a real-world object in the display. Therefore, when the user approaches the position of the indicator, the parking light display will become smaller and continue to grow.

[0059] In step 212, the method renders the sub-image using the distortion function. As described above, the method may display the sub-image on an augmented reality display placed inside the vehicle, such as on the vehicle windshield. Alternatively, the method may display the sub-image on augmented reality glasses worn by the user when driving, which will reduce the modification to the vehicle.

[0060] In step 214, the method continues to determine whether the sub-image is still valid. As used herein, a valid sub-image refers to a sub-image that is still relevant to the vehicle and / or the vehicle operator. For example, a speed limit sign may be valid only when the user is located in the road section associated with the indicator. In some embodiments, step 214 simply monitors the presence of the indicator and whether the time-to-live (TTL) of the indicator has expired.

[0061] In step 216, the method recalculates the distortion of the sub - picture when the sub - picture is valid. This recalculation uses the distortion function identified in step 212 and includes re - evaluating the function with the updated position and / or predicted future position.

[0062] In step 218, the method records the validity of the sub - picture after determining that the sub - picture is no longer being displayed.

[0063] In one embodiment, the recording in step 218 is first performed locally and can be exported to an external source. In one embodiment, the recorded data is included in a response packet transmitted from the vehicle to the road and is ultimately recorded in a blockchain data structure. The validity of the sub - picture generally refers to the performance data of the vehicle collected during the lifetime of the sub - picture. For example, for a speed - limit sub - picture, the performance data will include the actual speed, or to reduce bandwidth congestion, an indication of when (if ever) the user exceeded the speed limit. For a dangerous - curve indicator, the validity may also include the above - mentioned speed data and may also include whether the user crossed into another lane or onto the shoulder of the road. In some embodiments, the validity may include a manual identification of a failed sub - picture. For example, an error indicator may be marked by an operator (e.g., an incorrect identification of a construction zone).

[0064] It should be noted that the above method is described with respect to a single indicator. However, the method can operate in parallel for any number of indicators present at a given moment. Thus, the method can simultaneously display a speed - limit sign, a dangerous - curve graphic, an exit - ramp sign, and various other sub - pictures. The method can employ various rules that govern how such sub - pictures are aligned simultaneously during display.

[0065] Figure 3 A flowchart illustrating a method for responding to grouped indicators according to some embodiments of the present disclosure.

[0066] In step 302, the method retrieves vehicle operation data.

[0067] In the illustrated embodiment, the vehicle operation data includes data on how the vehicle performs during the display of the indicator. This data includes data such as the vehicle's speed, direction, acceleration, position, etc. during the indicator period.

[0068] In step 304, the method optionally receives Figure 2 the display validity generated in step 218 of

[0069] In step 306, the method packs the operation data and, if available, packs the validity data into one or more packets. As Figure 5 described in

[0070] In step 308, the method signs the packaged data using the private key. This operation is described in more detail in Figure 1 and, for the sake of clarity, is not repeated herein.

[0071] In step 310, the method transmits the signed packet data to the road via the secure channel discussed in Figure 1 .

[0072] Figure 3 The foregoing method in Figure 3 can be performed periodically during the display of the indicator sub-screen. Alternatively or in combination with the foregoing, the method in Figure 2 can be performed in batch after the indicator becomes invalid or expires, as described in

[0073] In some embodiments, steps 302 and 304 may be omitted. In this embodiment, the method may actually generate a request for information from the road. In this way, the vehicle itself can actively request indicator data from the road. This query can be broadly formatted to request all applicable indicators for a given section of the road. Alternatively, it can be narrowly defined as a request for the speed limit on a given section of the road.

[0074] Figure 4 Flowchart illustrating a method for processing vehicle data generated in response to an enhanced display.

[0075] In step 402, the method receives the packaged data. In the illustrated embodiment, the method described in Figure 4 can be performed by the road or lane. In this embodiment, the packaged data includes the data generated in the method of Figure 3 .

[0076] In step 404, the method determines whether the signature of the packaged data is valid. Details of checking the packet signature are described elsewhere and are not repeated herein, but are incorporated by reference in their entirety.

[0077] In step 406, if the signature verification fails, the method discards the message and records a failure message. In one embodiment, the recording of the message includes presenting a new response block to a blockchain as described in Figure 6 .

[0078] In step 408, if the signature verification is successful, the method records the received data. In one embodiment, the recording of the message includes presenting a new response block to a blockchain as described in Figure 6 .

[0079] In step 410, the method analyzes the validity of the data.

[0080] As described above, the data returned by the vehicle can include the operating data of the vehicle during the display of an indicator. For example, the user speed during the display of a speed limit indicator. In step 410, the method compares the actual data with the expected data. For example, the speed limit is compared with the actual user speed. If the data indicates that the user has not noticed the indicator, it means that the indicator may be potentially ineffective (e.g., not significantly located). Alternatively, if the user complies with the indicator, it indicates that it is an effective indicator.

[0081] In some embodiments, the above analysis can be performed on a per-user basis. That is, the responses of the user to all speed limit signs can be aggregated to determine whether a particular indicator is effective relative to a particular operator. Alternatively or in combination with the foregoing, the effectiveness data can be aggregated across multiple users to address the effectiveness of the indicator itself.

[0082] In step 412, the method updates the indicator data based on the effectiveness.

[0083] As described above, the method can mark those ineffective indicators for a particular user or for all users. The method can then modify the sub-screen associated with the indicator to enhance its effectiveness. For example, if a particular user routinely exceeds the speed limit, the method can automatically increase the size of the indicator. Alternatively, the method can forward the result to a team of human editors, who can replace the sub-screen with a new one based on their own expertise.

[0084] Figure 5 It is a block diagram of a grouping according to some embodiments of the present disclosure.

[0085] In the illustrated embodiment, two groupings (502a, 502b) are illustrated. The first grouping (502a) describes the data transmitted from the road to the vehicle, while the second grouping (502b) describes the data transmitted from the vehicle back to the road.

[0086] The packet (502a) includes an identification portion (502 to 508), a payload (510), and a signature (512). The identification portion (502 to 508) includes a road or lane identifier (502). This identifier (502) includes an encrypted-generated identifier of the road or lane. In one embodiment, the RIoT core generates this identifier (502) using an asymmetric key generator seeded with a CDI. The identification portion (502 to 508) further includes a road or lane identifier number (504). This identifier (504) includes a textual or numerical identifier of the road or lane (e.g., "US RTE1", "US RTE1, segment 123", "US RTE1, segment 123, lane 2") as compared to the identifier (502). Other coding systems may be used to code individual roads or lanes. The identification portion (502 to 508) further includes a certificate (506) and a public key (508). The certificate (506) and the public key (508) are also generated by the RIoT core of the road. In one embodiment, the public key (508) is generated using an asymmetric key generator seeded with a random number. In one embodiment, the certificate is formed by encoding the public key using a private key that is the output of the aforementioned asymmetric key generator seeded with a CDI. This encrypted output is used as data to be encrypted using a private key generated by an asymmetric key generator seeded with a random number. The packet (502a) further includes a payload portion (510). This payload includes, for example, the indicator information described previously. Finally, the sender signs the above fields (502 to 510) using the sender private key (generated by an asymmetric key generator seeded with a random number), and the signature is included within a digital signature (512). In some embodiments, the packet may further include a freshness indicator (monotonic counter, timestamp, etc.) to avoid replay attacks.

[0087] Grouping (502b) is structured similarly to grouping (502a), and the overlapping details will not be repeated herein. Generally, grouping (502b) includes acknowledgment packets sent by the vehicle to the road. In some embodiments, grouping (502b) may include status packets transmitted by the vehicle. Grouping (502b) contains an encrypted vehicle identifier (514), a vehicle certificate (518), and a vehicle public key (520), which is generated by the vehicle in the same manner as described in connection with grouping (502a). In the illustrated embodiment, the identification portion (514 to 520) of grouping (502b) contains an extended vehicle identifier (516) similar to a road / lane identifier number, including a human-readable identifier such as a vehicle identifier number or a similar serial number. Grouping (502b) further contains a payload (522), which contains response data transmitted by the vehicle to the road, as previously discussed. Finally, the sender signs the above fields (514 to 522) using a sender private key (generated by an asymmetric key generator seeded with a random number), and the signature is included within a digital signature (524). In some embodiments, the packet may further contain a freshness indicator (e.g., a monotonic counter, a timestamp, etc.) to avoid replay attacks.

[0088] Figure 6 A diagram of a blockchain and individual blocks therein according to some embodiments of the present disclosure.

[0089] In the illustrated embodiment, blockchain (600) contains a plurality of blocks, e.g., blocks 610a to 610c, 612, 614a to 612n, and 616. In one embodiment, blocks 610a to 610c and 614a to 612n include vehicle-proposed blocks (described in more detail in connection with block 603). Road / lane-proposed blocks (612, 616) are interleaved with vehicle-proposed blocks (610a to 610c and 614a to 612n). These blocks are generated (and signed) by the road device itself. As illustrated, vehicle-proposed blocks (610a to 610c and 614a to 612n) may include response blocks to an initial road / lane block.

[0090] Alternatively, blockchain (600) is illustrated using a plurality of blocks (601 to 604), where the block structure of two blocks (602, 603) is expanded. In one embodiment, blocks (602, 603) correspond to blocks (612, 614a).

[0091] In the illustrated embodiment, two blocks (602, 603) contain main ledger block headers (602a, 603a), which contain the hash of the current block and the hash of the previous block. In the illustrated embodiment, main ledger block headers (602a, 603a) include standard blockchain headers for creating a linked list of transactions.

[0092] Each block contains an encrypted sender identifier (602b, 603b), an extended identifier (602c, 603c), a sender public key (602d, 603d), and a sender certificate (602e, 603e). Details of these fields are described in Figure 5 and are not repeated herein for clarity. As illustrated, in block (602), the public key (602d) may be optional. Blocks (602a, 602b) also contain data (602f, 603f) corresponding to the payloads (510, 522).

[0093] Finally, each block is signed by the sender, and the signature is packaged and included in the block as a packaged digital signature (602g, 603g). In the illustrated embodiment, the signature (602g, 603g) is signed using the sender's private key and verified by the receiver using the sender's public key.

[0094] In the illustrated embodiment, the method records data into the blockchain data structure in multiple steps. In each step, as described above, recording the data includes adding a block to the blockchain data structure. In the illustrated embodiment, a block is identified as part of the block (i.e., included in the ledger) if two conditions are met. First, during insertion, the hash of the previous block must match the hash of the previous block as previously calculated. That is, before inserting block 603, the "Hash (Previous Local Block)" of header 603a must match the "Hash (Current Local Block)" of the previous block 602a. In addition to this requirement, the block must meet a second non-standard condition that the block is generated by an authorized host of the system. This particular requirement is actually an alternative to the proof-of-work in a traditional blockchain system (i.e., the mathematical competition between nodes). Specifically, in the illustrated blockchain, the nodes participating in the system attempt to guess the right-hand signature to insert a block into the blockchain. However, in the system, one entity in the system has the private key for generating the signature and can thus quickly "win" the competition (while ensuring that it can be verified by any other node using the public key of the private key user). In this way, only the true signer can win the competition, thereby preventing unauthorized entry of blocks into the blockchain.

[0095] Figure 7 Block diagram of an autonomous vehicle according to some embodiments of the present disclosure.

[0096] Figure 7 The system illustrated in can be fully installed inside the vehicle. In some embodiments, some components (e.g., components and subsystems other than subsystem (704)) may include existing autonomous vehicle subsystems.

[0097] The system includes an autonomous vehicle subsystem (702). In the illustrated embodiment, the autonomous vehicle subsystem (702) includes a map database (702A), a radar device (702B), a lidar device (702C), a digital camera (702D), a sonar device (702E), a GPS receiver (702F), and an inertial measurement unit (702G). Each of the components of the autonomous vehicle subsystem (702) includes standard components provided in the latest autonomous vehicles. In one embodiment, the map database (702A) stores a plurality of high-definition three-dimensional maps for route selection and navigation. The radar device (702B), lidar device (702C), digital camera (702D), sonar device (702E), GPS receiver (702F), and inertial measurement unit (702G) may include various corresponding devices located at various positions throughout the autonomous vehicle as known in the art. For example, these devices may be mounted along the perimeter of the autonomous vehicle to provide position awareness, collision avoidance, and other standard autonomous vehicle functionality.

[0098] A vehicle subsystem (706) is additionally included within the system. The vehicle subsystem (706) includes various antilock braking systems (706A), an engine control unit (706B), and a transmission control unit (706C). These components may be used to control the operation of the autonomous vehicle in response to the streaming data generated by the autonomous vehicle subsystem (702). The standard autonomous vehicle interactions between the autonomous vehicle subsystem (702) and the vehicle subsystem (706) are generally known in the art and are not described in detail herein.

[0099] The processing side of the system includes one or more processors (710), a short-term memory (712), an RF system (714), a graphics processing unit (GPU) (716), a long-term storage device (718), and one or more interfaces (720).

[0100] One or more processors (710) may include a central processing unit, an FPGA, or any range of processing devices required to support the operation of the autonomous vehicle. The memory (712) includes DRAM or other suitable volatile RAM for temporarily storing data required by the processor (710). The RF system (714) may include a cellular transceiver and / or a satellite transceiver. The long-term storage device (718) may include one or more large-capacity solid-state drives (SSDs). Generally, the long-term storage device (718) may be used to store, for example, high-definition maps, route selection data, and any other data that requires permanent or semi-permanent storage. The GPU (716) may include yet another high-throughput GPU device for processing the data received from the autonomous vehicle subsystem (702). Finally, the interface (720) may include various display units (e.g., built-in screens) located within the autonomous vehicle.

[0101] The system further includes an enhancer subsystem (704) that performs operations required for the methods illustrated in the foregoing figures. The enhancer subsystem (704) includes a subpicture database (704a) that stores (either permanently or as a cache of a cloud-based system) subpicture data to be used for generating augmented reality or head-up display data.

[0102] The enhancer subsystem (704) further includes a parser (704b). In one embodiment, the parser (704b) is configured to receive messages from the road (e.g., using the DICE-RIoT protocol) and extract the data required for the subpictures stored in the database (704a). The parser (704b) is further configured to insert the data into the subpictures.

[0103] The enhancer subsystem (704) further includes a renderer (704c). In one embodiment, the renderer (704c) is responsible for obtaining the completed subpictures and generating a distortion function. Additionally, the renderer (704c) is responsible for monitoring the position of the vehicle and the validity of the subpictures, and refreshing the display of the subpictures on the display (via interface 720).

[0104] Each of the devices is connected via a bus (708). In one embodiment, the bus (708) may include a Controller Area Network (CAN) bus. In some embodiments, other bus types (e.g., FlexRay or MOST bus) may be used. Additionally, each subsystem may include one or more additional buses (e.g., a LIN bus for low-bandwidth communication) for handling internal subsystem communication.

[0105] However, the subject matter disclosed above may be implemented in a variety of different forms, and thus, the subject matter covered or claimed is intended to be understood as not limited to any of the example embodiments set forth herein; the example embodiments are provided only for illustrative purposes. Similarly, a reasonably broad scope of the claimed or covered subject matter is contemplated. Among other things, for example, the subject matter may be embodied as a method, apparatus, component, or system. Thus, an embodiment may, for example, take the form of hardware, software, firmware, or any combination thereof (other than software itself). Accordingly, the following detailed description is not to be taken in a limiting sense.

[0106] Throughout this specification and the claims, the terms may have nuanced meanings that are suggested or implied in the context beyond the explicitly stated meanings. Similarly, as used herein, the phrase "in one embodiment" does not necessarily refer to the same embodiment, and the phrase "in another embodiment" does not necessarily refer to a different embodiment. It is contemplated that, for example, the claimed subject matter includes combinations of whole or partial example embodiments.

[0107] In general, terms may be understood, at least in part, based on their use in context. For example, terms such as "and", "or", or "and / or" as used herein may have multiple meanings, which may depend, at least in part, on the context in which such terms are used. Generally, "or" when used in connection with a list, such as A, B, or C, means A, B, and C, here used in an inclusive sense; and A, B, or C, here used in an exclusive sense. Additionally, depending at least in part on the context, the term "one or more" as used herein may be used to describe any feature, structure, or property in a singular sense, or may be used to describe a combination of features, structures, or properties in a plural sense. Similarly, depending at least in part on the context, terms such as "a / an" or "the" may also be understood to convey a singular use or to convey a plural use. Additionally, the term "based on" may be understood to not necessarily be intended to express a set of exclusive factors, and instead, may, at least in part, depending on the context, allow for additional factors that are not necessarily explicitly described.

[0108] The present disclosure is described with reference to block diagrams and operational descriptions of methods and apparatuses. It should be understood that each block of the block diagrams or operational descriptions, and combinations of blocks in the block diagrams or operational descriptions, can be implemented by means of analog or digital hardware and computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an ASIC, or another programmable data processing device for altering its functionality as detailed herein, such that the instructions executed via the processor of the computer or another programmable data processing device implement the functions / actions specified in the block diagram or one or more operational blocks. In some alternative implementations, the functions / actions labeled in the blocks may not occur in the order labeled in the operational descriptions. For example, depending on the functionality / action involved, two consecutively presented blocks may actually be executed substantially simultaneously or the blocks may sometimes be executed in the reverse order.

[0109] According to embodiments herein, these computer program instructions can be provided to a processor of: a general-purpose computer for altering its functionality to be dedicated; a special-purpose computer; an ASIC; or another programmable digital data processing device, such that the instructions executed via the processor of the computer or another programmable data processing device implement the functions / actions specified in the block diagram or one or more operational blocks, thereby transforming its functionality.

[0110] For the purposes of this disclosure, a computer-readable medium (or computer-readable storage medium) stores computer data, which may include computer program code (or computer-executable instructions) that can be executed by a computer in machine-readable form. By way of example and not limitation, a computer-readable medium may include a computer-readable storage medium for tangible or fixed storage of data, or a communication medium for transient interpretation of signals carrying code. As used herein, a computer-readable storage medium refers to physical or tangible storage (as opposed to signals) and includes, but is not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for physical storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technologies, CD-ROM, DVD or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other physical or material medium that can be used to tangibly store the desired information or data or instructions and that can be accessed by a computer or processor.

[0111] For the purposes of this disclosure, a module is a software, hardware, or firmware (or combination thereof) system, process, or functionality, or components thereof, that performs or facilitates the processes, features, and / or functions described herein (with or without human interaction or augmentation). A module may include sub-modules. The software components of a module may be stored on a computer-readable medium for execution by a processor. A module may be integrated with one or more servers, or loaded and executed by one or more servers. One or more modules may be grouped into an engine or application.

[0112] Those skilled in the art will recognize that the methods and systems of this disclosure may be implemented in many ways and are thus not limited to the foregoing exemplary embodiments and examples. In other words, the functional elements, performed by single or multiple components, implemented in various combinations of hardware and software or firmware, and performing individual functions, may be distributed among software applications at the client level or the server level or both. In this regard, any number of features of the different embodiments described herein may be combined into a single or multiple embodiments, and alternative embodiments with fewer or more than all of the features described herein are possible.

[0113] Functionality may also be distributed, in whole or in part, among multiple components in ways that are now known or will become known. Thus, a variety of software / hardware / firmware combinations may achieve the functions, features, interfaces, and preferences described herein. In addition, the scope of this disclosure encompasses conventional and known ways of performing the described features and functions and interfaces, as well as those changes and modifications to the hardware or software or firmware components described herein that will be understood by those skilled in the art now and in the future.

[0114] In addition, embodiments of the methods presented and described as flowcharts in this disclosure are provided by way of example to provide a more complete understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are envisioned, where the order of various operations is changed and where sub-operations described as parts of larger operations are performed independently.

[0115] Although various embodiments have been described for the purposes of this disclosure, such embodiments should not be considered as limiting the teachings of this disclosure to those embodiments. Various changes and modifications can be made to the elements and operations described above to obtain results that remain within the scope of the systems and processes described in this disclosure.

Claims

1. A method for providing an enhanced display, comprising: Establishing a secure connection with an object selected from a group consisting of a road and lanes of the road; Receiving a packet from the object, the packet describing a condition of the object; Verifying the packet; Generating an enhanced display using a sprite associated with data within the packet; Displaying the enhanced display in a vehicle; Recording the effectiveness of the sprite based on a comparison between performance data and expected data recorded when the sprite is displayed; And In response to the effectiveness indicating that the sprite is ineffective, modifying an attribute of the sprite to enhance the effectiveness of the sprite.

2. The method according to claim 1, wherein establishing the secure connection includes establishing the secure connection using the DICE-RIoT protocol.

3. The method according to claim 1, wherein generating the enhanced display includes: Selecting the sprite associated with the data; Modifying the sprite based on the data; And Generating a distortion function based on a position associated with the packet.

4. The method according to claim 3, wherein generating the enhanced display further includes displaying an overlay of the vehicle.

5. The method according to claim 1, transmitting data regarding vehicle operations performed in response to the enhanced display.

6. The method according to claim 1, wherein generating the enhanced display includes generating a display selected from a group consisting of an augmented reality display or a head-up display.

7. The method according to claim 1, further comprising updating an Advanced Driver Assistance System (ADAS) using the packet.

8. A non-transitory computer-readable storage medium for storing computer program instructions executable by a computer processor, the computer program instructions defining the following steps: Establishing a secure connection with an object selected from a group consisting of a road and lanes of the road; Receiving a packet from the object, the packet describing a condition of the object; Verifying the packet; Generating an enhanced display using a sprite associated with data within the packet; Displaying the enhanced display in a vehicle; Recording the effectiveness of the sprite based on a comparison between performance data and expected data recorded when the sprite is displayed; And In response to the effectiveness indicating that the sprite is ineffective, modifying an attribute of the sprite to enhance the effectiveness of the sprite.

9. The non-transitory computer-readable storage medium according to claim 8, wherein establishing the secure connection includes establishing the secure connection using the DICE-RIoT protocol.

10. The non-transitory computer-readable storage medium according to claim 8, wherein generating the enhanced display includes: Selecting the sprite associated with the data; Modifying the sprite based on the data; And Generating a distortion function based on a position associated with the packet.

11. The non-transitory computer-readable storage medium according to claim 10, wherein generating the enhanced display further includes displaying an overlay of the vehicle.

12. The non-transitory computer-readable storage medium according to claim 8, transmitting data regarding the vehicle operations performed in response to the enhanced display.

13. The non-transitory computer-readable storage medium according to claim 8, wherein the generating the enhanced display includes generating a display selected from the group consisting of an augmented reality display or a head-up display.

14. The non-transitory computer-readable storage medium according to claim 8, wherein the instructions further define steps of using the packet to update an advanced driver assistance system (ADAS).

15. An apparatus for providing an enhanced display, comprising: a processor; and a storage medium for storing program logic thereon for execution by the processor, the stored program logic including: logic executed by the processor for establishing a secure connection with an object selected from the group consisting of a road and lanes of the road; logic executed by the processor for receiving a packet from the object, the packet describing a condition of the object; logic executed by the processor for verifying the packet; logic executed by the processor for generating an enhanced display using a sprite associated with data within the packet; logic executed by the processor for displaying the enhanced display in a vehicle; logic executed by the processor for recording the effectiveness of the sprite based on a comparison between performance data and expected data recorded when the sprite is displayed; and logic executed by the processor for modifying an attribute of the sprite to enhance the effectiveness of the sprite in response to the effectiveness indicating that the sprite has failed.

16. The apparatus according to claim 15, wherein the logic for establishing a secure connection includes logic executed by the processor for establishing a secure connection using the DICE-RIoT protocol.

17. The logic for generating an enhanced display according to claim 15, includes: logic executed by the processor for selecting a sprite associated with the data; logic executed by the processor for modifying the sprite based on the data; and logic executed by the processor for generating a distortion function based on a position associated with the packet.

18. The apparatus according to claim 15, transmitting data regarding the vehicle operation performed in response to the enhanced display.

19. The logic for generating an enhanced display according to claim 15, includes logic executed by the processor for generating a display selected from the group consisting of an augmented reality display or a head-up display.

20. The stored program logic according to claim 15, further includes logic executed by the processor for using the packet to update an advanced driver assistance system (ADAS).

Images