Map features delivered via augmented reality (AR)

Abstract

This application discloses map features delivered via augmented reality (AR). Techniques are disclosed for conveying graphical representations of different types of information to a user using an augmented reality (AR) display. The information can be obtained from a variety of sources, such as objects and / or features dynamically detected by a vehicle or user-generated content. The graphical representations can also be based on information that can be obtained via use of crowd-sourced AV map data. The graphical representations can be presented in a medium (e.g., a vehicle windshield) as an AR view that matches the user's field of view, thereby precisely blending holographic information with physical objects in the scene.

Description

[0001] This application is a divisional application of the invention patent application entitled "Map features transmitted via augmented reality (AR)" with PCT international application number PCT / IB2022 / 058631, international application date of September 13, 2022, and Chinese national phase application number 202280075298.5.

[0002] Cross-references to related applications

[0003] This application claims the benefit and priority of U.S. Provisional Application No. 63 / 243,484, filed September 13, 2021, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0004] The aspects described in this article generally relate to techniques for presenting graphical representations via augmented reality (AR), and in particular, to presenting graphical representations of road maps, features, and other information to vehicle passengers in an AR view. Background Technology

[0005] Road Experience Management (REM) is a technology designed to deliver roadbook map features using crowdsourced data. Roadbook map features are supported by REM technologies and functionalities, along with other technologies, enabling autonomous vehicles (AVs) to deploy in new orientations almost instantly. Roadbook maps are generated independently of third-party mapping utilities and rely on specific mapping features. Unlike traditional static maps, roadbook maps encompass the dynamic history of how a driver navigates on any given road segment, better informing the AV decision-making process. Current systems using roadbook map delivery utilize traditional in-vehicle displays. Traditional displays do not consider the driver's field of view (FoV) when presenting information. Attached Figure Description

[0006] The accompanying drawings, which are incorporated herein and form part of the specification, together with the description, illustrate aspects of this disclosure and further serve to explain the principles of the aspects and enable those skilled in the art to make and use the aspects.

[0007] Figure 1 Exemplary vehicles according to one or more aspects of this disclosure are shown.

[0008] Figure 2 Various exemplary electronic components of a vehicle safety system according to one or more aspects of this disclosure are shown; Figure 3 A block diagram of an exemplary system including an AR device according to one or more aspects of this disclosure is shown; Figure 4A process flow of an exemplary delay compensation technique according to one or more aspects of this disclosure is shown; Figure 5 Example AR display frames are shown in accordance with one or more aspects of this disclosure; Figure 6A An example process flow is shown for presenting an AR display frame using windshield projection technology according to one or more aspects of this disclosure; Figure 6B An example process flow is shown for presenting an AR display frame using AR device projection technology according to one or more aspects of this disclosure; and Figure 7 A block diagram of an exemplary computing device according to one or more aspects of this disclosure is shown.

[0009] Exemplary aspects of this disclosure will be described with reference to the accompanying drawings. The first appearance of an element in the drawings is generally indicated by one or more leftmost numerals in the corresponding reference numerals. Detailed Implementation

[0010] Numerous specific details are set forth in the following description to provide a thorough understanding of the various aspects of this disclosure. However, it will be apparent to those skilled in the art that various aspects, including structures, systems, and methods, can be practiced without these specific details. The descriptions and representations herein are common means by which those experienced or skilled in the art most effectively communicate the substance of their work to others skilled in the art. In other instances, well-known methods, procedures, components, and circuit systems have not been described in detail to avoid unnecessarily obscuring the various aspects of this disclosure.

[0011] Figure 1 The following are examples of security systems 200 shown in accordance with various aspects of this disclosure (see also: Figure 2Vehicle 100. Vehicle 100 and safety system 200 are exemplary in nature and therefore can be simplified for illustrative purposes. The locations and relative distances of elements (as discussed herein, figures are not to scale) are provided by way of example rather than limitation. Safety system 200 may include a variety of components (depending on the requirements of a particular implementation and / or application) and may facilitate the navigation and / or control of vehicle 100. Vehicle 100 may be an autonomous vehicle (AV), which may include any level of automation (e.g., levels 0 to 5), including no automation or full automation (level 5). Vehicle 100 may implement safety system 200 as part of any suitable type of autonomous or driver assistance control system (e.g., including AV and / or advanced driver assistance systems (ADAS)). Safety system 200 may include one or more components that are integrated into vehicle 100 during manufacturing, are part of an add-on or aftermarket device, or a combination thereof. Thus, as Figure 2 The various components of the safety system 200 shown can be integrated as part of the system of the vehicle 100 and / or as part of the aftermarket system installed in the vehicle 100.

[0012] One or more processors 102 may be integrated with or separated from the electronic control unit (ECU) or engine control unit (which may be considered herein as a dedicated type of ECU) of vehicle 100. Safety system 200 may generate data to control or assist in controlling the ECU and / or other components of vehicle 100 to directly or indirectly control the driving of vehicle 100. However, the aspects described herein are not limited to implementations within autonomous or semi-autonomous vehicles, as these are provided by way of example. The aspects described herein can be implemented as part of any suitable type of vehicle capable of driving in a specific driving environment with or without any suitable level of human assistance. Therefore, in each aspect, multiple vehicle components (e.g., such as those referenced herein) Figure 2 One or more of the aforementioned safety systems may be implemented as part of a standard vehicle (i.e., a vehicle that does not use autonomous driving functions), a fully autonomous vehicle, and / or a semi-autonomous vehicle. In aspects implemented as part of a standard vehicle, it should be understood that safety system 200 may perform alternative functions, and therefore, according to such aspects, safety system 200 may alternatively represent any suitable type of system that can be implemented by a standard vehicle without having to utilize autonomous or semi-autonomous control-related functions.

[0013] Regardless of Figure 1 and Figure 2Regardless of the specific implementation of the vehicle 100 and the accompanying safety system 200 shown, the safety system 200 may include one or more processors 102, one or more image acquisition devices 104 (such as, for example, one or more vehicle cameras or any other suitable sensor configured to perform image acquisition within any suitable wavelength range), one or more position sensors 106 (which may be implemented as a position and / or orientation identification system, such as a Global Navigation Satellite System (GNSS), e.g., a Global Positioning System (GPS)), one or more memories 202, one or more map databases 204, one or more user interfaces 206 (such as, for example, a display, touchscreen, microphone, speaker, one or more buttons and / or switches, etc.), and one or more wireless transceivers 208, 210, 212. Alternatively or additionally, one or more user interfaces 206 may be identified as other components communicating with the safety system 200, such as an AR device and / or an external computing device, such as a mobile phone, as further discussed herein.

[0014] Wireless transceivers 208, 210, and 212 can be configured to operate according to any suitable number and / or type of desired radio communication protocol or standard. For example, a wireless transceiver (e.g., the first wireless transceiver 208) can be configured to operate according to short-range mobile radio communication standards, such as Bluetooth, Zigbee, etc. As another example, a wireless transceiver (e.g., the second wireless transceiver 210) can be configured to operate according to medium-range or wide-range mobile radio communication standards, such as 3G (e.g., Universal Mobile Telecommunications System - UMTS), 4G (e.g., Long Term Evolution - LTE), or 5G mobile radio communication standards conforming to the corresponding 3GPP (3rd Generation Partnership Project) standards (the latest version at the time of writing is 3GPP Release 16 (2020)).

[0015] As another example, the wireless transceiver (e.g., the third wireless transceiver 212) can be configured according to a wireless LAN communication protocol or standard, such as, for example, according to the IEEE 802.11 working group standard, the latest version of which at the time of writing is IEEE Std 802.11™-2020 published on February 26, 2021 (e.g., 802.11, 802.11a, 802.11b, 802.11g, 802.11n, 802.11p, 802.11-12, 802.11ac, 802.11ad, 802.11ah, 802.11ax, 802.11ay, etc.). One or more wireless transceivers 208, 210, 212 can be configured to transmit signals via an antenna system (not shown) using an air interface. As an additional example, one or more of transceivers 208, 210, and 212 may be configured to implement one or more vehicle-to-everything (V2X) communication protocols, which may include vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), vehicle-to-network (V2N), vehicle-to-pedestrian (V2P), vehicle-to-device (V2D), vehicle-to-grid (V2G), and any other suitable communication protocols.

[0016] One or more of the wireless transceivers 208, 210, and 212 may be additionally or alternatively configured to enable communication between the vehicle 100 and one or more other remote computing devices 150 via one or more wireless links 140. For example, this could include communication with a remote server or such... Figure 1 Communication with other suitable computing systems shown. Figure 1 The example shown illustrates such a remote computing system 150 as a cloud computing system, but this is by way of example and not a limitation, and the computing system 150 can be implemented according to any suitable architecture and / or network, and can comprise one or more physical computers, servers, processors, etc., including such a system. As another example, the remote computing system 150 can be implemented as an edge computing system and / or network.

[0017] One or more processors 102 can implement any suitable type of processing circuitry, other suitable circuitry, memory, etc., and use any suitable type of architecture. One or more processors 102 can be configured by vehicle 100 as a controller to perform various vehicle control functions, navigation functions, etc. For example, one or more processors 102 can be configured to act as a controller for vehicle 100 to analyze sensor data and received communications, calculate specific actions for vehicle 100 to perform for navigation and / or control of vehicle 100, and cause the corresponding actions to be performed, which may be based on, for example, an AV or ADAS system. One or more processors 102 and / or safety system 200 can form an integral or part of an advanced driver assistance system (ADAS).

[0018] Furthermore, one or more of the processors 214A, 214B, 216, and / or 218 in one or more processors 102 may be configured to operate collaboratively with each other and / or with other components of vehicle 100 to collect information about the environment (e.g., sensor data, such as images, depth information (for example, LiDAR)). In this context, one or more of the processors 214A, 214B, 216, and / or 218 in one or more processors 102 may be referred to as a “processor”. Thus, the processor may be implemented (independently or together) to create map information from the collected data, such as road segment data (RSD) information that may be used for Road Experience Management (REM) mapping technology, details of which are further described below. As another example, the processor may be implemented to process map information (e.g., roadbook information for REM mapping technology) received from a remote server via a wireless communication link (e.g., link 140) to locate vehicle 100 on an AV map, which may be used by the processor to control vehicle 100.

[0019] One or more processors 102 may include one or more application processors 214A, 214B, image processor 216, communication processor 218, and may additionally or alternatively include any other suitable processing means, circuitry, components, etc., not shown in the drawings for simplicity. Similarly, depending on the requirements of a particular application, image acquisition device 104 may include any suitable number of image acquisition means and components. Image acquisition device 104 may include one or more image capture means (e.g., a camera, charge-coupled device (CCD), or any other type of image sensor). Security system 200 may also include a data interface for communicatively connecting one or more processors 102 to one or more image acquisition means 104. For example, a first data interface may include any wired and / or wireless first link 220 or multiple wired and / or wireless first links 220 for transmitting image data acquired by one or more image acquisition means 104 to one or more processors 102, for example, to image processor 216.

[0020] Wireless transceivers 208, 210, and 212 may be coupled to one or more processors 102, such as communications processor 218, for example, via a second data interface. The second data interface may include any wired and / or wireless second link 222 or multiple wired and / or wireless second links 222 for transmitting radio transmission data acquired by wireless transceivers 208, 210, and 212 to one or more processors 102, such as communications processor 218. Such transmissions may also include (one-way or two-way) communication between vehicle 100 and one or more other (target) vehicles in the environment of vehicle 100 (e.g., to facilitate coordination of navigation of vehicle 100 based on other (target) vehicles in or with vehicle 100's environment), or even broadcast transmissions directed to unspecified receivers near the transmitting vehicle 100.

[0021] The memory 202 and one or more user interfaces 206 may be coupled to each of the one or more processors 102, for example, via a third data interface. The third data interface may include one or more suitable wired and / or wireless third links 224. Furthermore, the position sensor 106 may be coupled to each of the one or more processors 102, for example, via the third data interface.

[0022] Each of the one or more processors 102, 214A, 214B, 216, 218, can be implemented as any suitable number and / or type of hardware-based processing device (e.g., processing circuitry system) and can collectively, i.e., together with the one or more processors 102, form one or more types of controllers as discussed herein. Provided as Figure 2 The architecture shown is for ease of explanation and as an example, and the vehicle 100 may include any suitable number of one or more processors 102, each of which may be similarly configured to utilize data received via a variety of interfaces and perform one or more specific tasks.

[0023] For example, one or more processors 102 may form a controller configured to perform various control-related functions of the vehicle 100, such as the calculation and execution of specific vehicle following speed, rate, acceleration, braking, steering, trajectory, etc. As another example, as a supplement to or alternative to one or more processors 102, the vehicle 100 may implement other processors (not shown) that may form controllers of different types configured to perform additional or alternative types of control-related functions. Each controller may be responsible for controlling specific subsystems and / or controls associated with the vehicle 100. According to such aspects, each controller may be able to access, via a corresponding interface (e.g., 220, 222, 224, 232, etc.) from, Figure 2The corresponding coupled components shown receive data, wherein wireless transceivers 208, 210 and / or 212 provide data to the corresponding controllers via a second link 222, which in this example serves as a communication interface between the corresponding wireless transceivers 208, 210 and / or 212 and each corresponding controller.

[0024] To provide another example, application processors 214A, 214B may individually represent corresponding controllers that operate in conjunction with one or more processors 102 to perform specific control-related tasks. For example, application processor 214A may be implemented as a first controller, while application processor 214B may be implemented as a second, different type of controller configured to perform other types of tasks as further discussed herein. According to such aspects, one or more processors 102 may access the system via various interfaces 220, 222, 224, 232, etc., from sources such as… Figure 2 The corresponding coupled components shown receive data, and the communication processor 218 can provide each controller with communication data received from (or to be transmitted to) other vehicles via corresponding coupled links 240A, 240B, which in this example serve as the communication interface between the corresponding application processors 214A, 214B and the communication processor 218. Of course, as a supplement or alternative to the control-based functions, the application processors 214A, 214B can perform other functions, such as the image processing functions discussed herein, providing warnings about potential collisions, etc.

[0025] One or more processors 102 may be additionally implemented to communicate with any other suitable components of vehicle 100 to determine the state of the vehicle while it is being driven or at any other suitable time. For example, vehicle 100 may include one or more vehicle computers, sensors, ECUs, interfaces, etc., which may be collectively referred to as vehicle components 230, such as Figure 2 As shown. One or more processors 102 are configured to communicate with vehicle component 230 via an additional data interface 232, which can represent any suitable type of link and operate according to any suitable communication protocol (e.g., CAN bus communication). Using data received via data interface 232, one or more processors 102 can determine any suitable type of vehicle status information, such as current drive gear, current engine speed, vehicle 100 acceleration capability, etc.

[0026] One or more processors 102 may include any suitable number of other processors 214A, 214B, 216, 218, each of which may include processing circuitry systems such as subprocessors, microprocessors, preprocessors (such as image preprocessors), graphics processors, central processing units (CPUs), support circuitry, digital signal processors, integrated circuits, memory, or any other type of means suitable for running applications and performing data processing (e.g., image processing, audio processing, etc.) and analysis and / or enabling vehicle control functionally. In some aspects, each processor 214A, 214B, 216, 218 may include any suitable type of single-core or multi-core processor, microcontroller, central processing unit, etc. These processor types may each include multiple processing units with local memory and instruction sets. Such processors may include video input for receiving image data from multiple image sensors and may also include video output capabilities.

[0027] Any of the processors 214A, 214B, 216, and 218 disclosed herein can be configured to perform certain functions according to program instructions, which can be stored in the local memory of each respective processor 214A, 214B, 216, and 218, or accessed via another memory that is part of or outside the security system 200. This memory may include one or more memories 202. Regardless of the specific type and location of the memory, it may store software and / or executable (i.e., computer-readable) instructions that, when executed by the relevant processor (e.g., one or more of processors 102, 214A, 214B, 216, and 218), control the operation of the security system 200 and perform other functions, such as those associated with any aspect described further in detail below. This may include, for example, identifying relevant objects and features to be displayed via an AR display, which may include using interface 206 or any other suitable display device for this purpose.

[0028] The associated memory (e.g., one or more memories 202) accessed by one or more processors 214A, 214B, 216, 218 may also store one or more databases and image processing software, as well as a trained system (such as a neural network, or, for example, a deep neural network) that can be used to perform tasks according to any of the aspects discussed herein. The associated memory (e.g., one or more memories 202) accessed by one or more processors 214A, 214B, 216, 218 may be implemented as any suitable number and / or type of non-transitory computer-readable medium, such as random access memory, read-only memory, flash memory, disk drives, optical storage devices, magnetic tape storage devices, removable storage devices, or any other suitable type of storage device.

[0029] Show and such Figure 2 The components associated with the security system 200 shown are for illustrative purposes and are not intended to be limiting. The security system 200 may include additional, fewer, or alternative components, as referenced herein. Figure 2 As shown and discussed. Furthermore, one or more components of the security system 200 may be integrated or otherwise combined into or with common processing circuitry system components. Figure 2 The separations shown form unique and independent components. For example, one or more components of safety system 200 may be integrated onto a common die or chip. As an illustrative example, one or more processors 102 and associated memories (e.g., one or more memories 202) accessed by one or more processors 214A, 214B, 216, 218 may be integrated onto a common chip, die, package, etc., and together constitute a controller or system configured to perform one or more specific tasks or functions. Again, such a controller or system may be configured to perform various functions related to display features, objects, or other information obtained from road map data as part of an AR display, as discussed in further detail herein, thereby controlling the state of the vehicle in which safety system 200 is implemented, etc.

[0030] In some aspects, safety system 200 may further include components for measuring the speed of vehicle 100, such as speed sensor 108 (e.g., speedometer). Safety system 200 may also include one or more inertial measurement unit (IMU) sensors, such as, for example, accelerometers, magnetometers, and / or gyroscopes (single-axis or multi-axis) for measuring the acceleration of vehicle 100 along one or more axes, and additionally or alternatively one or more gyroscope sensors, which may be implemented alone or in combination with other suitable vehicle sensors, for example, to calculate the vehicle's self-motion, as discussed herein. For example, these IMU sensors may be part of position sensor 105, as discussed herein. Safety system 200 may further include additional sensors or different sensor types, such as ultrasonic sensors, thermal sensors, one or more radar sensors 110, one or more LiDAR sensors 112 (which may be integrated into the headlights of vehicle 100), digital compasses, etc. Radar sensor 110 and / or LiDAR sensor 112 may be configured to provide preprocessed sensor data, such as a radar target list or a LiDAR target list. A third data interface (e.g., one or more links 224) can couple a velocity sensor 108, one or more radar sensors 110, and one or more LiDAR sensors 112 to at least one of one or more processors 102.

[0031] Safe driving model

[0032] Safety system 200 may implement a safe driving model or SDM (also known as a “driving strategy model,” “driving strategy,” or simply a “driving model”), which may be used and / or executed as part of the ADAS or AV system discussed herein. For example, safety system 200 may include a computer implementation of a formal model, such as a safe driving model (e.g., as part of a driving strategy). A safe driving model may include an implementation in digital computer hardware of a formalized mathematical model interpreting applicable laws, standards, policies, etc., applicable to automated driving (e.g., ground-based) vehicles. In some embodiments, the SDM may include a standardized driving strategy, such as a responsibility-sensitive safety (RSS) model. However, embodiments are not limited to this particular example, and an SDM may be implemented using any suitable driving strategy model that defines various safety parameters that the AV should adhere to to facilitate safe driving.

[0033] For example, SDM can be designed to achieve, for example, three objectives: First, the interpretation of the law should be reasonable, i.e., the interpretation should conform to how humans interpret the law; second, the interpretation should produce useful driving strategies, meaning the interpretation will produce flexible driving strategies rather than overly defensive driving, which inevitably confuses other human drivers and obstructs traffic, thereby limiting the scalability of system deployment; and third, the interpretation should be efficiently verifiable, i.e., it can be rigorously proven that the autonomous vehicle correctly implements the interpretation of the law. The implementation of the safe driving model (e.g., vehicle 100) in the main vehicle can be or include the implementation of a mathematical model for safety assurance, which enables the identification and execution of correct responses to hazardous situations, thereby avoiding self-inflicted accidents.

[0034] The safe driving model can implement logic to apply driving behavior rules, such as the following five rules: - Don't rear-end other vehicles.

[0035] - Do not overtake recklessly.

[0036] - Give up rather than seize the initiative.

[0037] - Be aware of areas with limited visibility.

[0038] - You must do this when you can avoid an accident without causing another one.

[0039] It should be noted that these rules are neither restrictive nor exclusive, but can be modified in various aspects as needed. Therefore, the rules represent a social driving "contract," which may vary from region to region and may evolve over time. While these five rules currently apply to most countries, the rules may not be complete or identical in each region or country, and may be subject to modification.

[0040] As described above, vehicle 100 may include or refer to Figure 2 The safety system 200 is described herein. Therefore, the safety system 200 can generate data to control or assist in controlling the ECU of vehicle 100 and / or other components of vehicle 100 to directly or indirectly navigate and / or control the driving operations of vehicle 100, such navigation including driving vehicle 100 or other suitable operations as further discussed herein. This navigation may optionally include adjusting one or more SDM parameters, which may occur in response to the detection of any suitable type of feedback obtained via image processing, sensor measurements, etc. Feedback used for this purpose may be collectively referred to herein as “environmental data measurement results” and includes any suitable type of data identifying states associated with the external environment, vehicle passengers, vehicle 100, and / or the cabin environment of vehicle 100.

[0041] For example, environmental data measurement results can be used to identify longitudinal and / or lateral distances between vehicle 100 and other vehicles, the presence of objects on the road, hazard orientations, etc. Environmental data measurement results can be obtained via any suitable component of vehicle 100, and / or can be the result of data analysis obtained via any suitable component of vehicle 100, such as one or more image acquisition devices 104, one or more sensors 105, position sensors 106, speed sensors 108, one or more radar sensors 110, one or more LiDAR sensors 112, etc. To provide an illustrative example, environmental data can be used to generate an environmental model based on any suitable combination of environmental data measurement results. Vehicle 100 can utilize the environmental model to perform various navigation-related operations within the framework of a driving strategy model.

[0042] Therefore, environmental data measurements can be referenced by safety system 200 based on SDM parameters to estimate or predict hazardous situations, and in response, to perform actions such as changing vehicle navigation-related operations or issuing warnings. For example, safety system 200 can predict hazardous situations based on the current position, orientation, speed, driving direction, etc. of vehicle 100, indicating that a collision will occur within a predefined future time period unless additional intervention is taken. As another example, hazardous situations can be estimated based on one or more SDM parameters (e.g., minimum longitudinal distance, minimum lateral distance, etc.) being temporarily changed to a minimum (or maximum) threshold value due to the actions of another driver, cyclist, etc. As part of this process, safety system 200 can identify hazardous orientations relative to the position and orientation of vehicle 100, identified using estimated or predicted hazardous situations, and execute trajectories or other control-based functions to avoid or at least mitigate the occurrence of hazardous situations.

[0043] For example, navigation-related operations can be performed by generating an environmental model and combining it with a driving strategy model to determine the actions the vehicle must perform. That is, a driving strategy model can be applied based on the environmental model to determine one or more actions (e.g., navigation-related operations) the vehicle is about to perform. SDM can be used (as part of or as an add-on layer) in conjunction with the driving strategy model to ensure the safety of the actions the vehicle is about to perform at any given time. For example, ADAS can utilize or reference SDM parameters defined by a safe driving model to determine the navigation-related operations of vehicle 100 based on environmental data measurements and specific driving scenarios. Therefore, navigation-related operations can cause vehicle 100 to perform specific actions based on the environmental model to conform to the SDM parameters defined by the SDM model as discussed herein. For example, navigation-related operations may include steering vehicle 100, changing the acceleration and / or rate of vehicle 100, performing predetermined trajectory control, etc. In other words, an environmental model can be generated using acquired sensor data, and then an applicable driving strategy model can be applied together with the environmental model to determine the navigation-related operations or other actions (e.g., issuing a warning) to be performed by the vehicle.

[0044] Autonomous Vehicle (AV) Map Data and Road Experience Management (REM)

[0045] The data, referred to as REM map data (or alternatively, roadbook map data or AV map data), may also be stored in associated memory (e.g., one or more memories 202) accessed by one or more processors 214A, 214B, 216, 218, or stored in any suitable location and / or in any suitable format (such as stored locally or in a cloud-based database), accessed via communication between the vehicle and one or more external components (e.g., via transceivers 208, 210, 212), and so on. It should be noted that although referred to herein as “AV map data,” the data can be implemented in any suitable vehicle platform, which may include vehicles with any suitable level of automation (e.g., level 0-5), as mentioned above.

[0046] Regardless of where REM map data is stored and / or accessed, AV map data can include the geographic orientation of known landmarks that are easily identifiable in the navigation environment in which vehicle 100 is traveling. The orientation of a landmark can be generated from historical accumulations from other vehicles driving on the same road, which collect data on the appearance and / or orientation of the landmark (e.g., “crowdsourcing”). Therefore, each landmark can be associated with a predetermined set of geographic coordinates that have already been established. Thus, in addition to using orientation-based sensors (such as GNSS), the database of landmarks provided by REM map data enables vehicle 100 to identify landmarks using one or more image acquisition devices 104. Once identified, vehicle 100 can utilize images from other sensors (such as LIDAR, accelerometers, speedometers, etc.) or from image acquisition devices 104 to assess the position and orientation of vehicle 100 relative to the identified landmark location.

[0047] Furthermore, as mentioned above, vehicle 100 can determine its own motion, referred to as "vehicle motion." Vehicle motion is commonly used in computer vision algorithms and other similar algorithms to represent the motion of a vehicle's camera across multiple frames, providing a baseline (i.e., spatial relationships) that can be used to calculate the 3D structure of a scene based on the corresponding images. Vehicle 100 can analyze vehicle motion to determine its position and orientation relative to identified known landmarks. Because landmarks are identified using predetermined geographic coordinates, vehicle 100 can determine its geographic orientation and location on a map based on determining its position relative to the identified landmarks using landmark-related geographic coordinates. This offers the significant advantage of combining the benefits of smaller-scale location tracking with the reliability of GNSS positioning systems while avoiding the drawbacks of both. It should be further noted that this analysis of vehicle motion serves as an example of an algorithm that can be implemented using monocular imaging to determine the relationship between the vehicle's orientation and the known orientations of one or more known landmarks, thereby assisting the vehicle in locating itself. However, vehicle motion is not necessary or relevant for other types of technologies, and therefore not essential for localization using monocular imaging. Therefore, based on the aspects described herein, vehicle 100 can utilize any suitable type of positioning technology.

[0048] Therefore, AV map data is typically constructed as part of a series of steps, which may involve selecting any suitable number of vehicles to participate in the data collection process. As each vehicle collects data, the data is categorized into labeled data points and then launched to the cloud or another suitable external location. A suitable computing device (e.g., a cloud server) then analyzes the individual driving data points from the same road, aggregating and adjusting these data points together. After adjustment, the data points are used to define the precise outline of the road infrastructure. Next, the relevant semantics that enable vehicles to understand the real-time driving environment are identified, defining the features and objects linked to the categorized data points. For example, features and objects defined in this way can include traffic lights, road arrows, signs, road edges, drivable paths, lane dividers, stop lines, lane markings, etc., allowing vehicles to easily identify these features and objects using REM data. This information is then compiled into a "roadbook map," which consists of a set of driving paths, semantic road information (such as features and objects), and aggregated driving behaviors.

[0049] For example, the map database 204, which may be stored as part of one or more memories 202 or accessed via computing system 150 and via link 140, may include any suitable type of database configured to store (digital) map data of vehicle 100, such as (digital) map data of safety system 200. One or more processors 102 may download information to map database 204 via wired or wireless data connections (e.g., one or more links 140) using suitable communication networks (e.g., via cellular networks and / or the Internet). Again, map database 204 may store AV map data, which includes data relating to the location of various landmarks (such as objects and other information items, including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, etc.) in a reference coordinate system.

[0050] Therefore, map database 204 can not only store the orientation of such landmarks as part of the AV map data, but also store descriptors associated with these landmarks, including, for example, names associated with any stored features, and can also store information related to details of the items, such as the precise location and orientation of the items. In some cases, AV map data can store sparse data models, including certain road features of vehicle 100 (e.g., lane markings) or polynomial representations of the target trajectory. AV map data can also include stored representations of various identified landmarks, which can be used to determine or update the known position of vehicle 100 relative to the target trajectory. Landmark representations can include data fields such as landmark type, landmark orientation, etc., as well as other possible identifiers. AV map data can also include non-semantic features (including point clouds of certain objects or features in the environment) as well as feature points and descriptors.

[0051] Map database 204 can be augmented with data other than AV map data, and / or map database 204 and / or AV map data can reside partially or entirely as part of remote computing system 150. As discussed herein, the orientation and map database information of known landmarks that can be stored in map database 204 and / or remote computing system 150 can form content referred to herein as “AV map data,” “REM map data,” or “route map data.” One or more processors 102 can process sensory information about the environment of vehicle 100 (e.g., images, radar signals, depth information from LIDAR, or stereo processing of two or more images) and location information (e.g., GPS coordinates, vehicle motion, etc.) to determine the current orientation, position, and / or orientation of vehicle 100 relative to known landmarks using information contained in the AV map. Thus, the determination of vehicle orientation can be refined in this way. Certain aspects of this technique can be additionally or alternatively incorporated into positioning techniques such as mapping and route planning models.

[0052] The aspects described herein further utilize REM map data to identify road features and objects, as mentioned above, and optionally identify other types of information, as mentioned herein, to enhance driving safety and convenience by selectively displaying such features and objects to users (e.g., vehicle passengers, such as the driver or another passenger). Again, AV map data can be stored in any suitable location and can be downloaded and stored locally as part of map database 204, for example, said map database can be stored as part of one or more storage devices 202 or accessed via computing system 150 and via link 140.

[0053] Furthermore, the AV map data discussed in this paper is primarily described in terms of the geographic location of known landmarks and the use of other types of information that can be identified by those landmarks. However, this is by way of example and not limitation, and AV maps can be identified with any suitable content that can be associated with accurate geographic location. In this way, the graphical representation of various features, objects, and other information utilizing location and vehicle and user FoV tracking, as further discussed in this paper, may include third-party content or other suitable content, which may include portions of the AV map data.

[0054] Use AV map data to selectively present road objects and features.

[0055] As mentioned above, AV map data can be used by safety system 200 to track the geographic location of vehicle 100 relative to identified landmarks. Therefore, AV map data can be used by safety system 200 to determine the location of vehicle 100 on the road with high accuracy. Furthermore, the vehicle's self-movement can be further used with the sensor data mentioned above to determine the position and orientation of vehicle 100 corresponding to the determined location. The aspects described herein utilize safety system 200 and AV map data to selectively present various graphical representations of objects and features to the user within an augmented reality (AR) view. For example, the aspects described herein can present filled graphical representations (i.e., graphical elements) representing any suitable type of information on a display (e.g., projected onto a vehicle windshield or wearable device), which can be obtained from AV map data and, alternatively, any other suitable data source, as further discussed herein. This can include, for example, presenting features and / or objects obtained via AV map data and, alternatively, other features and / or objects based on sensor data, the state of vehicle 100, etc., as further discussed herein.

[0056] In order to do so, the aspects described herein can be used with any suitable type of device capable of providing an AR view to the user. For example, Figure 2 One or more user interfaces 206 shown may include an in-vehicle display, which may include a vehicle head-up display for selectively projecting graphical representations of road objects and features discussed herein within an AR view, such as... Figure 3 As shown in the figure. For example, this can be implemented by using a projector (which may form part of one or more user interfaces 206) or other suitable display technologies that enable a graphical representation of road objects and features to be presented within the inner surface of the windshield of the vehicle 100.

[0057] As further discussed below, aspects include processing data, displaying and selectively presenting a graphical representation within the user's Field of View (FoV) such that the graphical representation is aligned relative to the user's viewpoint (i.e., focus). The user can be a passenger of vehicle 100, such as the driver of vehicle 100, or another user such as someone wearing AR glasses (including the driver or other passengers of vehicle 100), a motorcyclist wearing a helmet-mounted display, etc., as further mentioned herein. Regardless of the specific user, the presentation of the graphical representation within the detected FoV can be achieved by one or more processors 102 first locating the vehicle using AV map data, which includes landmarks and corresponding predetermined geographic orientations. This can be achieved using any suitable technology, such as using orientation-based sensors like GNSS to determine the geographic orientation of vehicle 100 and using vehicle motion to determine the vehicle's position and orientation, as described herein. As further discussed below, this information can then be used to calculate the vehicle's field of view (FoV), which corresponds to the front of the vehicle and spans any suitable three-dimensional angular range based on the vehicle 100's position and orientation at a particular geographic orientation.

[0058] Next, the field of view (FoV) of the vehicle occupants is calculated, which can be performed using any suitable type of image or other sensor-based analysis in any suitable manner. For example, one or more processors 102 can utilize image data acquired from one or more image acquisition devices 104 (which may include in-vehicle cameras) to detect the position and orientation of the driver's head and eyes in three-dimensional space. From said image data, the position of the user's head and eyes relative to the AR view projection surface (e.g., the windshield) and the direction of gaze (e.g., by locating the user's pupils) can be determined. Alternatively or additionally, the position and orientation of the user's head and eyes can be identified from any other suitable type of sensor, such as the sensors described herein with respect to one or more sensors 105.

[0059] It should be noted that graphical representations of features and objects derived from AV map data, as well as any other suitable icons, features, objects, etc., may be presented in different ways as part of an AR view, depending on the aspects described herein. As discussed herein, the term "graphical representation" encompasses any suitable type of visual information that can be presented as part of an AR view, excluding "real-world" objects or features that may exist in other ways. One example of the presentation of an AR view is relative to the windshield of vehicle 100 (i.e., the "windshield projection embodiment"). As another example, AR device 301 may include any suitable display configured to present an AR view to its wearer (i.e., the "AR device projection embodiment"). Thus, each of these aspects can facilitate the presentation of any suitable type of graphical representation in an AR view, but may differ in the manner in which the passenger FoV is tracked and the graphical representation is selectively presented within the passenger FoV.

[0060] Furthermore, in the windshield and AR device projection embodiment, the AR view can include any suitable number of AR display frames, each presented sequentially over time to track changes in the relative motion of features and objects with respect to the passenger's FoV. Therefore, each AV display frame can contain a graphical representation aligned with the passenger's current FoV at the same time. In the windshield and AR device projection embodiment, the orientation and position of the graphical representation within the AR view can be determined at least initially based on the vehicle's FoV. Similarly, the vehicle FoV can be calculated via the vehicle's self-motion and can be identified using any suitable angular FoV pointing forward of the vehicle, based on the vehicle's current geographic orientation, position, and orientation on the road.

[0061] In other words, because the vehicle's self-movement indicates the position and orientation of vehicle 100, safety system 200 can identify one or more features and objects contained within the vehicle's FoV and included in the AV map data based on the vehicle's self-movement. Furthermore, since the geographic orientation of vehicle 100 is also known, safety system 200 can also use the AV map data to determine the relative orientation of the identified features and objects with respect to the vehicle's geographic orientation. Finally, once the features and objects and their relative positions with respect to vehicle 100 are determined, safety system 200 can generate AR display frames by filtering the identified features and objects contained within the vehicle's FoV, thereby presenting those features and objects contained within the passenger FoV. In other words, safety system 200 generates AV display frames based on the vehicle's geographic orientation, the identified FoV of the vehicle's passengers, and the relative orientation of one or more identified features and objects with respect to vehicle 100. In this way, as the vehicle's position and orientation change, as indicated by the vehicle's own motion, the graphical representation within the passenger's FoV is also updated to reflect new features and objects, features and objects moving out of the passenger's FoV, and the movement of features and objects within the passenger's FoV. Therefore, for example, each AR display frame presented in the AR view can include a graphical representation associated with one or more identified features and objects contained within the passenger's FoV at a given time.

[0062] Therefore, in the windshield projection embodiment, sensor 105 can be used to determine the passenger's FoV. That is, as part of a calculation performed by one or more components of the safety system 200, such as processor 102, the passenger FoV can be determined using sensor 105. The passenger FoV calculated in this way is then used to project a graphic representation onto the windshield of vehicle 100. In this way, the position and orientation of the passenger's head relative to other components in the passenger compartment (e.g., the windshield) can be tracked via acquired sensor data and / or image processing techniques, allowing the passenger's FoV to be tracked over time. For example, according to the windshield projection embodiment, the passenger's FoV can be used to identify a smaller portion of the windshield used to project the graphic representation within the AR view. Furthermore, according to the windshield projection embodiment, the passenger FoV can be used to determine the orientation of the graphic representation to be projected onto the windshield, ensuring that the graphic representation is aligned with the user's focus. That is, without considering the passenger's FoV in this way, the AR display frame will remain unchanged when the passenger moves relative to the distance between the user's eyes and the projection surface, potentially leading to misalignment of the graphic representation. Therefore, as the position and orientation of the vehicle and passengers change, using the passenger FoV in this way allows the graphical representation contained within the passenger FoV to be updated in each AR display frame. In this manner, the graphical representation of the FoV relative to the vehicle and passengers is overlaid at the corresponding orientation where the associated physical objects, physical features, etc., physically reside. An example of a single AR display frame generated according to such an embodiment is shown in... Figure 3 The image is shown as a projection onto the windshield inside the vehicle 100.

[0063] However, for AR device projection embodiments (e.g., AR glasses, helmets with AR masks, etc.), the AR device may additionally or alternatively implement one or more sensor and / or processing circuitry systems to enable tracking of the user's head position and posture, thereby tracking the passenger's FoV. Figure 3 An example of such a system is shown, demonstrating an (optional) AR device 301 inside vehicle 100, and implementing one or more processors 102 as part of the safety system 200 discussed above. Although in Figure 3 The AR device 301 is shown inside the vehicle, but this is by way of example and not a limitation, and the AR device 301 can be implemented as any suitable type of AR device, as discussed herein, which may include AR glasses worn by the driver (or another passenger) of the vehicle 100, an integrated AR device manufactured with the vehicle 100, an AR device installed aftermarket in the vehicle 100, a motorcycle helmet visor, etc.

[0064] Continue to refer to Figure 3The AR device 301 may include any suitable type of hardware and / or software that enables the graphical representation to be projected onto a suitable medium (e.g., a windshield, motorcycle helmet visor, AR mask, AR glasses, etc.), as discussed herein. For example, the AR device 301 may implement one or more processors 302, which may include any suitable number and / or type of processors or processing circuitry systems. In various aspects, the processor 302 may be configured as any suitable number and / or type of computer processor for controlling the AR device 301 and / or components of the AR device 301. The processor 302 can be identified by one or more processors (or suitable portions thereof) implemented by the AR device 301. For example, the processor 302 can be identified by one or more processors, such as a host processor, digital signal processor, one or more microprocessors, a graphics processor, a microcontroller, an application-specific integrated circuit (ASIC), a partial (or complete) field-programmable gate array (FPGA), etc.

[0065] In any case, aspects including processor 302 are configured to execute instructions to perform arithmetic, logical, and / or input / output (I / O) operations and / or control the operation of one or more components of AR device 301 to perform various functions associated with aspects described herein. For example, processor 302 may include one or more microprocessor cores, memory registers, buffers, clocks, etc., and may generate electronic control signals associated with components of AR device 301 to control and / or modify the operation of these components. For example, aspects include processor 302 communicating with memory 303, sensor 304, display 306, and / or data interface 308 and / or controlling functions associated with them. Processor 302 may additionally perform various operations described herein with reference to one or more processors of the safety system 200 for identification of vehicle 100 to selectively present a graphical representation derived from AV data (or other data sources mentioned herein) in an AR view.

[0066] Memory 303 is configured to store data and / or instructions such that when the instructions are executed by processor 302, AR device 301 performs the various functions described herein. Memory 303 can be implemented as any well-known volatile and / or non-volatile memory, including, for example, read-only memory (ROM), random access memory (RAM), flash memory, magnetic storage media, optical disc, erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), etc. Memory 303 can be non-removable, removable, or a combination of both. For example, memory 303 can be implemented as a non-transitory computer-readable medium storing one or more executable instructions, such as logic, algorithms, code, etc.

[0067] As further discussed herein, the instructions, logic, code, etc., stored in memory 303 enable the functional implementation of the aspects disclosed herein. Therefore, the aspects include processor 302, which, in conjunction with one or more hardware components, executes the instructions stored in memory 303 to perform various functions associated with the aspects, as further discussed herein.

[0068] Sensor 304 can be implemented as any suitable number and / or type of sensor configured to track the position and orientation of the AR device 301 relative to the vehicle 100, and thus the position and orientation of the user wearing the AR device 301 relative to the vehicle 100. Therefore, sensor 304 can be configured to perform tracking of the position and orientation between the AR device 301 and a specific surface or orientation within the vehicle 100 on which the AR display will be presented. To do this, one or more sensors 304 may include GPS, inertial measurement unit (IMU) sensors such as accelerometers, magnetometers and / or gyroscopes, cameras, depth sensors, etc. Thus, when the AR device 301 is used inside the vehicle 100, the relative position and orientation between the AR device 301 and the presentation plane within the vehicle 100 can be determined using any suitable type of tracking and positioning technology. Furthermore, the vehicle's own motion can be used to "filter out" vehicle motion from the user's head motion in this way, since the position and orientation of both the passenger's head and the vehicle are tracked.

[0069] Therefore, AR device sensor 304 can generate sensor data indicating the position and orientation of AR device 301 relative to various vehicle components in three-dimensional space. As further discussed herein, AR device 301 can also receive data from safety system 200 indicating the vehicle's motion. Therefore, processor 302 can use any suitable technique, including known techniques, and use each set of data to calculate the relative motion between AR device 301 and vehicle 100, and thus filter out the motion of vehicle 100. In this way, AR device 301 can determine where the wearer (e.g., vehicle driver or other passenger) is looking and the relevant user FOV for displaying content. This tracking can be provided by the manufacturer of AR device 301 (e.g., AR glasses manufacturer), and therefore can be achieved using tools provided by the AR device 301 manufacturer. Alternatively, external device 350 and / or vehicle processor 102 can receive sensor data from AR device 301 and perform these tracking and filtering operations using any suitable technique.

[0070] As an illustrative example, AR device 301 may use sensor data generated by sensor 304 to perform wearer tracking and positioning, which may include the identification of anchor objects. In this context, for example, anchor objects may include predefined objects or objects dynamically detected for this purpose. As some examples, anchor objects may include parts of vehicle 100, predefined markers, or any other suitable objects located inside or outside vehicle 100. Anchor objects can be identified using identifiable objects, and processor 302 (which may also be implemented via AR device 301 or external device 350) and / or on-board processor 102 may use the calculated relative position of the anchor object with respect to AR device 301 and vehicle 100, as well as the vehicle's own motion, which may be received by AR device 301 via cooperation with safety system 200, as discussed herein.

[0071] AR device 301 also includes a display 306, which can be implemented as any suitable type of display depending on a specific embodiment of AR device 301, and configured to display AR display frames to the wearer in an AR view. For example, AR device 301 can be implemented as a wearable head-mounted device, and therefore display 306 can be implemented as a display device used with such a head-mounted device. Additional examples of display 306 may include a face mask, a portion of a motorcycle helmet, etc.

[0072] AR device 301 further includes a data interface 308 configured to enable communication between AR device 301, security system 200 (e.g., wireless transceivers 208, 210, 212), or other components of vehicle 100 and / or external devices 350. Therefore, data interface 308 can be implemented as one or more transceivers, receivers, transmitters, communication interfaces, etc. Thus, data interface 308 can be implemented as any suitable number and / or type of components configured to transmit and / or receive data and / or wireless signals according to any suitable number and / or type of communication protocol. Data interface 308 can include any suitable type of components for facilitating this functionality, including components associated with the operation, configuration, and implementation of known transceivers, transmitters, and / or receivers. For example, data interface 308 can include components typically identified with an RF front end and includes, for example, antennas, power amplifiers (PAs), RF filters, mixers, local oscillators (LOs), low-noise amplifiers (LNAs), up-converters, down-converters, channel tuners, etc. Alternatively, the data interface 308 may include any suitable number and / or type of components configured to enable data transmission between the wireless link and the AR device 301, such as ports, modulators, demodulators, buffers, drivers, memory, etc.

[0073] Figure 3 The external device 350 shown is optional and, when present, can be implemented with components similar to or the same as those in AR device 301. Alternatively, external device 350 can be implemented with a subset of the components shown and described herein with reference to AR device 301. AR device 301 and external device 350 can each implement corresponding components to cooperate with each other to facilitate the embodiments discussed herein. For example, external device 350 can be implemented as a mobile device, and / or can implement one or more processors similar to processor 302 in AR device 301, as discussed in further detail herein. According to such an arrangement, AR device 301 can offload processing functions to external device 350 and / or security system 200, processing functions that would otherwise be performed via processor 302 discussed herein. Thus, AR device 301 can implement processor 302 to perform processing operations, or alternatively, external device 350 can perform these processing operations. In any case, the AR device 301, various components of the vehicle safety system 200 (e.g., one or more wireless transceivers 208, 210, 212, etc.) and external devices 350 (when present) can be communicatively coupled to each other via any suitable number of wireless links, such that the AR device 301 and / or external devices 350 (as the case may be) can receive graphic content to be displayed that is recognized by the safety system 200 or locally, as further discussed herein.

[0074] Alternatively or additionally, the offloading process performed via external device 350 can be offloaded to security system 200. To support this functionality, in various embodiments, AR device 301, external device 350, and / or security system 200 can communicate with each other unidirectionally or bidirectionally via any suitable number and / or type of wireless link. For example, AR device 301 can transmit sensor data and / or other suitable types of data to external device 350 and / or security system 200. AR device 301 and / or external device 350 can additionally or additionally receive any suitable type of information from security system 200, such as AV map data and / or data indicating the vehicle 100's self-movement, which can be used to define the user's FoV, as discussed herein. Similarly, this may be the case, for example, when such processing is not performed holistically within AR device 301.

[0075] Regardless of the specific device implemented, the identified orientation and orientation of the user's head and eyes relative to known geometry (e.g., seating position, distance from the user's eyes to the display (e.g., windshield)) can be used to calculate a viewpoint encompassing the user's FoV relative to the user's current position and orientation. This user FoV can then be mapped to or associated with a specific geographic location of vehicle 100 and a vehicle FoV for or related to the geographic location, position, and orientation of vehicle 100, to identify a graphical representation contained within the vehicle FoV from AV map data (or other suitable data sources). These graphical representations can then be further selected over a range aligned with the user's FoV. It should be noted that while both windshield and AR device projection embodiments can acquire the user FoV for this purpose, the windshield projection embodiment may not require the user's gaze direction (i.e., user head pose) because the position of the user's eyes in three-dimensional space can be used alternatively to calculate the projection plane (e.g., the windshield). In other words, for the windshield projection embodiment, the user's FoV is aligned with the vehicle's FoV, assuming that the user's FoV differs from the vehicle's FoV only in its distance from the windshield. Therefore, this distance can be taken into account to present a graphical representation at the correct orientation of the user's FoV relative to the user's FoV in the AR display frame within the plane.

[0076] However, for the AR device projection embodiment, the passenger's head position and orientation obtained from sensor data generated via sensor 304 can be implemented to further map the graphical representation from the vehicle FoV to the user FoV. For example, this can handle cases where the user turns his or her head to one side. To provide an illustrative example, if the wearer turns her head to the left, the user's FoV changes, and therefore the AR device 301 can update the AR display frames from being displayed in the center of display 306 to being displayed on the right side of display 306 to maintain alignment with their actual position relative to the vehicle FoV.

[0077] In other words, the spatial layout of the graphical representation of one or more road objects and features within the user's FOV is determined based on the position of each feature, object, etc., relative to the user (e.g., relative to vehicle 100 and the user's FOV relative to a projection plane or surface, such as a windshield or other display based on the application). Similarly, the position of each feature, object, etc., presented within each AV display frame is derived from the geographic orientation of vehicle 100, the vehicle's self-movement, and AV map data. The information included in the AV map data can be used in this way to identify the orientation of static features, objects, etc., and the positioning of vehicle 100 (and / or another device, depending on the specific implementation) relative to these features and objects.

[0078] Therefore, for windshield-based and AR-device-based projection embodiments, once the vehicle FoV is identified, the relevant processor (e.g., processor 102 or processor 302) can cause the corresponding display to present a graphical representation of one or more identified features and objects within the user's FoV in the AR view. In this way, one or more identified features, objects, etc., within the vehicle FoV can be overlaid at their corresponding "real-world" orientation relative to the user's corresponding physical object or feature. Furthermore, aspects include optionally displaying a graphical representation of one or more identified features and objects within the user's FoV in a 3D or 3D-like environment. Thus, changes in the orientation of the vehicle 100 and / or the user's head posture can be tracked and processed, and the 3D shape of the relevant graphical representation can be adjusted accordingly (e.g., using projection geometry).

[0079] Delay compensation

[0080] Similarly, each AR display frame contains graphical representations of various features, objects, etc., that can be derived from AR map data or other data sources, as further discussed herein. Therefore, AR display frames contain graphical representations based on the vehicle's current FoV, which are then selectively presented to align with the user's FoV. However, it should be noted that the processing used to generate AR display frames requires a certain processing time, thus introducing computational latency relative to the initial time for performing vehicle localization and (if present) AR device tracking, and the time when the graphical information is actually presented within the AR view. Therefore, embodiments include techniques to compensate for such latency and can be implemented according to windshield and AR device projection embodiments. The following will discuss... Figure 4 These technologies will be discussed further.

[0081] Figure 4 A process flow for an exemplary delay compensation technique according to one or more aspects of this disclosure is shown. For example... Figure 4 As shown, box 402 represents the vehicle localization process. This may include the safety system 200 determining the geographic orientation of the vehicle 100, and then, as part of the vehicle motion calculation, using AV map data to calculate the position and orientation of the vehicle 100 at that geographic orientation. Similarly, this may include referencing the geographic orientation of known landmarks relative to the vehicle 100's orientation. Additionally, the vehicle motion calculation may include, as part of the localization process, using position sensor 105 to identify the vehicle's motion.

[0082] In block 404, the position and orientation of the vehicle passengers are calculated. For AR device projection embodiments, this can be performed by AR device 301 alone or in combination with other devices such as external device 350. According to such embodiments, this can include analysis of sensor data provided by sensor 304, as mentioned above. For example, for windshield projection embodiments, safety system 200 can calculate the passenger's position and orientation based on vehicle sensors such as an in-vehicle camera. In any case, block 404 represents the passenger positioning process, in which the passenger's orientation and position relative to vehicle 100 are calculated to determine the user's FoV, as mentioned above.

[0083] Next, in box 406, an AR display frame is generated. For the AR device projection embodiment, this can be performed by the AR device 301 alone or in conjunction with other devices, such as external device 350. To do this, the applicable device (e.g., AR device 301) can receive the position and orientation of the vehicle 100 from the safety system 200, which has already been calculated as part of the vehicle positioning calculation. The applicable device can also receive AV map data or, alternatively, identified graphic representations recognized by the safety system 200 and their corresponding orientations mapped to the vehicle's FoV. That is, the corresponding orientations of the graphic representations are mapped relative to the position and orientation of the vehicle 100. According to the AR device projection embodiment, the applicable device can receive this information via a wireless communication link between the AR device 301 (or other suitable device) and the vehicle 100, as mentioned above.

[0084] Next, in this example, AR device 301 can determine which graphic representations are included in the AR display frame generated for the user's FOV and their corresponding orientations within the user's FOV. For example, this could include converting the orientation of each graphic representation within the vehicle's FOV to an updated orientation relative to the user's FOV. This process can utilize any suitable technique, including known techniques, to perform this coordinate transformation based on vehicle and passenger positioning processes and an analysis of their corresponding positions and orientations in three-dimensional space. Therefore, as... Figure 4 As shown, the computation delay represents the time between the initial execution of vehicle localization (i.e., calculating the position and orientation of vehicle 100) and the generation of an AR display frame containing a graphical representation.

[0085] For the AR device projection embodiment, this delay can be greater than that of the windshield projection embodiment because additional latency is introduced when the AR device 301 receives data from the safety system 200 (or other suitable computing device) via a communication link. However, for both the AR device and windshield projection embodiments, this computational delay can be calculated using any suitable technique, including known techniques. For example, the vehicle 100, safety system 200, and any components involved in the initial AR display frame generation can be synchronized with or otherwise referenced to a global clock. For example, this global clock can be derived via a GNSS system or other suitable time-based components of the vehicle 100. In any case, the global clock can be used as a timestamp to identify when each computation process is completed.

[0086] For example, timestamps can be used to identify when vehicle positioning (box 402) is completed and when the AR display frame is generated (box 406). Therefore, by using the differences between these timestamps, safety system 200 or applicable devices (e.g., AR device 301, external device 350, etc.) (as applicable) can calculate computational latency. Using timestamps in this manner can be particularly useful for implementations where computational latency is inherently dynamic. For other implementations where computational latency is known a priori or not expected to deviate significantly over time, computational latency can be estimated by other means, such as through calibration or other suitable testing procedures.

[0087] Regardless of how the computational latency is calculated, embodiments include using computational latency compensation to render the graphical representation within an AR display frame. For example, in some cases, the computational latency may be approximately 100 milliseconds, and the position and orientation of vehicle 100 and the passenger's FoV may change during this computational latency period. Therefore, not compensating for computational latency may result in the graphical representation in the AR view being rendered with features and objects aligned to them in their previous position and orientation relative to vehicle 100, rather than in their current position and orientation; thus, the graphical representation is not aligned with the corresponding orientation of features and objects relative to the user's current FoV.

[0088] Therefore, to address this issue, the embodiment includes calculating the movement of vehicles and passengers during the calculation delay period. This is performed as part of box 408, as... Figure 4 As shown. For example, in the windshield projection embodiment, vehicle motion can be calculated continuously, and this data can be readily available. Therefore, the latency of generating and accessing vehicle motion data can be significantly less than the computation latency. Similarly, passenger position and orientation can be tracked via in-cabin sensors and can be obtained without significant latency. Therefore, the safety system 200 can be integrated with... Figure 4The process flow shown tracks the vehicle's own motion and the passenger's position and orientation in parallel. Therefore, when the AR display frame is generated, the position and orientation of vehicle 100 and the passenger can be accessed with minimal latency (e.g., aside from the small latency introduced by processing and memory retrieval steps). Furthermore, because the recognition and display of the graphic representations to be presented in the AR view results in considerable computational latency, updated (i.e., current) position and orientation data of vehicle 100 and the passenger (box 410) can be applied to further move (e.g., via coordinate transformation) the relative position and orientation of each graphic representation, thereby aligning it with the actual physical orientation of features, objects, etc., presented in the AR view.

[0089] It should be noted that the AR device projection embodiment can operate in a similar manner, but the applicable device (e.g., AR device 301) wirelessly receives vehicle motion data from safety system 200 and calculates the passenger's position and orientation locally. Therefore, according to both embodiments, the motion of the vehicle and passengers during the calculation delay is used to compensate for changes in the position and orientation of the vehicle and passengers during the delay period. Once motion compensation is applied in this manner, an AR display frame (box 412) is presented in the AR view, as discussed herein.

[0090] In this way, the processing circuitry and / or applicable devices of the safety system 200 (as applicable) are configured to compensate for changes in the position and orientation of the vehicle and the user during the delay period by tracking the vehicle's own movement and the position and orientation of the user's head. Therefore, the vehicle's FoV and the passenger's FoV can be adjusted using this information to match the updated position and orientation of the vehicle and the user's head corresponding to the time the AR display frame was generated.

[0091] In some windshield and AR device projection embodiments, motion compensation can be performed consistently based on computational latency. However, in other windshield and AR device projection embodiments, motion compensation due to computational latency can be performed selectively. That is, for some embodiments, the computational latency can (e.g., consistently) be less than a threshold latency value, such that the difference between the motion during the period of compensation and non-compensation of computational latency is negligible. In other words, this effect is negligible relative to any change in how the graphical representation is presented in the AR view.

[0092] However, in other embodiments, computational delay can be monitored to determine whether the motion compensation techniques discussed herein should be performed. For example, the processing circuitry of safety system 200 and / or applicable devices (as applicable) can be configured to selectively compensate for changes in the position and orientation of the vehicle (and passengers) during the computational delay if the delay period exceeds a predetermined threshold time period, and not to perform motion compensation otherwise. Furthermore, it should be noted that the aspects discussed herein can be dynamically adjusted in terms of how computational delay is implemented and whether computational delay is required, based on the specific device used for the AR device. For example, in the case of using different AR devices in a single phase (e.g., using an AR windshield display first and then switching to an AR device display at some point), various delay parameters further discussed herein can be adjusted accordingly. These delay parameters can include any suitable parameters used as part of the delay compensation techniques discussed herein, such as, for example, a predetermined threshold for computational delay, image frame rate frequency, and sampling rate relative to sensor data acquired via safety system 200, AR device 301, etc.

[0093] Alternatively or concurrently, the motion compensation techniques discussed herein can be used to proactively reduce computational latency in various ways. This can also be a selective process, triggered only when the computational latency exceeds a predetermined threshold. This can include, for example, the safety system 200 adjusting one or more parameters regarding how it calculates the vehicle's motion to reduce computational latency. Such parameters may be referred to herein as "data acquisition parameters" and can be considered as an example of the aforementioned latency parameters. For example, this can be triggered via the safety system 200 or, for an AR device projection embodiment, via an applicable device, thereby transmitting (e.g., via data interface 308) a control signal to the safety system 200 to indicate this change.

[0094] As an illustrative example, vehicle motion can be calculated by safety system 200 based on the image frame rate, for example, when a vehicle camera is used for this purpose. Therefore, the frequency of this frame rate can be an example of a data acquisition parameter that can be adjusted in this way. That is, the frequency of this frame rate can be further increased to reduce the latency between the calculation of vehicle motion and the calculation, thereby further reducing computational latency.

[0095] As another illustrative example, data acquisition parameters may include any suitable number, type, and / or combination of parameters identified by the wireless link between the applicable device (e.g., AR device 301) and the security system 200. For example, such parameters may include variations in the frequency of the wireless link, variations in the communication rate provided by the wireless link, thereby altering the communication protocol and / or the number of frequencies used. As another illustrative example, data acquisition parameters may include the sampling rate relative to sensor data acquired via the security system 200, AR device 301, etc.

[0096] Additional graphical representations displayed in the AR view

[0097] Figure 5 Example AR display frames according to one or more aspects of this disclosure are shown. Similarly, embodiments described herein include the sequential presentation of such AR frames within an AR view. The AR view, and each AR display frame presented within the AR view, can contain any suitable type of graphical representation that can be derived from any suitable type of data source. For example, the examples discussed above describe graphical representations typically presented relative to features and objects on a road, but this is merely illustrative and not limiting. In practice, embodiments described herein can present alternative or additional graphical representations within an AR view based on AV map data or other suitable data sources.

[0098] For example, the graphical representation displayed in the AR view may also include data derived from sensors identified by vehicle 100 or other suitable devices. Therefore, the graphical representation of features, objects, etc., can be further enhanced using data derived from sensor measurements, enabling the display of information about dynamic objects. Furthermore, the graphical representation displayed in the AR view can further encompass other information that is neither stored in the AV map data nor derived from sensor measurements. For example, the positioning techniques discussed herein, which also utilize AR map data, can be implemented to precisely align the graphical representation of any suitable type of information to be displayed with any suitable type of location-based content (e.g., points of interest along the route, advertisements, navigation information, etc.).

[0099] As an illustrative example, in addition to or alternative to the use of AV map data for feature and object identification, safety system 200 may identify one or more features and objects contained within the vehicle's FoV (Field of View) detected by one or more vehicle sensors. For example, this may include safety system 200 performing object detection using sensor data from cameras and other sensors that can operate in the invisible spectrum, such as LIDAR and RADAR sensors. The embodiments described herein facilitate the presentation of any suitable portions of such data computation, sensor data, and / or data generated from any vehicle system as part of an AR view that might otherwise be unseen by vehicle occupants.

[0100] As another illustrative example, AR map data may include landmarks and road features such as lane markings and road signs. As yet another example, safety system 200 may perform various types of calculations based on the SDM system to autonomously or semi-autonomously identify and execute driving maneuvers. As yet another example, safety system 200 may perform image processing and machine vision algorithms to classify objects, identify trajectories, or perform maneuvers, etc.

[0101] Alternatively, AR display frames can be associated with any suitable number of AR information layers, graphic information layers, etc., such that each AR frame can display any suitable combination of graphic representations and / or types, each derived from one or more data sources that may be the same or different from each other. For example, in addition to a view of the environment surrounding vehicle 100, an AR frame may also include AR objects derived from navigation information created using AV map data. In another example, in addition to a view of the environment surrounding vehicle 100, an AR frame may also include AR objects derived from SDM, such as a safety envelope around a car, pedestrians, or cyclists in the AR frame. Therefore, each AR information layer may also have its own unique display characteristics, such as animation, color palettes, dynamic behaviors, etc.

[0102] In any case, regardless of how items are detected and identified using graphical representations, once detected, such features can be presented in the AR view in the same manner as objects and features discussed above. That is, the graphical representation of any identified object and / or feature detected by the relevant sensors can be presented in the AR view as a FoV overlay relative to the vehicle occupants at the corresponding location of the detected physical object and / or physical feature. Similarly, such sensors may include, for example, RADAR sensors, LIDAR sensors, cameras, etc. Therefore, such embodiments may be particularly useful for enhancing the driver's contextual awareness by presenting information invisible to the human eye or difficult to detect under certain environmental conditions in the AR view.

[0103] The specific graphical representation to be presented in the AR view can be determined using information obtained from AV map data and sensors of the vehicle 100's safety system 200. For example, this could include determining which objects should be displayed within the user's field of view (FoV) based on current camera feed data, which could be particularly useful when the vehicle 100 uses camera feed data for autonomous driving (e.g., ADAS) functions. For instance, the safety system 200's camera feed data can identify visible objects, and then, based on the AV map data, the safety system 200 can identify which objects are occluded or invisible. Continuing this example, only visible objects can be displayed in the generated AR display frame, or, if relevant, a graphical representation highlighting one or more invisible objects (i.e., objects not in the camera feed but detected by other sensors) can be displayed in the AR display frame.

[0104] As an example, when the camera feed does not include roads, it may be preferable not to display road markings. However, in the same scenario, it may be preferable to present icons or other indicators graphically to highlight traffic lights that are not visible in the camera view. This can be achieved by identifying the orientation of the traffic lights from AV map data, which can be compared to the expected orientation of the traffic lights in the camera feed.

[0105] As another example, AV map data can include the position of stop sign lines relative to a stop sign. A stop sign may appear in the camera feed, but the stop sign lines may have disappeared or otherwise not be present in the camera feed. In this case, an AR display frame including a graphical representation of the stop sign lines can be presented in the user's FoV, aligning the stop sign lines with the correct or "expected" orientation of the real-world stop sign.

[0106] To provide an additional example, the trajectory of vehicle 100 calculated by safety system 200 (e.g., as part of ADAS), i.e., the vehicle's future path, can be presented as a graphical representation in an AR view in a similar manner, such as... Figure 5As shown. Regarding the vehicle's trajectory, it should be noted that the processing circuitry of safety system 200 is configured to identify hazardous situations estimated or predicted based on the applied Safe Driving Model (SDM), as mentioned above. That is, as part of its autonomous function, safety system 200 can typically identify such situations based on the environment of vehicle 100 indicated by vehicle sensors and the implementation of the SDM. Therefore, safety system 200 can generate an AR display frame (or transmit this information to AR device 301 or another suitable device for generating the AR display frame) by adding any suitable overlay features relative to such operation of safety system 200 in the AV view. For example, this can include generating an AR display frame that includes a graphical representation of warning display features when a hazardous situation is identified. Thus, such warning features can advantageously be displayed at a location relative to the user's FoV within the generated AR display frame, for example, overlaid in the AR display frame at a location associated with the estimated focus position of the vehicle's occupants. Furthermore, such warning features can be presented to correspond to the orientation of objects associated with the identified hazardous situation. For example, such warning features can include graphical icons highlighting vehicles on one side of the road, icons highlighting debris in the road, etc.

[0107] As an additional example, AR views can be used to graphically represent, for example, the location of available driving routes, buildings, parking lots, available parking spaces (determined through crowdsourcing), potholes or other hazards, road edges, shoulders, etc.

[0108] As an additional example, a user or third-party provider can create, store, and / or link content to AV map data, which the security system 200 or other suitable device (e.g., AR device 301) can then provide to the user when relevant (e.g., when the linked content is within the user's FoV). For example, an external content provider can link their content to AV map data, which can then be located and augmented in the scene corresponding to the user's FoV using the techniques discussed herein. As an illustrative example, a user can add the addresses of friends she frequently visits, so that when it is determined that the friend's house is within the user's FoV, an appropriate icon indicating the friend's address is subsequently displayed as a graphical representation in the AR view.

[0109] For example, such embodiments may be particularly useful for enabling third-party providers to supplement AV map data via an application running on external device 350. Thus, by using a custom executable application, external computing device 350 can also provide data for a graphical representation to be presented within the AR view. For example, such an application can provide a graphical layout (e.g., cartoon characters, shapes, icons, etc.) of content about a specific geographic location, or content to be presented or overlaid. This could include, for example, the presentation of advertisements, notifications, etc., related to a specific geographic location.

[0110] To provide another example, content concerning moving objects in a driving scene can be displayed as a graphical representation in the AR view, which can be detected and identified as part of the operation of safety system 200. This can include dynamically identified objects or features that are detected "in real time" but are not included or otherwise identified as AV map data. Therefore, if pedestrians, cyclists, safety-related information, or any other suitable data are detected while driving via safety system 200, this can be graphically presented in the AR view with an appropriate representation, such as bounding boxes, text, lines, outlines, etc.

[0111] Similarly, for example, additional or alternative graphical representations presented in the AR view in this manner can therefore include identified objects, such as pedestrians, examples of which are shown in... Figure 5 The figure shows pedestrian 502. The graphic representation may also include identified vehicles, such as... Figure 5 The vehicle 506 is shown. Furthermore, the graphical representation may include virtual overlays corresponding to specific road features or warnings to be conveyed to the user, but not necessarily related to actual physical objects, such as... Figure 5 The boundary marker shown is 504.

[0112] Furthermore, although discussed in the context of a vehicle, the aspects described herein are not limited to vehicle AR displays and can be used with any suitable type of device capable of presenting AR or non-AR views to a user. For example, a graphical representation of road objects, features, or any other information discussed herein can be displayed via AR device 301, which can be implemented independently of vehicle 100, for example, as an AR glasses display, a motorcycle helmet visor AR display, etc. As an additional example, aspects may include other types of displays, such as overlays on navigation application interfaces, which may include vehicle navigation, street-level pedestrian navigation, etc. The use of computing devices that can be implemented independently of vehicle 100 is referenced below. Figure 7 To elaborate further.

[0113] In any case, it should be noted that the use of the positioning techniques described herein facilitates the precise “blending” of holographic information (e.g., the projected graphic representations discussed herein) with physical objects, features, or any other suitable type of information corresponding to geographic location within a specific scene, matching the user’s FoV in various environments. Conventional systems cannot achieve this level of precision in tracking the orientation of real-world objects and are therefore limited to presenting information only in general locations on the display (e.g., in predetermined portions of the display, such as the side, top, etc.). Due to this limitation, conventional systems cannot accurately “blend” or combine the projection of information so that the graphic representations of features, objects, etc., are aligned with or overlapped with the real-world location of the object in the tracked user’s FoV.

[0114] Process flow

[0115] Figure 6A An example process flow is shown for presenting an AR display frame using windshield projection technology according to one or more aspects of this disclosure. Figure 6A An example overall process flow for generating AR display frames that include any suitable number and / or type of graphical representations of objects, features, etc., is shown, as discussed in this paper. References Figure 6A Stream 600 can be a computer-implemented method executed and / or otherwise associated with one or more processors (processing circuitry), components, and / or storage devices. These processors, components, and / or storage devices can be associated with one or more computing components (e.g., one or more processors 102, 214A, 214B, 216, 218, etc.) identified herein by the safety system 200 of vehicle 100.

[0116] One or more processors identified as discussed herein can execute instructions stored on other computer-readable storage media (not shown in the figures), which may be locally stored instructions and / or as part of the processing circuitry itself. Stream 600 may include, for brevity, […]. Figure 6A Alternative or additional steps not shown in the text, and may be performed in a manner different from the one described above. Figure 6A The steps shown are executed in the order indicated.

[0117] Stream 600 may begin with one or more processors using AV map data to determine (box 602) the geographic location of a vehicle. For example, this may include initiating the vehicle localization process discussed herein, which may include using GNSS and geographic locations of nearby landmarks or other objects in the AV map data to accurately identify the vehicle's geographic location on the road.

[0118] Process flow 600 may further include one or more processors calculating (box 604) the vehicle FoV associated with the vehicle's position and orientation at the determined geographic orientation. For example, this could include completing the vehicle localization process performed in box 402, as referenced herein. Figure 4 As discussed above. For example, a vehicle's FoV can be determined by relating the vehicle's orientation and position to the range of angles identified in front of the vehicle. Furthermore, the range of angles determined in this way can be further mapped to the specific geographic location of the vehicle referenced in the AV map data.

[0119] Process flow 600 may further include one or more processors calculating (box 606) the passenger FoV. For example, this may correspond to the user FoV of the driver or other vehicle passengers for whom an AR view is to be generated. For example, the passenger FoV can be calculated by identifying the orientation and position of the passenger's head to determine the direction of gaze. Similarly, for example, this can be calculated using sensor data acquired via an in-vehicle camera or other sensors, wherein the sensor data is provided to the safety system 200 for this purpose.

[0120] Process flow 600 may further include one or more processors identifying (box 608) one or more features and / or objects contained within the vehicle's FoV from the AV map data. For example, since the vehicle FoV has been calculated, objects and / or features contained within this vehicle FoV can be identified from the AV map data. For example, based on the geographic location of vehicle 100, the angular range of the vehicle FoV can be correlated with the AV map data to determine which objects, features, etc., are contained within the vehicle FoV.

[0121] Process flow 600 may further include one or more processors determining (box 610) the relative orientation of identified (box 608) features and / or objects relative to the vehicle. This may include, for example, calculating the position and orientation of each identified feature, object, etc., relative to the vehicle's FoV. For example, since the geographic orientation of each feature, object, etc., is known from AV map data, the safety system 200 may use this information to plot the position and orientation of each object, feature, etc., contained within the vehicle's FoV relative to the center of the vehicle's FoV or other reference points identified by the vehicle 100. In this way, each feature, object, etc., contained within the vehicle's FoV can be positioned relative to the vehicle 100.

[0122] Process flow 600 may further include one or more processors generating (box 612) AR display frames based on the relative orientations of the vehicle FoV, passenger FoV, and identified features, objects, etc., contained within the vehicle FoV. For example, the vehicle FoV is calculated based on the vehicle's geographic orientation, and the identified features and / or objects have already been positioned relative to the vehicle FoV based on the vehicle's location and orientation. Once identified and positioned in this manner, the relative orientations of one or more features and objects identified within the FoV can be further filtered based on those also contained within the passenger FoV. Therefore, as part of this process, adapting the graphical representations of features and / or objects in the generated AR display frames to align with the passenger's FoV can also be performed as part of a three-dimensional coordinate transformation process.

[0123] Process flow 600 may further include one or more processors that render an AR display frame (box 614) in an AR view. Similarly, the AR display frame includes a graphical representation of the identified features and / or objects, and may additionally or alternatively include any other suitable type of information, as discussed herein. For example, for process flow 600, the AR display frame may be displayed via projection onto the windshield of vehicle 100, for example... Figure 3 As shown.

[0124] Figure 6B An example process flow is shown for presenting an AR display frame using AR device projection technology according to one or more aspects of this disclosure. Figure 6B An example overall process flow for generating AR display frames that include any suitable number and / or type of graphical representations of objects, features, etc., is shown, as discussed in this paper. References Figure 6B Stream 650 may be a computer-implemented method executed and / or otherwise associated with one or more processors (processing circuitry), components, and / or storage devices. These processors, components, and / or storage devices may be associated with one or more computing components (e.g., one or more processors 102, processors 214A, 214B, 216, 218, etc.) identified by an AR device (e.g., AR device 301), another suitable computing device (e.g., external computing device 350), and / or the safety system 200 of the vehicle 100 discussed herein.

[0125] One or more processors identified as discussed herein can execute instructions stored on other computer-readable storage media (not shown in the figures), which may be locally stored instructions and / or as part of the processing circuitry itself. Stream 650 may include, for brevity, […]. Figure 6B Alternative or additional steps not shown in the text, and may be performed in a manner different from the one described above. Figure 6B The steps shown are executed in the order indicated.

[0126] Process flow 650 in Figure 6B The box shown is similar to... Figure 6A Those processes performed as part of the process flow 600 shown. However, because process flow 650 relates to an AR device projection embodiment, process flow 650 may begin with one or more processors receiving (box 652) data via a communication link. This data may include any suitable portion of the data used to generate AR display frames, as discussed herein. For example, data received in this manner may include the vehicle's geographic location, AV map data including landmarks and corresponding predetermined geographic locations, the vehicle's position and orientation, a graphical representation to be displayed in the AR view, and so on. Thus, data may be received from the safety system 200, other suitable components of the vehicle 100, and / or other suitable computing devices (e.g., external computing device 350), and may include, for example, data as a result of calculations performed via the safety system 200 as discussed herein with respect to, for example, process flow 600.

[0127] Process flow 650 may further include one or more processors calculating (box 654) the vehicle FoV. For example, this vehicle FoV may be calculated by safety system 200, as discussed above with respect to box 604 of process flow 600, and received as part of data via a communication link. Alternatively, the vehicle FoV may be calculated locally via AR device 301 or other suitable means. In this case, the position and orientation of vehicle 100 and any other suitable data discussed herein for use by safety system 200 to calculate the vehicle FoV may be received as part of data via a communication link, and / or calculated locally via AR device 301 or other suitable computing means to facilitate the calculation.

[0128] Process flow 650 may further include one or more processors calculating (box 656) the passenger FoV. For example, this may include one or more processors using sensor data indicating the position and orientation of AR device 301 or other suitable device, as discussed herein.

[0129] Process flow 650 may further include one or more processors identifying (box 658) one or more features and / or objects contained within the vehicle FoV from AV map data. For example, one or more features and / or objects may be calculated by safety system 200, as discussed above with respect to box 608 of process flow 600, and received as part of data via a communication link. Alternatively, one or more features and / or objects may be calculated locally via AR device 301 or other suitable means. In this case, AV map data, vehicle FoV, and any other suitable data discussed herein for use by safety system 200 to identify one or more features and / or objects may be received as part of data via a communication link, and / or calculated locally via AR device 301 or other suitable computing means to facilitate the calculation.

[0130] Process flow 650 may further include one or more processors determining (box 660) the relative orientation of identified (box 608) features and / or objects relative to vehicle 100. For example, the relative orientation of one or more features and / or objects may be calculated by safety system 200, as discussed above with respect to box 610 of process flow 600, and received as part of data via a communication link. Alternatively, the relative orientation of one or more features and / or objects may be calculated locally via AR device 301 or other suitable means. In this case, the position and orientation of each identified feature, object, etc., relative to the vehicle FoV, the geographic orientation of each feature object, etc., and any other suitable data discussed herein for use by safety system 200 to determine the relative orientation of one or more features and / or objects may be received as part of data via a communication link, and / or calculated locally via AR device 301 or other suitable computing means to facilitate the calculation.

[0131] Process flow 600 may further include one or more processors generating (box 662) AR display frames based on the relative orientation of the vehicle FoV, passenger FoV, and identified features, objects, etc., contained within the vehicle FoV. For example, the AR display frames may be calculated by safety system 200, as discussed above in box 612 regarding process flow 600, and received as part of data via a communication link. Alternatively, the AR display frames may be calculated locally via AR device 301 or other suitable means. In this case, any suitable data discussed herein for use by safety system 200 to generate the AR display frames may be received as part of data via a communication link, and / or calculated locally via AR device 301 or other suitable computing means to facilitate the calculation.

[0132] Process flow 650 may further include presenting (box 664) an AR display frame in the AR view. Similarly, the AR display frame includes a graphical representation of the identified features and / or objects, and may additionally or alternatively include any other suitable type of information, as discussed herein. For example, for process flow 650, the AR display frame may be displayed via projection onto a suitable surface, such as a face mask, motorcycle helmet, etc.

[0133] Figure 7 Block diagrams of exemplary computing devices according to various aspects of this disclosure are shown. On one hand, regarding... Figure 7 The computing device 700 shown and described can be identified by a separate device that can be used to perform the aspects described herein, either inside or outside a vehicle environment. Therefore, the computing device 700 can perform the functions described herein with respect to the AR device 301, the external device 350, and / or the onboard processor 102, as described above with respect to… Figure 3 As discussed above. The computing device 700 may represent various components of a single hardware device implementing each component shown herein, or alternatively, the various components of the computing device 700 may be divided into any suitable number of hardware devices capable of communicating with each other (e.g., AR device 301, external device 350, and / or vehicle processor 102), as referenced above. Figure 3 As mentioned above. For example, sensor 704 and display 710 can be identified by AR device 301, such as AR glasses, while processing circuitry system 702 and memory 703 can be identified by external computing device 350. Data interface 708, including transceiver 708A and communication interface 708B, can be identified by both AR device 301 and external device 350 to support communication between these devices.

[0134] In any case, computing device 700 is configured to perform the various functions discussed herein to present a graphical representation of features, objects, etc., in an AR view associated with the user's FoV. This may include identifying the orientation (e.g., relative to a carriage), position, and orientation of computing device 700; determining the current FoV of the user of computing device 700 based on the current orientation, position, and orientation of computing device 700; determining features, objects, or other suitable information derived from AV map data or other suitable data sources to be displayed within the user's FoV; and causing display 706 to present a graphical representation of features, objects, or other suitable information in the AR view.

[0135] To do this, the computing device 700 may include a processing circuitry system 702, a sensor 704, a data interface 708 including a transceiver 708A and a communication interface 708B, a display 706, and a memory 703. Figure 7 The components shown are for ease of explanation, and aspects including the computing device 700 are implemented relative to... Figure 7 Additional, fewer, or alternative components to the components shown.

[0136] In this embodiment, various components of the computing device 700 can be identified using those previously described with respect to the AR device 301. For example, the processing circuitry 702, sensor 704, data interface 708, display 706, and memory 703 can be identified using those described herein with respect to the AR device 301. Figure 3 The processing circuitry 302, sensor 304, data interface 308, display 306, and memory 303 of the AR device 301 discussed herein can be identified as being identical to or substantially identical to those components. Alternatively, components of the computing device 700 can be identified as being identical to those of the external device 350.

[0137] In various aspects, the processing circuitry system 702 can be configured as any suitable number and / or type of computer processors that can be used to control the computing device 700 and / or components of the computing device 700. The processing circuitry system 702 can be identified as one or more processors (or suitable portions thereof) implemented in the computing device 700. For example, the processing circuitry system 702 can be identified as one or more processors, such as a host processor, a digital signal processor, one or more microprocessors, a graphics processor, a baseband processor, a microcontroller, an application-specific integrated circuit (ASIC), a partial (or complete) field-programmable gate array (FPGA), etc.

[0138] In any case, aspects including processing circuitry 702 are configured to execute instructions to perform arithmetic, logical, and / or input / output (I / O) operations, and / or control the operation of one or more components of computing device 700 to perform various functions associated with aspects described herein. For example, processing circuitry 702 may include one or more microprocessor cores, memory registers, buffers, clocks, etc., and may generate electronic control signals associated with components of computing device 700 to control and / or modify the operation of those components. For example, aspects include processing circuitry 702 communicating with and / or controlling associated functions with sensor 704, data interface 708, display 706, and / or memory 703. Processing circuitry 702 may additionally perform various operations as described herein with respect to one or more processors of the safety system 200 for vehicle 100 identification, to selectively present graphical representations of features, objects, or other suitable information derived from AV map data (or other data sources mentioned herein) in an AR view.

[0139] On one hand, sensor 704 can be implemented as any suitable number and / or type of sensor that can be used to determine the geographic location of computing device 700 and the position and orientation of computing device 400 worn by the user, and can be used to calculate the corresponding passenger FoV. Examples of such sensors may include positioning systems (e.g., GNSS systems) for positioning, cameras, IMU sensors such as compasses, gyroscopes, accelerometers, etc., ultrasonic sensors, infrared sensors, thermal sensors, digital compasses, RADAR, LIDAR, optical sensors, etc.

[0140] On one hand, the data interface 708 may include a transceiver 708A, which may be implemented as any suitable number and / or type of components configured to transmit and / or receive data and / or wireless signals according to any suitable number and / or type of communication protocol. The transceiver 708A may include any suitable type of components for facilitating this function, including components associated with the operation, configuration, and implementation of known transceivers, transmitters, and / or receivers. Although in Figure 7 While described as a transceiver, transceiver 708A may include any suitable number of transmitters, receivers, or combinations thereof, which may be integrated into a single transceiver or integrated as multiple transceivers or transceiver modules. For example, transceiver 708A may include components typically identified by RF front-ends, and includes, for example, antennas, power amplifiers (PAs), RF filters, mixers, local oscillators (LOs), low-noise amplifiers (LNAs), upconverters, downconverters, channel tuners, etc.

[0141] Regardless of the specific implementation, transceiver 708A may include one or more components configured to transmit and / or receive AV map data, as discussed herein, which may be used to determine the geographic location of computing device 700 relative to various known landmarks, and to identify features, objects or other information to be displayed within the user's FoV, as mentioned herein.

[0142] On the one hand, the communication interface 708B can be configured as any suitable number and / or type of components configured to facilitate the transceiver 708A in receiving and / or transmitting data and / or signals according to one or more communication protocols, as discussed herein. For example, the communication interface 708B can be implemented as any suitable number and / or type of components for interfacing with the transceiver 708A, such as analog-to-digital converters (ADCs), digital-to-analog converters, intermediate frequency (IF) amplifiers and / or filters, modulators, demodulators, baseband processors, etc. Therefore, the communication interface 708B can cooperate with the transceiver 708A and form part of the overall communication circuitry system implemented by the computing device 700.

[0143] On one hand, display 706 can be implemented as any suitable type of display, which can be controlled and / or modified by processing circuitry system 702 to selectively present features, objects, or other information to be displayed within the user's field of view, as mentioned herein. Display 706 can have any suitable shape and operate according to any suitable type of resolution and color palette. For example, display 706 can be recognized using AR glasses or a motorcycle face mask, as discussed herein. Alternatively, display 706 can be recognized using a projection system that projects images onto a suitable medium, such as a vehicle windshield.

[0144] On one hand, for example, memory 703 stores data and / or instructions such that when the instructions are executed by processing circuitry system 702, computing device 700 performs the various functions described herein, such as those described above. Memory 703 can be implemented as any well-known volatile and / or non-volatile memory, including, for example, read-only memory (ROM), random access memory (RAM), flash memory, magnetic storage media, optical disc, erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), etc. Memory 703 can be non-removable, removable, or a combination of both. For example, memory 703 can be implemented as a non-transitory computer-readable medium storing one or more executable instructions, such as logic, algorithms, code, etc.

[0145] As discussed further below, the instructions, logic, code, etc., stored in memory 703 are represented as Figure 7 The various modules shown enable the functional implementation of the aspects disclosed herein. Alternatively, if the aspects described herein are implemented via hardware, then... Figure 7 The module associated with memory 703 shown may include instructions and / or code for facilitating control and / or monitoring of the operation of such hardware components. In other words, it provides... Figure 7 The modules shown are for ease of explanation regarding the functional relationships between hardware and software components. Therefore, each aspect, including the processing circuitry system 702, combines with one or more hardware components to execute instructions stored in these respective modules to perform various functions associated with that aspect, as further discussed herein.

[0146] On one hand, executable instructions stored in the anchoring calculation module 713 can be combined with execution via the processing circuitry system 702 to facilitate the computing device 700 in identifying one or more anchoring objects. Similarly, anchoring objects can include parts of vehicle 100, predefined markings, or any other suitable object located inside or outside vehicle 100. The anchoring calculation process can include the processing circuitry system 702 implementing any suitable type of image and / or object identification process to identify anchoring objects based on comparisons with known object types or shapes. Alternatively, anchoring objects can be calculated based on prior knowledge of specific anchoring objects inside or outside vehicle 100 for a predetermined purpose within the vehicle.

[0147] On the one hand, the executable instructions stored in the tracking module 715 can be combined with the processing circuitry system 702 to facilitate the computing device 700 to calculate the user's FoV based on the relative position and orientation between the computing device 700 and the AR display plane (e.g., the windshield) within the vehicle 100, using the relative position of each identified anchored object, as referenced above. Figure 3 The discussion.

[0148] It should be noted that, for reference Figure 7 The described example includes using anchored object tracking as an example to calculate and track the relative position and orientation between the AR device 301 and the presentation plane within the vehicle 100. However, as mentioned herein, this is an example, and any suitable alternative tracking and positioning techniques can be implemented for this purpose.

[0149] On one hand, the executable instructions stored in the FoV recognition module 717 can be combined with the processing circuitry system 702 to facilitate the computing device 700 in recognizing the user's FoV relative to the current position and orientation of the user's head and eyes, which can be used to calculate the user's FoV, as mentioned herein. For example, this may include analyzing sensor data collected via sensor 704 to identify the orientation and position of the user's head in three-dimensional space, the orientation of the user's eyes, the direction of the user's gaze, etc.

[0150] On one hand, executable instructions stored in the feature / object correlation model 719 can be combined with execution via the processing circuitry system 702 to facilitate the determination of specific features, objects, or other suitable information from AV map data or other suitable data sources corresponding to the identified user's FoV, as mentioned herein. For example, this could include performing a filtering operation that masks other graphic objects, features, etc., in the vehicle FoV view that are outside the user's FoV. For example, this filtering process could be performed by selecting a set of pixels based on the angle of the user's FoV and comparing the extracted objects at relevant angles. Alternatively or additionally, clipping can be used based on the range of objects and a predefined relevance or preference that can be identified a priori, so that the display is not unnecessarily cluttered with irrelevant and / or distant objects. In this way, only the range of pixels corresponding to features and objects that are relevant and close (i.e., within a predetermined threshold distance from the user's FoV) contained within the user's FoV are displayed.

[0151] On one hand, executable instructions stored in the hologram display module 721 can be combined with execution via the processing circuitry system 702 to facilitate the generation of AR display frames, which include graphical representations corresponding to specific features, objects, or other suitable information of the identified user's FoV. The hologram display module 721 may include instructions that enable the processing circuitry system 702 to implement a display 706, which may also be a projector or other suitable type of display, to ensure that the AR display frames are presented at an appropriate size and contain appropriate graphical representations. This may include and include content received via the vehicle processor 102, such as... Figure 3 and 5 As shown, the content may include dynamic data detected by the operation of the security system 200 as supplementary or alternative to information derived from the AV map data, as mentioned herein.

[0152] Example

[0153] The following examples cover further aspects.

[0154] Example (e.g., Example 1) relates to a vehicle. The vehicle includes a processing circuitry system configured to determine the vehicle's geographic location using autonomous vehicle (AV) map data that includes landmarks and corresponding predetermined geographic orientations; Calculate the vehicle FoV associated with the vehicle's position and orientation at the vehicle's geographic location; calculate the passenger field of view (FoV) associated with the vehicle's passengers; identify one or more features and / or objects contained within the vehicle FoV as identified by the AV map data; determine the relative orientation of the identified one or more features and / or objects relative to the vehicle using the AV map data; and generate an augmented reality (AR) display frame based on the vehicle's geographic location, the passenger FoV, and the relative orientation of the identified one or more features and objects; and a display configured to present the AR display frame in an AR view, the AR display frame including a graphical representation associated with the one or more identified features and / or objects contained within the passenger FoV.

[0155] Another example (e.g., Example 2) relates to the previously described example (e.g., Example 1), where the display includes a vehicle head-up display.

[0156] Another example (e.g., Example 3) relates to the previously described examples (e.g., one or more of Examples 1 to 2), wherein the vehicle head-up display includes the inner surface of the vehicle's windshield.

[0157] Another example (e.g., Example 4) relates to the previously described examples (e.g., one or more of Examples 1 to 3), wherein the one or more identified features and / or objects are overlaid relative to the passenger FoV at the corresponding location of the associated physical object or physical feature.

[0158] Another example (e.g., Example 5) relates to the previously described examples (e.g., one or more of Examples 1 to 4), wherein the processing circuitry is configured to calculate the vehicle's self-movement and use the self-movement to determine the vehicle's position and orientation.

[0159] Another example (e.g., Example 6) relates to the previously described examples (e.g., one or more of Examples 1 to 5), wherein the display is configured to present, in the AR view, a graphical representation of the one or more identified features and / or objects contained within the passenger FoV, based on the vehicle’s location and orientation.

[0160] Another example (e.g., Example 7) relates to the previously described examples (e.g., one or more of Examples 1 to 6), wherein the processing circuitry is configured to: calculate a time delay between (i) the time of calculating the vehicle’s position and orientation, and (ii) the time of generating the AR display frame; and to render a graphical representation of the one or more identified features and / or objects within the vehicle’s FoV in the AR view by selectively compensating for changes in the vehicle’s position and orientation during the time delay.

[0161] Another example (e.g., Example 8) relates to the previously described examples (e.g., one or more of Examples 1 to 7), wherein the processing circuitry is configured to selectively compensate for changes in the vehicle's position and orientation during the delay period by tracking the vehicle's self-movement and adjusting the vehicle's FoV to match the updated position and orientation of the vehicle corresponding to the time the AR display frame is generated.

[0162] Another example (e.g., Example 9) relates to the previously described examples (e.g., one or more of Examples 1 to 8), wherein the processing circuitry is configured to compensate for changes in the vehicle’s position and orientation during the delay period when the delay period exceeds a predetermined threshold time period.

[0163] Another example (e.g., Example 10) relates to the previously described examples (e.g., one or more of Examples 1 to 9), wherein the processing circuitry is configured to: calculate a delay period between (i) the time for calculating the vehicle’s position and orientation, and (ii) the time for generating the AR display frame; and when the delay period exceeds a predetermined threshold time period, adjust one or more parameters relating to the manner in which the vehicle’s motion is calculated to reduce the delay.

[0164] Another example (e.g., Example 11) relates to the previously described examples (e.g., one or more of Examples 1 to 10), wherein the processing circuitry is further configured to identify one or more additional features and / or objects contained within the vehicle FoV, the one or more additional features and / or objects being detected via one or more sensors, wherein the display is configured to present a graphical representation of the identified one or more additional features and / or objects within the passenger FoV in the AR view.

[0165] Another example (e.g., Example 12) relates to the previously described examples (e.g., one or more of Examples 1 to 11), wherein one or more additional features and / or objects identified are covered by the FoV of the passenger of the vehicle at the corresponding location of the associated physical object or physical feature.

[0166] Another example (e.g., Example 13) relates to the previously described examples (e.g., one or more of Examples 1 to 12), wherein the one or more sensors include one or more light detection and ranging (LIDAR) sensors.

[0167] Another example (e.g., Example 14) relates to the previously described examples (e.g., one or more of Examples 1 to 13), wherein the one or more additional features and / or objects include one or more pedestrians.

[0168] Another example (e.g., Example 15) relates to the previously described examples (e.g., one or more of Examples 1 to 14), wherein one or more additional features and / or objects identify the trajectory of the vehicle according to an advanced driver assistance system (ADAS).

[0169] Another example (e.g., Example 16) relates to the previously described examples (e.g., one or more of Examples 1 to 15), wherein the processing circuitry is further configured to identify hazardous situations estimated or predicted based on an applied Safe Driving Model (SDM), and to generate the AR display frame by adding a warning display feature to the AR frame when the hazardous situation is identified.

[0170] Another example (e.g., Example 17) relates to the previously described examples (e.g., one or more of Examples 1 to 16), wherein the warning display feature is overlaid in the AR display frame relative to the passenger FoV at a location corresponding to the orientation of an object associated with the identified hazardous situation.

[0171] Another example (e.g., Example 18) relates to the previously described examples (e.g., one or more of Examples 1 to 17), wherein the orientation of the warning display feature is overlaid in the AR display frame at an orientation associated with the estimated focus orientation of the passenger in the vehicle.

[0172] Examples (e.g., Example 19) relate to a non-transitory computer-readable medium. The non-transitory computer-readable medium is configured to store instructions, which, when executed by a processing circuitry system of an augmented reality (AR) device, cause the AR device to: receive data via a communication link with a vehicle, the data including (i) the geographic location of the vehicle, (ii) autonomous vehicle (AV) map data including landmarks and corresponding predetermined geographic locations, and (iii) the position and orientation of the vehicle; calculate a passenger field of view (FoV) using sensor data indicating the position and orientation of the AR device, the passenger FoV being associated with a passenger wearing the AR device in the vehicle; and, based on the vehicle's geographic location within the vehicle... The system calculates the vehicle's Field of View (FoV) based on the location and orientation at the vantage point; identifies one or more features and / or objects contained within the vehicle's FoV, as identified by the AV map data; determines the relative orientation of the identified one or more features and / or objects relative to the vehicle using the AV map data; generates an AR display frame based on the vehicle's geographic location, the passenger's FoV, and the relative orientation of the identified one or more features and objects; and causes the display of the AR device to present the AR display frame, which includes a graphical representation of the one or more identified features and objects, within the passenger's FoV in an AR view.

[0173] Another example (e.g., Example 20) relates to the previously described example (e.g., Example 19), where the display includes an AR glasses display.

[0174] Another example (e.g., Example 21) relates to the previously described examples (e.g., one or more of Examples 19 to 20), wherein the display includes a motorcycle helmet visor display.

[0175] Another example (e.g., Example 22) relates to the previously described examples (e.g., one or more of Examples 19 to 21), wherein one or more of the identified features and / or objects are overlaid relative to the passenger FoV at the corresponding location of the associated physical object or physical feature.

[0176] Another example (e.g., Example 23) relates to the previously described examples (e.g., one or more of Examples 19 to 22), wherein the instructions, when executed by the processing circuitry system, further cause the AR device to: calculate a delay period between (i) the time for calculating the vehicle's position and orientation, and (ii) the time for generating the AR display frame; and to present the graphical representation of the one or more identified features and / or objects in the AR view by selectively compensating for changes in the vehicle's position and orientation during the delay period.

[0177] Another example (e.g., Example 24) relates to the previously described examples (e.g., one or more of Examples 19 to 23), wherein the instructions, when executed by the processing circuitry system, further cause the AR device to selectively compensate for changes in the vehicle's position and orientation during the delay period by tracking the received position and orientation of the vehicle and adjusting the vehicle's FoV to match the updated position and orientation of the vehicle corresponding to the time the AR display frame is generated.

[0178] Another example (e.g., Example 25) relates to the previously described examples (e.g., one or more of Examples 19 to 24), wherein the instructions, when executed by the processing circuitry system, further cause the AR device to compensate for changes in the vehicle's position and orientation during the delay period when the delay period exceeds a predetermined threshold time period.

[0179] Another example (e.g., Example 26) relates to the previously described examples (e.g., one or more of Examples 19 to 25), wherein the instructions, when executed by the processing circuitry system, further cause the AR device to: calculate a delay period between (i) the time for calculating the vehicle's position and orientation, and (ii) the time for generating the AR display frame; and when the delay period exceeds a predetermined threshold time period, adjust one or more parameters regarding the manner of receiving data via the communication link to reduce the delay.

[0180] Another example (e.g., Example 27) relates to the previously described examples (e.g., one or more of Examples 19 to 26), wherein the instructions, when executed by the processing circuitry system, further cause the AR device to: calculate a delay period between (i) the time for calculating the vehicle's position and orientation, and (ii) the time for generating the AR display frame; and when the delay period exceeds a predetermined threshold time period, transmit a request to the vehicle via the communication link to adjust one or more parameters regarding the manner of calculating the vehicle's position and orientation, thereby reducing the delay.

[0181] Another example (e.g., Example 28) relates to the previously described examples (e.g., one or more of Examples 19 to 27), wherein the instructions, when executed by the processing circuitry system, further cause the AR device to: identify one or more additional features and / or objects contained within the vehicle FoV, said one or more additional objects being detected via one or more vehicle sensors; and configure the display to present a graphical representation of said identified one or more additional features and objects in the AR view.

[0182] Another example (e.g., Example 29) relates to the previously described examples (e.g., one or more of Examples 19 to 28), wherein one or more additional features and / or objects identified are overlaid relative to the passenger FoV at the corresponding location of the associated physical object or physical feature.

[0183] Another example (e.g., Example 30) relates to the previously described examples (e.g., one or more of Examples 19 to 29), wherein the one or more vehicle sensors include one or more light detection and ranging (LIDAR) sensors.

[0184] Another example (e.g., Example 31) relates to the previously described examples (e.g., one or more of Examples 19 to 30), wherein the one or more additional features and / or objects include one or more pedestrians.

[0185] Another example (e.g., Example 32) relates to the previously described examples (e.g., one or more of Examples 19 to 31), wherein one or more additional features and / or objects identify the trajectory of the vehicle according to an advanced driver assistance system (ADAS).

[0186] Another example (e.g., Example 33) relates to the previously described examples (e.g., one or more of Examples 19 to 32), wherein the instructions, when executed by the processing circuitry system, further cause the AR device to identify a hazardous situation estimated or predicted according to the applied Safe Driving Model (SDM), and generate the AR display frame by adding a warning display feature to the AR frame when the hazardous situation is identified.

[0187] Another example (e.g., Example 34) relates to the previously described examples (e.g., one or more of Examples 19 to 33), wherein the warning display feature is overlaid in the AR display frame relative to the passenger FoV at a location corresponding to the orientation of an object associated with the identified hazardous situation.

[0188] Another example (e.g., Example 35) relates to the previously described examples (e.g., one or more of Examples 19 to 34), wherein the orientation of the warning display feature is overlaid in the AR display frame at an orientation associated with the estimated focus orientation of the passenger in the vehicle.

[0189] Examples (e.g., Example 36) relate to an augmented reality (AR) device. The AR device includes: a data interface configured to receive data via a communication link with a vehicle, the data including (i) the geographic location of the vehicle, (ii) autonomous vehicle (AV) map data including landmarks and corresponding predetermined geographic locations, and (iii) the position and orientation of the vehicle; a processing circuitry configured to: calculate a passenger field of view (FoV) using sensor data indicating the position and orientation of the AR device, the passenger FoV being associated with a passenger wearing the AR device in the vehicle; calculate a vehicle FoV based on the vehicle's position and orientation at the geographic location of the vehicle; identify one or more features and / or objects contained within the vehicle FoV as identified by the AV map data; determine the relative orientation of the identified one or more features and / or objects relative to the vehicle using the AV map data; generate an AR display frame based on the vehicle's geographic location, the passenger FoV, and the relative orientation of the identified one or more features and objects; and a display configured to present the AR display frame, including a graphical representation of the one or more identified features and objects, within the passenger FoV in an AR view.

[0190] Another example (e.g., Example 37) relates to the previously described example (e.g., Example 36), where the display includes an AR glasses display.

[0191] Another example (e.g., Example 38) relates to the previously described examples (e.g., one or more of Examples 36 to 37), wherein the display includes a motorcycle helmet visor display.

[0192] Another example (e.g., Example 39) relates to the previously described examples (e.g., one or more of Examples 36 to 38), wherein the one or more identified features and / or objects are overlaid relative to the passenger FoV at the corresponding location of the associated physical object or physical feature.

[0193] Another example (e.g., Example 40) relates to the previously described examples (e.g., one or more of Examples 36 to 39), wherein the processing circuitry is further configured to: calculate a time delay between (i) the time for calculating the vehicle’s position and orientation, and (ii) the time for generating the AR display frame; and to present the graphical representation of the one or more identified features and / or objects in the AR view by selectively compensating for changes in the vehicle’s position and orientation during the time delay.

[0194] Another example (e.g., Example 41) relates to the previously described examples (e.g., one or more of Examples 36 to 40), wherein the instructions, wherein the processing circuitry system is further configured to selectively compensate for changes in the vehicle's position and orientation during the delay period by tracking the received position and orientation of the vehicle and adjusting the vehicle's FoV to match the updated position and orientation of the vehicle corresponding to the time the AR display frame is generated.

[0195] Another example (e.g., Example 42) relates to the previously described examples (e.g., one or more of Examples 36 to 41), wherein the instructions, wherein the processing circuitry system is further configured to cause the AR device to compensate for changes in the vehicle's position and orientation during the delay period when the delay period exceeds a predetermined threshold time period.

[0196] Another example (e.g., Example 43) relates to the previously described examples (e.g., one or more of Examples 36 to 42), wherein the processing circuitry is further configured to: calculate a delay period between (i) the time for calculating the vehicle’s position and orientation, and (ii) the time for generating the AR display frame; and when the delay period exceeds a predetermined threshold time period, adjust one or more parameters regarding the manner in which data is received via the communication link to reduce the delay.

[0197] Another example (e.g., Example 44) relates to the previously described examples (e.g., one or more of Examples 36 to 43), wherein the processing circuitry is further configured to: calculate a delay period between (i) the time of calculating the vehicle’s position and orientation, and (ii) the time of generating the AR display frame; and when the delay period exceeds a predetermined threshold time period, transmit a request to the vehicle via the communication link to adjust one or more parameters regarding the manner of calculating the vehicle’s position and orientation, thereby reducing the delay.

[0198] Another example (e.g., Example 45) relates to the previously described examples (e.g., one or more of Examples 36 to 44), wherein the processing circuitry is further configured to: identify one or more additional features and / or objects contained within the vehicle FoV, said one or more additional objects being detected via one or more vehicle sensors; and configure the display to present a graphical representation of said identified one or more additional features and objects in the AR view.

[0199] Another example (e.g., Example 46) relates to the previously described examples (e.g., one or more of Examples 36 to 45), wherein one or more additional features and / or objects identified are overlaid relative to the passenger FoV at the corresponding location of the associated physical object or physical feature.

[0200] Another example (e.g., Example 47) relates to the previously described examples (e.g., one or more of Examples 36 to 46), wherein the one or more vehicle sensors include one or more light detection and ranging (LIDAR) sensors.

[0201] Another example (e.g., example 48) relates to the previously described examples (e.g., one or more of examples 36 to 47), wherein the one or more additional features and / or objects include one or more pedestrians.

[0202] Another example (e.g., Example 49) relates to the previously described examples (e.g., one or more of Examples 36 to 48), wherein one or more additional features and / or objects identify the trajectory of the vehicle according to an advanced driver assistance system (ADAS).

[0203] Another example (e.g., Example 50) relates to the previously described examples (e.g., one or more of Examples 36 to 49), wherein the processing circuitry is further configured to identify hazardous situations estimated or predicted according to an applied Safe Driving Model (SDM), and to generate the AR display frame by adding a warning display feature to the AR frame when the hazardous situation is identified.

[0204] Another example (e.g., Example 51) relates to the previously described examples (e.g., one or more of Examples 36 to 50), wherein the warning display feature is overlaid in the AR display frame relative to the passenger FoV at a location corresponding to the orientation of an object associated with the identified hazardous situation.

[0205] Another example (e.g., Example 52) relates to the previously described examples (e.g., one or more of Examples 36 to 49), wherein the orientation of the warning display feature is overlaid in the AR display frame at an orientation associated with the estimated focus orientation of the passenger in the vehicle.

[0206] A device as shown and described in the figure.

[0207] A method shown and described in the figure.

[0208] in conclusion

[0209] The foregoing description of the specific aspects so fully reveals the general nature of this disclosure that others can easily modify and / or adapt various applications of such specific aspects by applying knowledge within the art, without excessive experimentation and without departing from the general conception of this disclosure. Therefore, based on the teachings and guidance presented herein, such adaptations and modifications are intended to fall within the meaning and scope of equivalents of the disclosed aspects. It should be understood that phrases or terms in this document are for descriptive and not limiting purposes, and that the terminology or phrases in this specification will be interpreted by those skilled in the art based on the teachings and guidance.

[0210] References to "an aspect," "one aspect," "exemplary aspect," etc., in this specification indicate that the described aspect may include a particular feature, structure, or characteristic, but not every aspect necessarily includes that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same aspect. Moreover, when a particular feature, structure, or characteristic is described in conjunction with an aspect, whether explicitly stated or not, it is assumed that the influence of such feature, structure, or characteristic on such feature, structure, or characteristic in conjunction with other aspects is within the knowledge of those skilled in the art.

[0211] The exemplary aspects described herein are provided for illustrative purposes and not for limitation. Other exemplary aspects are possible, and modifications may be made to the exemplary aspects. Therefore, this specification is not intended to limit this disclosure. Rather, the scope of this disclosure is defined solely by the appended claims and their equivalents.

[0212] The aspects can be implemented in hardware (e.g., circuitry), firmware, software, or any combination thereof. The aspects can also be implemented as instructions stored on a machine-readable medium, which can be read and executed by one or more processors. The machine-readable medium can include any mechanism for storing or transmitting information in a machine-readable form (e.g., a computing device). For example, a machine-readable medium can include: read-only memory (ROM); random access memory (RAM); disk storage media; optical storage media; flash memory devices; electrical, optical, acoustic, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.), and so on. Furthermore, firmware, software, routines, and instructions may be described herein as performing certain actions. However, it should be understood that such descriptions are merely for convenience, and such actions are actually generated by a computing device, processor, controller, or other means of executing firmware, software, routines, instructions, etc. Moreover, any variation of the implementation can be implemented by a general-purpose computer.

[0213] For the purposes of this discussion, the term "processor circuit system" or "processor circuit system" should be understood as one or more circuits, one or more processors, logic, or a combination thereof. For example, a circuit may include analog circuits, digital circuits, state machine logic, other structured electronic hardware, or a combination thereof. A processor may include a microprocessor, a digital signal processor (DSP), or other hardware processor. A processor may be "hard-coded" with instructions to perform one or more corresponding functions according to the aspects described herein. Alternatively, a processor may access internal and / or external memory to retrieve instructions stored in memory that, when executed by the processor, perform one or more corresponding functions associated with the processor and / or one or more functions and / or operations related to the operation of components having a processor included therein.

[0214] In one or more of the exemplary aspects described herein, the processing circuitry may include memory storing data and / or instructions. The memory may be any well-known volatile and / or non-volatile memory, including, for example, read-only memory (ROM), random access memory (RAM), flash memory, magnetic storage media, optical disk, erasable programmable read-only memory (EPROM), and programmable read-only memory (PROM). The memory may be non-removable, removable, or a combination of both.

Claims

1. A vehicle comprising: The processing circuit system is configured as follows: The geographic location of the vehicle is determined using autonomous vehicle (AV) map data that includes landmarks and corresponding predetermined geographic locations. Calculate the vehicle FoV associated with the vehicle's position and orientation at the vehicle's geographic location; Calculate the passenger field of view (FoV) associated with the passengers of the vehicle. Identify one or more features and / or objects contained within the vehicle's field of view (FoV) as identified by the AV map data; The AV map data is used to determine the relative orientation of one or more identified features and / or objects with respect to the vehicle. as well as An augmented reality (AR) display frame is generated based on the vehicle's geographic location, the passenger's field of view (FoV), and the relative location of one or more identified features and objects. as well as A display configured to present the AR display frame in an AR view, the AR display frame including a graphical representation associated with one or more of the identified features and / or objects contained within the passenger FoV.

2. The vehicle of claim 1, wherein the display comprises a vehicle head-up display.

3. The vehicle of claim 2, wherein the vehicle head-up display includes the inner surface of the windshield of the vehicle.

4. The vehicle of claim 1, wherein one or more of the identified features and / or objects are covered relative to the passenger FoV at the corresponding location of the associated physical object or physical feature.

5. A non-transitory computer-readable medium configured to store instructions that, when executed by a processing circuitry system of an augmented reality (AR) device, cause the AR device to: Data is received via a communication link with the vehicle, the data including (i) the geographical location of the vehicle, (ii) autonomous vehicle (AV) map data including landmarks and corresponding predetermined geographical locations, and (iii) the position and orientation of the vehicle. The passenger field of view (FoV) is calculated using sensor data indicating the position and orientation of the AR device, and the passenger FoV is associated with the passenger in the vehicle wearing the AR device; The vehicle FoV is calculated based on the vehicle's position and orientation at the vehicle's geographic location; Identify one or more features and / or objects contained within the vehicle's field of view (FoV) as identified by the AV map data; The AV map data is used to determine the relative orientation of one or more identified features and / or objects with respect to the vehicle. An AR display frame is generated based on the vehicle's geographic location, the passenger's field of view (FoV), and the relative location of one or more identified features and objects; as well as The display of the AR device presents, within the passenger's FoV, an AR display frame that includes a graphical representation of one or more of the identified features and objects.

6. The non-transitory computer-readable medium of claim 5, wherein the display includes an AR glasses display.

7. The non-transitory computer-readable medium of claim 5, wherein the display comprises a motorcycle helmet visor display.

8. The non-transitory computer-readable medium of claim 5, wherein one or more of the identified features and / or objects are overlaid relative to the passenger FoV at the corresponding location of the associated physical object or physical feature.

9. An augmented reality (AR) device comprising: A data interface configured to receive data via a communication link with the vehicle, the data including (i) the geographic location of the vehicle, (ii) autonomous vehicle (AV) map data including landmarks and corresponding predetermined geographic locations, and (iii) the position and orientation of the vehicle. The processing circuit system is configured as follows: The passenger field of view (FoV) is calculated using sensor data indicating the position and orientation of the AR device, and the passenger FoV is associated with the passenger in the vehicle wearing the AR device; The vehicle FoV is calculated based on the vehicle's position and orientation at the vehicle's geographic location; Identify one or more features and / or objects contained within the vehicle's field of view (FoV) as identified by the AV map data; The AV map data is used to determine the relative orientation of one or more identified features and / or objects with respect to the vehicle. An AR display frame is generated based on the vehicle's geographic location, the passenger's field of view (FoV), and the relative location of one or more identified features and objects; as well as A display configured to present, within the passenger FoV, an AR display frame that includes a graphical representation of one or more of the identified features and objects.

10. The AR device of claim 9, wherein the display includes an AR glasses display.

Images