Systems and methods for augmented reality in vehicles

Abstract

This document provides systems and methods for presenting information in a vehicle. In one example, a method includes: determining a route from a source location to a destination location; obtaining one or more features of the route; enhancing a map of the route with the one or more features to generate an enhanced map; and displaying the enhanced map on the windshield of the vehicle.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to Indian Provisional Application No. 202041015794, filed on April 11, 2020, entitled "Systems and methods for augmented reality in a vehicle". The entire contents of the application listed above are incorporated herein by reference for all purposes. Technical Field

[0003] This disclosure relates to an in-vehicle augmented reality projector and related operations. Background Technology

[0004] Vehicles provide various information and services to in-vehicle infotainment systems, such as streaming services. Information received from these services, as well as vehicle operating parameters and navigation information, can be presented to vehicle occupants via one or more displays. However, vehicle displays may be small and positioned outside the driver's line of sight. Summary of the Invention

[0005] In one embodiment, a method for presenting information in a vehicle includes: determining a route from a source location to a destination location; obtaining one or more features of the route; enhancing a map of the route with the one or more features to generate an enhanced map; and displaying the enhanced map on the windshield of the vehicle.

[0006] The above and other advantages and features of this specification will readily become apparent from the following detailed description, either alone or in conjunction with the accompanying drawings.

[0007] It should be understood that the above overview is provided to present the concept choices further described in the detailed embodiments in a simplified form. This is not intended to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims following the detailed embodiments. Furthermore, the claimed subject matter is not limited to implementations that address any shortcomings pointed out above or in any part of this disclosure. Attached Figure Description

[0008] Figure 1 An example partial view of a carriage according to one or more embodiments of the present disclosure is shown;

[0009] Figure 2 An example in-vehicle computing system according to one or more embodiments of the present disclosure is shown;

[0010] Figure 3An example augmented reality (AR) view presented on the windshield of a vehicle according to one or more embodiments of the present disclosure is shown;

[0011] Figure 4 This is a block diagram illustrating an example AR system according to one or more embodiments of the present disclosure;

[0012] Figure 5 This is a block diagram illustrating an example enhanced generation system according to one or more embodiments of the present disclosure;

[0013] Figure 6 This is a block diagram illustrating an example AR projector according to one or more embodiments of the present disclosure;

[0014] Figure 7 Another example AR view presented on the windshield of a vehicle according to one or more embodiments of the present disclosure is shown;

[0015] Figure 8 and Figure 9 Is using Figures 1 to 2 and Figures 4 to 6 A flowchart illustrating an example method for presenting information in a vehicle using an example system;

[0016] Figure 10 This is a block diagram illustrating another example of an enhanced generation system according to one or more embodiments of the present disclosure; and

[0017] Figures 11A to 11C The process for determining data coverage along a route is illustrated schematically according to one or more embodiments of this disclosure. Detailed Implementation

[0018] Now go to Figure 1 The image shows an example partial view of an environment for one type of infotainment system, which includes an image enhancement module configured to output augmented reality to an augmented reality projector. The system includes the interior of a passenger compartment 100 of a vehicle 102, where a driver and / or one or more passengers can be seated. Figure 1Vehicle 102 may be a motor vehicle including drive wheels (not shown) and an internal combustion engine 104. The internal combustion engine 104 may include one or more combustion chambers that receive intake air via an intake passage and exhaust combustion gases via an exhaust passage. Vehicle 102 may be a road vehicle, as well as other types of vehicles. In some examples, vehicle 102 may include a hybrid propulsion system including an energy conversion device operable to absorb energy from vehicle motion and / or the engine and convert the absorbed energy into an energy form suitable for storage by an energy storage device. Vehicle 102 may include an all-electric vehicle incorporating a fuel cell, solar capture elements, and / or other energy storage systems for powering the vehicle.

[0019] As shown in the figure, the dashboard 106 may include various displays and controls accessible to the human driver (also known as the user) of the vehicle 102. For example, the dashboard 106 may include a touchscreen 108 of an in-vehicle computing system 109 (e.g., an infotainment system), an audio system control panel, and an instrument cluster 110. The touchscreen 108 can receive user input from the in-vehicle computing system 109 for controlling audio output, visual display output, user preferences, control parameter selection, etc. Although... Figure 1The example system shown includes audio system controls that can be executed via a user interface of the in-vehicle computing system 109 (such as a touchscreen 108 without a separate audio system control panel). However, in other embodiments, the vehicle may include an audio system control panel, which may include controls for conventional vehicle audio systems such as a radio, CD player, MP3 player, etc. The audio system controls may include features for controlling one or more aspects of audio output via the speakers 112 of the vehicle speaker system. For example, the in-vehicle computing system or audio system controls may control the volume of the audio output, the sound distribution between the individual speakers of the vehicle speaker system, the equalization of the audio signal, and / or any other aspect of the audio output. In another example, the in-vehicle computing system 109 may adjust radio station selection, playlist selection, audio input source (e.g., from a radio, CD, or MP3), etc., based on user input received directly via the touchscreen 108 or based on data about the user (such as the user's physical condition and / or environment) received via external devices 150 and / or mobile devices 128. The vehicle's audio system may include amplifiers (not shown) coupled to multiple speakers (not shown). In some embodiments, one or more hardware components of the in-vehicle computing system 109 (such as touchscreen 108, display screen 111, various control dials, knobs and buttons, memory, processor, and any interface components (e.g., connectors or ports)) may form an integrated head unit mounted in the vehicle's dashboard 106. The head unit may be fixedly or removably attached to the dashboard 106. In additional or alternative embodiments, one or more hardware components of the in-vehicle computing system 109 may be modular and may be mounted in multiple locations within the vehicle.

[0020] The vehicle may include one or more sensors for monitoring the vehicle, the user, and / or the environment. For example, sensors may be positioned in the powertrain compartment, on the exterior surface of the vehicle, and / or other suitable locations to provide information about the vehicle's operation, environmental conditions, the user, etc. Information about the vehicle's environmental conditions, vehicle status, or the vehicle's driver may also be received from sensors external to the vehicle or separate from the vehicle (i.e., not part of the vehicle system) such as sensors coupled to external device 150 and / or moving device 128.

[0021] The vehicle may include one or more cameras for monitoring the vehicle's surroundings, traffic information, and / or environment. For example, the cameras may be positioned at the front, sides, rear, top, and / or any other location on the vehicle. Image information captured by one or more cameras may be displayed on the device display described herein. For example, when the vehicle is reversing, video feeds from one or more rear-facing cameras may be displayed on the device display.

[0022] The vehicle compartment 100 may also include one or more user objects, such as mobile devices 128, stored in the vehicle before, during, and / or after travel. Mobile devices 128 may include smartphones, tablets, laptops, portable media players, and / or any suitable mobile computing device. Mobile devices 128 may be connected to an in-vehicle computing system via a communication link 130. The communication link 130 may be wired (e.g., via Universal Serial Bus [USB], Mobile High Definition Link [MHL], High Definition Multimedia Interface [HDMI], Ethernet, etc.) or wireless (e.g., via BlueTooth, Wi-Fi, Wi-Fi Direct, Near Field Communication [NFC], cellular connectivity, etc.) and configured to provide bidirectional communication between the mobile device and the in-vehicle computing system. Mobile devices 128 may include one or more wireless communication interfaces for connecting to one or more communication links (e.g., one or more of the example communication links described above). The wireless communication interface may include one or more physical devices (such as antennas or ports coupled to data lines to transmit or receive data), and one or more modules / drivers for operating the physical devices according to other devices within the mobile device. For example, communication link 130 can provide sensor and / or control signals from various vehicle systems (such as vehicle audio systems, climate control systems, etc.) and touchscreen 108 to mobile device 128, and can provide control and / or display signals from mobile device 128 to the in-vehicle system and touchscreen 108. Communication link 130 can also provide power to mobile device 128 from the vehicle power supply to charge the mobile device's internal battery.

[0023] The in-vehicle computing system 109 can also be communicatively coupled to additional devices, such as one or more external devices 150, that are operated and / or accessed by a user but located outside the vehicle 102. In the depicted embodiments, the external devices are located outside the vehicle 102; however, it will be understood that in alternative embodiments, the external devices may be located inside the vehicle compartment 100. External devices may include server computing systems, personal computing systems, portable electronic devices, electronic wristbands, electronic headbands, portable music players, electronic activity trackers, pedometers, smartwatches, GPS systems, etc. As discussed with reference to communication link 130, external devices 150 may be connected to the in-vehicle computing system via a communication link 136, which may be wired or wireless, and are configured to provide bidirectional communication between the external device and the in-vehicle computing system. For example, external device 150 may include one or more sensors, and communication link 136 may transmit sensor outputs from external device 150 to the in-vehicle computing system 109 and touchscreen 108. External device 150 can also store and / or receive information about contextual data, user behavior / preferences, operating rules, etc., and can transmit such information from external device 150 to in-vehicle computing system 109 and touchscreen 108. As described herein, communication links may be limited in some locations, referred to as black spots.

[0024] The in-vehicle computing system 109 can analyze input received from external devices 150, mobile devices 128, and / or other input sources and select settings for various in-vehicle systems (such as audio systems), provide output via touchscreen 108 and / or speaker 112, communicate with mobile devices 128 and / or external devices 150, and / or perform other actions based on evaluation. In some embodiments, all or part of the evaluation can be performed by mobile devices 128 and / or external devices 150.

[0025] In some implementations, one or more of the external devices 150 may be communicatively coupled to the in-vehicle computing system 109 indirectly via mobile device 128 and / or another of the external devices 150. For example, communication link 136 may communicatively couple an external device 150 to mobile device 128, such that output from external device 150 is relayed to mobile device 128. Data received from external device 150 may then be aggregated with data collected by mobile device 128 at mobile device 128, and the aggregated data may be transmitted to in-vehicle computing system 109 and touchscreen 108 via communication link 130. Similar data aggregation may occur at a server system and then be transmitted to in-vehicle computing system 109 and touchscreen 108 via communication links 136 / 130.

[0026] Figure 2A block diagram of an in-vehicle computing system 109 configured and / or integrated within vehicle 102 is shown. In some embodiments, the in-vehicle computing system 109 may perform one or more of the methods described herein. In some examples, the in-vehicle computing system 109 may be a vehicle infotainment system configured to provide information-based media content (audio and / or visual media content, including entertainment content, navigation services, etc.) to vehicle users to enhance the operator's in-vehicle experience. The vehicle infotainment system may include or be coupled to various vehicle systems, subsystems, hardware components, and software applications and systems integrated into or potentially integrated into vehicle 102 to enhance the in-vehicle experience for the driver and / or passengers.

[0027] The in-vehicle computing system 109 may include one or more processors, including an operating system processor 214 and an interface processor 220. The operating system processor 214 can execute the operating system on the in-vehicle computing system and control the input / output, display, playback, and other operations of the in-vehicle computing system. The interface processor 220 can interface with the vehicle control system 230 via the inter-vehicle system communication module 222.

[0028] The inter-vehicle system communication module 222 can output data to other vehicle systems 231 and vehicle control elements 261, and also receive data input from other vehicle components and systems 231, 261, for example, via the vehicle control system 230. When outputting data, the inter-vehicle system communication module 222 can provide signals via a bus corresponding to any state of the vehicle, the vehicle's surrounding environment, or the output of any other information source connected to the vehicle. Vehicle data outputs may include, for example, analog signals (such as current speed), digital signals provided by various information sources (such as clocks, thermometers, position sensors such as GPS sensors, etc.), and digital signals transmitted via vehicle data networks (such as the engine CAN bus through which engine-related information can be transmitted, the climate control CAN bus through which climate control-related information can be transmitted, and the multimedia data network through which multimedia data can be transmitted between the vehicle's multimedia components). For example, the on-board computing system 109 can retrieve the vehicle's current speed estimated by wheel sensors, the vehicle's power status via the vehicle's battery and / or power distribution system, the vehicle's ignition status, etc., from the engine CAN bus. Furthermore, other interfacing devices such as Ethernet may be used without departing from the scope of this disclosure.

[0029] Non-volatile storage device 208 may be included in the in-vehicle computing system 109 to store data such as instructions executable by processors 214 and 220 in a non-volatile form. Storage device 208 may store application data including pre-recorded audio to enable the in-vehicle computing system 109 to run applications for connecting to and / or collecting information for transmission to a cloud-based server. The applications may retrieve information collected by vehicle systems / sensors, input devices (e.g., user interface 218), data stored in volatile storage device (e.g., memory) 219A or non-volatile storage device (e.g., memory) 219B, devices communicating with the in-vehicle computing system (e.g., mobile devices connected via a Bluetooth link), etc. The in-vehicle computing system 109 may also include volatile memory 219A. Volatile memory 219A may be random access memory (RAM). Non-transitory storage devices such as non-volatile storage device 208 and / or non-volatile memory 219B can store instructions and / or code that, when executed by a processor (e.g., operating system processor 214 and / or interface processor 220), control the on-board computing system 109 to perform one or more of the actions described in this disclosure.

[0030] Microphone 202 may be included in the in-vehicle computing system 109 to receive voice commands from a user, measure ambient noise in the vehicle, determine whether to tune audio from the vehicle speakers according to the vehicle's acoustic environment, etc. Voice processing unit 204 may process voice commands, such as those received from microphone 202. In some embodiments, the in-vehicle computing system 109 may also be able to use microphones included in the vehicle's audio system 232 to receive voice commands and sample ambient vehicle noise.

[0031] The sensor subsystem 210 of the vehicle computing system 109 may include one or more additional sensors. For example, the sensor subsystem 210 may include cameras, such as a rearview camera to assist a user in parking the vehicle and / or a cabin camera to identify the user (e.g., using facial recognition and / or user gestures). The sensor subsystem 210 of the vehicle computing system 109 can communicate with and receive input from various vehicle sensors, and may also receive user input. For example, the input received by the sensor subsystem 210 may include transmission gear position, transmission clutch position, accelerator pedal input, brake input, transmission selector position, vehicle speed, engine speed, airflow through the engine, ambient temperature, intake air temperature, etc., as well as input from climate control system sensors (such as heat transfer fluid temperature, antifreeze temperature, fan speed, passenger compartment temperature, desired passenger compartment temperature, ambient humidity, etc.), audio sensors that detect voice commands issued by the user, remote key sensors that receive commands from the vehicle's remote key and optionally track its geographical location / proximity, etc. While some vehicle system sensors can communicate independently with sensor subsystem 210, others can communicate with both sensor subsystem 210 and vehicle control system 230, or indirectly via vehicle control system 230 to sensor subsystem 210. The navigation subsystem 211 of the onboard computing system 109 can generate and / or receive navigation information, such as location information (e.g., via GPS sensors and / or other sensors from sensor subsystem 210), route guidance, traffic information, points of interest (POI) identification, and / or provide other navigation services to the driver.

[0032] The external device interface 212 of the in-vehicle computing system 109 can be coupled to and / or communicate with one or more external devices 150 located outside the vehicle 102. Although the external devices are shown as being located outside the vehicle 102, it should be understood that they may be temporarily housed within the vehicle 102, such as when a user operates the external device while operating the vehicle 102. In other words, the external device 150 is not integrated with the vehicle 102. The external device 150 may include a mobile device 128 (e.g., connected via Bluetooth, NFC, Wi-Fi Direct, 4G LTE, 5G connectivity, or other wireless connectivity) or alternatively, a Bluetooth-enabled device 252. The mobile device 128 may be a mobile phone, a smartphone, a wearable device / sensor that can communicate with the in-vehicle computing system via wired and / or wireless communication, or other portable electronic device. Other external devices include external services 246. For example, external devices may include vehicle-exterior devices that are detached from the vehicle and located outside the vehicle. Other external devices also include external storage devices 254, such as solid-state drives, pen drives, USB drives, etc. External device 150 may communicate with vehicle computing system 109 wirelessly or via a connector without departing from the scope of this disclosure. For example, external device 150 may communicate with vehicle computing system 109 via external device interface 212 through network 260, universal serial bus (USB) connection, direct wired connection, direct wireless connection and / or other communication links.

[0033] External device interface 212 may provide a communication interface to enable the in-vehicle computing system to communicate with mobile devices associated with the driver's contacts. For example, external device interface 212 may enable the establishment of telephone calls and / or the sending (e.g., SMS, MMS, etc.) of text messages to mobile devices associated with the driver's contacts (e.g., via cellular communication networks). External device interface 212 may additionally or alternatively provide a wireless communication interface to enable the in-vehicle computing system to synchronize data with one or more devices in the vehicle (e.g., the driver's mobile device) via Wi-Fi Direct, as described in more detail below.

[0034] One or more applications 244 may operate on mobile device 128. As an example, mobile device application 244 may be operated to aggregate user data regarding user interactions with the mobile device. For instance, mobile device application 244 may aggregate data regarding music playlists listened to by the user on the mobile device, phone call logs (including the frequency and duration of phone calls received by the user), location information (including frequently visited locations and the amount of time spent at each location), etc. The collected data may be transmitted by application 244 to external device interface 212 via network 260. Furthermore, specific user data requests may be received at mobile device 128 from in-vehicle computing system 109 via external device interface 212. Specific data requests may include requests to determine the user's geographic location, ambient noise levels and / or music type at the user's location, ambient weather conditions (temperature, humidity, etc.) at the user's location, etc. Mobile device application 244 may send control commands to components of mobile device 128 (e.g., microphones, amplifiers, etc.) or other applications (e.g., navigation applications) to enable the requested data to be collected on the mobile device or to make the requested adjustments to the components. Then, the mobile device application 244 can relay the collected information back to the vehicle computing system 109.

[0035] Similarly, one or more applications 248 can operate on external service 246. As an example, external service application 248 can be operated to aggregate and / or analyze data from multiple data sources. For instance, external service application 248 can aggregate data from one or more of a user's social media accounts, data from in-vehicle computing systems (e.g., sensor data, log files, user input, etc.), data from internet queries (e.g., weather data, POI data), etc. The collected data can be transferred to another device and / or analyzed by the application to determine the context of the driver, vehicle, and environment, and perform actions based on said context (e.g., requesting / sending data to other devices).

[0036] The vehicle control system 230 may include controls for controlling aspects of various vehicle systems 231 involved in different in-vehicle functions. These may include, for example, aspects of a vehicle audio system 232 for providing audio entertainment to vehicle occupants, aspects of a climate control system 234 for meeting the cooling or heating needs of the vehicle occupants' cabin, and aspects of a telecommunications system 236 for enabling vehicle occupants to establish telecommunications connections with others.

[0037] Audio system 232 may include one or more acoustic reproduction devices, including electromagnetic transducers such as speaker 235. Vehicle audio system 232 may be passive or active, such as by including a power amplifier. In some examples, in-vehicle computing system 109 may be the sole audio source for the acoustic reproduction devices, or other audio sources (e.g., external devices such as mobile phones) may be connected to the audio reproduction system. The connection of any such external device to the audio reproduction devices may be analog, digital, or any combination of analog and digital technologies.

[0038] The climate control system 234 can be configured to provide a comfortable environment within the passenger compartment or cabin of vehicle 102. The climate control system 234 includes components capable of achieving controlled ventilation, such as vents, heaters, air conditioners, integrated heater and air conditioning systems, etc. Other components linked to the heating and air conditioning settings may include a windshield defrosting and defogging system capable of cleaning the windshield and a ventilation air filter for cleaning outside air entering the passenger compartment through fresh air inlets.

[0039] The vehicle control system 230 may also include controls for adjusting settings of various vehicle controls 261 (or vehicle system control elements) related to the engine and / or auxiliary components within the vehicle cabin, such as steering wheel controls 262 (e.g., steering wheel-mounted audio system controls, cruise control, wiper controls, headlight controls, turn signal controls, etc.), instrument panel controls, microphones, accelerator / brake / clutch pedals, gearshift levers, door / window controls located in the driver's or passenger's door, seat controls, cabin light controls, audio system controls, cabin temperature controls, etc. Vehicle controls 261 may also include internal engine and vehicle operation controls (e.g., engine controller module, actuators, valves, etc.), configured to receive instructions via the vehicle's CAN bus to change the operation of one or more of the engine, exhaust system, transmission, and / or other vehicle systems. Control signals may also control audio output at one or more speakers 235 of the vehicle audio system 232. For example, control signals can adjust audio output characteristics such as volume, equalization, audio image (e.g., configuring audio signals to produce audio output that appears to the user to originate from one or more defined locations), audio distribution among multiple speakers, etc. Similarly, control signals can control the vents, air conditioning, and / or heaters of the climate control system 234. For example, control signals can increase the delivery of cooling air to specific parts of the vehicle compartment.

[0040] Control elements located outside the vehicle (e.g., controls for safety systems) may also be connected to the computing system 109, for example, via communication module 222. The control elements of the vehicle control system may be physically and permanently located on and / or within the vehicle to receive user input. In addition to receiving control commands from the onboard computing system 109, the vehicle control system 230 may also receive input from one or more external devices 150 operated by the user (such as from mobile device 128). This allows for control of various aspects of the vehicle system 231 and vehicle controls 261 based on user input received from external devices 150.

[0041] The in-vehicle computing system 109 may also include an antenna 206. Antenna 206 is shown as a single antenna, but in some embodiments it may include one or more antennas. The in-vehicle computing system can obtain broadband wireless internet access via antenna 206 and can also receive broadcast signals such as radio, television, weather, traffic, etc. The in-vehicle computing system can receive location signals such as GPS signals via one or more antennas 206. The in-vehicle computing system can also receive wireless commands via FR (e.g., via antenna 206 or via infrared or other means) through a suitable receiving device. In some embodiments, antenna 206 may be included as part of audio system 232 or telecommunications system 236. Additionally, antenna 206 may provide AM / FM radio signals to external device 150 (such as mobile device 128) via external device interface 212.

[0042] One or more components of the in-vehicle computing system 109 can be controlled by a user via a user interface 218. The user interface 218 may include a touchscreen (such as...) Figure 1 The graphical user interface (GUI) presented on the touchscreen 108, and / or user-actuated buttons, switches, knobs, dials, sliders, etc. For example, user-actuated elements may include steering wheel controls, door and / or window controls, dashboard controls, audio system settings, climate control system settings, etc. Users can also interact with one or more applications of the in-vehicle computing system 109 and the mobile device 128 via the GUI. In addition to receiving the user's vehicle setting preferences on the GUI, vehicle settings selected by the in-vehicle control system can also be displayed to the user on the GUI. Notifications and other messages (e.g., received messages) and navigation assistance can be displayed to the user on the GUI's display. User preferences / information and / or responses to presented messages can be performed via user input to the GUI.

[0043] In some examples, vehicle 102 can operate in one or more autonomous modes, in which some or all vehicle operations (e.g., acceleration, braking, steering) are automatically controlled without driver input. To facilitate autonomous or semi-autonomous operation, the vehicle can utilize outputs from various sensors described herein (e.g., radar sensors, machine vision cameras) to identify and track vehicles, pedestrians, cyclists, rough roads, potholes, and other objects and report these objects to the autonomous control module. The autonomous control module may be part of vehicle control system 230.

[0044] For example, radar sensors can communicate with the autonomous control module via vehicle data networks such as CAN bus, Flexray, or Ethernet. Machine vision cameras can also identify lane markings and report the curvature of the road ahead to the autonomous control module. It should be understood that the radar sensors and machine vision cameras mentioned here are exemplary to represent any number of possible sensors. In reality, a vehicle can have more sensors than the two discussed herein. For example, a vehicle can utilize multiple radar sensors and cameras facing different directions, with different ranges and different fields of view.

[0045] The autonomous control module can process information received from vehicle sensors (e.g., radar sensors and machine vision cameras) and calculate vehicle control actions in response. If sensor data indicates the presence of an object in front of and in the path of the main vehicle, the autonomous control module can communicate with the vehicle's brakes to initiate braking. The autonomous control module can also communicate with the vehicle's steering system to apply torque to the steering mechanism and prevent the vehicle from drifting out of its lane or around objects in its path.

[0046] As augmented reality (AR) enters the world of autonomous vehicles and automobiles, it offers the opportunity to realize a variety of scenarios without distracting the driver. AR can be used to virtually create any landscape and display its appearance without building anything. Therefore, this paper discloses implementation schemes for creating and displaying 5G signal strength Quality of Service (QoS) parameters, smart destinations, points of interest, charging stations, tourist attractions on a given route in an autonomous vehicle via AR.

[0047] Drivers using mobile phones and other forms of driver multitasking in vehicles can lead to distraction and contribute to road accidents. Conversely, as will be discussed in more detail below, AR projection units mounted behind the steering wheel and projected onto the vehicle's windshield can reduce driver distraction while informing the driver and / or any passengers of desired / relevant information for efficient navigation, road hazard awareness, and more, thus improving the driving experience.

[0048] This method relates to systems, devices, and methods for AR projection used in vehicle driver assistance. One of the main aspects of this method is receiving data from the vehicle, analyzing it, and sending the enhanced image back to the vehicle for projection onto a display. For example, the enhanced image includes images generated from information received from the vehicle, pre-stored images of popular destinations, and generated images for displaying QoS parameters.

[0049] In AR projection, the most common method is to use a projection display with touchscreen functionality. An advantageous feature of this disclosure is the use of the windshield as the plane on which the augmented image is projected. If too many AR elements are presented, or if they are presented temporarily, the resulting real-world view can be cluttered, potentially blurring the driver's observation of objects and hazards. Therefore, augmented images can be created with concise information without distracting the driver or affecting system functionality.

[0050] Figure 3 An example augmented reality (AR) view 300 is schematically shown, including an augmentation displayed on a flat surface of a vehicle (in this document, the windshield 302). This can be achieved from installations such as... Figure 1 An augmented reality projector 306 (AR projector 306) on the steering wheel 304 of a vehicle such as vehicle 102 projects an augmentation. As used herein, an augmentation can refer to an augmented reality display element (e.g., an image) projected onto a surface that is transparent or partially transparent (such as the surface of a windshield), thereby enabling the augmentation to blend into the real-world environment surrounding and behind the surface.

[0051] exist Figure 3In the example shown, different sets of enhancements are displayed at different locations on the windshield 302. For example, a first set of enhancements 308 is displayed near the steering wheel 304 (e.g., above the steering wheel, immediately to the left and / or right of the steering wheel, etc.). The first set of enhancements 308 includes vehicle information (e.g., vehicle speed, navigation information, vehicle-based alerts such as diagnostic lights) and is therefore positioned near the driver's field of vision (FOV), which can help facilitate the driver's monitoring of vehicle information without requiring the driver to take his or her attention away from the road. A second set of enhancements 310 is displayed in a different display area than the first set of enhancements 308, along the right-hand side of the windshield 302 (e.g., the passenger side). The second set of enhancements 310 may include emergency notifications (e.g., detected obstacles, impending severe weather, impending accidents), weather information, points of interest along the vehicle's route, etc. In some examples, the second set of enhancements 310 may be displayed only when the vehicle is operating in autonomous driving mode, in which the vehicle automatically controls vehicle handling, acceleration, and braking, etc., without input from the driver. In this way, a second enhanced set 310 can be displayed only when the driver does not need to be fully focused on operating the vehicle, and can be positioned in locations that might confuse the driver's field of view (FOV).

[0052] AR projector 306 can receive the enhancements to be displayed from an in-vehicle computing system (such as in-vehicle computing system 109). Figure 4 A block diagram of an example AR system 400 is shown. The AR system 400 includes an AR projector 306 coupled to a High Definition Multimedia Interface (HDMI) transmitter 402. The AR projector 306 can receive enhancements (e.g., images) to be projected and instructions regarding where to project the enhancements (e.g., display coordinates of each enhancement) from the HDMI transmitter 402. In some examples, the HDMI transmitter can wirelessly transmit the enhancements and any additional information to the AR projector 306, or the HDMI transmitter 402 can transmit the enhancements and / or any additional information via a wired connection (e.g., an HDMI cable / connector).

[0053] HDMI transmitter 402 receives enhancements from an in-vehicle infotainment (IVI) system, such as the aforementioned infotainment system. An IVI system is a collection of hardware and software in a vehicle that provides audio or video entertainment, and may include a combination of vehicle systems used to deliver entertainment and information to the driver and passengers via audio / video interfaces, control elements such as touchscreen displays, button panels, voice commands, gestures, etc. IVI systems can connect with and integrate with intelligent vehicle technologies such as ADAS systems, V2X / V2I connectivity solutions (e.g., vehicle-to-vehicle connectivity, vehicle-to-infrastructure connectivity, etc.), telematics devices, smartphones, sensors, etc., to provide a safe driving experience.

[0054] like Figure 4 As shown, the IVI system includes an IVI controller 404, which may be a system-on-a-chip (SoC) including a central processing unit (CPU), memory, input / output ports, auxiliary storage devices, etc. The IVI controller 404 is a non-limiting example of the in-vehicle computing system 109. As described above, the IVI controller 404 can receive signals from one or more cameras and other sensors. The IVI controller 404 can communicate with other devices via USB and Ethernet connections and can output audio signals to one or more speakers, also as described above. The IVI controller 404 can also output images, videos, etc., on the display unit 406. The display unit 406 may be a touchscreen (and therefore a non-limiting example of a user interface 218) and / or a standard display device, and may be a head-up unit display (e.g., touchscreen 108 or display screen 111) and / or a display for an instrument cluster (e.g., instrument cluster 110). The display unit 406 may include an LCD (liquid crystal display) or TFT (thin-film transistor) equipped with display technologies for complex and dynamic graphics. The IVI controller can transmit audio and video content to the display screen, speakers, and headphones via Bluetooth, HDMI cable, and USB. Images in JPEG, PNG, and BMP formats may be suitable for infotainment systems.

[0055] Figure 5 This is a schematic block diagram illustrating an example enhanced generation system 500. System 500 includes a vehicle 501, which includes a collection 502 of vehicle sensors and communication modules, which may include one or more cameras, a GPS system, and communication modules (e.g., for communicating with other devices via WiFi and cellular communication networks), as described above regarding... Figure 2As explained, cameras can provide real-time obstacle detection to facilitate lane departure and track road information (such as road signs). GPS systems can use satellites to triangulate the vehicle's position. Communication modules may include GPS, Wi-Fi, near field communication (NFC), and / or Bluetooth modules to provide connectivity to external networks and devices. These communication modules can help establish services such as navigation, internet connectivity, and integration with smartphones and infotainment systems.

[0056] Cameras (a key component of autonomous vehicles) rely more heavily on software for understanding the captured video frames. Automotive sensors, such as proximity sensors, gesture recognition sensors for detecting ambient light, camera sensors, and other onboard sensors, are integrated with the infotainment system to provide safety-related information to the driver and passengers. Connectivity components like Wi-Fi, 5G / LTE, and GPS modules are used in various applications, such as path planning, autonomous driving, obstacle detection, and parking assistance. The infotainment system supports advanced vehicle features like daytime running light indicators, in-vehicle climate controls, and voice assistants to control system functionality.

[0057] Vehicle 501 also includes an AR projector 306. To generate enhancements sent to the AR projector 306, system 500 includes a data acquisition module 504. Data acquisition module 504 receives outputs / signals from a collection 502 of vehicle sensors and communication modules. The outputs / signals from the collection 502 may include images and / or videos captured by cameras, location information from a GPS system (e.g., the vehicle's current location), and environmental and / or additional information from WiFi and / or cellular network communication modules (e.g., local weather conditions, nearby accidents, points of interest, WiFi and / or cellular network coverage, etc.). Separation and analysis module 506 may receive the outputs / signals obtained by data acquisition module 504 and may process the outputs / signals to determine various parameters along the vehicle's current route (such as weather conditions along the route, cellular network coverage along the route, whether the images / videos from the cameras include vehicles or other objects, etc.). The separation and analysis module 506 can transmit parameters determined along the vehicle's current route to the image enhancement module 508, which can determine which enhancements to display and send the enhancements to the AR projector based on the determined parameters. The image enhancement module 508 can retrieve enhancements (e.g., images) from the storage module 510 (which may be an auxiliary storage module of the IVI system). In some examples, the image enhancement module 508 can generate enhancements on the fly, for example, by modifying the enhancements obtained from the storage module 510. For example, the separation and analysis module 506 can determine that a pedestrian is positioned crossing the street in front of the vehicle based on the output / signal obtained by the data capture module 504. The image enhancement module 508 can retrieve pedestrian warning enhancements (such as...) from the storage device 510. Figure 3 (The pedestrian warning enhancement shown) is sent to the AR projector.

[0058] In some examples, predefined enhancements / images may be stored in storage device 510. Image enhancement module 508 may adjust a selected predefined image (e.g., pedestrian warning) based on information obtained from one or more vehicle sensors (e.g., the location of detected pedestrians, the size of detected pedestrians, ambient light levels, weather conditions, etc.) so that the displayed enhancement matches / appears to be part of the real world. In some examples, image enhancement module 508 may include one or more artificial intelligence-based models (such as CNNs) that can be trained to adjust the selected image based on information obtained from vehicle sensors.

[0059] In some examples, real-time vehicle information, such as current speed, can be determined from a Vehicle Information Processor (VIP) module that interfaces with the IVI controller. For example, as mentioned above... Figure 2As explained, the IVI controller can interface with a VIP (which can be...). Figure 2 A non-limiting example of a vehicle control system 230 may acquire analog signals (such as current speed), digital signals from various information sources (such as clocks, thermometers, position sensors such as GPS sensors, etc.), and digital signals transmitted via a vehicle data network (such as an engine CAN bus through which engine-related information can be transmitted). Furthermore, information related to direction, motion tracking, driver actions, and / or driver eye movements may be acquired via an Advanced Driver Assistance System (ADAS) chip that interfaces with an IVI controller or an operational telematics unit. As an illustrative and non-limiting example, the ADAS chip may handle processing related to ADAS (such as front-facing cameras, short / long-range radar, eMirror, etc.), as well as DMS / OMS, automated parking, highway navigation, evasive steering assist, intelligent intersections, surround view / parking, etc. The ADAS chip may be part of the aforementioned autonomous control module.

[0060] This information can be quickly absorbed and fed back to the augmentation module for image generation. Finally, the AR projection coordinates on a given projection area are programmable and can be changed within a given scene. The system can employ a client-server architecture, where the application communicates with the augmentation module (AM). For example, the real-time information and augmentation markers (AR markers) can be sent as a request to the AM, which returns the corresponding metadata (projection coordinates) as a response.

[0061] In this way, various parameters that may contribute to understanding smooth driving can be determined, including the location of the car or objects around the car, quality of service (QoS) parameters in the environment (e.g., 4G / 5G coverage), the fastest and safest route to the destination, and the ability to assist the driver in emergency situations. These parameters are then used to obtain / generate augmentations displayed via an AR projector. All data objects relating to personally identifiable information and privacy data can be encrypted and shared through a secure channel.

[0062] Figure 6 A block diagram schematically illustrating components of an AR projector 600 (such as AR projector 306) is shown. The AR projector 600 includes a processor 602 (e.g., a system-on-a-chip) that receives enhancements transmitted via HDMI and received by an HDMI receiver 604. The AR projector 600 includes multiple light emitters 606 such as light-emitting diodes (LEDs) and lenses 608. Through various electronic devices (e.g., transistors, resistors, etc.), in... Figure 6(Referred to as a discrete device in the text), the processor 602 can control which light emitters are activated, the intensity of the light output by the light emitters, etc. The AR projector 600 can be powered by the vehicle's power supply (e.g., a vehicle battery). About Figure 6 The described AR projector can be mounted behind and / or on top of the steering wheel and positioned to use the windshield as the projection plane, as per [reference to...]. Figure 3 The enhanced image received from the vehicle controller / computing system is then projected onto the windshield.

[0063] Figure 6 The projector shown may include a System-on-a-Chip (SoC) with the ability to convert incoming RGB signals from digital image frames (received at an HDMI interface) into a beam of light, which is then focused onto a plane to display a color image (e.g., image enhancement). Colors in the digital image are stored and processed using only three colors: red, green, and blue, typically written as RGB colors for this projection unit. Millions of colors can be produced on the projection plane when the three primary RGB colors are combined or added in different proportions.

[0064] Figure 7 An example AR view 700, generated using the AR system described herein, is shown. AR view 700 includes a vehicle's windshield 702 and steering wheel 704. An AR projector 706 is mounted above and / or behind the steering wheel 704. The AR projector 706 projects an enhancement 707 onto the windshield 702. The enhancement 707 includes the route the vehicle is currently following (e.g., from point A to point B). The enhancement 707 also includes indicators of weather conditions along the route. For example, a first area 708 of the route includes sunny weather conditions, a second area 710 includes potentially problematic weather conditions (e.g., snow, heavy rain, fog, ice), and a third area 712 includes sunny weather conditions. By highlighting weather conditions throughout the route, potential driving problems can be communicated to the driver and / or any passengers before they occur, providing an opportunity to reschedule or otherwise prepare for upcoming weather conditions. For example, autonomous driving can be deactivated when weather conditions include heavy rain or fog, and the driver can be notified so they can prepare to regain full control of the vehicle. As the weather changes and the vehicle continues along the route, areas with different weather conditions can be updated in real time. Furthermore, by graphically displaying real-time weather conditions along the entire route via Enhanced 707, drivers can quickly assess the weather and driving conditions without having to navigate through multiple pages of information (e.g., if a driver were looking up weather conditions on a smartphone or other device), or zoom in / out or manually navigate to a map displaying weather conditions. This could improve driver safety and enhance the driver's experience with the vehicle systems. Figure 7In the example shown, reinforcement 707 spans most of the extension of windshield 702, but reinforcement 707 can actually be displayed at any part of the windshield without departing from the scope of this disclosure.

[0065] Now go to Figure 8 The diagram illustrates a flowchart of a method 800 for displaying information in a vehicle. For example, it can be used... Figure 4 Method 800 may be executed by components of the vehicle computing system 109 and / or AR system 400. Method 800 may be executed according to instructions stored in the non-transitory memory of the vehicle computing device (such as the vehicle computing system 109 or IVI controller 404).

[0066] At point 802, determine the current operating parameters. These parameters may include current vehicle operating parameters such as vehicle speed, vehicle assistance mode (e.g., autonomous, semi-autonomous, or fully driver-controlled), current in-vehicle infotainment settings, current route, etc. This can be based at least in part on the above information regarding... Figure 2 and Figure 4 The described sensors and / or communication modules determine current operating parameters. At 804, method 800 determines whether the operator of the in-vehicle infotainment system (e.g., the driver or passenger of the vehicle) has requested an augmented reality view (AR view) to be presented via the vehicle's windshield. For example, the operator may enter input via the vehicle's user interface (e.g., user interface 218) to request the display of at least some requested information via augmentation on the windshield.

[0067] If the operator has not yet requested an AR view on the windshield (e.g., if the operator has not typed a request for an AR view, or if the operator requests to view additional information only via the infotainment system's device display), method 800 proceeds to 806 to display any requested information on the device display, which may include a head-up unit display (e.g., touchscreen 108 or display screen 111) and / or a display for the instrument cluster (e.g., instrument cluster 110). The additional information displayed may include the same information that would be displayed if an AR view were selected (described in more detail below) and / or additional information such as vehicle climate control information, vehicle audio system information, etc. Method 800 then returns.

[0068] If the operator has requested an AR view on the windshield, method 800 proceeds to 808 to project an augmented image onto the windshield upon instruction, which will be discussed below. Figure 9Description. In short, the AR projector can receive enhancements to be displayed on the windshield from the onboard computing system based on operator input, data collected by vehicle sensors and / or communication modules, etc. The enhancements may include information related to various aspects of the current vehicle route, such as route-specific emergency notifications, communication coverage, points of interest, etc. Furthermore, in some examples, the displayed enhancements and / or their position on the windshield may be adjusted based on whether the vehicle is operating in autonomous mode. Method 800 then returns.

[0069] Figure 9 This is a flowchart illustrating a method 900 for displaying enhancements on the windshield of a vehicle. For example, it can be used... Figure 4 Method 900 may be executed by components of the in-vehicle computing system 109 and / or the AR system 400. Method 900 may be executed according to instructions stored in the non-transitory memory of the in-vehicle computing device (such as the in-vehicle computing system 109 or the IVI controller 404). In some examples, method 900 may be executed as part of method 800 (e.g., in response to an operator's request for an AR view on the windshield).

[0070] At 902, method 900 determines one or more selected enhancement categories to be displayed. For example, an operator of the infotainment system can type in an input specifying one or more information categories to view via the displayed enhancements. The categories may include vehicle operation information (e.g., vehicle speed), navigation information, detected obstacles, and route-specific information. Route-specific information may include emergency notifications along the route, as shown at 904. Emergency notifications along the route may be displayed to inform the driver and passengers of dangerous road conditions, route detours, and emergency notifications. When an emergency notification is selected as an enhancement category, information may be displayed regarding obstacles, conditions such as ongoing roadworks, weather notifications in rain, floods, or torrential rain, and the distance to the destination along the given route.

[0071] Route-specific information may also include communication disruptions along the route, as illustrated at 906. As previously explained, vehicles may include communication modules that communicate with external devices (e.g., other vehicles, roadside infrastructure) to support various vehicle and / or infotainment features. However, when a vehicle travels beyond the range of any basic unit (also referred to herein as a roadside unit (RSU)), connectivity (e.g., 5G coverage) and data / connectivity loss may occur. Predicted coverage loss along the route can be identified and displayed as an enhancement to provide information about such instances of connectivity loss, strength, bandwidth-related network characteristics, and so on.

[0072] Route-specific information can include points of interest (POIs) along the route, as shown at position 908. POIs can include gas stations or charging stations, landmarks, tourist destinations, restaurants, distances from the destination along the route, and so on. When a POI is selected as an enhancement category, enhancements indicating the identified POIs along the route can be displayed.

[0073] Although the various enhancement categories are described here as being selected by the user, it should be understood that in some examples, categories can be selected automatically, all categories can be selected (and different enhancements can be displayed in a loop), more than one category can be selected at any given time, etc.

[0074] At point 910, relevant vehicle sensor / communication data and the current route are obtained. For example, a user can enter an input specifying a destination location, and a dynamic route can be calculated based on the source (e.g., current location) and destination location coordinates. Based on one or more augmentation categories that will be displayed, the obtained relevant vehicle sensor / communication data may include vehicle data, weather information along the route from a weather service, image information from one or more cameras (e.g., this can be used to determine if there are any obstacles within the vehicle's range), such as WiFi and / or cellular network coverage along the route determined based on WiFi and / or cellular communication modules / predicted outages, etc.

[0075] At point 912, one or more enhancements are selected based on one or more selected categories and the obtained sensor / communication data. For example, enhancements could include a determined dynamic route presented as a map, and these enhancements could be modified to display weather conditions along the route (e.g., weather conditions along the route). Figure 7 (as shown), network interruptions along the route, points of interest along the route, etc. One or more selected enhancements may also include enhancements representing vehicle operation information (such as current vehicle speed).

[0076] At 914, method 900 includes determining whether the vehicle is operating in autonomous mode, in which the vehicle operates without driver input. Autonomous mode operation can be performed in response to a user request for autonomous mode operation (e.g., based on user input to a user interface such as a touchscreen of an infotainment system, a steering wheel, or buttons on a dashboard).

[0077] If the vehicle is operating in autonomous mode, method 900 proceeds to 916 to instruct the AR projector to project selected augmentations onto one or more indicated locations on the windshield. Due to operation in autonomous mode, the entire displayable area of ​​the windshield can be used as the display area for projecting the augmentations, as driver distraction or confusion of the driver's field of view (FOV) is no longer an issue during autonomous operation. (In some examples, the entire windshield may be displayable, meaning augmentations can be projected onto any portion of the windshield. In other examples, only one or more portions of the windshield may be displayable, meaning augmentations can be projected onto only certain areas of the windshield instead of the entire windshield.) Therefore, each augmentation can be displayed at a corresponding location on the windshield, i.e., based on user preference, the type of augmentation being displayed (e.g., a route map including weather conditions may be displayed in a first display location (such as in front of the passenger seat), while vehicle operating parameters may be displayed in a second display location (such as above the instrument cluster)), current vehicle orientation, current driver FOV, and / or combinations thereof.

[0078] For example, enhancements can be displayed in a screen-locked manner, where the enhancement is displayed at specified display coordinates that remain unchanged even when the vehicle orientation changes, the driver's field of view (FOV) changes, etc. In some examples, enhancements can be displayed in a world-locked manner, where the enhancement is displayed in the same location relative to a real-world object or location, such as a pedestrian warning enhancement displayed near a detected pedestrian. In such configurations, world-locked enhancements may move with the movement of real-world objects, and / or the enhancement's display coordinates may change with changes in vehicle orientation to maintain a fixed relationship between the enhancement and the real-world object. In further examples, enhancements can be displayed in a body-locked manner, where the enhancement is displayed at display coordinates that remain in a fixed position within the driver's or passenger's FOV, even when the FOV changes. In some examples, enhancements can be displayed in a combination of screen-locked, world-locked, and / or body-locked configurations, such as displaying a first enhancement (e.g., a pedestrian warning enhancement) at a world-locked location and a second enhancement (e.g., an enhancement displaying vehicle operating parameters) at a screen-locked location.

[0079] When augmentations are displayed on the windshield during autonomous operation, some similar images can be shown on other in-vehicle displays, such as vehicle operating parameters displayed on both the windshield and the instrument cluster displays. However, during autonomous operation, some augmentations can only be displayed on the windshield and not on any other in-vehicle displays, such as maps showing weather conditions, points of interest, etc.

[0080] At 918, method 900 determines whether a request to stop the AR view has been received. This request can be received via user input (e.g., a user can request that the enhancements no longer be displayed on the windshield). In some examples, the request to stop the AR view can be output by the onboard computing system, such as in response to detecting that the vehicle is no longer operating in autonomous mode or in response to detecting an adverse condition indicating the driver's attention to the road. If no request to stop the AR view has been received, method 900 returns to 902 to continue evaluating the requested enhancements and real-time sensor / communication data to continue selecting and displaying enhancements based on the requested enhancements and sensor / communication data.

[0081] If a request to stop the AR view is received, method 900 proceeds to 920 to deactivate the AR projector or otherwise stop the display of enhancements on the windshield, and in some examples, to display the enhancements on one or more in-vehicle displays. In this way, the enhancements currently displayed on the windshield can instead be displayed on the in-vehicle displays. For example, alternatively, a map of the weather conditions along the route previously displayed on the windshield can be displayed on an infotainment system display (e.g., display screen 111). Doing so provides a seamless display of the desired information by allowing information to be displayed across multiple display areas (including the windshield if desired). Method 900 then returns.

[0082] Returning to 914, if it is determined that the vehicle is not operating in autonomous mode—for example, if the vehicle is fully operated by the driver (e.g., without cruise control, without steering assist, etc.) or partially operated by the driver (e.g., adaptive cruise control is enabled, but the driver still has full control of steering)—then method 900 proceeds to 922 to instruct the AR projector to project selected augmentations to one or more locations outside the driver's FOV. For example, the augmentation might be displayed at the bottom or top corner of the windshield (e.g., on the passenger side), rather than across the middle of the windshield. Therefore, when operating in autonomous mode, in some examples, the augmentation may be displayed within the driver's FOV. Conversely, when operating in semi-autonomous or non-autonomous mode, the augmentation may not be displayed within the driver's FOV (assuming a standard FOV for driving, where the driver's FOV includes a fixed area of ​​the windshield in front of the driver). In some examples, when the driver is looking directly at the road, the augmentation may be displayed in a fixed area of ​​the windshield, which is assumed to be outside the driver's FOV. In other examples, the enhancements can be displayed based on the actual driver's FOV, such that the display coordinates of the enhancements can be changed based on vehicle orientation (e.g., this could indicate that the vehicle is turning or going through a curve, and therefore the driver's FOV may have moved) or the direction of the driver's line of sight.

[0083] At 924, method 900 optionally includes: adjusting which enhancements to display (as opposed to which enhancements to display if the vehicle were operating in autonomous mode) in response to determining that the vehicle is not operating in autonomous mode. To reduce driver distraction, it may be beneficial to limit the enhancements displayed to the driver while controlling the vehicle. Therefore, the size of the enhancements can be adjusted and / or fewer enhancements can be displayed. For example, vehicle operating parameter enhancements can be kept on the windshield (so that the driver can more easily assess vehicle operating parameters than by looking down at the instrument cluster), while various maps discussed herein can be removed (and displayed on the in-vehicle display in some examples). As explained above, method 900 then proceeds to 918.

[0084] Therefore, method 900 provides the use of an AR projector to display one or more enhancements on the windshield during autonomous operation mode or during semi-autonomous or non-autonomous operation mode. The displayed enhancements may include route-specific notifications such as weather conditions, emergency notifications, cellular or Wi-Fi data coverage, points of interest, etc. To generate route-specific enhancements, the user can enter their destination location into the onboard computing system to enable information about emergency notifications, points of interest, communication coverage, etc., along the route. Based on the source and destination location coordinates, a dynamic route is calculated and a route image is generated.

[0085] In the example, points of interest (POIs) can be calculated from the coordinates of all locations within the route coverage area using a map / navigation database (described below). If a POI falls within the coordinate range between the source and destination locations, the navigation route map is enhanced with the POI's marker and distance from the current route. The final enhanced image is sent to the vehicle's infotainment system for AR projection and display on the windshield. A similar approach can be taken to enhance the generated route image with weather conditions (e.g., by contacting weather services and enhancing the image with visual cues such as colors representing different weather conditions), accidents along the route, road construction along the route, and so on.

[0086] In another example, once the image of the route is calculated, QoS parameters are calculated for all location coordinates within the route's coverage area. If the QoS parameters are within acceptable limits, the generated image of the navigation route map is enhanced and represented using a primary visual representation (such as green). If the QoS parameters are outside acceptable limits, the generated image is enhanced and represented using a different visual representation (such as red) to indicate signal outages and poor coverage in these areas. The final enhanced image is sent to the vehicle's infotainment system for AR projection and display on the windshield. Additional details regarding the acquisition of QoS parameters are provided below.

[0087] Therefore, the above method can be deployed to display enhanced images of the following: obstacles, conditions such as ongoing road construction, weather notifications in the event of rain, floods, torrents, etc., debris, and distances to the destination along a given route.

[0088] Figure 10 This is a schematic block diagram of system 1000 used to generate enhancements that show points of interest along the route. System 1000 includes some of the same components as system 500, and these components are numbered the same and will not be repeated.

[0089] System 1000 includes a navigation and data analysis module 1002 that receives output / signals from a vehicle communication module (e.g., WiFi, cellular data, GPS). The navigation and data analysis module 1002 can determine source and destination location coordinates and / or determine a dynamic route based on these coordinates. An image of the route map can be retrieved from a navigation database 1004. Points of interest (POIs) are calculated for all location coordinates along the route from the navigation database 1004. The navigation database 1004 can also store enhancements of POIs along the route (and other routes) that can also be retrieved. The image of the route map can be enhanced with POIs along the route at a POI enhancement module 1010 (which may be part of or a non-limiting example of an image enhancement module 508). For example, enhancements to fuel pumps can indicate gas stations along the route, and enhancements to silverware and / or plates can indicate restaurants along the route. In some examples, an image 1006 of the vehicle's surrounding environment captured by a vehicle camera can be provided to the POI enhancement module 1010. The final augmented reality image 1012 (e.g., a route map, including enhancements to some or all points of interest along the route) can be sent to an AR projector for display.

[0090] Images captured from vehicle cameras can be used in the backend to apply traditional and evolving data augmentation techniques to create new training data from existing training data implemented using modern deep learning algorithms like convolutional neural networks (CNNs). Image data augmentation is a type of data augmentation that involves creating transformed versions of images (input images) from the training dataset that belong to the same class as the original images, where the original images refer to a predefined set of images such as pedestrian crossings, road signs, tourist destination images, and weather images.

[0091] Augmented reality (AR) technology allows users to view the real world and virtual objects together by overlaying virtual objects onto the real world. AR algorithms fall into two categories: label-based AR methods that use human-labeled data, and label-free AR methods that use natural features instead of human-labeled data. Image processing techniques such as adaptive thresholding, contour detection, and edge detection, along with AR algorithms, can be used to combine real-world images (captured images) and virtual objects.

[0092] Figures 11A to 11C The diagram illustrates how data coverage along a route (e.g., cellular data) can be determined in order to generate an enhancement showing the data coverage along the route, as described above. Figure 11A Example 1100 of a vehicle connected to a Roadside Unit (RSU) is shown. In a 5G network environment, base stations such as femtocells, picocells, microcells, and macrocells are used for signal coverage. Base stations used for road coverage are called Roadside Units (RSUs).

[0093] Base station type Coverage area (km) Bandwidth (MHz) Micrometer Community 0.01 to 0.1 10 Weiwei Community 0.1 to 0.2 20 Micro-community / Metropolitan Area Community 1 to 2 20 to 40 Hongxiaoqu 5 to 32 60 to 75

[0094] Table 1.

[0095] Table 1 shows the base station types based on operator deployment, with typical RSU coverage ranging from 0.01Kms to 32Kms.

[0096] Figure 11B Example scenario 1120 is shown, illustrating how vehicle 1122 can move from a first RSU coverage area 1124 to a second RSU coverage area 1126, with a coverage gap between them. Figure 11B As shown, vehicle 1122 begins within the first RSU coverage area 1124, which represents an area with sufficient signal coverage from the first RSU (RSU-A). As vehicle 1122 moves along the road in the indicated direction of travel, vehicle 1122 eventually leaves the first RSU coverage area 1124. When vehicle 1122 leaves the first RSU coverage area 1124, vehicle 1122 is outside the range of any RSU and therefore lacks 5G network coverage. Finally, vehicle 1122 enters the second RSU coverage area 1124, which represents an area with sufficient signal coverage from the second RSU (RSU-B).

[0097] Example scenario 1120 highlights poor QoS when the vehicle travels beyond the range of the installed RSU, and 5G coverage and data / connectivity loss issues when the connection with the RSU is lost. The above-mentioned deployments can be made... Figure 9The AR projection method discussed is used to provide vehicle operators / passengers with information about such instances of QoS loss, strength, bandwidth-related network characteristics, etc.

[0098] In the background, the vehicle's infotainment system (e.g., in-vehicle computing system) continuously sends real-time 5G QoS parameters to the data acquisition and enhancement module in a proprietary VINQ data format. QoS parameters related to the 5G network can include latency, peak data rate, available spectrum, and connection density, and are packetized and transmitted to the analysis module for storage and processing. In a 5G network scenario, this method can provide information about QoS parameters along a given route to a destination.

[0099] Figure 11C An example VINQ packet 1140 is shown. A VINQ packet may include a vehicle ID (for uniquely identifying the vehicle), a packet ID (for identifying the current packet), current vehicle coordinates (longitude and latitude), and 5G QoS parameters (such as latency, peak data rate, available spectrum, etc.). The VINQ packet is sent to a data acquisition and enhancement module (e.g., data capture module 504 and image enhancement module 508). Once a VINQ packet is received, it is decoded and 5G QoS data for the current location coordinates is stored. The navigation analysis database can then be updated.

[0100] As an example, return to the reference. Figure 11B When vehicle 1122 is in the first RSU coverage area 1124, the vehicle communication module (e.g., Figure 5 and Figure 10 The 5G module shown can communicate with the RSU-A. As described above, based on this communication, 5G QoS parameters can be determined and sent in the VINQ packet. However, when the vehicle 1122 moves outside the coverage area 1124 of the first RSU, the VINQ packet can indicate weak signal or no signal.

[0101] However, the systems and methods described herein can allow for advance perception of impending loss of communication data coverage, such as perception of the gap between the first RSU coverage area 1124 and the second RSU coverage area 1126.

[0102] To identify areas along a route with weak or no data coverage, the vehicle may include a navigation transceiver configured to receive wireless data including wireless black spot data (e.g., the location / boundaries of the black spot, wireless network coverage parameters within the black spot, etc.) from a remote service, such as a cloud storage device located remotely from the navigation system and / or from a device including the navigation system (e.g., a smartphone, handheld device, in-vehicle computing system, etc.). In some examples, the transceiver may receive route and / or black spot data (e.g., areas without cellular data coverage) information from a navigation data server. After receiving the wireless data, the transceiver may send the received data to a processor for processing and storage.

[0103] The processor can receive the vehicle's current location from a GPS system, such as wireless black dot data information from a transceiver and / or from onboard memory, receive destination input from a user, and generate route data, which is then stored in a data memory. Thus, the data may include black dot data. Furthermore, for example, the data may include a map of the generated route. In some examples, the route may be displayed as an enhanced image on the windshield. In the absence of cellular coverage, the processor can retrieve the map from a map data memory. The processor can also cache the processed map along with additional data related to the black dots in the data. In some examples, vehicle-to-vehicle communication or other configurations can be used to receive information from other vehicles about upcoming data coverage black dots.

[0104] A key advantage of this approach is that all services relying on this connectivity can plan and cache data in advance within a specific distance, thereby improving QoS. For example, a navigation application, knowing that GPS is lost within a certain distance (the area shown in red on the route map, as illustrated above), can cache navigation data before the journey begins. In another example, if any 5G service application, such as a media streaming app, is aware of signal loss, the system can pre-buffer media content for the duration of that loss and provide a seamless user experience while watching the content.

[0105] The technological advantage of displaying enhancements on the windshield is that it can inform vehicle occupants of upcoming road conditions, points of interest, communication interruptions, etc., which can improve vehicle safety and / or the occupant's riding experience. The technological advantage of generating enhancements on an in-vehicle computing system is that enhancements can be displayed on the windshield during certain situations (e.g., autonomous driving) and then seamlessly switched to display on one or more in-vehicle displays (e.g., infotainment system displays) during other situations (e.g., non-autonomous driving), thereby continuously providing vehicle occupants with the desired information without confusing the driver's field of view or causing excessive distraction.

[0106] Descriptions of embodiments have been presented for illustrative and descriptive purposes. Appropriate modifications and variations to the embodiments may be performed based on the above description or obtained through practice of the methods described. The methods may be performed by executing stored instructions using one or more logical means (e.g., a processor) in conjunction with one or more additional hardware elements (such as storage devices, memories, image sensor / lens systems, light sensors, hardware network interfaces / antennas, switches, actuators, clock circuits, etc.). The described methods and associated actions may also be performed in various sequences other than those described in this application, in parallel, and / or simultaneously. Furthermore, the described methods may be repeated. The described systems are exemplary in nature and may include additional and / or omitted elements. The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of various systems and configurations, as well as other features, functions, and / or properties disclosed.

[0107] As used herein, elements or steps described in the singular and beginning with the word "a" or "an" should be understood to not exclude a plurality of said elements or steps unless such exclusion is stated. Furthermore, references to "an embodiment" or "an example" in this disclosure are not intended to exclude the existence of additional embodiments that also include said features. The terms "first," "second," and "third," etc., are used merely as labels and are not intended to impose numerical requirements or a particular order of position on their objects. The appended claims specifically point to subject matter from the foregoing disclosure that is considered novel and non-obvious.

Claims

1. A method for presenting information in a vehicle, comprising: Determine the route from the source location to the destination location; Obtain one or more features of the route; The map of the route is enhanced using one or more of the features mentioned above to generate an enhanced map; During autonomous vehicle mode, the enhanced map is displayed at a first display position on the windshield and one or more operating parameters are displayed at a second display position on the windshield of the vehicle; as well as In response to switching to non-autonomous vehicle mode, the one or more operating parameters are maintained on display in an area outside the driver's field of view on the windshield, and the enhanced map is switched to be displayed on the vehicle display.

2. The method of claim 1, wherein one or more features include weather conditions along the route, and wherein enhancing the map includes: One or more visual indicators of the determined weather conditions along the route are superimposed at the location corresponding to each determined weather condition.

3. The method of claim 1, wherein one or more features include data communication service quality parameters along the route, and wherein enhancing the map includes: One or more visual indicators of the determined data communication quality of service parameters along the route are superimposed at the location corresponding to each determined data communication quality of service parameter.

4. The method of claim 1, wherein one or more features include points of interest along the route, and wherein enhancing the map includes: One or more visual indicators of the identified points of interest along the route are superimposed at the location corresponding to each identified point of interest.

5. The method of claim 1, wherein during the autonomous vehicle mode the vehicle operates without direct input from the driver.

6. A system comprising: Augmented reality projector; In-vehicle computing systems, and A vehicle display, operatively coupled to the in-vehicle computing system, the in-vehicle computing system storing instructions capable of executing: Determine the route from the source location to the destination location; Obtain one or more features of the route; The map of the route is enhanced using one or more of the features mentioned above to generate an enhanced map; During autonomous vehicle mode, the augmented map is displayed on the windshield of the vehicle at a first display position via the augmented reality projector and one or more operating parameters are displayed at a second display position; In response to switching to non-autonomous mode, the display of one or more operating parameters is maintained in an area outside the driver's field of view on the windshield, the display of the enhanced map is removed from the first display position, and the enhanced map is displayed on the vehicle display.

7. The system of claim 6, wherein the enhanced map is displayed on the windshield during autonomous vehicle mode operation of the vehicle without direct input from the driver.

8. A system for a vehicle, comprising: Augmented reality projector; Vehicle display screen; as well as An in-vehicle computing system, operatively coupled to the augmented reality projector and the vehicle display, the in-vehicle computing system storing instructions capable of executing: During autonomous vehicle mode, an augmented map is displayed on the windshield of the vehicle via the augmented reality projector. The augmented map includes one or more features that determine the route the vehicle is to travel. The augmented reality projector is also configured to display the augmented map at a first display position on the windshield and to display one or more vehicle operating parameters at a second display position on the windshield. In response to the switch from the autonomous vehicle mode to the non-autonomous vehicle mode, the augmented map is displayed on the vehicle display instead of via the augmented reality projector, and the display of the one or more vehicle operating parameters is maintained in an area outside the driver's field of view on the windshield.

9. The system of claim 8, wherein the one or more features include weather conditions along the route, and wherein the enhanced map includes one or more visual indications of the determined weather conditions along the route at locations corresponding to each determined weather condition.

10. The system of claim 8, wherein the one or more features include data communication quality of service parameters along the route, and wherein the enhanced map includes one or more visual indications of the determined data communication quality of service parameters along the route at locations corresponding to each determined data communication quality of service parameter.

11. The system of claim 8, wherein the one or more features include points of interest along the route, and wherein the enhanced map includes one or more visual indications of the determined points of interest along the route at locations corresponding to each determined point of interest.

Images