Control device and method for controlling a display unit arranged in a vehicle

Abstract

Various aspects relate to a control device (101) and a method (1100) for controlling a display unit (304) arranged in a vehicle (100), the control device comprising a processor (102) which is configured to: receive first data (402) representing an environment of a vehicle (100); determine a travel trajectory (404) of the vehicle (100) using the first data; receive second sensor data (406) representing a head movement of an occupant (302) of the vehicle; determine a time-dependent relative movement (410) between the head movement of the occupant and a vehicle movement according to the travel trajectory using a speed (408) of the vehicle;to determine a time-dependent change in the position of display information to be displayed on a display unit arranged in the vehicle such that the change in the position of the display information fully or partially compensates for the relative movement; and to generate control instructions (414) for controlling the display unit to display the display information in accordance with the time-dependent change in the position.

Description

Technical field

[0001] Various aspects of this disclosure generally relate to a control apparatus and method for controlling a display unit disposed in a vehicle. background

[0002] When an occupant (e.g., a driver, a front passenger, a passenger, a passenger) moves in a vehicle, they are subject to vertical, longitudinal, lateral, and rotational movements as a result of the vehicle's motion. For various applications, it can be advantageous to display information, such as text and / or a video, on a display unit that is world-fixed to the occupant. Clearly, a world-fixed display of information for the occupant can be independent of vehicle motion. Brief description of the revelation

[0003] According to various embodiments, a control device and a method for controlling a display unit arranged in a vehicle are provided, which enable such a world-fixed display of display information for a vehicle occupant. According to various aspects, a position of the display information on the display unit is temporally changed in such a way that a relative movement between a movement of the vehicle and a head movement of the occupant is reduced, minimized, or even compensated. Short description of the characters

[0004] In the drawings, reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments of the invention are described with reference to the following drawings, in which: Fig. 1 shows a vehicle according to different aspects. Fig. 2 shows various electronic components of the vehicle. Fig. Figure 3 shows various electronic components of the vehicle and an illustration of an occupant. Fig. 4 shows a flowchart for controlling a display unit arranged in a vehicle according to various aspects. Fig. 5 and Fig. 6 each show an exemplary representation of text on a display unit according to various aspects. Fig. 7 shows an exemplary representation of display information in a section of the display unit according to various aspects. Fig. 8A to Fig. 9B show various aspects for displaying display information. Fig. 10 shows a flowchart for controlling a display unit arranged in a vehicle according to various aspects. Fig. 11 shows a flowchart of a method for controlling a display unit arranged in a vehicle. Detailed description

[0005] The following detailed description refers to the accompanying drawings which, by way of illustration, show specific details and embodiments in which the invention may be practiced.

[0006] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be considered preferred or advantageous over other embodiments.

[0007] The terms "at least one" and "one or more" may be understood to mean a numerical quantity greater than or equal to one (e.g., one, two, three, four, [...], etc.). The term "a plurality" may be understood to mean a numerical quantity greater than or equal to two (e.g., two, three, four, five, [...], etc.).

[0008] The terms "multiple" and "plurality" expressly refer to a set greater than one. Accordingly, all expressions that expressly refer to the foregoing words (e.g., a plurality of elements, multiple elements) refer to a set of elements, expressly referring to more than one of those elements. The terms "group (of)," "set (of)," "collection (of)," "series (of)," "sequence (of)," "grouping (of)," etc., and similar expressions in the description and claims, refer to a set equal to or greater than one, i.e., one or more.

[0009] The phrase "at least one of" with respect to a group of elements may be used herein to mean at least one element from the group comprising the elements. For example, the phrase "at least one of" with respect to a group of elements may be used herein to mean a selection of: one of the listed elements, a plurality of one of the listed elements, a plurality of individual listed elements, or a plurality of a multiple of individual listed elements.

[0010] The term "data" as used herein can be understood to include information in any suitable analog or digital form, e.g., in the form of a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. In addition, the term "data" can also be used to refer to information, e.g., in the form of a pointer. However, the term "data" is not limited to the aforementioned examples and can take various forms and represent any information as understood in the art.

[0011] The term "processor," as used herein, can be understood as any type of entity that allows the processing of data or signals. The data or signals can, for example, be processed according to at least one (i.e., one or more than one) specific function performed by the processor. A processor can include or be formed from an analog circuit, a digital circuit, a mixed-signal circuit, a logic circuit, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA), an integrated circuit, or any combination thereof. Any other type of implementation of the respective functions, which are described in more detail below, can also be understood as a processor or logic circuit.It is understood that one or more of the method steps described in detail herein may be executed (e.g., realized) by a processor through one or more specific functions performed by the processor. The processor may therefore be configured to perform one of the methods described herein or its components for information processing.

[0012] The distinctions between software- and hardware-implemented computing can be blurred. A processor, security system, computing system, and / or other aspects described herein may be implemented in software, hardware, and / or as a hybrid implementation with both software and hardware.

[0013] The term "memory" is understood herein to mean a computer-readable medium in which data or information can be stored for retrieval. Memory used in the embodiments may be volatile memory, for example, DRAM (Dynamic Random Access Memory), or non-volatile memory, for example, PROM (Programmable Read-Only Memory), EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or flash memory, such as a floating-gate memory device, a charge-trapping memory device, MRAM (Magnetoresistive Random Access Memory), or PCRAM (Phase Change Random Access Memory). Memory may be flash memory, solid-state memory, magnetic tape, a hard disk drive, an optical drive, etc., or any combination thereof. Registers, shift registers, processor registers, data buffers, etc. are also included in the term "memory."The term “software” refers to all types of executable instructions, including firmware.

[0014] The term "system" (e.g., a computing system, an automated driving system, a safety system, etc.), as further discussed herein, may be understood as a set of interacting elements, where the elements may be, by way of example and not limitation, one or more mechanical components, one or more electrical components, one or more instructions (e.g., encoded in storage media), and / or one or more processors, and the like.

[0015] Unless explicitly stated, the term "transmit" includes both direct (point-to-point) and indirect transmission (via one or more intermediate points). Similarly, the term "receive" includes both direct and indirect reception. Furthermore, the terms "send," "receive," "communicate," and similar terms encompass both physical transmission (e.g., the transmission of radio signals) and logical transmission (e.g., the transmission of digital data over a logical software-level connection).For example, a processor or controller may send or receive data over a software-level connection with another processor or controller in the form of radio signals, with the physical transmission and reception being handled by radio-layer components such as RF transceivers and antennas, and the logical transmission and reception over the software-level connection being performed by the processors or controllers. The term "communicate" encompasses both sending and receiving, i.e., unidirectional or bidirectional communication in one or both directions, i.e., inbound and outbound.

[0016] The term “calculate” includes both “direct” calculations using a mathematical expression / formula / relationship and “indirect” calculations using lookup or hash tables and other array indexing or search operations.

[0017] A "vehicle" can be any type of powered or drivable object. For example, a vehicle can be a powered object with an internal combustion engine, a reaction engine, an electrically powered object, a hybrid powered object, or a combination thereof. A vehicle can be or include an automobile, a bus, a minibus, a van, a truck, a camper van, a vehicle trailer, a motorcycle, a bicycle, a tricycle, a train locomotive, a train car, a moving robot, a personnel carrier, a boat, a ship, a submersible, a submarine, a drone, an aircraft, a rocket, and the like.

[0018] According to various aspects, the vehicle may be a ground-moving vehicle. A "ground-moving vehicle" (also referred to as a ground vehicle) may be understood as any type of vehicle, as described above, that is configured to move or be driven on the ground, e.g., on a road, path, track, one or more rails, off-road, etc. An "air vehicle" may be understood as any type of vehicle, as described above, that is capable of maneuvering above the ground for any period of time, e.g., a drone. Similar to how a ground vehicle is equipped with wheels, belts, etc., to move over terrain, an "air vehicle" may have one or more propellers, wings, fans, etc., to enable maneuvering in the air. A "water vehicle" may be understood as any type of vehicle, as described above, that can maneuver on or below the surface of a liquid, e.g.,a boat on the water's surface or a submarine below the water's surface. It is clear that some vehicles may be configured to operate as ground, air, and / or water vehicles.

[0019] The term "at least partially automated vehicle" can describe a vehicle that is capable of making at least one navigation change without driver input. A navigation change can describe or include a change in the steering, braking, or acceleration / deceleration of the vehicle. A vehicle can be described as autonomous if it is fully automated (e.g., fully functional without driver input). Partially automated vehicles can include vehicles that can operate under driver control during certain periods and without driver control during other periods. At least partially automated vehicles can also include vehicles that control only some aspects of the vehicle's navigation, such as steering (e.g.,to maintain a vehicle's course between lane boundaries) or some steering operations under certain circumstances (but not all circumstances), but leave other aspects of vehicle navigation to the driver (e.g., braking or deceleration under certain circumstances). At least partially automated vehicles may also include vehicles that share control of one or more aspects of vehicle navigation under certain circumstances (e.g., "hands-on", i.e., in response to driver input), and vehicles that control one or more aspects of vehicle navigation under certain circumstances (e.g., "hands-off", i.e., independent of driver input). At least partially automated vehicles may also include vehicles that control one or more aspects of vehicle navigation under certain circumstances, e.g., under certain environmental conditions (e.g., spatial areas, roadway conditions).In various aspects, at least partially automated vehicles may assume responsibility for some or all aspects of braking, speed control, cruise control, and / or steering of the vehicle. Autonomous vehicles may include those capable of driving without a driver. The level of automation of a vehicle may be described or determined by the Society of Automotive Engineers (SAE) Level of Automation (e.g., as defined by the SAE, e.g., in SAE J3016 2018: Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles) or by other relevant professional organizations. The SAE Level may have a value ranging from a minimum level, e.g., Level 0 (illustrative: substantially no driving automation), to a maximum level, e.g., Level 5 (illustrative: full driving automation).Therefore, an at least partially automated vehicle can be an automated vehicle or an autonomous vehicle. For example, the at least partially automated vehicle can be a partially automated vehicle (according to Level 2), a highly automated vehicle (according to Level 3), a fully automated vehicle (according to Level 4), or an autonomous vehicle (according to Level 5).

[0020] A model (e.g., a machine learning-based model (also referred to as a machine learning model)) may, for example, comprise or be a reinforcement learning model (e.g., using Q-learning, temporal difference (TD), deep adversarial networks, etc.) and / or a classification model (e.g., a linear classifier (e.g., a logistic regression classifier or a naive Bayes classifier), a support vector machine, a decision tree, a boosted tree classifier, a random forest classifier, a neural network, or a nearest neighbor model). A neural network can be or comprise any type of neural network, such as a convolutional neural network (CNN), a variational autoencoder network (VAE), a sparse autoencoder network (AAN), or a multi-level neural network (MNE).sparse autoencoder network (SAE), a recurrent neural network (RNN), a deconvolutional neural network (DNN), a generative adversarial network (GAN), a forward-thinking neural network, a sum-product neural network, a transformer-based network, etc.

[0021] A "control device" as used herein can be understood as any type of entity (e.g., implementing logic) that allows the processing of data or signals. The control device can, for example, comprise one or more processors that can execute software stored in a storage device (in some aspects also referred to as a storage medium), in firmware, or in a combination thereof, and issue instructions based thereon. The control device can, for example, be configured by means of code segments (e.g., software) to control the operation of a system. For example, the data or signals can be handled according to at least one (i.e., one or more than one) specific function performed by the processor.

[0022] The term “image” as used herein may be any type of digital image data that can represent a pictorial representation, such as a digital RGB image, a digital RGB-D image, a binary image, a 3D image, a point cloud, a time series, a semantic segmentation image, etc.

[0023] The expression that an element, a parameter, etc. "represents" another element, another parameter, etc. can be understood to mean that these are linked to each other, e.g. the element and / or the parameter is a (e.g. unique, e.g. one-to-one) function of the other element and / or parameter.

[0024] Fig. 1 shows a vehicle 100 with a mobility system 120 and a control system 200 (see also Fig. 2) according to various aspects. It is understood that the vehicle 100 and the control system 200 are exemplary and may therefore be simplified for explanatory purposes. For example, while the vehicle 100 is illustrated as a ground vehicle, aspects of this disclosure may equally or analogously apply to aircraft, such as drones, or watercraft, such as boats. Furthermore, the quantities and positions of the elements, as well as the relative spacing (as described above, the illustrations are not to scale), are illustrated as examples and are not limited thereto. The components of the vehicle 100 may be arranged around a vehicle body of the vehicle 100, mounted on or outside the vehicle body, enclosed within the vehicle body, or in another arrangement relative to the vehicle body in which the components move with the moving vehicle 100.The vehicle body may be, for example, a car body, a drone body, an airplane or helicopter fuselage, a boat hull, or a similar type of vehicle body, depending on the type of vehicle 100.

[0025] The control system 200 may include a control device 101. The control device 101 may include one or more processors 102. Reference herein to processing by the control device 101 may be made to one or more of the one or more processors 102.

[0026] In addition to a control system 200, the vehicle 100 may also include a mobility system 120. The mobility system 120 may include components of the vehicle 100 related to the steering and movement of the vehicle 100. For example, in some cases where the vehicle 100 is an automobile, the mobility system 120 may include wheels and axles, a suspension, an engine, a transmission, brakes, a steering wheel, associated electrical circuits and wiring, and any other components used in powering an automobile. In some cases where the vehicle 100 is an aircraft, the mobility system 120 may include one or more rotors, propellers, jet engines, wings, rudder or flaps, air brakes, a yoke or cycle, associated electrical circuits and wiring, and any other components used in flying an aircraft.In some cases where the vehicle 100 is a watercraft or an underwater vehicle, the mobility system 120 may include one or more of the following elements: rudders, engines, propellers, a steering wheel, associated electrical circuits and wiring, and any other components used for steering or moving a watercraft. In various aspects, the mobility system 120 may also include autonomous driving functionality and, accordingly, interface with one or more processors 102 configured to perform calculations and decisions for autonomous driving, and a series of sensors for motion and obstacle detection. In this sense, the mobility system 120 may be provided with instructions for controlling the navigation and / or mobility of the vehicle 100 from one or more components of the control system 200.The at least partially automated driving components of the mobility system 120 may also be connected to one or more radio frequency (RF) transceivers 108 to facilitate coordination of mobility with other nearby vehicle communication devices and / or central network components that perform decisions and / or computations related to autonomous driving.

[0027] The control system 200 may include various components depending on the requirements of a particular implementation. As shown in Fig. 1 and Fig. 2, the control system 200 may include one or more processors 102, one or more storage devices (memory for short) 104, an antenna system 106, which may include one or more antenna arrays at various locations on the vehicle for radio frequency coverage, one or more radio frequency transceivers 108, one or more data acquisition devices 112, one or more positioning devices 114, which may include components and circuitry for receiving and determining a position based on a global navigation satellite system (GNSS) and / or a global positioning system (GPS), and one or more measurement sensors 116, e.g., speedometer, altimeter, gyroscope, speed sensors, etc.

[0028] The control system 200 may be configured to control the mobility of the vehicle 100 via the mobility system 120 and / or the interaction with its environment, e.g., communication with other devices or network infrastructure elements (NIEs) such as base stations, via the data acquisition devices 112 and the radio frequency communication arrangement comprising the one or more RF transceivers 108 and the antenna system 106.

[0029] The one or more processors 102 may include a data acquisition processor 214, an application processor 216, a communications processor 218, and / or any other suitable processing device. Each processor 214, 216, 218 of the one or more processors 102 may include various types of hardware-based processing devices. For example, each processor 214, 216, 218 may include a microprocessor, pre-processors (e.g., an image pre-processor), graphics processors, a central processing unit (CPU), support circuits, digital signal processors, integrated circuits, memory, or other types of devices suitable for executing applications and for image processing and analysis. In various aspects, each processor 214, 216, 218 may include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, etc.These processor types can each have multiple processing units with local memory and instruction sets. Such processors can have video inputs for receiving image data from multiple image sensors and also video output functions.

[0030] Each of the processors 214, 216, 218 may be configured to perform certain functions according to program instructions that may be stored in a memory of the one or more memories 104. In other words, a memory of the one or more memories 104 may store software that, when executed by a processor (e.g., by the one or more processors 102), controls the operation of the system, e.g., a driving and / or safety system. A memory of the one or more memories 104 may store one or more databases and image processing software, as well as a trained system, such as a neural network or a deep neural network. The one or more memories 104 may include any number of random access memories, read-only memories, flash memories, hard disk drives, optical storage, tape storage, removable storage, and other types of storage.Alternatively, each of the processors 214, 216, 218 may have internal memory for such storage.

[0031] The data acquisition processor 216 may include processing circuitry, such as a CPU, for processing the data acquired by the data acquisition devices 112. If one or more data acquisition devices are, for example, image acquisition units, e.g., one or more cameras, the data acquisition processor may include image processors for processing image data that use the information received from the image acquisition units as input. The data acquisition processor 216 may be configured to create voxel maps that depict the surroundings of the vehicle 100 in detail based on the data input from the data acquisition devices 112, e.g., from the cameras.

[0032] Various aspects relate to first sensor data representing an environment of the vehicle 100 (at least in the direction of travel). This first sensor data may include sensor data provided by one or more of the data acquisition devices 112.

[0033] The application processor 216 may be a CPU and may be configured to process the layers above the protocol stack, including the transport and application layers. The application processor 216 may be configured to execute various applications and / or programs of the vehicle 100 at an application layer of the vehicle 100, such as an operating system (OS), a user interface (UI) 206 for supporting user interaction with the vehicle 100, and / or various user applications. The application processor 216 may interface with the communications processor 218 and act as a source (in the transmit path) and sink (in the receive path) for user data such as voice data, audio / video / image data, message data, application data, basic internet / web access data, etc.In the transmit path, the communications processor 218 may therefore receive and process outgoing data provided by the application processor 216 according to the layer-specific functions of the protocol stack and provide the resulting data to the digital signal processor 208. The communications processor 218 may then process the received data at the physical layer to generate digital baseband samples, which the digital signal processor may forward to the RF transceiver(s) 108. The RF transceiver(s) 108 may then process the digital baseband samples to convert the digital baseband samples into analog RF signals, which the RF transceiver(s) 108 may transmit wirelessly via the antenna system 106.In the receive path, the RF transceiver(s) 108 may receive analog RF signals from the antenna system 106 and process the analog RF signals to obtain digital baseband samples. RF transceivers 108 may forward the digital baseband samples to the communications processor 218, which processes the digital baseband samples at the physical layer. The communications processor 218 may then forward the resulting data to other processors of the one or more processors 102, which may process the resulting data according to the layer-specific functions of the protocol stack and forward the resulting incoming data to the applications processor 216.The application processor 216 may then process the incoming data at the application layer, which may include executing one or more application programs with the data and / or presenting the data to a user via one or more user interfaces 206. The user interfaces 206 may include one or more screens, microphones, mice, touchpads, keyboards, or any other interface that provides a mechanism for user input.

[0034] The communication processor 218 may include a digital signal processor and / or a controller that can control communication functionality of the vehicle 100 according to the communication protocols associated with one or more radio access networks, and that can perform control of the antenna system 106 and the RF transceiver(s) 108 to transmit and receive radio signals according to the formatting and scheduling parameters defined by each communication protocol. Although various practical implementations may include separate communication components for each supported radio communication technology (e.g., a separate antenna, an RF transceiver, a digital signal processor, and a controller), the Fig. 1 and Fig. For reasons of clarity, the configuration of the vehicle 100 shown in Figure 2 represents only one component. The configuration of the vehicle 100 shown in Figures 1 and 2 represents only a single example of such components.

[0035] The vehicle 100 may transmit and receive wireless signals with the antenna system 106, which may be a single antenna or an antenna array with multiple antenna elements. In various aspects, the antenna system 202 may additionally include analog antenna combinations and / or beamforming circuits. In the receive path (RX), the RF transceiver(s) 108 may receive analog radio frequency signals from the antenna system 106 and perform analog and digital RF front-end processing of the analog radio frequency signals to generate digital baseband samples (e.g., in-phase / quadrature (IQ) samples) that are provided to the communications processor 218. RF transceivers 108 may include analog and digital receive components, including amplifiers (e.g., low-noise amplifiers (LNAs)), filters, RF demodulators (e.g.,RF I / O demodulators) and analog-to-digital converters (ADCs) that RF transceivers 108 may use to convert the received radio-frequency signals into digital baseband samples. In the transmit (TX) path, the RF transceiver(s) 108 may receive digital baseband samples from the communications processor 218 and perform analog and digital RF front-end processing on the digital baseband samples to generate analog radio-frequency signals that are provided to the antenna system 106 for wireless transmission. RF transceivers 108 may therefore include analog and digital transmission components, including amplifiers (e.g., power amplifiers (PAs), filters, RF modulators (e.g.,RF I / O modulators (RF-I / Q modulators) and digital-to-analog converters (DACs) that RF transceivers 108 may use to mix the digital baseband samples received by communications processor 218 and generate the analog radio frequency signals for wireless transmission by antenna system 106. In various aspects, communications processor 218 may control the radio transmission and reception of RF transceiver(s) 108, including determining the transmit and receive radio frequencies for operation of RF transceiver(s) 108.

[0036] According to various aspects, the communications processor 218 may include a baseband modem configured to perform physical layer (PHY, Layer 1) transmit and receive processing to prepare outgoing transmit data provided by the communications processor 218 in the transmit path for transmission via the RF transceiver(s) 108, and to prepare incoming received data provided by the RF transceiver(s) 108 in the receive path for processing by the communications processor 218. The baseband modem may include a digital signal processor and / or a controller.The digital signal processor may be configured to perform one or more of the following functions: error detection, forward error correction coding / decoding, channel coding and interleaving, channel modulation / demodulation, physical channel assignment, radio measurement and search, frequency and time synchronization, antenna diversity processing, power control and weighting, rate matching, forwarding processing, interference cancellation, and other physical layer processing functions. The digital signal processor may be structurally implemented as hardware components (e.g., as one or more digitally configured hardware circuits or FPGAs), as software-defined components (e.g., one or more processors configured to execute program code that defines arithmetic, control, and I / O instructions (e.g.,Software and / or firmware) stored in a non-transitory, computer-readable storage medium) or as a combination of hardware and software components. In various aspects, the digital signal processor may include one or more processors configured to retrieve and execute program code defining control and processing logic for physical layer processing operations. In various aspects, the digital signal processor may perform processing functions with software via the execution of executable instructions. In various aspects, the digital signal processor may include one or more dedicated hardware circuits (e.g.ASICs, FPGAs, and other hardware) digitally configured to perform specific processing functions, where the one or more processors of the digital signal processor may offload certain processing tasks to these dedicated hardware circuits, known as hardware accelerators. Example hardware accelerators may include fast Fourier transform (FFT) circuits and encoder / decoder circuits. In various aspects, the processor and hardware acceleration components of the digital signal processor may be implemented as a coupled integrated circuit.

[0037] The vehicle 100 may be configured for operation with one or more radio communication technologies. The digital signal processor of the communication processor 218 may be responsible for the lower layer (e.g., Layer 1 / PHY) processing functions of the radio communication technologies, while a controller of the communication processor 218 may be responsible for the upper layer (e.g., Data Link Layer / Layer 2 and / or Network Layer / Layer 3) protocol stack functions. The controller may thus be responsible for controlling the radio communication components of the vehicle 100 (antenna system 106, RF transceiver(s) 108, positioning device 114, etc.) in accordance with the communication protocols of each supported radio communication technology and may accordingly represent the Access Stratum and Non-Access Stratum (NAS) (which also includes Layer 2 and Layer 3) of each supported radio communication technology.The controller may be structurally implemented as a protocol processor configured to execute protocol stack software (retrieved from a memory of the controller) and subsequently control the radio communication components of the vehicle 100 to transmit and receive communication signals in accordance with the corresponding protocol stack control logic defined in the protocol stack software. The controller may include one or more processors configured to retrieve and execute program code defining upper-layer protocol stack logic for one or more radio communication technologies, which may include data link layer / Layer 2 and network layer / Layer 3 functions.The controller may be configured to perform both user-plane and control-plane functions to facilitate the transmission of application-layer data to and from the vehicle 100 according to the specific protocols of the supported radio communication technology. User-plane functions may include header compression and encapsulation, security, error checking and correction, channel multiplexing, scheduling, and priority, while control-plane functions may include radio bearer establishment and maintenance. The program code retrieved and executed by the controller of the communication processor 218 may include executable instructions defining the logic of such functions.

[0038] In various aspects, vehicle 100 may be configured to transmit and receive data according to multiple wireless communication technologies. Accordingly, in various aspects, one or more of antenna system(s) 106, RF transceiver(s) 108, and communication processor 218 may include separate components or instances dedicated to different wireless communication technologies and / or unified components shared by different wireless communication technologies. For example, in various aspects, multiple controllers of communication processor 218 may be configured to execute multiple protocol stacks, each dedicated to a different wireless communication technology and residing either on the same processor or on different processors.In various aspects, multiple digital signal processors of communication processor 218 may include separate processors and / or hardware accelerators, each associated with different radio communication technologies, and / or one or more processors and / or hardware accelerators shared by multiple radio communication technologies. In various aspects, RF transceiver(s) 108 may include separate RF circuitry portions associated with different respective radio communication technologies and / or RF circuitry portions shared by multiple radio communication technologies. In some cases, antenna system 106 may include separate antennas, each associated with different radio communication technologies, and / or antennas shared by multiple radio communication technologies.Accordingly, the antenna system 106, the RF transceiver(s) 108, and the communications processor 218 may include separate and / or shared components dedicated to multiple radio communication technologies.

[0039] The communications processor 218 may be configured to implement one or more vehicle-to-everything (V2X) communication protocols, which may include vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), vehicle-to-network (V2N), vehicle-to-pedestrian (V2P), vehicle-to-device (V2D), vehicle-to-grid (V2G), and other protocols. The communications processor 218 may be configured to transmit communications, such as communications (one-way or two-way) between the vehicle 100 and one or more other (target) vehicles in the vicinity of the vehicle 100 (e.g., to facilitate coordination of the navigation of the vehicle 100 with respect to or together with other (target) vehicles in the vicinity of the vehicle 100), or even a broadcast transmission to unspecified receivers in the vicinity of the transmitting vehicle 100.

[0040] The communication processor 218 may be configured to operate via a first RF transceiver of the one or more RF transceivers 108 in accordance with various desired wireless communication protocols or standards. For example, the communication processor 218 may be configured in accordance with a short-range cellular communication standard such as Bluetooth, Zigbee, and the like, and the first RF transceiver may conform to the corresponding short-range cellular communication standard. As another example, the communication processor 218 may be configured to operate via a second RF transceiver of the one or more RF transceivers 108 in accordance with a medium- or long-range cellular communication standard such as a 3G (e.g., Universal Mobile Telecommunications System - UMTS), a 4G (e.g.,Long Term Evolution (LTE) or an SG mobile communications standard in compliance with the corresponding 3GPP standards (3. rd Generation Partnership Project). As another example, the communications processor 218 may be configured to operate, via a third RF transceiver of the one or more RF transceivers 108, in accordance with a wireless local area network communications protocol or standard, such as IEEE 802.11 (e.g., 802.11, 802.11a, 802.11b, 802.11g, 802.11n, 802.11p, 802.11-12, 802.11ac, 802.11ad, 802.11ah, and the like). The RF transceiver(s) 108 may be configured to transmit signals via the antenna system 106 over an air interface. The RF transceivers 108 may each have a corresponding antenna element of the antenna system 106 or may share an antenna element of the antenna system 106.

[0041] The memory 214 may be a storage component of the vehicle 100, such as a hard drive or other permanent storage device. Although Fig. 1 and Fig. 2, the various other components of the vehicle 100, e.g., one or more processors 102, may additionally each have integrated permanent and non-permanent memory components, e.g., for storing software program code, buffering data, etc.

[0042] The antenna system 106 may include a single antenna or multiple antennas. In various aspects, each of the one or more antennas of the antenna system 106 may be placed at multiple locations on the vehicle 100 to ensure maximum RF coverage. The antennas may include a phased array, a switch beam array with multiple antenna elements, etc. The antenna system 106 may be configured to operate according to analog and / or digital beamforming schemes to maximize signal amplification and / or ensure a high level of information protection. The antenna system 106 may include separate antennas, each dedicated to different radio communication technologies, and / or antennas shared by multiple radio communication technologies. Although in Fig. 1 as a single element, the antenna system 106 may include a plurality of antenna elements (e.g., antenna arrays) positioned at various locations on the vehicle 100. The placement of the multiple antenna elements may be strategically chosen to provide a desired level of RF coverage. For example, additional antennas may be mounted on the front, rear, corners, and / or sides of the vehicle 100.

[0043] The data acquisition devices 112 may include any number of data acquisition devices and components, depending on the requirements of a particular application. These may include: image capture devices, proximity detectors, acoustic sensors, infrared sensors, piezoelectric sensors, etc., that provide data about the vehicle's surroundings. Image capture devices may be cameras (e.g., standard cameras, digital cameras, video cameras, SLR cameras, infrared cameras, stereo cameras, etc.), charge-coupled devices (CCDs), or any type of image sensor. Proximity detectors include radar sensors, LIDAR (light detection and ranging) sensors, mmWave radar sensors, etc. Acoustic sensors may include: microphones, sonar sensors, ultrasonic sensors, etc.Accordingly, each of the data acquisition devices may be configured to acquire a specific type of data from the environment of the vehicle 100 and forward the data to the data acquisition processor 214 to provide the vehicle with an accurate representation of the vehicle's environment. The data acquisition devices 112 may be configured to implement preprocessed sensor data, such as radar target lists or LIDAR target lists, in conjunction with the acquired data.

[0044] The measuring devices 116 may also include other devices for measuring vehicle state parameters, such as a speed sensor (e.g., a speedometer) for measuring the speed of the vehicle 100, one or more accelerometers (either single-axis or multi-axis) for measuring the accelerations of the vehicle 100 along one or more axes, a gyroscope for measuring orientation and / or angular velocity, odometer, altimeter, thermometer, etc. It is understood that the vehicle 100 may have different measuring devices 116 depending on the type of vehicle, e.g., car, drone, or boat.

[0045] Positioning devices 114 may include components for determining the position of the vehicle 100. These may include, for example, global positioning system (GPS) or other global navigation satellite system (GNSS) circuitry configured to receive signals from a satellite system and determine a position of the vehicle 100. Accordingly, the positioning devices 114 may provide the vehicle 100 with satellite navigation capabilities.

[0046] The one or more memories 104 may store data, e.g., in a database or other format, which may correspond to a map. The map may, for example, indicate the location of known landmarks, roads, trails, network infrastructure elements, or other elements of the environment of the vehicle 100. The one or more processors 102 may process sensory information (e.g., images, radar signals, depth information from LIDAR, or stereo processing of two or more images) of the environment of the vehicle 100 along with position information, e.g., a GPS coordinate, ego motion of the vehicle, etc., to determine a current position of the vehicle 100 relative to the known landmarks and to refine the determination of the vehicle position. Certain aspects of this technology may be incorporated into localization technology such as a mapping and routing model.

[0047] The map database (DB) 204 may include any type of database in which (digital) map data for the vehicle 100, e.g., for the control system 200, is stored. The map database 204 may include data relating to the position of various objects in a reference coordinate system, e.g., roads, water features, geographical features, businesses, landmarks, restaurants, gas stations, etc. Not only the positions of such objects may be stored in the map database 204, but also descriptors relating to these objects, e.g., names associated with one of the stored features. In various aspects, a processor of the one or more processors 102 may download information from the map database 204 via a wired or wireless data connection to a communications network (e.g., via a cellular network and / or the Internet, etc.).In some cases, map database 204 may store a sparse data model with polynomial representations of certain road features (e.g., lane markings) or target trajectories for vehicle 100. Map database 204 may also include stored representations of various detected landmarks that may be provided to determine or update a known position of vehicle 100 relative to a target trajectory. The landmark representations may include data fields such as the landmark type, the landmark location, and other possible identifiers.

[0048] In addition, the control system 200 may comprise a driving model, e.g., implemented in an advanced driver assistance system (ADAS) and / or a driver assistance and (at least partially) automated driving system. For example, the control system 200 may comprise (e.g., as part of the driving model) a computer implementation of a formal model such as a safety driving model. A safety driving model may be or include a mathematical model that formalizes an interpretation of the applicable laws, standards, guidelines, etc. applicable to self-driving vehicles. A safe driving model may be designed, e.g., to achieve three goals: First, the interpretation of the law should be sound in the sense that it is consistent with human interpretation of the law; second, the interpretation should lead to a sensible driving policy, i.e.,to an agile driving policy rather than an overly defensive driving style, which would inevitably confuse other human drivers and block traffic, thus limiting the scalability of the system's deployment; and third, the design should be efficiently verifiable, i.e., it should be possible to rigorously demonstrate that the self-driving (autonomous) vehicle correctly implements the interpretation of the law. A model for safe driving can, for example, be or include a mathematical model for ensuring safety that enables the detection and implementation of appropriate responses to dangerous situations, thus preventing self-inflicted accidents.

[0049] As described above, the vehicle 100 may include the control system 200, as also described with reference to Fig. 2. The vehicle 100 may include one or more processors 102 integrated with or separate from an engine control unit (ECU), which may be included in the mobility system 120 of the vehicle 100. The control system 200 may generally generate data to control or assist in controlling the ECU and / or other components of the vehicle 100 to directly or indirectly control the movement of the vehicle 100 via the mobility system 120. The one or more processors 102 of the vehicle 100 may be configured to implement the aspects and methods described herein.

[0050] The Fig. 1 and Fig. The components illustrated in Figure 2 may be interconnected via any suitable interfaces. Furthermore, not all connections between components are explicitly illustrated, and other interfaces between components may be present within the scope of this disclosure.

[0051] Various aspects herein relate to a display unit. In some embodiments, the display unit may be a display unit mounted in the vehicle 100. In other embodiments, the display unit may be a user device worn by an occupant of the vehicle 100. The user device may be, for example, a head-mounted display, a tablet, a smartphone, smart glasses, etc.

[0052] In some embodiments, vehicle 100 may be an at least partially automated vehicle (e.g., according to any of Levels 2 through 5). In this case, the occupant described herein may be the driver or a passenger of the vehicle. In other embodiments, vehicle 100 may be a non-automated vehicle. In this case, the occupant described herein may be a passenger (but not the driver).

[0053] Fig. 3 shows a section of the vehicle 100 with an occupant 302. In this example, the vehicle 100 may have a display unit 304 mounted in the vehicle 100. It is understood that what is described below as an example for the display unit 304 also applies accordingly to user devices worn by the occupant 302.

[0054] The vehicle 100 may include an imaging unit 306 (which may be any type of image capture device described herein). The imaging unit 306 may be configured to capture second sensor data representing a head movement and / or gaze direction of the occupant 302. Illustratively, the second sensor data may represent continuous image data (e.g., video data) showing a head movement and / or gaze direction of the occupant 302. If the display unit is a display unit worn on the head by the occupant, in some embodiments it may include an acceleration sensor configured to provide the second sensor data (as acceleration data). In this case, the imaging unit 306 is optional.

[0055] Fig. 4 shows a flowchart 400 for controlling the display unit 304 according to various aspects. It is understood that the control device 101 may be configured to implement these processes. The control device 101 may be configured to receive first data 402. This first data 402 may include the first sensor data 402 (e.g., from one or more data acquisition devices 112) and / or map data (e.g., based on terrain and other environmental information, e.g., road conditions, traffic, weather, etc.). For illustrative purposes, the first data 402 will be referred to below as first sensor data. It is understood that this is for illustrative purposes and that the first data may additionally or alternatively include other data, such as the map data (as long as the data represents the surroundings of the vehicle 100).The control device 101 may be configured to receive the second sensor data 406 from the imaging unit 306.

[0056] The control device 101 can be configured to determine (e.g., for a forecast period) a travel trajectory 404 of the vehicle 100 using the first (sensor) data 402. For this purpose, the control device 101 can, for example, implement a vehicle control model or be based on terrain and other environmental information mapped or acquired by sensors of the vehicle, e.g., road conditions, traffic, weather (also retrievable from other sources, e.g., the Internet). The travel trajectory can be a sequence of spatial-temporal states of the vehicle 100 in a collision-free space. In some aspects, the control device 101 can be configured to determine the travel trajectory using navigation data representing a route of the vehicle 100.

[0057] The control device 101 may be configured to receive sensor data regarding a speed 408 of the vehicle 100 (e.g., from one or more than one measuring sensor 116).

[0058] The control device 101 can be configured to determine or predict a time-dependent relative movement 410 between the head movement of the occupant 302 and the vehicle movement according to the travel trajectory 404 using the speed and / or acceleration 408 of the vehicle 100, the travel trajectory 404 and the head movement and / or viewing direction of the occupant 302 according to the second sensor data 406.

[0059] The relative movement 410 may comprise a longitudinal relative movement (in the direction of travel) and / or a lateral relative movement (transverse to the direction of travel) and / or a vertical relative movement, and / or a rotational movement.

[0060] The control device 101 can be configured to determine (e.g., for the prediction period) a time-dependent change 412 in a position and / or orientation of display information to be displayed on the display unit 304 such that the change 412 in the position and / or orientation of the display information at least partially (e.g., completely or partially) compensates for the (e.g., lateral and / or vertical) relative movement.

[0061] The control device 101 may be configured to generate control instructions 414 for controlling the display unit 304 to display the display information according to the time-dependent change 412 of the position and / or orientation.

[0062] In the following, various aspects are explained as to how a lateral and / or vertical relative movement 410 can be compensated by the time-dependent change 412 of the position and / or orientation of the display information, and / or by a controlled adjustment of a display unit (e.g. motor-controlled placement, or display unit on a controlled robot arm).

[0063] The display information described herein may be any type of information that can be displayed on a display device (e.g., a display). For example, the display information may include one or more of the following display information: text, one or more images, a video, a website, etc.

[0064] For example, the relative movement 410 can be the vertical time-dependent relative movement between the head movement and / or line of sight of the occupant 302 and the vehicle movement. In this case, the time-dependent change 412 in the position and / or orientation of the display information can be a vertical position change of the display information on the display. When driving on very bumpy roads, however, a permanent up-down shifting of the display information could occur. Optionally, the system can have a user interface by means of which the occupant can manually activate / deactivate the compensation function described herein. In some embodiments, the control device 101 can be configured to determine a rate of change with which the position is changed per unit of time and whether the rate of change is within a time window (e.g.of the prediction period) is greater than or equal to a predefined change rate threshold. If this is the case, the change in the position of the display information can be paused in this time window and continued as soon as the change rate is less than the predefined change rate threshold. Clearly, the control device 101 can recognize from the vehicle movement / course that the vehicle 100 is currently on a very bumpy road and can, in this case, pause the change in the position of the display information. Optionally, control instructions can also be generated to control the display unit 304 to inform the occupant 302 about the bumpy road and the associated pause.

[0065] For example, the relative movement 410 can be the lateral, time-dependent relative movement between the head movement of the occupant 302 and the vehicle movement. If the display information were only shifted laterally on the display unit, it could move out of the display area during curves, turning, etc. Various aspects that enable compensation for the lateral relative movement are described below.

[0066] If the display information contains text, this can be used, for example, in Fig. 5, can be displayed on the display unit 304 such that the text follows the driving trajectory 404. The text can be read from bottom to top. The text can be displayed line by line on the display unit 304, and one or more words of the text can be displayed in each line. This creates the impression that the vehicle 100 is driving over the text. To reinforce this impression, an image of the surroundings of the vehicle 100 in the direction of travel can be displayed as the background of the text. The image can serve as an artificial horizon, as in Fig. 6. In one example, the image may be an image captured by a data acquisition device 112, such as a camera oriented in the direction of travel. In another example, the image may be a virtual representation of the surroundings of the vehicle 100 in the direction of travel, wherein the virtual representation shows a road running along the travel trajectory 404.

[0067] According to various aspects, the control device 101 can further be configured to take into account a reading speed of the occupant 302. For this purpose, the control device 101 can be configured to determine a viewing direction of the occupant 302 using sensor data showing the eyes of the occupant 302. This sensor data can, for example, be the second sensor data (which shows the head of the occupant 302 and thus his eyes). Additionally or alternatively, third sensor data can be received which shows the eyes of the occupant 302. For example, in the case that the display unit 304 is a display of a head-worn device worn by the occupant, the head-worn device can, for example, have a camera directed at the eyes of the occupant 302, and this camera can be configured to provide the third sensor data.

[0068] The control device 101 can then determine the reading speed of the occupant 302 using the viewing direction of the occupant 302. The control device 101 can be configured to display the time-dependent change in the text displayed along the travel trajectory according to the reading speed of the occupant 302. In some aspects, the control device 101 can highlight a word and / or line of text that the occupant 302 is currently reading (e.g., using different colors, fonts, and / or other visual cues).

[0069] According to various aspects, the control device 101 can be configured to determine (using continuously provided sensor data) whether the reading speed of the occupant 302 is changing, and if this changes, to adapt the speed at which the displayed text changes accordingly. The control device 101 can be configured to determine whether the occupant 302 has read displayed and already hidden words of the text, and if the occupant has not yet read some words, to display them again. If the reader stops reading (e.g., detected by the fact that the line of sight is no longer in the direction of the display unit 304), the text can be hidden to prevent a constant repetition of the text.This can also be extended with an option for direct user interaction, for example with a text / scroll option to scroll through the text (forward and backward) and / or with an option to set a specific reading speed via a menu item.

[0070] In some aspects, a maximum position change of the display information may be present, since otherwise the display information can no longer be displayed within the display area, for example. This maximum position change may be associated with a predefined inclination angle of the travel trajectory 404 and stored as a predefined inclination angle threshold (for example, in the memory 104). This predefined inclination angle threshold may be exceeded, for example, in very steep curves or when turning at intersections. For example, the control device 101 may determine whether an inclination angle of the travel trajectory in a time window (e.g., the prediction period) is greater than or equal to a predefined inclination angle threshold and, if this is the case, may pause the change in the display information in the time window. After the time window, the display of the display information may continue.

[0071] According to various aspects, the display information as shown in Fig. 7, in a section 702 of the display unit 304. This section 702 can, for example, show a virtual screen (e.g., if the display information includes a video) or can show a virtual book page (e.g., if the display information includes text). The time-dependent change 412 in the position of the display information can, in this case, be a time-dependent change in the position of the section 702 on the display unit 304.

[0072] If the inclination angle of the travel trajectory 404 is greater than the predefined inclination angle threshold, the position of the section 702 can, in some embodiments, be reset to a predefined position such that compensation can be made in the subsequent time window of the prediction period in which the inclination angle of the travel trajectory 404 is greater than or equal to the predefined inclination angle threshold. In an illustrative example, the display information can comprise a video, and the control device 101 can determine that the inclination angle of the travel trajectory 404 in the time window is greater than or equal to the predefined inclination angle threshold. The control device 101 can determine a cut in the video that occurs within a predefined time interval (e.g., less than 5 seconds, e.g.,less than 1 second) before the start of the time window, and can reposition the position of the section 702 at a time of the cut in the video in such a way (e.g., laterally) that the lateral relative movement can be compensated by changing the position of the section 702 in the time window.

[0073] To reduce the frequency of such scenarios in which the inclination angle of the driving trajectory 404 in the time window is greater than or equal to the predefined inclination angle threshold, the control device 101, if it determines using the viewing direction of the occupant 302 that the occupant is currently looking at the display unit 304, and if the vehicle is an autonomous vehicle or the driver is using a navigation system, can determine a driving behavior of the vehicle 100 that has a reduced (predicted) lateral and / or vertical vehicle movement (e.g., a different route). If the vehicle is an autonomous vehicle (Level 5), the control device 101 can be configured to influence the motion control of the vehicle 100, for example, to prefer a smooth movement with lower accelerations compared to a higher vehicle speed.This limits the acceleration forces and movements acting on the body of the occupant 302.

[0074] With reference to Fig. 8A, the section 702 can, for example, show a virtual surface (also referred to as a virtual screen) 802 at a predefined distance 804 in front of the vehicle 100. This virtual screen 802 can be positioned along the centerline of the roadway and orthogonal to a tangent 806 of the lane. Optionally, the image (e.g., the camera image or the virtual representation) of the surroundings of the vehicle 100 in the direction of travel can be displayed as the background of the section.

[0075] The compensation for relative movement explained above ensures a world-fixed display of information. As used herein, "world-fixed" refers to a fixed position in an earth-based coordinate system. A world-fixed coordinate system can also be referred to as an earth-fixed coordinate system or geocentric coordinate system. Compensation for relative movement leads, for example, to the occupant's internal perception of movement (due to vehicle movement) and their perception of movement through their eyes (by looking at the display unit) being synchronized. This synchronization has the advantage, for example, of preventing the occupant from experiencing motion sickness.

[0076] As explained above, limitations may occur at tilt angles greater than or equal to the predefined tilt angle threshold. Regarding the example scenario of the virtual screen 802, Fig. 8B illustrates that, depending on the vehicle movement / lane shape, the virtual screen 802 may not be completely within the field of view of the display unit 304. In this case, portions would be projected outside the screen.

[0077] According to various aspects, a tolerance value representing a tolerance of an inertial system of the occupant 302 can be taken into account. In this case, the tolerance value can clearly indicate a margin for compensating the relative movement through the time-dependent change 412 in the position of the display information. The tolerance of a person's inertial system can be a movement tolerance of the human perception system. Clearly, minor discrepancies between eye and inertial measurements can be tolerated or not detected at all. In this way, virtual trajectories can be generated that are equivalent to the real movement and enable a displacement of the section 702 (even at inclination angles greater than or equal to the predefined inclination angle threshold). The displacement can occur both in the direction of the center of the trajectory and towards the tangent 806 of the trajectory.

[0078] In this regard, Fig. 9A a tolerated movement 902 determined using the tolerance value of the occupant 302 and Fig. 9B shows the correspondingly positioned virtual screen 802.

[0079] According to various aspects, the memory of the control device 101 can store, for one or more than one occupant, respectively associated identification data for identifying the occupant and an associated tolerance value. The control device 101 can be configured to determine the current occupant (e.g., the person looking at the display unit 304) using the second sensor data and / or the sensor data showing the occupant's eyes and / or other sensor data enabling identification of the occupant, and can determine the associated tolerance value from the memory. The control device 101 can determine the time-dependent change in the position of the display information such that a difference between the change in position and the relative movement is less than or equal to the associated tolerance value.

[0080] Fig. 10 shows an exemplary flowchart 1000 for controlling the display unit 304 to display text 1002 (as display information) according to various aspects.

[0081] In this exemplary embodiment, the control device 101 can generate the image (e.g., the camera image or the virtual representation) of the surroundings of the vehicle 100 in the direction of travel as a background image 1004 using the first (sensor) data.

[0082] The control device 101 can project the text 1002 onto the background image 1004 according to the position specified by the time-dependent change 412 in the position of the display information (i.e., the text 1002 in this case) and generate a corresponding rendered image 1006. The control device 101 can then generate the control instructions 414 for controlling the display unit 304 to display the rendered image.

[0083] Fig.11 shows a flowchart of a method 1100 for controlling a display unit arranged in a vehicle according to various aspects.

[0084] The method 1100 may include (in 1102) receiving first data (e.g., the first sensor data) representing an environment of a vehicle (e.g., in the direction of travel).

[0085] The method 1100 may include (in 1104) determining a travel trajectory of the vehicle (e.g., for a prediction period) using the first data.

[0086] The method 1100 may include (in 1106) receiving second sensor data representing a head movement of an occupant of the vehicle.

[0087] The method 1100 may include (in 1108) determining a time-dependent relative motion between the head motion of the occupant and a vehicle motion according to the travel trajectory using a speed of the vehicle.

[0088] The method 1100 may include (in 1110) determining a time-dependent change in a position of display information to be displayed on a display unit arranged in the vehicle (e.g., for the prediction period) such that the change in the position of the display information at least partially (e.g., completely or partially) compensates for the relative movement.

[0089] The method 1100 may include (in 1112) generating control instructions for controlling the display unit to display the display information according to the time-dependent change in position.

[0090] Various examples are provided below that describe one or more aspects of the control device 101 and the method 1100. It is understood that aspects described with respect to the control device 101 may also apply to the method 1100, and vice versa.

[0091] Example 1 is a control device for controlling a display unit arranged in a (e.g. autonomous or semi-autonomous) vehicle, the control device comprising a processor which is configured to: receive first sensor data representing an environment of a vehicle (e.g. in the direction of travel); using the first data, determine a (predicted) travel trajectory of the vehicle (e.g. for a prediction period); receive second sensor data representing a head movement of an occupant of the vehicle; using a speed of the vehicle, determine a time-dependent relative movement between the head movement of the occupant and a vehicle movement according to the (predicted) travel trajectory; (e.g. for the prediction period) a time-dependent change in a position of display information (e.g.To determine the information (text and / or video) to be displayed on a display unit arranged in the vehicle, such that the change in the position of the display information fully or partially compensates for the relative movement; and to generate control instructions for controlling the display unit to display the display information according to the time-dependent change in position. For example, the occupant can use their time in an at least partially automated vehicle to read, work, watch a film, etc.

[0092] Example 2 is a control device for controlling a display unit arranged in a (e.g. autonomous or semi-autonomous) vehicle, the control device comprising a processor which is configured to: receive (first) (sensor) data representing an environment of a vehicle (e.g. in the direction of travel); using the (first) data, determine a (predicted) travel trajectory of the vehicle (e.g. for a prediction period); using a speed of the vehicle, determine a time-dependent vehicle movement according to the (predicted) travel trajectory; (e.g. for the prediction period) a time-dependent change in a position of display information (e.g.text and / or a video) to be displayed on a display unit arranged in the vehicle, such that the change in the position of the display information compensates for the vehicle movement; and to generate control instructions for controlling the display unit to display the display information in accordance with the time-dependent change in position.

[0093] Example 3 is configured according to Example 2, wherein the processor is further configured to: receive second sensor data representing a head movement of an occupant of the vehicle; determine, using the speed of the vehicle, a time-dependent relative movement between the head movement of the occupant and the vehicle movement; and determine (e.g., for the prediction period) the time-dependent change in the position of the display information (e.g., text and / or a video) such that the change in the position of the display information at least partially (e.g., completely or partially) compensates for the relative movement.

[0094] Example 4 is configured according to example 1 or 3, wherein the processor is further configured to: determine, using the first data, a predicted vertical movement of the vehicle, wherein the time-dependent relative movement comprises a time-dependent vertical relative movement between the head movement of the occupant and the predicted vertical movement of the vehicle; and / or determine, using the first data, a predicted lateral movement of the vehicle, wherein the time-dependent relative movement comprises a time-dependent lateral relative movement between the head movement of the occupant and the predicted lateral movement of the vehicle.

[0095] Example 5 is configured according to any one of examples 1 to 4, wherein the display unit is mounted in the vehicle; or wherein the display unit is a display unit of a user device worn by the occupant (e.g., a head-mounted display, a tablet, or smart glasses).

[0096] Example 6 is configured according to any one of examples 1 to 5, wherein the display information comprises a text; wherein the processor is configured to: determine the time-dependent change in the position of the text such that the text to be displayed on the display unit follows the driving trajectory.

[0097] Example 7 is configured according to example 6, wherein the processor is configured to determine the time-dependent change in the position of the text such that the text to be displayed on the display unit is to be read from bottom to top.

[0098] Example 8 is configured according to example 6 or 7, wherein the processor is configured to determine the time-dependent change in the position of the text such that the text is to be displayed line by line on the display unit and that one or more than one word of the text is to be displayed in each line.

[0099] Example 9 is configured according to any one of examples 6 to 8, wherein the first sensor data comprises image data showing an image in the direction of travel of the vehicle; wherein the processor is configured to: generate the control instructions for displaying at least a portion of the image on the display unit as a background of the text (e.g., as an artificial horizon).

[0100] Example 10 is configured according to any one of examples 6 to 8, wherein the processor is configured to: generate a virtual representation of an environment of the vehicle in the direction of travel, the virtual representation showing a road running along the travel trajectory; generate the control instructions for displaying the virtual representation as a background of the text (e.g., as an artificial horizon).

[0101] Example 11 is configured according to any one of Examples 1 to 10, wherein the display information comprises text; wherein the second sensor data comprises image data showing at least eyes of the occupant and / or wherein the processor is configured to receive third sensor data comprising image data showing at least the occupant's eyes; wherein the processor is configured to: determine a viewing direction of the occupant using the image data; determine a reading speed of the occupant using the viewing direction of the occupant; and determine the time-dependent change in the position of the display information using the reading speed of the occupant (e.g., by correlating vehicle speed with reading speed).

[0102] Example 12 is configured according to Example 11, wherein the processor is configured to: determine whether the occupant is looking at the display unit using the occupant's gaze direction; if it is determined that the occupant is looking at the display unit, determine the occupant's reading speed.

[0103] Example 13 is configured according to example 11 or 12, wherein the processor is configured to: generate the control instructions for controlling the display unit such that a (e.g., colored) highlighting of one or more words of the text is temporally varied according to the reading speed of the occupant.

[0104] Example 14 is configured according to any one of Examples 11 to 13, wherein the processor is configured to: continuously receive image data showing at least the eyes of the occupant; using the continuous image data, determine whether the reading speed of the occupant changes; and if it is determined that the reading speed of the occupant changes, adjust the control instructions to adjust the time-dependent change in the position of the display information to the changed reading speed.

[0105] Example 15 is configured according to Example 14, wherein the processor is configured to: determine whether the occupant has read displayed and already hidden words of the text; if it is determined that the occupant has not yet read displayed and already hidden words of the text, adjust the control instructions to redisplay the already hidden words.

[0106] Example 16 is configured according to any one of Examples 1 to 15, wherein the processor is configured to: determine whether a tilt angle of the travel trajectory in a time window (e.g., the prediction period) is greater than or equal to a predefined tilt angle threshold (e.g., due to a turn at an intersection); and if it is determined that the tilt angle of the travel trajectory in a time window is greater than or equal to the predefined tilt angle threshold, determine the time-dependent change in the position of the display information such that a change (e.g., both the position and the text itself) of the display information is paused in the time window.

[0107] Example 17 is configured according to any one of Examples 1 to 16, wherein the display information comprises a video; wherein the control instructions indicate that the video is to be displayed in a section of the display unit (e.g., as a virtual screen); wherein the processor is configured to: determine whether an inclination angle of the travel trajectory in a time window (e.g., the prediction period) is greater than or equal to a predefined inclination angle threshold; and if it is determined that the inclination angle of the travel trajectory in a time window (e.g., the prediction period) is greater than or equal to the predefined inclination angle threshold, to determine a cut in the video that lies within a predefined time interval before the start of the time window; and to determine the position of the section at a time of the cut in the video in such a way (e.g.,lateral) so that a lateral relative movement can be compensated by changing the position of the section in the time window.

[0108] Example 18 is configured according to any one of examples 1 to 17, wherein the control instructions indicate that the display information is to be displayed in a section of the display unit (e.g., as a virtual screen in the case of a video or a virtual book page in the case of a text); wherein the time-dependent change in the position of the display information is a time-dependent change in the position of the section (on the display unit).

[0109] Example 19 is configured according to any one of Examples 1 to 18, wherein the display information comprises a text (to be read) and / or a video.

[0110] Example 20 is configured according to any one of Examples 1 to 19, wherein the processor is configured to determine the travel trajectory of the vehicle (e.g., for the prediction period) using navigation data representing a route of the vehicle.

[0111] In Example 21, the control device according to any one of Examples 1 to 20 can optionally further comprise: a storage device which stores, for one or more than one occupant, respectively assigned identification data for identifying the occupant and an assigned tolerance value, wherein the tolerance value represents a tolerance of an inertial system of the occupant, wherein the second sensor data comprises image data showing at least a head of the occupant; wherein the processor is configured to: determine a (current) occupant and the assigned tolerance value using the image data and the identification data; determine the time-dependent change in the position of the display information such that a difference between the change in position and the relative movement is less than or equal to the assigned tolerance value.

[0112] Example 22 is configured according to any one of Examples 1 to 21, wherein the processor is configured to: determine, using the time-dependent change in the position, a rate of change at which the position is changed per unit of time; determine whether the rate of change in a time window (e.g., the prediction period) is greater than or equal to a predefined rate of change threshold; and if it is determined that the rate of change in a time window (e.g., the prediction period) is greater than or equal to a predefined rate of change threshold, determine the control instructions such that a change (e.g., both in the position and in the text itself) of the display information in the time window is paused.

[0113] Example 23 is configured according to any one of Examples 1 to 22, wherein the second sensor data comprises image data showing at least eyes of the occupant and / or wherein the processor is configured to receive third sensor data comprising image data showing at least the occupant's eyes; wherein the processor is configured to: determine a viewing direction of the occupant using the image data; determine whether the occupant is looking at the display unit using the occupant's viewing direction; if it is determined that the occupant is looking at the display unit, determine a driving behavior of the vehicle having a reduced (predicted) lateral and / or vertical vehicle movement (e.g., a different route and / or with a reduced acceleration and / or a reduced top speed, etc.); and generate control instructions for controlling the vehicle according to the (determined) driving behavior.

[0114] Example 24 is an (at least partially automated) vehicle having the display unit and the control device according to any one of Examples 1 to 23.

[0115] Example 25 is configured according to Example 24, wherein the vehicle further comprises: a first imaging unit (e.g., comprising a camera, a lidar sensor, and / or a radar sensor) configured to acquire the first sensor data; and / or a second imaging unit (e.g., a camera) configured to acquire the second sensor data.

[0116] Example 26 is a method for controlling a display unit arranged in a (e.g., autonomous or semi-autonomous) vehicle, the method comprising: receiving first data representing an environment of a vehicle (e.g., in the direction of travel); using the first data, determining a (predicted) travel trajectory of the vehicle (e.g., for a prediction period); receiving second sensor data representing a head movement of an occupant of the vehicle; using a speed of the vehicle, determining a time-dependent relative movement between the head movement of the occupant and a vehicle movement according to the (predicted) travel trajectory; determining a time-dependent change in a position of display information (e.g., text and / or a video) to be displayed on a display unit arranged in the vehicle (e.g.,for the prediction period) such that the change in the position of the display information at least partially (e.g., completely or partially) compensates for the relative movement; and generating control instructions for controlling the display unit to display the display information in accordance with the time-dependent change in position.

[0117] Example 27 is a method for controlling a display unit arranged in a (e.g., autonomous or semi-autonomous) vehicle, the method comprising: receiving (first) data representing an environment of a vehicle (e.g., in the direction of travel); using the (first) data, determining a (predicted) travel trajectory of the vehicle (e.g., for a prediction period); using a speed of the vehicle, determining a time-dependent vehicle movement according to the (predicted) travel trajectory; determining a time-dependent change in a position of display information (e.g., text and / or a video) to be displayed on a display unit arranged in the vehicle (e.g.,for the prediction period) such that the change in the position of the display information compensates for the vehicle movement; and generating control instructions for controlling the display unit to display the display information according to the time-dependent change in the position.

[0118] Example 28 is the method of Example 27, further comprising: receiving second sensor data representing a head movement of an occupant of the vehicle; using the speed of the vehicle, determining a time-dependent relative movement between the head movement of the occupant and the vehicle movement; and determining the time-dependent change in the position of the display information (e.g., text and / or a video) (e.g., for the prediction period) such that the change in the position of the display information at least partially (e.g., completely or partially) compensates for the relative movement.

[0119] Example 29 is the method of example 26 or 28, further comprising: using the first data, determining a predicted vertical motion of the vehicle, wherein the time-dependent relative motion comprises a time-dependent vertical relative motion between the head motion of the occupant and the predicted vertical motion of the vehicle; and / or using the first data, determining a predicted lateral motion of the vehicle, wherein the time-dependent relative motion comprises a time-dependent lateral relative motion between the head motion of the occupant and the predicted lateral motion of the vehicle.

[0120] Example 30 is configured according to any one of examples 26 to 29, wherein the display unit is mounted in the vehicle; or wherein the display unit is a display unit of a user device worn by the occupant (e.g., a head-mounted display, a tablet, or smart glasses).

[0121] Example 31 is configured according to any one of Examples 26 to 30, wherein the display information comprises text; the method further comprising: determining the time-dependent change in the position of the text such that the text to be displayed on the display unit follows the driving trajectory.

[0122] Example 32 is configured according to Example 31, wherein the time-dependent change in the position of the text is determined such that the text to be displayed on the display unit is to be read from bottom to top.

[0123] Example 33 is arranged according to example 31 or 32, wherein the time-dependent change in the position of the text is determined such that the text is to be displayed line by line on the display unit and that one or more than one word of the text is to be displayed in each line.

[0124] Example 34 is configured according to any one of examples 31 to 33, wherein the first sensor data comprises image data showing an image in the direction of travel of the vehicle; generating the control instructions for displaying at least a portion of the image on the display unit as a background of the text (e.g., as an artificial horizon).

[0125] Example 35 is the method according to any one of examples 31 to 33, further comprising: generating a virtual representation of an environment of the vehicle in the direction of travel, wherein the virtual representation shows a road running along the travel trajectory; generating the control instructions for displaying the virtual representation as a background of the text (e.g., as an artificial horizon).

[0126] Example 36 is configured according to any one of Examples 26 to 35, wherein the display information comprises text, wherein the second sensor data comprises image data showing at least eyes of the occupant and / or wherein the processor is configured to receive third sensor data comprising image data showing at least the eyes of the occupant; wherein the method further comprises: using the image data, determining a viewing direction of the occupant; using the viewing direction of the occupant, determining a reading speed of the occupant; and wherein the time-dependent change in the position of the display information is determined using the reading speed of the occupant (e.g., by correlating vehicle speed with reading speed).

[0127] Example 37 is the method of Example 36, further comprising: using the gaze direction of the occupant, determining whether the occupant is looking at the display unit; if it is determined that the occupant is looking at the display unit, determining the reading speed of the occupant.

[0128] Example 38 is configured according to example 36 or 37, wherein the control instructions for controlling the display unit are generated such that a (e.g., colored) highlighting of one or more words of the text is temporally varied according to the reading speed of the occupant.

[0129] Example 39 is the method of any one of Examples 36 to 38, further comprising: receiving continuous image data showing at least the eyes of the occupant; using the continuous image data, determining whether the reading speed of the occupant is changing; and if it is determined that the reading speed of the occupant is changing, adjusting the control instructions to adjust the time-dependent change in the position of the display information to the changed reading speed.

[0130] Example 40 is the method of Example 39, further comprising: determining whether the occupant has read displayed and already hidden words of the text; if it is determined that the occupant has not yet read displayed and already hidden words of the text, adjusting the control instructions to redisplay the already hidden words.

[0131] Example 41 is the method of any one of Examples 26 to 40, further comprising: determining whether a tilt angle of the travel trajectory in a time window (e.g., the prediction period) is greater than or equal to a predefined tilt angle threshold (e.g., due to a turn at an intersection); and wherein, if it is determined that the tilt angle of the travel trajectory in a time window (e.g., the prediction period) is greater than or equal to the predefined tilt angle threshold, the time-dependent change in the position of the display information is determined such that a change (e.g., both the position and the text itself) of the display information in the time window is paused.

[0132] Example 42 is configured according to any one of Examples 26 to 41, wherein the display information comprises a video; wherein the control instructions indicate that the video is to be displayed in a section of the display unit (e.g., as a virtual screen); wherein the method further comprises: determining whether an inclination angle of the travel trajectory in a time window (e.g., the prediction period) is greater than or equal to a predefined inclination angle threshold; and if it is determined that the inclination angle of the travel trajectory in a time window (e.g., the prediction period) is greater than or equal to the predefined inclination angle threshold, determining a cut in the video that lies within a predefined time interval before the start of the time window; and (e.g.,lateral) repositioning of the position of the section at a time of the cut in the video in such a way that a lateral relative movement can be compensated by changing the position of the section in the time window.

[0133] Example 43 is configured according to any one of examples 26 to 42, wherein the control instructions indicate that the display information is to be presented in a section of the display unit (e.g., as a virtual screen in the case of a video or a virtual book page in the case of a text); wherein the time-dependent change in the position of the display information is a time-dependent change in the position of the section (on the display unit).

[0134] Example 44 is configured according to any one of examples 26 to 43, wherein the display information comprises a text (to be read) and / or a video.

[0135] Example 45 is configured according to any one of examples 26 to 44, wherein the travel trajectory of the vehicle (e.g., for the forecast period) is determined using navigation data representing a route of the vehicle.

[0136] Example 46 is configured according to any one of Examples 26 to 45, wherein a storage device stores, for one or more than one occupant, respectively associated identification data for identifying the occupant and an associated tolerance value, wherein the tolerance value represents a tolerance of an inertial system of the occupant; wherein the second sensor data comprises image data showing at least a head of the occupant; wherein the method further comprises: using the image data and the identification data, determining a (current) occupant and the associated tolerance value; wherein the time-dependent change in the position of the display information is determined such that a difference between the change in position and the relative movement is less than or equal to the associated tolerance value.

[0137] Example 47 is the method of any one of Examples 26 to 46, further comprising: using the time-dependent change in position, determining a rate of change at which the position is changed per unit of time; determining whether the rate of change in a time window (e.g., the prediction period) is greater than or equal to a predefined rate of change threshold; and if it is determined that the rate of change in a time window (e.g., the prediction period) is greater than or equal to a predefined rate of change threshold, determining the control instructions such that a change (e.g., both in position and in the text itself) of the display information in the time window is paused.

[0138] Example 48 is configured according to any one of Examples 26 to 47, wherein the second sensor data comprises image data showing at least eyes of the occupant and / or wherein the processor is configured to receive third sensor data comprising image data showing at least the occupant's eyes; the method further comprising: using the image data, determining a gaze direction of the occupant; using the gaze direction of the occupant, determining whether the occupant is looking at the display unit; if it is determined that the occupant is looking at the display unit, determining a driving behavior of the vehicle having a reduced (predicted) lateral and / or vertical vehicle movement (e.g., a different route and / or with a reduced acceleration and / or a reduced top speed, etc.); and generating control instructions for controlling the vehicle according to the (determined) driving behavior.

[0139] Example 49 is a (e.g., non-transitory) computer-readable medium (e.g., a computer program product, a non-transitory storage medium, a non-transitory storage medium, or a non-volatile storage medium) storing instructions that, when executed by a processor, cause the processor to control an apparatus to perform a method according to any one of Examples 26 to 48.

[0140] Example 50 is one or more means for performing the method of any one of Examples 26 to 48.

[0141] In the foregoing description and the accompanying figures, the components of electronic devices are sometimes depicted as separate elements. In this regard, it is understood that discrete elements may be combined or integrated into a single element. This includes combining two or more circuits into a single circuit, mounting two or more circuits on a common chip or chassis to form an integrated element, executing discrete software components on a common processor core, etc.Conversely, it is understood that a single element can be divided into two or more discrete elements, such as dividing a single circuit into two or more separate circuits, dividing a chip or chassis into discrete elements originally intended on it, dividing a software component into two or more sections and executing them on a separate processor core, etc.

[0142] It is understood that the implementations of the methods described herein are demonstrative in nature and can therefore be implemented in a corresponding device. Likewise, it is understood that implementations of the device described herein can be implemented as a corresponding method. It is therefore understood that a device corresponding to a method described herein can include one or more components configured to perform each aspect of the corresponding method.

[0143] While the invention has been particularly shown and described with reference to specific embodiments, it is understood that various changes in design may be made without departing from the scope of the invention as defined by the appended claims. The scope of the invention is thus determined by the appended claims, and all changes which come within the meaning and range of equivalence of the claims are therefore intended to be embraced. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] SAE J3016 2018

[0019]

Claims

[1] Control device (101) for controlling a display unit (304) arranged in a vehicle (100), the control device (101) comprising a processor (102) which is configured: • receive first data (402) representing an environment of a vehicle (100); • to determine a travel trajectory (404) of the vehicle (100) using the first data (402); • receive second sensor data (406) representing a head movement of an occupant (302) of the vehicle (100); • using a speed (408) of the vehicle (100) to determine a time-dependent relative movement (410) between the head movement of the occupant (302) and a vehicle movement according to the travel trajectory (404); • to determine a time-dependent change (412) in a position of display information to be displayed on a display unit (304) arranged in the vehicle (100) in such a way that the change in the position of the display information compensates for the relative movement (410) in whole or in part; and • to generate control instructions (414) for controlling the display unit (304) to display the display information according to the time-dependent change (412) of the position. [2] Control device (101) according to claim 1, wherein the processor (102) is further configured to: • to determine a predicted vertical movement of the vehicle (100) using the first data (402), wherein the time-dependent relative movement (410) comprises a time-dependent vertical relative movement (410) between the head movement of the occupant (302) and the predicted vertical movement of the vehicle (100); and / or • using the first data (402) to determine a predicted lateral movement of the vehicle (100), wherein the time-dependent relative movement (410) comprises a time-dependent lateral relative movement (410) between the head movement of the occupant (302) and the predicted lateral movement of the vehicle (100). [3] Control device (101) according to claim 1 or 2, • wherein the display unit (304) is mounted in the vehicle (100); or • wherein the display unit (304) is a display unit of a user device worn by the occupant (302). [4] Control device (101) according to one of claims 1 to 3, wherein the display information comprises a text (1002); and wherein the processor (102) is configured to determine the time-dependent change (412) in the position of the text (1002) such that the text (1002) to be displayed on the display unit (304) follows the travel trajectory (404); wherein the processor (102) is preferably configured to determine the time-dependent change (412) in the position of the text such that the text (1002) to be displayed on the display unit (304) can be read from bottom to top. [5] Control device (101) according to claim 4, wherein the first data (402) comprises image data showing an image in the direction of travel of the vehicle (100); and wherein the processor (102) is configured to generate the control instructions (414) for displaying at least a portion of the image on the display unit (304) as the background of the text; or wherein the processor (102) is configured to: generate a virtual representation of an environment of the vehicle (100) in the direction of travel, the virtual representation showing a road running along the travel trajectory (404); and generate the control instructions (414) for displaying the virtual representation as the background of the text. [6] Control device (101) according to one of claims 1 to 5, wherein the display information comprises a text (1002); wherein the second sensor data (406) comprises image data showing at least eyes of the occupant (302) and / or wherein the processor (102) is configured to receive third sensor data comprising image data showing at least the eyes of the occupant (302); wherein the processor (102) is configured: • to determine a viewing direction of the occupant (302) using the image data; • to determine a reading speed of the occupant (302) using the viewing direction of the occupant (302); and • to determine the time-dependent change (412) of the position of the display information using the reading speed of the occupant (302). [7] Control device (101) according to one of claims 1 to 6, • wherein the control instructions (414) indicate that the display information is to be displayed in a section (702) of the display unit (304); • wherein the time-dependent change (412) in the position of the display information is a time-dependent change (412) in the position of the section (702). [8] Vehicle (100) comprising the display unit (304) and the control device (101) according to one of claims 1 to 7. [9] Method (1100) for controlling a display unit (304) arranged in a vehicle (100), the method (1100) comprising: • Receiving (1102) first data representing an environment of a vehicle; • using the first data, determining (1104) a travel trajectory (404) of the vehicle; • receiving (1106) second sensor data representing a head movement of an occupant (302) of the vehicle; • using a speed (408) of the vehicle, determining (1108) a time-dependent relative movement (410) between the head movement of the occupant (302) and a vehicle movement according to the travel trajectory (404); • Determining (1110) a time-dependent change in the position of display information to be displayed on a display unit (304) arranged in the vehicle (100), such that the change in the position of the display information compensates for the relative movement (410) in whole or in part; and • Generating (1112) control instructions (414) for controlling the display unit (304) to display the display information according to the time-dependent change in the position. [10] A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to control a device to perform a method (1100) according to claim 9.

Images