Method for controlling a mixed-reality function and motor vehicle

Abstract

Method for controlling a mixed-reality application function, MR application function inside a motor vehicle during a mixed-reality application, MR application, characterized in that - the motor vehicle has at least one control panel located behind a decorative element (24), and an MR application function is controlled by an operating input on the control panel, - wherein the MR application outputs optical information to a user (16) of the MR application by means of data glasses (18), - wherein the MR application includes a virtual control element (10), - wherein the virtual control element is at least partially overlaid with at least one section (30) of the control panel for displaying the virtual control element (10) by means of the data glasses (18), wherein the section overlaps with a part of the decorative element (24), - wherein a sensor device (26) included in the control panel detects an operating input on the control panel, and checks whether a position of the operating input corresponds to a position of the virtual control element (10) and whether the operating input corresponds to an actuation possibility of the virtual control element (10), and - when both the position of the user input and the position of the virtual control element (10) match, and the user input and the possibility of actuating the virtual control element (10) match, optical feedback is output to the user (16) via the data glasses (18).

Description

[0001] The invention relates to a method for controlling a mixed-reality function and a motor vehicle.

[0002] The use of mixed reality (MR) systems in motor vehicles has increased significantly recently. One example is augmented reality (AR) systems that utilize a head-up display (HUD), allowing the user to maintain their gaze and head position because information is projected into their field of vision, for example, via a semi-transparent mirror. Using a head-up display eliminates the need for smart glasses, but the disadvantage is that the user's head must be positioned within a specific area to view the information.

[0003] Furthermore, virtual reality (VR) systems offer the option of using smart glasses, which present a user with a simulated reality. When using a VR system in a vehicle, for example, it is also possible to combine VR content with the vehicle's real-time motion and navigation data. If, for instance, the vehicle is turning left, the user of a VR application can be shown the corresponding behavior of their virtual vehicle. The speed and acceleration of the real vehicle thus influence the user's virtual experience.By combining physical feedback such as speed and steering with driving routes and times, an experience of immersion can be created or enhanced for the user, meaning that the virtual environment is perceived as real and the user's awareness of being exposed to illusory stimuli recedes into the background.

[0004] In the past, such systems were controlled using dedicated control devices or other special operating elements. However, there is currently a trend toward reducing expensive switches and using virtual controls instead. This is achieved primarily through a combination of blinking and gaze direction, finger movements, gaze directions, and / or pointing. However, command-less (i.e., purely virtual) operation of MR systems is usually not intuitive and requires a high degree of user concentration. Particularly in dynamic situations (i.e., a moving environment and changing accelerations), it is undesirable for a vehicle occupant to have to operate an MR application through focused gaze movements.

[0005] For example, DE 10 2019 212 278 A1 discloses a vehicle operating system for controlling content displayed via a screen in the form of virtual or augmented reality. The operating system features a haptically perceptible input unit with touch-sensitive user interfaces. This input unit is either mounted at a specific location within the vehicle or held in the user's hand. However, providing a dedicated input unit within a vehicle is complex and costly. Furthermore, the operating system is limited to a few functions and, due to its specific design, only configurable to a limited extent for additional tasks.

[0006] German patent application DE 10 2020 100 040 A1 discloses an interior component of a vehicle with a haptically perceptible control element integrated into its surface. The control element is only visible during an operating state, with its visibility being controlled by the state of a proximity sensor. However, providing a dedicated control element with a proximity sensor is complex, costly, and offers limited reconfigurability. Furthermore, it is unclear how such a control element could be integrated into a magnetic resonance (MR) application.

[0007] Furthermore, German patent application DE 10 2019 127 309 A1 discloses a cover element for the interior of a motor vehicle. The cover element features a three-dimensional LED structure in conjunction with capacitive sensors and can be used as a control unit. However, the independent provision of a cover element with a three-dimensional LED structure is extremely complex and costly.

[0008] Furthermore, the publication DE 10 2018 208 827 A1 describes a user interface, a means of transport and a method for determining user input, the publication DE 10 2017 220 475 A1 describes an interactive decorative element for arrangement in the interior of a vehicle, the publication DE 10 2014 014 336 A1 describes an operating device for a motor vehicle and the publication DE 10 2020 108 986 A1 describes a means of transport, a device and a method for controlling a computer game on board a means of transport.

[0009] It is therefore an object of the present invention to create a control element that is cost-effective, intuitive to operate, flexibly reconfigurable and easy to integrate into an MR application.

[0010] The invention is based on the idea of ​​arranging a touch-sensitive touch panel sublayer beneath an otherwise non-functional decorative element. This sublayer is larger than the decorative element itself, and a virtual control element is created by means of a display in the smart glasses. Because the touch-sensitive touch panel sublayer is larger than the virtual control element, multiple virtual control elements can be easily and flexibly arranged or defined on it, allowing for simple reconfiguration. The touch panel sublayer can transmit information about touch and pressure intensity to the MR application, which can then provide the user with visual feedback on the operation via the smart glasses.

[0011] The invention solves this problem by providing a method for controlling an MR function by means of a control element inside a motor vehicle in a mixed-reality application according to claim 1. Furthermore, the invention solves the problem by providing a motor vehicle according to a further independent claim. Advantageous embodiments of the invention are disclosed in the dependent claims, the following description, and the figures.

[0012] According to the invention, a method for controlling a mixed-reality (MR) application function inside a motor vehicle is provided, wherein the motor vehicle has at least one control panel which, from the user's perspective, is arranged behind a decorative element and preferably within the user's reach. The user can, for example, be a passenger in the motor vehicle. Furthermore, the method can, for example, control a vehicle function. The MR application can, for example, be executed by a control unit of the motor vehicle. The MR application function is controlled by an input on the control panel. The MR application outputs optical information to the user of the MR application via smart glasses worn by the user.The MR application includes a virtual control element, meaning that the MR application presents the user with a virtual control element. This virtual control element is at least partially overlaid with at least one section of the control panel for display via the smart glasses, with the control panel section overlapping with a portion of the decorative element from the user's perspective. The virtual control element could be, for example, a button or a toggle switch, and the optical information is used to overlay the virtual control element, at least partially, onto the section of the control panel to display it in the MR application. In other words, to control a mixed-reality application function, a virtual control element can be coupled with a haptic command transmitter, thus enabling haptic operability.The control panel can be larger than the virtual control element. The display of the virtual control element depends, for example, on the user's viewing direction, the vehicle's status, vehicle acceleration, the state of the MR application, and a 3D model of the control panel within the MR application. A sensor integrated into the control panel detects user input and checks whether the input position corresponds to a position of the virtual control element and whether the input corresponds to an activation option of the virtual control element. If both the input position and the virtual control position match, visual feedback is provided to the user via the smart glasses.This allows the activation of the virtual control element to be registered and visually signaled to the user. It is also possible to provide additional acoustic feedback to the user, such as a click, a noise, and / or a tone.

[0013] To give just a few examples, a push button can be represented in the MR application, which can assume two states that can be visually represented as different colors, brightness levels, or lighting conditions. Furthermore, a decorative strip can be visualized as a toggle switch that can be pressed in two directions to control a continuous value. Pressure on the decorative strip can be detected by the sensor, thus making it possible to capture a targeted user input from the virtual control element.

[0014] The control panel section can be part of a decorative element, such as a strip, a panel, a cover, and / or a relief. The decorative element can serve to embellish or enhance a surface and / or component of the vehicle. Furthermore, the decorative element can be distinguished by a special surface and / or a particular feel. The virtual control element can thus be displayed to the user via the smart glasses in such a way that it appears to be located at the same position as the decorative element.

[0015] Control devices serve as interfaces between a person and a machine, for example, as switches or pushbuttons. A sensible and intuitive arrangement of control devices can be considered under the heading of ergonomics. Most control devices today are electrical or electronic because they are generally cost-effective in this form. However, there are also control devices that are pneumatic, hydraulic, or mechanical. Control devices can be classified according to their actuation method.

[0016] The term "interior" generally refers to all components visible to the driver and passengers within the vehicle's interior. Decorative elements and trim pieces in the vehicle interior can fulfill not only aesthetic but also functional requirements, such as high scratch resistance. Depending on the specific needs, various materials can be used, including aluminum, carbon fiber, wire mesh, fine wood, leather, textiles, and various plastics.

[0017] Virtual reality, or VR for short, can be described as the representation and simultaneous perception of an apparent reality and its physical properties in a real-time, computer-generated, interactive virtual environment. To create a sense of immersion, special output devices (so-called virtual reality headsets or data glasses) are required to display virtual worlds. To convey a spatial impression, two images from different perspectives are generated and displayed (stereo projection).

[0018] Real-time processing, in this context, refers to the operation of a computer system in which programs for processing incoming data are constantly ready to run, ensuring that the processing results are available within a predetermined timeframe. Depending on the application, the data may be generated randomly over time or at predetermined intervals. Hardware and software ensure that no delays occur that could prevent this condition from being met. The data processing itself does not necessarily have to be exceptionally fast; it simply needs to be guaranteed to be fast enough for the specific application.

[0019] Augmented reality (AR) can be understood as the computer-aided enhancement of our perception of reality. This information can address all human sensory modalities. However, AR is often understood to mean only the visual presentation of information, i.e., the supplementation of images or videos with computer-generated additional information or virtual objects through overlay and / or superimposition.

[0020] Mixed Reality (MR) encompasses environments or systems that blend a user's natural perception with an artificial (computer-generated) perception. Besides the primarily computer-generated virtual reality, this includes, in particular, augmented reality and augmented virtuality systems. Mixed Reality covers the entire "reality-virtuality continuum" with the exception of "only reality" and "only virtuality": Between these two extremes, there are seamless intermediate stages that blend both. Augmented Reality and Augmented Virtuality, in particular, are specific manifestations of the Mixed Reality principle.

[0021] The invention provides a solution that does not require eye contact for operation. The virtual control element offers intuitive haptic feedback and visual feedback via the smart glasses. This improves both the sense of safety during operation and the user's comfort experience.

[0022] The functionality provided in this way enables a high degree of speed and accuracy for precise action detection. The communication concept utilizes existing channels of the smart glasses, which can optically track the user's hand movements. The invention is therefore more cost-effective and flexible than previous MR control devices. Operation is also safer, more familiar, and more comfortable than with other MR control devices.

[0023] The method according to the invention thus has the advantage that a virtual control element can be easily arranged inside a motor vehicle and that it is very easy to define or configure a virtual control element on the control panel. Because the control panel is a large panel in which a sensor device is arranged, a virtual control element can be easily defined that provides a special haptic feel and optical feedback to the user via the smart glasses.

[0024] The invention also includes further embodiments which offer additional advantages.

[0025] One embodiment provides that the virtual control element is displayed to the user via optical information using smart glasses, as if the virtual control element were located on the physical control panel. Various methods can be used for this, such as LED indicators or displays placed on the haptic functional surfaces. These visual elements can display information about the type of touch, pressure intensity, or other relevant parameters. Thus, the virtual control element is clearly displayed to the user, and optical feedback via the smart glasses clearly signals its presence.

[0026] Another advantageous embodiment provides that the control panel has a haptically perceptible surface structure suitable for haptically supporting the representation of the position and actuation options of the virtual control element in the MR application. This means that the virtual control element is arranged such that a haptically perceptible surface structure is used on the control panel to realize the virtual control element. A special embossing, pattern, material, or texture can be used for this purpose. For example, a decorative strip can also be represented as a slide or toggle switch. This embodiment has the advantage that existing surface structures can be used as virtual control elements, thus providing a particularly beneficial immersive experience for the user.

[0027] Another advantageous embodiment provides that when both the position of the user input coincides with the position of the virtual control element and the user input coincides with the activation capability of the virtual control element, activation of the virtual control element is registered, and an MR application function associated with the virtual control element and / or the interaction is executed. This means that not only is visual feedback provided, but also an MR application function associated with the virtual control element is executed. This has the advantage that the user is shown a user input on the control panel as activation of the virtual control element, and feedback from the MR application is returned to the user by means of the execution of the associated application function.

[0028] A particularly advantageous embodiment provides that the control panel section is a part of the control panel configured to detect touch input at any position within the panel and to allow reconfiguration or arbitrary placement of individual virtual controls within the control panel. This means that the control panel is configured to allow multiple individual virtual controls within it, and it is possible to easily reconfigure these as needed.

[0029] Another advantageous embodiment provides that the sensor device uses resistive, capacitive, projected-capacitive, inductive, projected-inductive, optical, surface wave-based, and / or volume wave-based touch detection technology. For example, rectangular resistive layers are readily available and inexpensive to install, making it easy to provide a control panel with a large surface area. This therefore represents a particularly simple implementation of the sensor device.

[0030] According to a further advantageous embodiment, further processing of the captured user input is provided within the MR application or an external MR application. This means that the MR application can be processed on an electronic control unit of the vehicle, or that the captured user input can be forwarded to an external control unit that executes the MR application. "External" in this context can also mean that processing is performed by a server located outside the vehicle, with wireless communication between the smart glasses and the server. This offers the advantage of flexible data processing and the ability to easily handle particularly high processing loads.

[0031] According to a further advantageous embodiment, the control panel of the MR application provides information about the touch history of the user input. This means that, for example, gesture recognition can enable multiple assignments to the user input, thus allowing for better use of the virtual control element or the control panel.

[0032] According to a further advantageous embodiment, the control panel of the MR application provides information about the pressure applied to the user input. This means that improved feedback can be achieved and multiple functions can be assigned to the user input, for example, through different pressure levels. Thus, a better user experience of the virtual control element or the control panel can be realized.

[0033] According to the invention, a motor vehicle is also provided with at least one control panel inside the vehicle, which is configured to control a vehicle function and / or a mixed-reality application function (MR application function) in a mixed-reality application (MR application) by means of an input on the control panel. The control panel is arranged behind a decorative element, and the motor vehicle has a control unit configured to execute the MR application and output optical information to a user of the MR application via smart glasses.The MR application includes a virtual control element, the control unit being configured to cause the data glasses to output optical information in such a way that it overlaps at least partially with at least one section of the control panel for displaying the virtual control element in the MR application, the control panel section being designed as part of the decorative element. The control panel includes a sensor device configured to detect an input on the control panel and to check whether a position of the input corresponds to a position of the virtual control element and whether the input corresponds to an actuation option of the virtual control element.The control unit is designed to activate or cause the data glasses to output optical feedback to the user when both the position of the user input and the position of the virtual control element match, as well as when the user input and the virtual control element can be actuated.

[0034] For use cases or application situations that may arise during the procedure and are not explicitly described here, it may be provided that, according to the procedure, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0035] The invention also includes the control unit for the motor vehicle. The control unit can comprise a data processing device or a processor unit configured to carry out an embodiment of the method according to the invention. For this purpose, the processor unit can comprise at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor unit can comprise program code configured to carry out the embodiment of the method according to the invention when executed by the processor unit. The program code can be stored in a data memory of the processor unit.The processor setup can be based on at least one circuit board and / or at least one SoC (System on Chip).

[0036] The invention also includes further developments of the method according to the invention, which have features as already described in connection with the further developments of the method according to the invention.

[0037] For this reason, the corresponding further developments of the motor vehicle according to the invention are not described again here.

[0038] The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0039] As a further solution, the invention also includes a computer-readable storage medium comprising program code which, when executed by a computer or a computer network, causes it to execute an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data storage medium (e.g., as flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data storage medium (e.g., as RAM - random access memory). The storage medium can be located within the computer or computer network. However, the storage medium can also be operated, for example, as an app store server and / or cloud server on the internet. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code can be provided as binary code and / or as assembly code and / or as source code of a programming language (e.g. C) and / or as a program script (e.g. Python).

[0040] The invention also includes combinations of the features of the described embodiments. The invention therefore also includes realizations that each exhibit a combination of the features of several of the described embodiments, provided that the embodiments have not been described as mutually exclusive.

[0041] The following are exemplary embodiments of the invention described. This is illustrated by: Fig. 1. A schematic side view into the interior of a motor vehicle; and Fig. 2 A schematic representation of a control panel during user input.

[0042] The exemplary embodiments described below are preferred embodiments of the invention. In these exemplary embodiments, the described components each represent individual features of the invention, which can be considered independently of one another and each further develops the invention independently. Therefore, the disclosure is intended to include combinations of features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0043] In the figures, identical reference symbols denote functionally equivalent elements.

[0044] Fig. Figure 1 shows a schematic side view of the interior of a motor vehicle, depicting a passenger or user 16 who is viewing the interior of the vehicle using smart glasses 18. The smart glasses 18 can display optical information, such as a virtual display 12, to reproduce virtual content. The smart glasses 18 can also display the dashboard in such a way that a virtual control element 10 is shown on a control panel of the vehicle. The control panel is equipped with a sensor layer that serves to detect user input from the user 16. The virtual control element 10 in the MR application overlaps at least partially with at least one section of the control panel in order to display the virtual control element 10 in the MR application using the smart glasses 18.The control panel section, which is represented as the virtual control element 10 by the data glasses 18, is designed as part of a decorative element. The sensor device integrated into the control panel can detect an input on the control panel and check whether the input position corresponds to a position of the virtual control element 10 and whether the input corresponds to an actuation option of the virtual control element 10. For example, if the virtual control element 10 is in the form of a push button, an actuation option of the virtual control element 10 is, for example, pressing this virtual control element 10. If, for example, an input is detected that does not correspond to an actuation option of the virtual control element 10, it can be ignored, and no visual feedback is output to the user 16 via the data glasses 18.Furthermore, no corresponding application function is executed. However, if both the position of the user input and the position of the virtual control element 10 coincide, and the user input also coincides with a possible activation of the virtual control element 10, optical feedback can be output to the user 16 via the data glasses 18. The virtual control element 10 can thus be visualized to the user 16 using optical information or the data glasses 18, as if the virtual control element 10 were located on the control panel section. If both the positions and the user input coincide, and the user input also coincides with the activation possibility of the virtual control element 10, an MR application function associated with the virtual control element 10 or with the interaction can be executed.The control panel section is a part of the larger control panel, which is designed to allow and detect touch operation at any position on the control panel, and the control panel is designed to allow reconfiguration of individual virtual controls within the control panel.

[0045] A recorded operation or activation of the virtual control element 10 can be carried out within the MR application or within an external MR application. Fig. Figure 1 also shows a line of sight 14, which corresponds to a line of sight from the data glasses 18 to the virtual control element 10. An input on the control panel or on the virtual control element 10 can be processed by an electronic control unit 22.

[0046] Fig.Figure 2 shows a schematic representation of a control panel comprising a decorative element 24 and a sensor layer 26. Using the data glasses 18, a virtual control element 10 can be displayed to the user 16 within or at least on a part of the decorative element 24. When the user touches the sensor, for example with a finger 28, the touch input can be detected by the sensor layer 26 located directly behind the decorative element. The virtual control element 10 corresponds to a section 30 of the sensor surface, which is the part of the sensor surface or control panel located behind the virtual control element 10. The sensor layer 26 can employ a resistive, capacitive, projected-capacitive, inductive, projected-inductive, optical, surface wave-based, and / or volume wave-based touch detection technology.Preferably, the sensor layer 26 is a resistive touch-sensitive layer. This has the advantage of being a cost-effective and easy-to-implement sensor layer 26. The sensor layer 26 can provide information about the touch profile of the user input. This enables, for example, the detection of gestures or multiple assignments to the user input. Furthermore, it can be provided that the MR application is supplied with information about the pressure applied to the user input. This enables, for example, multiple assignments to the user input of the virtual control element 10. It can also offer, for example, a continuously variable adjustment option where a parameter can be continuously adjusted via pressure.

[0047] Overall, the examples show how the invention can provide an improved control element 10 for an MR application.

[0048] The invention relates to a method for controlling a mixed-reality application function, MR application function, inside a motor vehicle during a mixed-reality application, MR application, wherein the motor vehicle has at least one control panel arranged behind a decorative element 24, and an MR application function is controlled by an input on the control panel, wherein the MR application outputs optical information to the user 16 of the MR application by means of data glasses 18, in the MR application comprising a virtual control element 10, the optical information being at least partially superimposed on at least one section 30 of the control panel for displaying the virtual control element 10 in the MR application by means of the data glasses 18, the section 30 being designed as part of a decorative element 24.A sensor device 26 included in the control panel detects an operating input on the control panel and, if a correct operating input is detected, optical feedback is output to the user 16 via the data glasses 18. Reference symbol list: 10 Control element 12 ads 14 Line of sight 16 users 18 data glasses 22 Control unit 24 decorative elements 26 Sensor device 28 fingers Section 30

Images