Method for controlling a mixed reality function, and motor vehicle

Abstract

The invention relates to a method for controlling a mixed reality application function, to an MR application function in the interior of a motor vehicle in a mixed reality application, and to an MR application, wherein the motor vehicle has at least one control panel which is located behind a decorative element (24), and an MR application function is controlled by an operating input on the control panel, wherein the MR application outputs visual information to the user (16) of the MR application by means of data glasses (18), a virtual operating element (10) is included in the MR application, the optical information is superimposed at least partially with at least one portion (30) of the control panel for displaying the virtual control element (10) in the MR application by means of the data glasses (18), the portion (30) is configured as part of a decorative element (24), an operating input on the control panel is detected by a sensor means (26) included in the control panel and, if a correct operating input is detected, visual feedback is output to the user (16) via the data glasses (18).

Description

[0001] Method for controlling a mixed reality function and motor vehicle

[0002] DESCRIPTION:

[0003] The invention relates to a method for controlling a mixed reality function and a motor vehicle.

[0004] The use of mixed reality (MR) systems in motor vehicles has recently increased significantly. One example is augmented reality (AR) systems that utilize a head-up display (literally "head-up display"), where the user can maintain their gaze direction and head position because information is projected into their field of vision, for example, via a semi-transparent mirror. Using a head-up display does not require data glasses, but the disadvantage is that the user's head must be within a certain range to perceive the information.

[0005] Furthermore, so-called virtual reality (VR) systems offer the possibility of using data glasses, which present a user with an apparent reality via data glasses. When using a VR system in a motor vehicle, for example, it is also possible to combine VR content in real time with the vehicle's movement and navigation data. If, for example, the vehicle makes a left turn, the user of a VR application can be shown the corresponding behavior of their virtual vehicle. The speed and acceleration of the real vehicle thus influence the user's virtual experience.

[0006] By combining physical feedback such as speed and steering with driving routes and times, an experience of immersion can be created or enhanced for the user, meaning that the virtual environment is perceived as real and the user's awareness of being exposed to illusory stimuli fades into the background.

[0007] In the past, such systems were controlled using dedicated controllers or other special operating elements to enable operation. However, there is currently a trend to reduce the use of expensive switches and instead use virtual operation. This is mainly achieved through a combination of blinking and gaze direction, finger movements, gaze directions, and / or finger pointing. However, command-less (i.e., purely virtual) operation of MR systems is usually not intuitive or requires the user to concentrate particularly on operating the control element. Particularly in moving situations (i.e., a moving external world and changing accelerations), it is undesirable for a vehicle occupant to have to operate an MR application using focused gaze directions.

[0008] For example, DE 10 2019 212 278 A1 discloses a vehicle operating system for operating content presented by a display device in the form of virtual or augmented reality. The operating system has a haptically perceptible input unit with touch-sensitive user interfaces. An input unit is mounted at a specific position in the vehicle or is located in the hand of a user. However, providing a dedicated input unit in a vehicle is complicated and time-consuming. Furthermore, the operating system is limited to a few functions and, due to its specific design, can only be configured to a limited extent for additional tasks.

[0009] Document DE 102020 100 040 A1 discloses an interior component of a vehicle with a haptically perceptible control element integrated into the surface. The control element is only visible during an operating state, with visibility being controlled depending on the state of a proximity sensor. However, providing a dedicated control element with a proximity sensor is complex and cost-effective, and only offers limited reconfigurability. Furthermore, it is unclear how such a control element could be integrated into an MR application.

[0010] Furthermore, DE 10 2019 127 309 A1 discloses a cover element for the interior of a motor vehicle. The cover element has a three-dimensional LED structure in conjunction with capacitive sensors and can be used as an operating unit. However, providing a cover element with a three-dimensional LED structure on its own is extremely complex and expensive.

[0011] It is therefore an object of the present invention to provide an operating element that is cost-effective, intuitive to use, flexibly reconfigurable and easy to integrate into an MR application.

[0012] The invention is based on the idea of ​​arranging a touch-sensitive touch panel sublayer beneath an essentially non-functional decorative element, which is also larger than the decorative element, and creating a virtual control element by means of a display in the data glasses. Because the touch-sensitive touch panel sublayer is larger than the virtual control element, multiple virtual control elements can be easily and flexibly arranged or defined on it, making them easily reconfigurable. The touch panel sublayer can provide information about a touch and pressure force to the MR application, and the application can provide the user with visual feedback about an operation via the data glasses.

[0013] The invention solves this problem by providing a method for controlling an MR function using a control element inside a motor vehicle in a mixed reality application according to claim 1. Furthermore, the invention solves this problem by providing a motor vehicle according to a further independent claim. Advantageous developments of the invention are disclosed in the dependent claims, the following description, and the figures.

[0014] According to the invention, a method is provided for controlling an MR application function inside a motor vehicle in a mixed reality application (MR application). The motor vehicle has at least one control panel which, from the perspective of a user, is arranged behind a decorative element and is preferably arranged within reach of the user. The user can be, for example, a passenger in the motor vehicle. Furthermore, the method can, for example, control a vehicle function. The MR application can, for example, be executed by a control device of the motor vehicle. The MR application function is controlled by an operating input on the control panel. The MR application outputs optical information to the user of the MR application by means of data glasses worn by the user.The MR application includes a virtual control element, meaning that the MR application presents the user with a virtual control element, wherein the virtual control element is at least partially overlaid with at least a portion of the control panel for displaying the virtual control element using the data glasses, wherein the control panel portion overlaps with a portion of the decorative element from the user's perspective. The virtual control element can be, for example, a button or a toggle switch, and the virtual control element is at least partially overlaid on the portion of the control panel using the optical information in order to display the virtual control element in the MR application. In other words, to control a mixed reality application function, a virtual control element can be coupled to a haptic command transmitter, and the virtual control element can thus have haptic operability.The control panel can be larger than the virtual control element. In this case, the display of the virtual control element depends, for example, on the user's viewing direction, the vehicle state, the vehicle acceleration, the state of the MR application, and a 3D model of the control panel in the MR application. A sensor device included in the control panel detects an operating input on the control panel and checks whether the position of the operating input matches the position of the virtual control element and whether the operating input matches the actuation option of the virtual control element. If both the position of the operating input matches the position of the virtual control element and the operating input matches the actuation option of the virtual control element, optical feedback is output to the user via the data glasses.This allows the activation of the virtual control element to be registered and visually signaled to the user. It is also possible to provide additional acoustic feedback to the user, such as a click, a noise, and / or a tone.

[0015] To name just a few examples, a push button can be represented that can assume two states in the MR application, for example, which can be represented as visually different colors, brightnesses, or illumination states in the MR application. Furthermore, a decorative strip can be visualized as a toggle switch that can be pressed in two directions to control a continuous value. Pressure on the decorative strip can be detected by the sensor device, thus making it possible to capture a targeted operating input of the virtual control element.

[0016] The control panel section can be part of the decorative element, i.e., a strip, a panel, a cover, and / or a relief. The decorative element can serve to decorate or enhance a surface and / or component of the motor vehicle. Furthermore, the decorative element can be distinguished by a special surface and / or a special haptic quality. The virtual control element can thus be presented to the user via the data glasses in such a way that it is attached or present at the location of the decorative element. Control devices exist as interfaces between a person and a machine, for example as switches or buttons. A sensible and intuitively understandable arrangement of control devices can be considered ergonomics. Most control devices today are electrical or electronic because they are generally cost-effective in this version.However, there are also pneumatic, hydraulic, or mechanical control devices. Control devices can be classified according to the type of actuation.

[0017] Interior trim generally refers to all components visible to the driver and passengers in the vehicle's interior. The trim parts and elements in the vehicle interior can meet not only aesthetic but also functional requirements, such as high scratch resistance. Depending on the requirements, various materials such as aluminum, carbon fiber, wire mesh, fine wood, leather, textiles, and various plastics can be used.

[0018] Virtual reality, or VR for short, can be described as the representation and simultaneous perception of an apparent reality and its physical properties in a real-time, computer-generated, interactive virtual environment. To create a sense of immersion, special output devices (so-called virtual reality headsets or data glasses) are required to display virtual worlds. To convey a spatial impression, two images from different perspectives are generated and displayed (stereo projection).

[0019] Real-time refers to the operation of a computer system in which programs for processing incoming data are constantly operational, ensuring that the processing results are available within a specified time period. Depending on the application, the data can be generated randomly or at predetermined times. Hardware and software can ensure that no delays occur that could prevent this condition from being met. Data processing does not have to be particularly fast; it can only be guaranteed to be fast enough for the respective application.

[0020] Augmented reality (AR) is the computer-assisted enhancement of a perception of reality. This information can address all human sensory modalities. However, AR is often understood only as the visual representation of information, i.e., the supplementation of images or videos with computer-generated additional information or virtual objects by means of overlay and / or overlay.

[0021] Mixed reality (MR) refers to environments or systems that combine a user's natural perception with an artificial (computer-generated) perception. In addition to the primarily computer-generated virtual reality, these include, in particular, augmented reality and augmented virtuality systems. Mixed reality encompasses the entire "reality-virtuality continuum" with the exception of "only reality" and "only virtuality": Between the two extremes of "only reality" and "only virtuality," there are infinitely variable intermediate states that blend both. In particular, augmented reality and augmented virtuality are specific manifestations of the mixed reality principle.

[0022] The invention provides a solution that does not require eye contact for operation. The virtual control element offers intuitive haptics and visual feedback via the smart glasses. This improves both the sense of safety during operation and a comfortable experience for the user.

[0023] The functionality provided in this way enables a high degree of speed and accuracy for precise detection of an actuation. The communication concept utilizes the existing channels of the data glasses, which can optically track the user's hand movements. The invention is thus more cost-effective and flexible than previous MR command devices. Operation is also safer, more familiar, and more comfortable than with other MR command devices.

[0024] The method according to the invention thus has the advantage that a virtual control element can be easily arranged in the interior of a motor vehicle, and it is very easy to define and configure a virtual control element on the control panel. Because the control panel is a large control panel in which a sensor device is arranged, a virtual control element can be easily defined that provides a special haptic and optical feedback to the user via the data glasses.

[0025] The invention also includes further embodiments which provide additional advantages.

[0026] One embodiment provides that the virtual control element is presented to the user using optical information via the data glasses, as if the virtual control element were located on the control panel section. Various methods can be used for this, such as LED indicators or displays placed on the haptic functional surfaces. These visual elements can display information about the type of touch, pressure force, or other relevant parameters. Thus, the virtual control element is clearly displayed to the user, and when optical feedback is provided via the data glasses, the virtual control element is clearly signaled.

[0027] A further advantageous embodiment provides that the control panel has a haptically perceptible surface structure that is suitable for haptically supporting a representation of the position and actuation options of the virtual control element in the MR application. This means that the virtual control element is arranged such that a haptically perceptible surface structure is used on the control panel to implement the virtual control element. A special embossing, a special pattern, a special material, or a special texture can be used here. For example, a decorative strip can also be represented as a slide switch or toggle switch. This embodiment has the advantage that existing surface structures can be used as virtual control elements, thus providing a particularly advantageous immersive experience for the user.

[0028] A further advantageous embodiment provides that if both the position of the operating input matches the position of the virtual operating element and the operating input matches the actuation option of the virtual operating element, an actuation of the virtual operating element is registered, and an MR application function associated with the virtual operating element and / or the interaction is executed. This means that not only is visual feedback provided, but an MR application function associated with the virtual operating element is also executed. This has the advantage that an operating input is displayed to the user on the control panel as an actuation of the virtual operating element, and feedback is returned to the user from the MR application by executing the associated application function.

[0029] A particularly advantageous embodiment provides that the control panel section is a part of the control panel that is configured to detect a touch operation at any position of the control panel and is configured to allow reconfiguration or any desired placement of individual virtual control elements within the control panel. This means that the control panel is configured to allow multiple individual virtual control elements within the control panel, and it is possible to easily reconfigure them as needed. A further advantageous embodiment provides that the sensor device uses a resistive, capacitive, projected-capacitive, inductive, projected-inductive, optical, surface wave-based, and / or bulk wave-based touch detection technology.For example, resistive layers in rectangular designs are readily available and inexpensive to install, making it easy to create a large-area control panel. This makes for a particularly simple implementation of the sensor device.

[0030] According to a further advantageous embodiment, further processing of the detected operation is provided within the framework of the MR application or an external MR application. This means that the MR application can be processed on an electronic control unit of the vehicle, or means that the detected operation can be forwarded to an external control unit that executes the MR application. External can also mean that processing is performed by a server device located outside the motor vehicle, and communication between the data glasses and the server device is wireless. This has the advantage that flexible data processing is possible and a particularly high processing load can be easily implemented.

[0031] According to a further advantageous embodiment, the control panel of the MR application provides information about the touch history of the operating input. This means that, for example, gesture recognition can be used to implement multiple assignments of the operating input, thus enabling better use of the virtual control element or the control panel.

[0032] According to a further advantageous embodiment, the control panel of the MR application provides information about the pressure strength of the operating input. This means that improved feedback can be achieved and multiple assignments of the operating input can be realized, for example, through different pressure strengths. This allows for a better user experience of the virtual control element or the control panel.

[0033] According to the invention, a motor vehicle is also provided with at least one control panel inside the motor vehicle, which is configured to control a vehicle function and / or a mixed reality application function (MR application function) in a mixed reality application through an operator input on the control panel. The control panel is arranged behind a decorative element, and the motor vehicle has a control device configured to execute the MR application and output optical information to a user of the MR application via data glasses.The MR application includes a virtual operating element, wherein the control device is configured to cause the data glasses to output optical information such that it at least partially overlaps with at least a portion of the operating panel for displaying the virtual operating element in the MR application, wherein the operating panel portion is formed as part of the decorative element. The operating panel includes a sensor device configured to detect an operating input on the operating panel and to check whether a position of the operating input corresponds to a position of the virtual operating element and whether the operating input corresponds to an actuation option of the virtual operating element.The control device is designed to control or cause the data glasses to output optical feedback to the user if both the position of the operating input matches the position of the virtual operating element and the operating input matches the actuation option of the virtual operating element.

[0034] For use cases or application situations that may arise during the method and which are not explicitly described here, it may be provided that, in accordance with the method, an error message and / or a request to enter user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0035] The invention also includes the control device for the motor vehicle. The control device can have a data processing apparatus or a processor device that is configured to carry out an embodiment of the method according to the invention. For this purpose, the processor device can have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor device can have program code that is configured to carry out the embodiment of the method according to the invention when executed by the processor device. The program code can be stored in a data memory of the processor device.The processor device can be based, for example, on at least one circuit board and / or on at least one SoC (System on Chip).

[0036] The invention also includes further developments of the method according to the invention that have features already described in connection with the further developments of the method according to the invention. For this reason, the corresponding further developments of the motor vehicle according to the invention are not described again here.

[0037] The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0038] As a further solution, the invention also encompasses a computer-readable storage medium comprising program code which, when executed by a computer or computer network, causes the computer to carry out an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data memory (e.g. as a flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data memory (e.g. as a RAM - random access memory). The storage medium can be arranged in the computer or computer network. However, the storage medium can also be operated, for example, as a so-called app store server and / or cloud server on the Internet. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code may be provided as binary code and / or as assembly code and / or as source code of a programming language (e.g. C) and / or as a program script (e.g. Python).

[0039] The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

[0040] Exemplary embodiments of the invention are described below. Shown are:

[0041] Fig. 1 is a schematic side view of the interior of a motor vehicle; and

[0042] Fig. 2 is a schematic representation of a control panel during an operator input.

[0043] The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0044] In the figures, the same reference symbols designate elements with the same function.

[0045] Fig. 1 shows a schematic side view of the interior of a motor vehicle, showing a passenger or user 16 looking at the interior of the motor vehicle using data glasses 18. The data glasses 18 can be used to display optical information, for example as a virtual display 12, in order to reproduce virtual content. The data glasses 18 can also be used to display the dashboard in such a way that a virtual operating element 10 is shown on a control panel of the motor vehicle. The control panel is provided with a sensor layer that serves to detect an operating input from the user 16. The virtual operating element 10 in the MR application at least partially overlaps with at least a section of the control panel in order to display the virtual operating element 10 in the MR application using the data glasses 18.The control panel section, which is represented as the virtual control element 10 by means of the data glasses 18, is designed as part of a decorative element. Using the sensor device included in the control panel, an operating input on the control panel can be detected, and it can be checked whether a position of the operating input corresponds to a position of the virtual control element 10, and whether the operating input corresponds to an actuation option of the virtual control element 10. If, for example, a virtual control element 10 is provided in the form of a push button, one actuation option of the virtual control element 10 is, for example, pressing this virtual control element 10. If, for example, an operating input is detected that does not correspond to an actuation option of the virtual control element 10, this can be ignored, and no visual feedback is output to the user 16 via the data glasses 18.Furthermore, no corresponding application function is executed. However, if both the position of the operator input matches the position of the virtual operating element 10 and the operator input matches a possible actuation of the virtual operating element 10, optical feedback can be output to the user 16 via the data glasses 18. The virtual operating element 10 can thus be visualized to the user 16 using optical information or using the data glasses 18, as if the virtual operating element 10 were arranged on the control panel section. If both the positions match and the operator input matches the actuation possibility of the virtual operating element 10, an MR application function associated with the virtual operating element 10 or an interaction can be executed.The control panel section is a part of the larger control panel that is configured to allow and detect touch operation at any position of the control panel, and the control panel is configured to allow reconfiguration of individual virtual control elements within the control panel.

[0046] A detected operation or actuation of the virtual control element 10 can be performed within the MR application or within an external MR application. Fig. 1 also shows a line of sight 14, which corresponds to a line of sight from the data glasses 18 to the virtual control element 10. An operating input on the control panel or on the virtual control element 10 can be processed by an electronic control unit 22.

[0047] Fig. 2 shows a schematic representation of a control panel, which in this case comprises a decorative element 24 and a sensor layer 26. By means of the data glasses 18, a virtual control element 10 can be displayed to the user 16 within or at least on a part of the decorative element 24. In the case of a touch input, for example by means of a finger 28, a touch actuation on the sensor layer 26 arranged directly behind the decorative element can be detected and detected by the sensor layer 26. The virtual control element 10 corresponds to a section 30 of the sensor surface, which is the part of the sensor surface or control panel that lies behind the virtual control element 10. The sensor layer 26 can use a resistive, capacitive, projected-capacitive, inductive, projected-inductive, optical, surface wave-based and / or bulk wave-based touch detection technology.The sensor layer 26 is preferably a resistive touch-sensitive layer. This has the advantage of being a cost-effective, easy-to-implement sensor layer 26. The sensor layer 26 can provide information about the touch history of the operating input. This enables, for example, the detection of gestures or multiple assignments of the operating input. Furthermore, it can be provided that information about the pressure strength of the operating input is provided to the MR application. This enables, for example, multiple assignments of the operating input of the virtual control element 10. Furthermore, this can, for example, offer a continuously running adjustment option in which a parameter can be continuously adjusted via a pressure strength.

[0048] Overall, the examples show how the invention can provide an improved control element 10 for an MR application.

[0049] The invention relates to a method for controlling a mixed reality application function, MR application function in the interior of a motor vehicle in a mixed reality application, MR application, wherein the motor vehicle has at least one control panel arranged behind a decorative element 24, and an MR application function is controlled by an operating input on the control panel, wherein the MR application outputs optical information to the user 16 of the MR application by means of data glasses 18, a virtual control element 10 is included in the MR application, the optical information is at least partially superimposed on at least one section 30 of the control panel for displaying the virtual control element 10 in the MR application by means of the data glasses 18, the section 30 is designed as part of a decorative element 24,an operating input on the operating panel is detected by a sensor device 26 included in the operating panel and, upon detection of a correct operating input, an optical feedback is output to the user 16 via the data glasses 18.

[0050] List of reference symbols:

[0051] 10 Control element

[0052] 12 Display 14 Line of sight

[0053] 16 users

[0054] 18 data glasses

[0055] 22 Control device

[0056] 24 Decorative element 26 Sensor device

[0057] 28 fingers

[0058] Section 30

Claims

PATENT CLAIMS:

1. Method for controlling a mixed reality application function, MR application function inside a motor vehicle in a mixed reality application, MR application, - wherein the motor vehicle has at least one control panel arranged behind a decorative element (24), and an MR application function is controlled by an operating input on the control panel, - wherein the MR application outputs optical information to a user (16) of the MR application by means of data glasses (18), - wherein the MR application includes a virtual control element (10), - wherein the virtual control element is at least partially superimposed with at least one section (30) of the control panel for displaying the virtual control element (10) by means of the data glasses (18), wherein the section is overlapped with a part of the decorative element (24), - wherein an operating input on the operating panel is detected by a sensor device (26) included in the operating panel, and it is checked whether a position of the operating input corresponds to a position of the virtual operating element (10) and whether the operating input corresponds to an actuation possibility of the virtual operating element (10), and - if both the position of the operating input matches the position of the virtual operating element (10) and the operating input matches the actuation option of the virtual operating element (10), optical feedback is output to the user (16) via the data glasses (18).

2. Method according to one of the preceding claims, wherein the virtual control element (10) is displayed to the user (16) with the aid of optical information by means of the data glasses (18) in such a way as if the virtual control element (10) is arranged on the control panel section.

3. Method according to one of the preceding claims, wherein the control panel has a haptically perceptible surface structure which is suitable for haptically supporting a representation of the position and the actuation options of the virtual control element (10) in the MR application.

4. Method according to one of the preceding claims, wherein if both the position of the operating input matches the position of the virtual operating element (10) and the operating input matches the actuation possibility of the virtual operating element (10), an MR application function associated with the virtual operating element (10) and / or an interaction is executed.

5. The method according to any one of the preceding claims, wherein the control panel section is a part of the control panel that is configured to detect a touch operation at any position of the control panel and is configured to allow reconfiguration of individual virtual control elements (10) within the control panel.

6. The method according to any one of the preceding claims, wherein the sensor device (26) uses a resistive, capacitive, projected capacitive, inductive, projected inductive, optical, surface wave-based and / or bulk wave-based touch detection technique.

7. Method according to one of the preceding claims, wherein further processing of the detected operation is provided within the framework of the MR application or an external MR application.

8. The method according to any one of the preceding claims, wherein the control panel of the MR application provides information about a touch history of the operating input.

9. The method according to any one of the preceding claims, wherein the control panel of the MR application provides information about a pressure strength of the operating input.

10. Motor vehicle with at least one control panel inside the motor vehicle, which is configured to control a vehicle function and / or a mixed reality application function, MR application function, in a mixed reality application, MR application, by means of an operating input on the control panel, - wherein the control panel is arranged behind a decorative element (24), - wherein the motor vehicle has a control device (22) which is configured to execute the MR application and to output optical information to a user (16) of the MR application by means of data glasses (18), - wherein the MR application includes a virtual control element (10), - wherein the control device (22) is configured to cause the data glasses (18) to output optical information such that it at least partially overlaps with at least one section (30) of the control panel for displaying the virtual control element (10) in the MR application, wherein the control panel section is formed as part of the decorative element (24), - wherein the control panel comprises a sensor device (26) which is configured to detect an operating input on the control panel and to check whether a position of the operating input corresponds to a position of the virtual operating element (10) and whether the operating input corresponds to an actuation possibility of the virtual operating element (10), and - the control device (22) is designed to, if both the position of the operating input matches the position of the virtual operating element (10) and the operating input matches the actuation option of the virtual operating element (10), To control data glasses (18) to output optical feedback to the user (16).

Images