Method, operator control system, motor vehicle, and computer program for coupling a portable mobile terminal to an operator control device of a motor vehicle

Abstract

The invention relates to a method for coupling a portable mobile terminal (24) to an operator control device of a motor vehicle (10), wherein the operator control device operates a screen (13) and the terminal (24) runs an application software (29) that provides a graphical user interface. According to the invention, a raw data channel to a screen control unit (18) of the screen (13) is provided by coupling software (19) of the operator control device of the motor vehicle (10) via a wired and / or radio-based and / or optical data connection (25) and exchange of raw data with the screen control unit (18) is carried out by means of the application software (29) via the raw data channel, by means of which data exchange a design of the user interface is directly defined on the screen (13) and / or processing of touch inputs is carried out directly by the application software (29).

Description

[0001] AUDI AG

[0002] Method, operating system, motor vehicle and computer program for coupling a portable, mobile terminal with an operating device of a motor vehicle

[0003] DESCRIPTION:

[0004] The invention relates to a method for coupling a portable, mobile terminal, such as a smartphone, to an operating device of a motor vehicle, for example to an infotainment system (information and maintenance system), such as may be permanently installed in the motor vehicle. Through this coupling, application software of the terminal can display its graphical user interface (GUI) on a screen of the operating device of the motor vehicle, for example on a console screen. The invention also includes a computer program for implementing corresponding application software in the terminal, as well as a motor vehicle with an operating device and an operating system comprising the motor vehicle and the terminal.

[0005] When using a portable, mobile terminal, such as a smartphone, in a motor vehicle, it has proven desirable not to carry out the graphical outputs and the touchscreen inputs (touchscreen) in a motor vehicle on the terminal itself, but to couple the terminal with an operating device of the motor vehicle in order to use its screen, preferably a touchscreen, for the graphical user interface (GUI) of the application software.

[0006] A coupling technology is known for this purpose which enables a wired or wireless data exchange between the application software of a terminal device on the one hand and a control unit of the motor vehicle on the other, in order to display a graphical user interface on a screen of the control unit of the vehicle, via which the application software can be controlled. An example of such a solution is the CarPlay® product from Apple Inc.®. However, this graphical user interface is not a customizable component of the application software itself; rather, it consists of ready-made, unchangeable specifications, and the application software can only select from a limited number of control elements (templates) that the application software must accept as default.The application software can only select from the predefined user interface templates and receives notification of the resulting operating events, such as a signal indicating that the user has operated a control element displayed on the screen. This limitation to predefined control elements (from the templates) limits the design options when operating a mobile device via a vehicle's control device.

[0007] The invention is based on the object of making a graphical user interface of an application software of a portable, mobile terminal available on a screen of an operating device of a motor vehicle.

[0008] The problem is solved by the subject matter of the independent patent claims. Advantageous further developments of the invention are revealed by the dependent patent claims, the following description, and the figures.

[0009] As one solution, the invention comprises a corresponding method for coupling a portable, mobile terminal device to an operating device of a motor vehicle. As already described, the terminal device can be a smartphone, a tablet PC, a smartwatch, or data glasses, to name just a few examples. The said operating device of the motor vehicle can be a so-called infotainment system. The screen of the operating device can, for example, be arranged in a conventional manner on a center console of the motor vehicle (console screen) or it can be a component of a head-up display.

[0010] In order to enable application software running on the terminal device to display a freely configurable graphical user interface (GUI) on this screen, the invention provides that operating software or coupling software of the operating device, i.e. a coupling module, provides a "raw data channel" to a screen control unit of the screen via a data connection through the operating device of the motor vehicle. This data connection can, in a manner known per se, comprise a cable (e.g. USB - Universal Serial Bus) and / or a radio connection (e.g. using Bluetooth) and / or WLAN (Wireless Local Area Network) and / or an optical connection (e.g. using LiFi). According to the invention, a data transmission is thus provided, which is referred to here as a raw data channel.This couples the application software with the screen control unit of the monitor, for example with a graphics card and / or a graphics interface of the operating system of the operating device. This means that the application software has the option of setting the pixel values ​​of individual pixels on the screen in the screen control unit and / or receiving a touch signal from a touch sensor on a touchscreen or a touchpad on the operating device directly in the application software. This can of course be done with more complex commands or a graphics library, i.e. the raw data channel can be used directly to receive the pixel values ​​of images or finished control elements and / or the actual drawing commands for drawing or setting the pixel values, in particular drawing commands for graphics primitives such as lines, circles and / or rectangles and / or characters, to name just a few examples.The application software thus exchanges raw data with the screen control unit via the raw data channel. This "raw data" refers, as described, to the pixel-by-pixel control of the screen's pixel values ​​(as described above) and / or (in the opposite direction) to the touch signals from the screen's touch sensors, if the screen is a touchscreen. The coupling software required for this in the operating device itself therefore only passes data (namely the raw data) between the screen control unit of the operating device and the application software of the end device. In particular, the coupling software itself does not have to provide a predefined or predefined control element.Instead, the application software itself provides the drawing commands and / or the finished graphic data for displaying control elements (e.g., buttons, sliders, or text input fields) and / or images (e.g., rendered road maps) as part of the "raw data." Through the raw data exchange, the application software itself directly designs the user interface (with its control elements) and / or processes touch inputs (e.g., the touch signals from the touch sensors) on the screen.In other words, the control of the screen content and / or the processing of touch signals takes place directly in the application software. The coupling software in the operating device, on the other hand, merely represents a coupling to make the screen control unit of the screen accessible to the application software of the end device, so that the application software can directly control the screen control unit itself, as if it were built into the end device itself. The coupling software in the operating device can behave as if it were local application software of the operating device itself, which wants to set pixels directly in the screen control unit and / or interpret touch coordinates from touch signals, so that the error! Text mark is not defined.It behaves like local application software in relation to the operating system of the operating device, which directly defines the screen content, i.e., pixel by pixel, and / or checks the touch signals themselves to determine which operating event or touch event has occurred. However, the logic for generating the raw data (setting pixels and / or processing touch signals) is located in the application software, which is why the coupling software only passes the raw data through. Alternatively, the operating software can be designed as a component of the operating system of the operating device.

[0011] With respect to the application software, the coupling software acts like a gateway, a simulation, or an interface to the screen control unit of the display. For example, a software interface (API - Application Programming Interface) can be simulated by the coupling software via the raw data channel, such as one that may be provided in the operating system of the end device itself for manipulating or controlling the screen of the end device itself.

[0012] The invention provides the advantage that the application software itself can directly set or specify the screen content on the screen of the operating device via the raw data channel, by controlling the screen content pixel-by-pixel using drawing commands and / or graphic data from finished images from the application software itself. Thus, in particular, no prefabricated control elements are selected from a template; instead, the application software can, for example, specify or specify the appearance or layout of GUI control elements on the screen from the mobile device using its own software library.In the same way, a touch input, such as that signaled by a touch signal from a touch sensor on a touchscreen, can be checked in the application software itself to determine which touch event or operating event corresponds to it, for example, which operating element was selected. The touch signal therefore only needs to signal the touch coordinates. An evaluation as to which operating element was touched can be carried out in the application software in the end device. "Application software" is understood in particular to mean software that provides a function that can be used directly by a user, such as playing media (music and / or video), browsing the internet, reading emails, social media, or using a portal, to name just a few examples.In particular, in connection with the invention, it can be provided that a so-called health monitoring (health observation) is carried out as application software, wherein the characteristic data then describe at least one said biosensory and / or psychological characteristic and in this way the influence of a journey on the observed person, in particular the driver, is determined.

[0013] The invention also includes further developments which result in additional advantages.

[0014] According to a further development, the said raw data exchange involves the application software generating raw pixel data for setting pixel values ​​of screen pixels pixel by pixel, so that the color and / or brightness of the respective pixel can be set individually or independently of other pixels by the application software itself. Memory cells of a graphics memory can then be set from pixel values ​​of the screen via the raw data channel. The raw pixel data thus includes, as already described, drawing commands and / or graphics memory values ​​(pixmaps), both of which are also understood here as "raw pixel data," in contrast to the predefined control elements of a predefined template. Thus, the screen pixels either respond directly to the raw pixel data from the application software, or their pixel values ​​are set directly according to values ​​from the drawing command.This offers the advantage that the output area or image area controlled or managed by the application software on the screen can be defined pixel by pixel or pixel-precisely by the application software. Examples of raw pixel data include drawing commands (DrawLine, DrawRect, SetPixel) and pixel arrays (e.g., bitmaps). The screen output can be configured as a so-called view or framebuffer and / or made directly controllable or accessible to the application software via a system call (syscall) of the operating system by the coupling software. The application software can thus define graphic content in the screen output area by setting pixel values ​​and / or executing drawing commands within the application software itself.

[0015] A further development of this approach involves granting the raw data channel access only to a portion of the screen's graphics memory cells that is restricted to the application software. This advantageously prevents the application software from interfering with or distorting other image content displayed on the screen simultaneously alongside the application software's GUI.

[0016] A further development provides that the raw data exchange includes the application software receiving touch coordinates and mapping them to operating events of GUI operating elements. In other words, the application software itself enables the processing of the touch signal with the touch coordinates, so that it can derive or recognize operating events defined in the application software. Thus, one is advantageously not limited to those operating events that would result from predefined operating elements of the operating device itself. For this purpose, the raw data channel advantageously transmits the coordinate memory content of a coordinate memory of the touch coordinates of the screen's touch sensors, or this coordinate memory is read out via the raw data channel.The touch coordinates can be absolute coordinates on the screen or relative coordinates with respect to the output area on the screen accessible to the application software. Such a display area or output area is also referred to as a "window" of the user interface of an application software. This can be a subset of the total display area provided by the screen or the entire screen (so-called full-screen mode). The limitation can be implemented by allocating a memory frame for the application software, so that the remaining graphics memory is not addressable by the application software at all.

[0017] A further development involves the application software generating a sequence of multiple display pages of the user interface on the screen. In other words, the application software can now advantageously generate or program a user interface that changes or switches over time, since there is no longer any need to rely on predefined templates. This can be advantageously used to design a so-called user journey.

[0018] A further development involves making the screen output on the end device's screen independent of the user interface on the control device's screen, as the application software controls or supplies the raw data channel with its own graphics data, independent of the end device's screen. This is therefore not a simple mirroring or copying of screen content from the end device's screen to the control device's screen.

[0019] A further development involves providing one or more selectable templates for a respective standard gill with control elements. These templates are selected by the application software and then displayed by the control device via the screen control unit. In the case of a touchscreen, the touch coordinates of the touch sensors are converted by the control device into the control events of the control elements of the currently displayed standard gill, and only these are signaled to the application software. In other words, the conversion from touch coordinates to the evaluation of which control element was touched or operated takes place in the control device itself, and only the final control event (e.g., "Button XY was pressed") is signaled to the application software.This advantageously reduces data traffic between the operating device and the application software.

[0020] For use cases or application situations that may arise during the method and which are not explicitly described here, it may be provided that, in accordance with the method, an error message and / or a request to enter user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0021] In order to implement an embodiment of the method in a portable, mobile terminal, the invention comprises, as a further solution, a computer program for such a portable, mobile terminal, wherein the computer program comprises program instructions which, when executed by a processor circuit of the terminal, cause the latter to operate a graphical user interface on a screen of an operating device of a motor vehicle via a raw data channel, in which a raw data exchange is carried out with the screen control unit of the screen via the raw data channel, by means of which a design of the user interface and / or a processing of touch inputs is carried out directly on the screen by the application software itself.In the manner described, the application software itself controls the appearance and / or the pixel values ​​on the screen for displaying operating elements via the raw data channel by means of the raw data exchange. Touch coordinates are interpreted by the application software itself to determine which operating event (which operating element was operated) has occurred.

[0022] As a further solution, the invention also encompasses a computer-readable storage medium comprising the computer program which, when executed by a computer or computer network (generally a processor circuit), causes the computer to carry out an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data memory (e.g., as a flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data memory (e.g., as a RAM - random access memory). The storage medium can be arranged in the computer or computer network. However, the storage medium can also be operated on the Internet, for example, as a so-called app store server and / or cloud server. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code may be provided as binary code and / or as assembly code and / or as source code of a programming language (e.g. C) and / or as a program script (e.g. Python).

[0023] Conversely, as a further solution, the invention also encompasses the motor vehicle with the operating device, which has said screen, for example, in the center console with its screen control unit, and a processor circuit with coupling software comprising program instructions that, when executed by the processor circuit of the operating device, cause it to provide said raw data channel to the screen control unit of the screen to application software of a portable, mobile terminal device via a wired and / or radio-based and / or optical data connection. Thus, the terminal device can directly control the screen control unit itself via this raw data channel, i.e., set pixel values ​​for the screen and / or receive touch coordinates from a touch sensor system of the screen in the case of a touchscreen.

[0024] Furthermore, the invention comprises as a solution an operating system that is composed of the described motor vehicle and a mobile terminal with the described computer program for the terminal. The respective processor circuit can for this purpose have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit) or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor circuit can have program code that is configured to carry out the respective embodiment of the described method when executed by the processor circuit. The program code can be stored in a data memory of the processor circuit. The processor circuit can, for example,based on at least one circuit board and / or on at least one SoC (System on Chip).

[0025] The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0026] The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

[0027] Exemplary embodiments of the invention are described below. The single figure shows:

[0028] Fig. a schematic representation of an embodiment of the operating system according to the invention, which can carry out an embodiment of the method according to the invention.

[0029] The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0030] In the figures, the same reference symbols designate elements with the same function.

[0031] The figure shows a perspective view of the interior of a motor vehicle 10 as it might appear from the driver's perspective. For orientation, the following are shown: a windshield 11 and a steering wheel 12. For example, a screen 13 of an MMI (man-machine interface) operating device of the motor vehicle 10, such as may be implemented by an infotainment system, can be located in a center console of the motor vehicle 10. A user interface GUI can be shown on the screen 13, which can include, for example, an operating element 14 that a user can operate, for example, with their finger 15 by touching or touching. A touch sensor system 16 can detect touch coordinates of the touch point of the finger 15 on the screen 13.The control of the image content of the screen 13 for the GUI user interface and / or the reading of the touch sensor system 16 can be carried out, for example, by a screen control unit 18, which can be part of a processor circuit 17 of the MMI operating device. For example, the screen control unit 18 can comprise a graphics memory 20 for defining pixel values ​​of individual pixels of the screen 13 and / or a coordinate memory 21 for the touch coordinates (signaled by the touch signal of the touch sensor system 16). The GUI user interface can be defined in the motor vehicle 10 by a portable, mobile terminal 24, which can have a processor circuit 28, by which application software 29 can be executed, which can define the image content or the appearance of the GUI user interface on the screen 13 using graphics commands.For this purpose, the terminal device 24 can be coupled to the processor circuit 17 or generally to the MMI operating device via a data connection 25, via which raw pixel data 26, for example, for defining the memory content of the graphics memory 20, can be transmitted independently of one another in a raw data exchange for directly defining individual pixel values. Additionally or alternatively, touch coordinates 27 can be transmitted from the coordinate memory 21 or generally from the touch sensor system 16 for directly reading the touch coordinates (without interpretation as to which operating element 14 was touched). The data connection 25 thus represents a raw data channel.

[0032] Depending on the touch coordinates 27, the application software 29 itself can determine which operating element 14 displayed or drawn by the application software 29 using the raw pixel data 26 on the screen 13 has been operated and can signal a corresponding operating event 30 to an event controller 31 of the terminal device 24 so that the mobile terminal device 24 can be operated. This means that a program associated with the respective operating element 14 can be started depending on the event or depending on which operating element 14 has been operated. For example, a health monitoring system (health data monitoring) and / or a media player (e.g., a music streaming service) can be operated on the terminal device.

[0033] The data connection 25 can be provided by coupling software 19 of the processor circuit 17 to enable or provide the raw data channel between the screen control unit 18, on the one hand, and the application software 29, on the other. The coupling software 19 can be executed on an operating system OS of the processor circuit 17 or be part of the operating system OS (not shown in the figure).

[0034] Thus, it is possible for the application software 29 as a whole to design or define the shape and / or color of an operating element 14 or several such operating elements (for example buttons and / or text fields and / or sliders, to name just a few examples) on the screen 13 freely or independently of the operating device or a design or operating system OS of the operating device MMI and / or to define an assignment of touch coordinates 27 to an operating event 30 freely through the software or logic of the application software 29 itself.

[0035] For this purpose, the coupling software 19 can, for example, provide an API (Application Programming Interface) for the application software 29 via the data connection 25, in order to receive, for example, drawing commands from the application software 29 for so-called graphic primitives or basic shapes and to implement them in the graphics memory 20. Such a basic shape can be, for example: line, area, polygon, circle, shadow, font, to name just a few examples. In particular, these are not composite graphics, such as

[0036] Control elements. For this purpose, a library for graphical control elements can be stored in the application software 29, for example, through which these commands can then be combined or summarized into predefined control elements in the library of the application software 29. Thus, the application software 29 itself can determine which appearance or design should result in the user interface GUI on the screen 13. In particular, this design or these control elements do not have to be available or present in the processor circuit 17 of the MMI operating device itself.

[0037] In contrast, consider Apple Inc.'s CarPlay® technology, which allows smartphone app screens (application software) to be projected onto the screen of a vehicle's control panel. All standard CarPlay apps look largely the same, as there currently appear to be only five screen templates an app can follow to perform its task. This could lead to a poor user experience if the standard templates provided by Apple® are followed.

[0038] On the other hand, the app can now come with its own user interface design, meaning that user interfaces can be designed freely using pixels, whereas a normal CarPlay app has to use one of the existing templates.

[0039] The idea is to use a raw data channel to offer the user a new, immersive user experience that creates a better connection to the control device and avoids confusion with other apps. An application-specific screen design is provided, for example, to enable better user identification with the application software.

[0040] The solution is to use the application software authorization to obtain authorization for a raw data channel in the vehicle's control unit's operating system. This also allows for the extension of the access rights the app should have (e.g., access to health data on the phone or the car manufacturer's CarPlay®, or various others). If an app uses the car manufacturer's CarPlay functionality, additional settings are required to pair the app with the in-car entertainment device. Without this pairing, the CarPlay app (with manufacturer authorization) would not be displayed in the vehicle's control unit when the device is connected to the control unit. When all these steps are combined, a CarPlay car manufacturer app can be made available to the user, enabling customized screen design.At the code level, this requires / enables screens to be implemented individually and separately from phone screens, with similar freedom as one would when developing apps for smartphones themselves. This allows the user to be presented with a customized user journey (sequence of GUIs) on the screen in the vehicle.

[0041] Overall, the examples show how an adaptation of a user interface of a mobile device can be provided in a vehicle operating device.

Claims

PATENT CLAIMS:

1. Method for coupling a portable, mobile terminal (24) to an operating device of a motor vehicle (10), wherein the operating device (MMI) operates a screen (13) and the terminal (24) executes application software (29) which provides a graphical user interface, characterized in that a coupling software (19) of the operating device (MMI) of the motor vehicle (10) provides a raw data channel to a screen control unit (18) of the screen (13) via a wired and / or radio-based and / or optical data connection (25), and the application software (29) carries out a raw data exchange with the screen control unit (18) via the raw data channel, by means of which a design of the user interface is directly defined on the screen (13) and / or processing of touch inputs is directly carried out by the application software (29).

2. The method according to claim 1, wherein the raw data exchange comprises that pixel raw data (26) for setting pixel values ​​of pixels of the screen (13) pixel by pixel are generated by the application software (29) and memory cells of a graphics memory are set by pixel values ​​of the screen (13) via the raw data channel.

3. The method according to claim 2, wherein the raw data channel is authorized to access only a portion of the memory cells that is limited to the application software (29).

4. Method according to one of the preceding claims, wherein the raw data exchange comprises receiving touch coordinates (27) by the application software (29) and responding to operating events (30) of Control elements (14) of the user interface are mapped, and for this purpose a coordinate memory (21) of the touch coordinates (27) of a touch sensor (16) of the screen (13) is read out via the raw data channel.

5. Method according to one of the preceding claims, wherein the application software (29) outputs a sequence of several display pages of the user interface on the screen (13).

6. Method according to one of the preceding claims, wherein the application software (29) supplies the raw data channel with its own graphic data independently of a screen (13) of the terminal (24) and thus a screen output on the screen (13) of the terminal (24) is independent of the user interface on the screen (13) of the operating device.

7. Method according to one of the preceding claims, wherein one or more selectable templates are provided for a respective standard gill with operating elements (14), the selection of which is carried out by the application software (29) and the display of which is then carried out by the operating device via the screen control unit (18) and the touch coordinates (27) are converted by the operating device into the operating events (30) of the operating elements (14) of the currently displayed standard gill and only these are signaled to the application software (29).

8. Computer program for a portable, mobile terminal (24), wherein the computer program comprises program instructions which, when executed by a processor circuit (28) of the terminal (24), cause the latter to operate a graphical user interface on a screen (13) of an operating device of a motor vehicle (10) via a raw data channel, by carrying out a raw data exchange with a screen control unit (18) of the screen (13) via the raw data channel, by means of which on the Screen (13) a design of the user interface and / or a processing of touch inputs is carried out directly by the application software (29).

9. Motor vehicle (10) with an operating device which has a screen (13) with a screen control unit (18) and a processor circuit (17) with coupling software (19) comprising program instructions which, when executed by the processor circuit (17), cause it to provide a raw data channel to the screen control unit (18) of the screen (13) to application software (29) of a portable, mobile terminal (24) via a wired and / or radio-based and / or optical data connection (25).

10. An operating system comprising a motor vehicle (10) according to claim 9 and at least one portable mobile terminal (24) having a computer program according to claim 8.

Images