Method, operating system, motor vehicle and computer program for coupling a portable, mobile device with an operating device of a motor vehicle

Abstract

The invention relates to a method for coupling a portable, mobile device (24) with an operating unit of a motor vehicle (10), wherein the operating unit operates a screen (13) and the device (24) executes application software (29) that provides a graphical user interface. The invention comprises that, by means of coupling software (19) of the operating unit of the motor vehicle (10), a raw data channel is provided to a screen control unit (18) of the screen (13) via a wired and / or a wireless and / or an optical data connection (25), and that the application software (29) performs a raw data exchange with the screen control unit (18) via the raw data channel, thereby directly defining a user interface design on the screen (13) and / or directly processing touch inputs by the application software (29).

Description

[0001] The invention relates to a method for coupling a portable, mobile device, such as a smartphone, with a motor vehicle's control unit, for example, with an infotainment system (information and maintenance system) that may be permanently installed in the vehicle. Through this coupling, the device's application software can display its graphical user interface (GUI) on a screen of the vehicle's control unit, such as a console screen. The invention also includes a computer program for implementing the corresponding application software in the device, a motor vehicle with a control unit, and a control system comprising the motor vehicle and the device.

[0002] When using a portable, mobile device, such as a smartphone, in a motor vehicle, it has proven desirable not to perform the graphical outputs and touchscreen inputs (touchscreen) on the device itself, but to couple the device with a control unit of the motor vehicle in order to use its screen, preferably a touchscreen, for the graphical user interface (GUI) of the application software.

[0003] A known technology exists for this purpose, enabling wired or wireless data exchange between the application software of a terminal device and the vehicle's control unit. This allows a graphical user interface to be displayed on the vehicle's control unit screen, through which the application software can be controlled. An example of such a solution is Apple Inc.'s CarPlay® product. However, this graphical user interface is not a customizable component of the application software itself, but rather consists of predefined, unchangeable templates. The application software only allows selection from a limited set of controls (templates) that it must accept as given.The application software can only select from predefined user interface templates and receives notifications of resulting user events, such as a signal indicating that the user has interacted with a control element displayed on the screen. This limitation to fixed, predefined controls (from the templates) restricts the design possibilities when operating a mobile device via a vehicle's control system.

[0004] The invention is based on the objective of making a graphical user interface of application software of a portable, mobile device available on a screen of an operating device of a motor vehicle.

[0005] The problem is solved by the subject matter of the independent patent claims. Advantageous further developments of the invention result from the dependent patent claims, the following description, and the figures.

[0006] As one solution, the invention comprises a corresponding method for connecting a portable, mobile device to a motor vehicle's control unit. As already described, the device can be a smartphone, a tablet PC, a smartwatch, or smart glasses, to name just a few examples. The motor vehicle's control unit can be an infotainment system. The screen of the control unit can, for example, be arranged in a known manner on a center console of the motor vehicle (console screen), or it can be a component of a head-up display.

[0007] To enable application software running on the terminal device to display a freely configurable graphical user interface (GUI) on this screen, the invention provides that the vehicle's control unit provides operating software or coupling software of the control unit, i.e., a coupling module, via a data connection a "raw data channel" to a screen control unit of the display. This data connection can, in a manner known per se, comprise a cable (for example, USB - Universal Serial Bus) and / or a wireless connection (for example, via Bluetooth) and / or WLAN (Wireless Local Area Network) and / or an optical connection (for example, via LiFi). Thus, according to the invention, a data transmission is provided, which is referred to here as the raw data channel.This connects the application software to the screen's control unit, for example, a graphics card and / or a graphics interface of the operating system of the control device. This allows the application software to set the pixel values ​​of individual pixels on the screen in the control unit and / or to receive a touch signal from a touch sensor of a touchscreen or touchpad of the control device directly into the application software. This can, of course, be done with more complex commands or a graphics library; that is, the pixel values ​​of images or pre-drawn control elements can be directly transmitted via the raw data channel, and / or the actual drawing commands for drawing or setting pixel values ​​can be sent, especially drawing commands for graphic primitives such as lines, circles, rectangles, and / or characters, to name just a few examples.

[0008] The application software thus performs a raw data exchange with the screen control unit via the raw data channel. This "raw data" refers, as described above, to the pixel-by-pixel control of the screen's pixel values ​​and / or (in the opposite direction) to the touch signals from the screen's touch sensor, if it is a touchscreen. The coupling software required for this in the control unit itself therefore only transmits data (namely the raw data) between the screen control unit of the control unit and the application software of the terminal device. In particular, the coupling software itself does not need to provide a pre-defined or predefined control element.Instead, the application software itself provides the drawing commands and / or the finished graphic data for displaying controls (such as buttons, sliders, or text input fields) and / or images (e.g., rendered maps) as part of the "raw data." Through this raw data exchange, the application software directly designs the user interface (with its controls) and / or processes touch input (i.e., the touch signals from the touch sensors) on the screen.In other words, the application software directly controls the screen content and / or processes touch signals. The coupling software in the control unit, on the other hand, merely provides a connection to make the screen's control unit accessible to the application software of the terminal device. This allows the application software to directly control the screen's control unit, as if it were built into the terminal device itself. The coupling software in the control unit can behave as if it were a local application of the control unit itself, directly setting pixels in the screen's control unit and / or interpreting touch coordinates from touch signals. Therefore, the error message "! Bookmark not defined" is not displayed.The interface software behaves like local application software in relation to the operating system of the control device. It directly defines the screen content freely, i.e., pixel by pixel, and / or checks the touch signals to determine which operating event or touch event has occurred. However, the logic for generating the raw data (setting pixels and / or processing touch signals) is located in the application software, which is why the interface software only forwards the raw data. Alternatively, the interface software can be designed as a component of the operating system of the control device.

[0009] With regard to the application software, the coupling software acts as an access point, a simulation, or an interface to the screen's control unit. For example, the coupling software can emulate a software interface (API - Application Programming Interface) via the raw data channel, similar to the interface that the operating system of the end device itself might provide for manipulating or controlling the end device's screen.

[0010] The invention offers the advantage that the application software itself can directly set or define the screen content on the control device's screen via the raw data channel by controlling the screen content pixel by pixel using drawing commands and / or graphic data from finished images within the application software itself. In particular, no pre-defined control elements are selected from a template; instead, the application software can, for example, use its own software library to define or define the appearance, look, or layout of GUI controls on the screen from the mobile device.Similarly, a touch input, such as that signaled by a touch sensor on a touchscreen, can be checked within the application software itself to determine which touch event or operating event corresponds to it, for example, which control element was selected. The touch signal therefore only needs to indicate the touch coordinates. The evaluation to determine which control element was touched can then be performed within the application software on the end device.

[0011] The term "application software" refers in particular to software that provides a function which can be used directly by a user, such as, in particular, the playback of media (music and / or video), browsing the internet, reading emails, using social media, or operating a portal, to name just a few examples. In particular, in connection with the invention, it may be provided that the application software performs so-called health monitoring, whereby the characteristic data describes at least one of the aforementioned biosensory and / or psychological parameters, and the influence of a journey on the observed person, in particular the driver, is determined.

[0012] The invention also includes further developments that result in additional advantages.

[0013] According to a further development, the aforementioned raw data exchange involves the application software generating pixel raw data for setting pixel values ​​of screen pixels on a pixel-by-pixel basis. This allows the application software to define the color and / or brightness of each pixel individually or independently of other pixels. Memory cells of a graphics memory can then be set to pixel values ​​of the screen via the raw data channel. The pixel raw data thus includes, as already described, drawing commands and / or graphics memory values ​​(pixel maps), both of which are also referred to here as "pixel raw data," in contrast to the fixed, predefined controls of a predefined template. Therefore, the screen pixels react directly to the pixel raw data from the application software, or their pixel values ​​are set directly according to values ​​from the drawing command.This offers the advantage that the output area or image area controlled or managed by the application software on the screen can be defined pixel by pixel or with pixel-level precision. Examples of raw pixel data include drawing commands (DrawLine, DrawRect, SetPixel) and pixel arrays (e.g., bitmaps). The screen output can be implemented as a so-called view or framebuffer and / or be directly controllable or accessible to the application software via a system call (syscall) from the operating system. Thus, the application software can define graphical content in the screen output area by setting pixel values ​​and / or executing drawing commands within the application software itself.

[0014] A further development of this involves granting the raw data channel access only to a subset of the graphics memory cells of the screen, limited to the application software. This advantageously prevents the application software from interfering with or distorting other image content displayed on the screen simultaneously with the application software's GUI.

[0015] A further development process envisages that the raw data exchange includes the application software receiving touch coordinates and mapping them to user interaction events of GUI elements. In other words, the application software itself processes the touch signal along with the touch coordinates, enabling it to derive or recognize user interaction events defined within the application software. This advantageously avoids limiting the user interaction events to those that would result from the user interface's own predefined controls. For this purpose, the raw data channel advantageously transmits the contents of a coordinate memory containing the touch coordinates of the screen's touch sensor, or this coordinate memory is read directly from the raw data channel.The touch coordinates can be absolute coordinates on the screen or relative coordinates with respect to the output area on the screen that is accessible to the application software. Such a display or output area is also referred to as a "window" of the application software's user interface. This can be a portion of the total display area provided by the screen or the entire screen (so-called fullscreen mode). The limit can be implemented by allocating a memory frame to the application software, so that the remaining graphics memory is not accessible to the application software at all.

[0016] Further development involves the application software displaying a sequence of multiple user interface pages on the screen. In other words, the application software can now advantageously generate or program a user interface that changes or switches over time, as it is no longer dependent on fixed templates. This can be used to advantage in designing a so-called user journey.

[0017] A further development involves making the screen output on the terminal device's screen independent of the user interface on the control device's screen. This is achieved by the application software controlling or supplying the raw data channel with its own graphic data, independent of the terminal device's screen itself. Therefore, it is not simply a matter of mirroring or copying screen content from the terminal device's screen to the control device's screen.

[0018] A further development involves providing one or more selectable templates for a standard GUI with control elements. The application software selects these templates, and the operating device then displays them via the screen control unit. Furthermore, in the case of a touchscreen, the operating device converts the touch coordinates from the touch sensor into the operating events of the controls in the currently displayed standard GUI, and only these events are signaled to the application software. In other words, the conversion of touch coordinates to determine which control element was touched or operated takes place within the operating device itself, and only the final operating event (e.g., "Button XY was pressed") is signaled to the application software.This advantageously reduces data traffic between the operating device and the application software.

[0019] For use cases or application situations that may arise during the procedure and are not explicitly described here, it may be provided that, according to the procedure, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0020] To implement an embodiment of the method in a portable, mobile device, the invention further comprises a computer program for such a portable, mobile device, wherein the computer program includes program instructions which, when executed by a processor circuit of the device, cause it to operate a graphical user interface on a screen of an operating device of a motor vehicle via a raw data channel, in which a raw data exchange is carried out with the screen control unit of the screen via the raw data channel, through which a design of the user interface and / or a processing of touch inputs is carried out directly on the screen by the application software itself.In the manner described, the application software itself controls the appearance and / or pixel values ​​on the screen via the raw data channel using raw data exchange. These pixel values ​​are used to display control elements and / or touch coordinates, which are interpreted by the application software itself to determine which operating event (which control element was activated) has occurred.

[0021] As a further solution, the invention also includes a computer-readable storage medium comprising the computer program which, when executed by a computer or a computer network (generally a processor circuit), causes it to execute an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data storage medium (e.g., as flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data storage medium (e.g., as RAM - random access memory). The storage medium can be located within the computer or computer network. However, the storage medium can also be operated, for example, as an app store server and / or cloud server on the internet. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code can be provided as binary code and / or as assembly code and / or as source code of a programming language (e.g. C) and / or as a program script (e.g. Python).

[0022] Conversely, as a further solution, the invention also includes the motor vehicle with the operating device, which comprises the said screen, for example in the center console with its screen control unit, and a processor circuit with coupling software. This software includes program instructions which, when executed by the processor circuit of the operating device, cause it to provide the said raw data channel to the screen control unit via a wired and / or wireless and / or optical data connection to application software of a portable, mobile device. Thus, the device can directly control the screen control unit itself via this raw data channel, i.e., set pixel values ​​for the screen and / or, in the case of a touchscreen, receive touch coordinates from a touch sensor of the screen.

[0023] Furthermore, the invention comprises as a solution an operating system that consists of the described motor vehicle and a mobile terminal with the described computer program for the terminal.

[0024] The respective processor circuit can comprise at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor circuit can contain program code configured to execute the respective embodiment of the described method when executed by the processor circuit. The program code can be stored in a data memory of the processor circuit. The processor circuit can be based, for example, on at least one circuit board and / or on at least one SoC (System on Chip).

[0025] The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0026] The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

[0027] The following describes exemplary embodiments of the invention. The single figure illustrates this: Fig. a schematic representation of an embodiment of the operating system according to the invention, which can carry out an embodiment of the method according to the invention.

[0028] The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0029] In the figures, the same reference symbols designate elements with the same function.

[0030] The figure shows a perspective view of the interior of a motor vehicle 10, as it might appear from the driver's point of view. For orientation, a windshield 11 and a steering wheel 12 are shown. For example, a screen 13 of a human-machine interface (MMI) operating device, such as an infotainment system, may be located in the center console of the motor vehicle 10. The screen 13 may display a user interface (GUI) that may include, for example, a control element 14, which a user can operate, for example, with their finger 15 by touch. A touch sensor 16 can detect the coordinates of the point of contact of the finger 15 on the screen 13.The control of the screen content 13 for the user interface GUI and / or the reading of the touch sensors 16 can be carried out, for example, by a screen control unit 18, which can be part of a processor circuit 17 of the MMI operating device. For example, the screen control unit 18 can include a graphics memory 20 for setting pixel values ​​of individual pixels of the screen 13 and / or a coordinate memory 21 for the touch coordinates (signaled by the touch signal from the touch sensors 16).

[0031] The user interface (GUI) of the vehicle 10 can be defined by a portable, mobile device 24, which may have a processor circuit 28 that can execute application software 29. This software can define the image content or appearance of the GUI on the screen 13 using graphic commands. For this purpose, the device 24 can be connected to the processor circuit 17 or, more generally, to the MMI operating unit via a data connection 25. This connection allows raw pixel data 26, for example, to define the contents of the graphics memory 20, to be transmitted independently in a raw data exchange for directly defining individual pixel values. Additionally or alternatively, touch coordinates 27 from the coordinate memory 21 or, more generally, from the touch sensor 16 can be transmitted for direct reading of the touch coordinates (without interpreting which control element 14 was touched).Data connection 25 therefore represents a raw data channel.

[0032] Depending on the touch coordinates 27, the application software 29 can determine which control element 14, displayed or drawn on the screen 13 by the application software 29 using the raw pixel data 26, has been operated and can signal a corresponding operating event 30 to an event controller 31 of the terminal device 24, thus enabling operation of the mobile terminal device 24. In other words, depending on the event or which control element 14 has been operated, a program associated with the respective control element 14 can be started. For example, health monitoring and / or a media player (e.g., a music streaming service) can be operated on the terminal device.

[0033] The data connection 25 can be provided by coupling software 19 of the processor circuit 17 to enable or provide the raw data channel between the screen control unit 18 on the one hand and the application software 29 on the other. The coupling software 19 can run on an operating system (OS) of the processor circuit 17 or be part of the operating system (OS) (not shown in the figure).

[0034] Thus, the application software 29 is able to design or define the shape and / or color of a control element 14 or several such control elements (for example, buttons and / or text fields and / or sliders, to name just a few examples) on the screen 13 freely or independently of the operating device or a design or operating system OS of the MMI operating device and / or to freely define an assignment of touch coordinates 27 to an operating event 30 by the software or logic of the application software 29 itself.

[0035] The coupling software 19 can, for example, provide an API (Application Programming Interface) for the application software 29 via the data connection 25 in order to receive drawing commands for so-called graphic primitives or basic shapes from the application software 29 and implement them in the graphics memory 20. Such a basic shape could be, for example, a line, a shape, a polygon, a circle, a shadow, or text, to name just a few examples. These are specifically not composite graphics, such as user interface elements. The application software 29 can, for example, contain a library of graphical user interface elements, through which these commands can then be combined or grouped into predefined user interface elements in the application software 29's library.Thus, the application software 29 itself can determine which appearance or design should be displayed in the user interface (GUI) on the screen 13. In particular, this design or these controls do not need to be available or present in the processor circuit 17 of the MMI operating unit itself.

[0036] In contrast, Apple Inc.'s CarPlay® technology allows smartphone app screens to be projected onto the vehicle's control unit screen. All standard CarPlay apps look largely the same, as there currently appear to be only five screen templates that an app can follow to perform its function. This raises concerns about a poor user experience when using the standard templates provided by Apple®.

[0037] In contrast, the app can now come with its own user interface design, meaning that user interfaces can be freely designed using pixels, whereas a normal Carplay app has to use one of the existing templates.

[0038] The idea is to use a raw data channel to offer the user a new, immersive user experience that establishes a better connection to the user interface and avoids confusion with other apps. An application-specific screen design is provided to, for example, enable better user identification with the application software.

[0039] The solution involves using the application software's permissions to obtain access to a raw data channel within the vehicle's operating system. This also allows for the extension of access rights the app should have (e.g., access to health data on the phone or the car manufacturer's CarPlay®, or various others). If an app utilizes the car manufacturer's CarPlay functionality, additional settings are required to pair the app with the in-car entertainment system. Without this pairing, the CarPlay app (with manufacturer permission) would not appear on the vehicle's control panel when the user's device is connected. When all these steps are completed, a CarPlay-compatible car manufacturer app can be provided to the user, enabling a customized screen design.

[0040] At the code level, this requires / enables the implementation of screens individually and separately from phone screens, with similar freedom to that enjoyed when developing smartphone apps. This allows for a customized user journey (sequence of GUIs) to be presented to the user on the vehicle's screen.

[0041] Overall, the examples show how a user interface adaptation of a mobile device can be provided in a vehicle control unit.

Claims

[1] Method for coupling a portable, mobile terminal (24) to an operating device of a motor vehicle (10), wherein the operating device (MMI) operates a screen (13) and the terminal (24) executes application software (29) which provides a graphical user interface, characterized by , that a coupling software (19) of the operating device (MMI) of the motor vehicle (10) provides a raw data channel to a screen control unit (18) of the screen (13) via a wired and / or radio-based and / or optical data connection (25), and a raw data exchange is carried out by the application software (29) via the raw data channel with the screen control unit (18), by means of which a design of the user interface is directly defined on the screen (13) and / or a processing of touch inputs is directly carried out by the application software (29). [2] Method according to claim 1, wherein the raw data exchange comprises that pixel raw data (26) for pixel-by-pixel setting of pixel values ​​of pixels of the screen (13) are generated by the application software (29) and memory cells of a graphics memory are set of pixel values ​​of the screen (13) via the raw data channel. [3] Method according to claim 2, wherein the raw data channel is authorized to access only a portion of the memory cells that is limited to the application software (29). [4] Method according to one of the preceding claims, wherein the raw data exchange comprises that touch coordinates (27) are received by the application software (29) and mapped to operating events (30) of operating elements (14) of the user interface, and for this purpose a coordinate memory (21) of the touch coordinates (27) of a touch sensor system (16) of the screen (13) is read out via the raw data channel. [5] Method according to one of the preceding claims, wherein the application software (29) outputs a sequence of several display pages of the user interface on the screen (13). [6] Method according to one of the preceding claims, wherein the application software (29) supplies the raw data channel with its own graphic data independently of a screen (13) of the terminal (24) and thus a screen output on the screen (13) of the terminal (24) is independent of the user interface on the screen (13) of the operating device. [7] Method according to one of the preceding claims, wherein one or more selectable templates are provided for a respective standard GUI with operating elements (14), the selection of which is carried out by the application software (29) and the display of which is then carried out by the operating device via the screen control unit (18), and the touch coordinates (27) are converted by the operating device into the operating events (30) of the operating elements (14) of the currently displayed standard GUI, and only these are signaled to the application software (29). [8] Computer program for a portable, mobile terminal (24), wherein the computer program comprises program instructions which, when executed by a processor circuit (28) of the terminal (24), cause the latter to operate a graphical user interface on a screen (13) of an operating device of a motor vehicle (10) via a raw data channel by carrying out a raw data exchange with a screen control unit (18) of the screen (13) via the raw data channel, by means of which a design of the user interface and / or a processing of touch inputs by the application software (29) is carried out directly on the screen (13). [9] Motor vehicle (10) with an operating device which has a screen (13) with a screen control unit (18) and a processor circuit (17) with coupling software (19) which comprises program instructions which, when executed by the processor circuit (17), cause it to provide a raw data channel to the screen control unit (18) of the screen (13) to an application software (29) of a portable, mobile terminal (24) via a wired and / or a radio-based and / or an optical data connection (25). [10] Operating system comprising a motor vehicle (10) according to claim 9 and at least one portable mobile terminal (24) which has a computer program according to claim 8.

Images