Systems, methods, computer programs, mobile devices and kits for operating devices with low computing power

Abstract

Systems, methods, computer programs and devices are provided in which a virtual operating system application (22) that essentially operates a device with reduced computing power runs on a mobile device (16). The virtual operating system can allow an application (26A, 26B) for another device to run or act as a proxy for another device controlled by other applications running on the mobile device.

Description

[0001] The present application generally relates to the operation of devices having relatively low computing power.

[0002] Many devices are equipped with a display for displaying content and / or an input device allowing the user to make inputs. Such input devices can use a so-called touch screen in combination with a display. Due to space or cost constraints, some such devices have only relatively low available computing power. For example, in so-called smart glasses, the display can be incorporated into the glasses, but due to the space constraints of such smart glasses and also due to weight considerations, it is difficult to equip such smart glasses with high computing power. However, it is desirable, for example, to provide various information to the user via such a device with low computing power. Such a device with low computing power is also referred to herein as a thin device.

[0003] There are several ways to couple a relatively small device with a mobile device such as a smart phone or tablet computer to enhance capabilities. For example, in a smart watch, an application (referred to herein as an app) is installed in the smart watch, and a corresponding application is installed in the mobile device, and these apps communicate with each other to provide capabilities for the smart watch, such as accessing multimedia files (such as music or videos) of the mobile device, transferring calls of the mobile device, etc.

[0004] However, this solution requires the installation of a corresponding app on a small device, which may not be possible on devices with very low computing power or that otherwise do not allow the installation of apps.

[0005] Another conventional method is to use remote display technology. In this case, the content displayed on a mobile device such as a smart phone or tablet computer is basically mirrored to a device with low computing power and displayed there. For example, Bluetooth or other wireless communications can be used for such screen mirroring. However, this technology also has some disadvantages. For example, devices with low computing power and mobile devices usually have different screen sizes. The content shown on the mobile device is usually designed for the screen of the mobile device, and displaying the same content on other devices may lead to unsatisfactory results. In addition, this method may make it difficult to operate the mobile device essentially independently of the device with low computing power, or in other words, this solution may make it difficult to use different applications on the device.

[0006] Therefore, it is an object to provide an improved solution for operating such thin devices.

[0007] In this regard, there is provided a system as claimed in claim 1, a method as claimed in claim 12, a computer program as claimed in claim 18, a mobile device as claimed in claim 20 and a kit as claimed in claim 21. The dependent claims define further embodiments.

[0008] According to an embodiment, there is provided a system comprising:

[0009] A mobile device comprising an operating system and a first interface,

[0010] Another device, the other device includes a display and a second interface, wherein the mobile device further includes a virtual operating system application running under the operating system of the mobile device, wherein the virtual operating system application is configured to

[0011] - managing a connection between the mobile device and the other device via the first interface and the second interface,

[0012] - providing the screen frame to the other device for display on a display of the other device,

[0013] - receiving and processing input from the other device, and

[0014] - enabling execution of an application for the other device on the mobile device.

[0015] By running a virtual operating system for a device with low computing power on a mobile device, functionality can be provided to the device with low computing power while still being able to operate the mobile device using other applications. In this way, applications customized for devices with low computing power can be provided. In addition, such applications can use input (e.g., sensor input) from both the device with low computing power and from the mobile device (e.g., a sensor of the mobile device, such as a GPS sensor).

[0016] The further device may have less computing power than the mobile device, ie may be a "thin device" as explained above.

[0017] In some embodiments, the other device does not allow installation and running of additional applications. For such another device, applications can be run on the mobile device using the virtual operating system application, either within the scope of the virtual operating system, or as independent applications running with the virtual operating system but communicating with the virtual operating system using an inter-process communication mechanism that may be provided by the operating system of the mobile device. Thus, the virtual operating system can communicate with other applications on the mobile device.

[0018] The virtual operating system may also be configured to send commands and / or configuration information to the other device so that the other device may be substantially fully operated from the mobile device.

[0019] In some embodiments, the virtual operating system application enables the application to communicate with devices (components) of the mobile device so that these devices can be used by the application.

[0020] The virtual operating system application may be configured to receive screen content from the applications and provide the screen frames to the other device based on the screens.

[0021] These applications may include navigation applications.

[0022] The mobile device may be a smartphone or a tablet computer.

[0023] The other device may be smart glasses.

[0024] According to another embodiment, a method is provided, the method comprising:

[0025] On a mobile device, a virtual operating system application is run for another device, wherein the virtual operating system application performs the following operations:

[0026] - managing a connection between the mobile device and the other device via the first interface of the mobile device and the second interface of the other device,

[0027] - providing the screen frame to the other device for display on a display of the other device,

[0028] - receiving and processing input from the other device, and

[0029] - enabling execution of an application for the other device on the mobile device.

[0030] The virtual operating system applications may enable communication of these applications with devices (components) of the mobile device.

[0031] The virtual operating system may send commands and / or configuration information to the other device.

[0032] The virtual operating system can communicate with other applications on the mobile device so that the other applications can, for example, send screens to the other device or receive inputs from the other device via the virtual operating system.

[0033] The virtual operating system application may receive screen content from the applications and provide the screen frames to the other device based on the screen content.

[0034] These applications may include navigation applications.

[0035] According to another embodiment, a computer program is provided comprising a program code which, when executed on a processor, causes the performance of the method as explained above.

[0036] A storage medium, in particular a tangible storage medium, comprising a computer program is also provided.

[0037] Furthermore, a mobile device having a computer program stored thereon is provided.

[0038] According to another embodiment, there is provided a kit comprising:

[0039] The computer program or the storage medium as explained above, and

[0040] Another device operated using the virtual operating system application of the computer program.

[0041] Using such a kit, a user can authorize his or her mobile device to operate the other device by installing a computer program on the mobile device.

[0042] The other device may be smart glasses.

[0043] Hereinafter, in order to provide a more thorough understanding, embodiments will be described with reference to the accompanying drawings.

[0044] Figure 1 is a block diagram of a system according to an embodiment.

[0045] Figure 2 is a block diagram illustrating an operating system architecture of some embodiments including a virtual operating system.

[0046] Figure 3 is a simplified diagram of a system according to another embodiment.

[0047] Figure 4 is a flow chart illustrating a method according to an embodiment.

[0048] Figure 5 is a simplified diagram illustrating an example operation of the system according to an embodiment of a specific application.

[0049] In the following, various embodiments will be described in detail. It should be noted that these embodiments are given as examples only and should not be construed as being restrictive. Features from different embodiments may be combined to form further embodiments. Changes, modifications or details described about one of the embodiments may also be applied to other embodiments, and therefore will not be described repeatedly.

[0050] Before describing the embodiments in detail, some definitions of the terms used below will be given.

[0051] As used herein, a mobile device is a computing device designed to be carried by a person. Therefore, the weight of such a mobile device is less than 2 kg, typically less than 1 kg or less, and the size is less than 50 cm 50 cm 2 cm, typically less than 1 kg. A typical mobile device is a smart phone or a tablet computer. Many mobile devices have a touch screen for displaying content and receiving user input, as well as other sensors such as a GPS sensor. Mobile devices typically have computing power sufficient to run various custom applications (apps). In an embodiment, the mobile device typically runs an operating system that allows the installation of additional such applications.

[0052] Devices with low computing power, referred to herein as thin devices, are devices with lower computing power than mobile devices. Typically, thin devices have only firmware and do not allow installation of custom applications, i.e. additional applications that are not part of the program absolutely necessary to operate the device, or in other words, programs that are not related to the core functionality of the device. The processors used in such thin devices are not as powerful as those used in mobile devices and may be, for example, relatively simple microcontrollers or even simple logic circuits for handling basic input or output.

[0053] In an embodiment, the operating system of the thin device runs as an application on the mobile device. The operating system will also be referred to as a "virtual operating system application" because the operating system does not run on the device (thin device) on which it provides operating system functionality, but on another device (the mobile device). As will be explained in more detail, this allows installation of additional applications for the thin device and / or flexible use of the mobile device in applications at least partially independent of the thin device.

[0054] Figure 1 A system according to an embodiment is shown, and Figure 2 For further demonstration Figure 1 A logical diagram of the operating system structure of a system.

[0055] Figure 1 The system comprises a thin device 10 and a mobile device 16. The thin device 10 comprises a processor 13 coupled to a display 11, one or more input devices 12 and a memory 14. The processor used herein is not limited to a single processor, but may also include the possibility of providing multiple processors.

[0056] As used herein, a display is any device that can be used to display content. For example, in the case of the initially mentioned smart glasses, the display may include a projection system that projects content so that the user of the glasses can view it, or may include a more conventional display, such as a TFT display or a liquid crystal display. An input device may be any device that allows user input. In some embodiments, the input device may be implemented by providing a display 11 that is a so-called touch screen. Other examples of input devices include a camera sensor for sensing gestures of a person operating the thin device 10 or a microphone for receiving voice commands from a person. The memory 14 stores instructions necessary for the processor 13 to operate the thin device 10. In some embodiments, the thin device 10 does not allow custom applications to be installed in the memory 14 or elsewhere in the thin device 10.

[0057] The thin device 10 further comprises an interface 15 for communicating with a mobile device 16. For such communication, the mobile device 16 comprises a corresponding interface 17. Figure 1 In the embodiment of the invention, the interfaces 15, 17 are wireless interfaces. An example of a suitable wireless interface is a Bluetooth interface. However, other kinds of wireless interfaces may also be used. Such interfaces are usually present in commercially available mobile devices such as smartphones or tablet computers. In addition to the devices shown, the thin device 10 may include other components, such as sensors.

[0058] The mobile device 16 includes a processor 18 , one or more other devices 19 , and a memory 110 .

[0059] Other devices 19 may include any devices conventionally used in mobile devices (e.g., smart phones or tablet computers), such as sensors (e.g., GPS sensors), speakers, microphones, displays, touch screens, buttons, etc. Memory 110 includes writable memory, such as flash memory, in which custom applications can be installed. Figure 1 In a system such as the one described above, one of these custom applications is a virtual operating system application that essentially provides an operating system to the thin device 10. Figure 2 Explain this concept further.

[0060] On the side of the thin device 10, a corresponding "thin firmware" 20 is provided. The thin firmware is firmware on the thin device 10, which is essentially limited to providing various functions necessary for the operation of the thin device 10 in the thin device 10. In addition, in the mobile device 16, a virtual operating system application 22 running under the normal operating system (OS) 21 of the mobile device is installed. The thin firmware 20 interacts with the virtual operating system application 22 to provide various functions to the thin device 10.

[0061] The thin firmware 20 has the functionality to pair and unpair the thin device with the mobile device, i.e., use Figure 1 The interfaces 15, 17 in the wireless device establish or terminate a wireless connection. Pairing, especially for Bluetooth communication or other wireless applications, is the process of connecting two devices for the first time.

[0062] In addition, the thin firmware 20 has the function of connecting to and disconnecting from the virtual operating system application 22 running on the mobile device 16. The thin firmware 20 then receives screen frames from the virtual operating system application 22 to display on the display 11 and sends user input from the input device 12 or other data such as sensor data to the virtual operating system application 22. In some embodiments, upon request, the thin firmware 20 can also send the device status of the thin device 10, such as the battery status, to the virtual operating system application 22.

[0063] Thus, the thin firmware 20 sends input (user input and / or other sensor input) to the virtual operating system application 22 and receives screen frames to be displayed from the virtual operating system application 22 .

[0064] The virtual operating system application 22 includes a remote display and input / output handling program 23 for providing screen frames to the thin firmware 20 and receiving input from the thin firmware 20. In addition, commands and / or configuration information can be sent to the thin firmware 20 via the remote display and input / output handling program 23. Example commands can include screen on / off commands, audio capture on / off commands, disconnection commands, etc. Example configurations can include display off time threshold, display brightness, etc.

[0065] Input and output are further processed by a content provider interface unit 24, which converts screen content into a screen frame format for transmission to the thin firmware 20 via the remote display and input / output handling program 23 in one transmission direction (to the thin firmware 20), and receives input from the remote display and input / output handling program 23 in the other transmission direction (from the thin firmware 20) and makes these inputs available to other components, such as applications or operating system parts. The extension manager 25 is used to manage the installation / configuration / removal of mini-apps running "on" the virtual operating system, two of which are shown as applications 26A, 26B (referred to as mini-apps herein), and can also communicate with other devices of the mobile device 16 (e.g., other devices 19), such as providing sensor input from the mobile device 16 to the mini-apps 26A, 26B. It should be noted that the remote display and input / output handling program 23, the input / output abstraction component 24, and the extension manager 25 are shown as separate components of the virtual operating system application 22, but this is only for convenience and to provide a better understanding of certain functions provided by the virtual operating system application 22, and they do not need to be implemented as separate components.

[0066] The mini-apps 26A, 26B are applications designed for the thin device 10 to provide additional functionality to the thin device 10. Examples of such mini-apps are navigation apps that allow navigation using the display 11 of the thin device 10, or any other application for providing content on the thin device, such as a video player application. The mini-apps 26A, 26B are typically applications that are installed in addition to the basic virtual operating system application 22, and may be provided, for example, by other vendors than the virtual operating system application 22. In other words, the extension manager 25 enables installation / configuration / removal of mini-apps within the virtual operating system, and the content provider interface unit 24 provides an interface to the mini-apps 26A, 26B and other applications 28 on the mobile operating system 21 to allow these applications to access input from the thin firmware 20 or other devices 19 from the mobile device 16, and to allow these applications to modify content displayed on the display 11 of the thin device 10.

[0067] In addition, the virtual mobile operating system application 22 can communicate with other applications 28 (e.g., 3rd party applications) installed on the mobile device under the operating system 21 of the mobile device. The other applications 28 are independently developed mobile applications running on the mobile device, which can use the communication interface 27 provided by the operating system 21 of the mobile device. The communication interface 27 is generally used for communication between applications under the operating system 21 of the mobile device. Such a communication interface is generally provided by the operating system of the mobile device. Using the communication interface 27, the virtual mobile operating system application 22 can communicate with the other applications 28. In this way, the other applications 28 can receive sensor / user data from the thin device and send screen content to the thin device (when needed) without using Figure 1 The interface 17 of the mobile operating system is not directly connected to the thin device, but only communicates directly with the virtual mobile operating system application 22.

[0068] Other applications 28 typically have their own independent user interface and business logic, but in the manner explained above, they can also extend their user interaction model by sending content to the thin device and receiving input data from the thin device. Examples of such other applications 28 include sports apps (for example, they can display sports-related information on the thin device and receive sports data from the thin device), navigation apps developed independently of the virtual operating system (but providing scenarios similar to the navigation mini-app), social networking and messaging apps, etc.

[0069] In some embodiments, the thin device 10 can be provided together with the virtual operating system application 22 as software in a suite (on a storage medium or available for download) so that a user can install the virtual operating system application on his or her mobile device (e.g., a smart phone, tablet computer) to operate the thin device 10.

[0070] Figure 3 An example system is presented to illustrate examples of thin devices and mobile devices. Figure 3 In the example system of FIG. 3 , the thin device is implemented as smart glasses 30, wherein display units 31A, 31B are embedded in the glasses to display content to a person wearing the smart glasses 30. The mobile device is implemented as a smart phone or a tablet computer 32.

[0071] Figure 4 is a flow chart illustrating a method according to an embodiment, which illustrates the operation of a virtual operating system application.

[0072] It should be noted that although Figure 4 The method of is shown as a series of steps, but the order of the steps shown should not be interpreted as limiting. In particular, some steps can be performed in succession.

[0073] At 40, the method includes receiving input at the mobile device from the thin device, such as receiving input from a sensor of the thin device that detects user input. At 41, via a virtual operating system (e.g., Figure 2 22), optionally using a mini-app installed in the virtual operating system application and / or a device of the mobile device (e.g., a sensor of the mobile device) to process the input. At 42, the method includes sending the screen frame to the thin device for display on the thin device. In addition, as shown in reference Figure 2 As explained, commands or configuration information may be sent to a thin device.

[0074] Now refer to Figure 5 Explains the sample operation of the virtual OS application and mini-app. Figure 5 In , it is assumed that the user has installed a virtual operating system on his or her mobile device and has a thin device that has been paired with the mobile device, e.g. Figure 3 Smart glasses 30.

[0075] Furthermore, it is assumed that the user has installed the navigation mini-app on the virtual operating system application of the mobile device.

[0076] Figure 5 An example timeline of how a thin device may be operated using a virtual operating system application and a navigation mini-app installed thereon is shown from top to bottom.

[0077] First, at 50, the user opens (starts) a virtual operating system application on the mobile device. The virtual operating system application then starts at 51 and searches for a thin device at 52. Communication is then initiated, for example using a handshake process 53, and a start screen frame (also referred to as a home screen) is sent from the virtual operating system application to the thin device for display at 54. At 55, the user then opens a navigation mini-app running under the virtual operating system on the mobile device. After the launcher and initialization of the navigation mini-app at 56, the start screen of the mini-app is presented at 57 and transmitted as a screen frame to the thin device for display at 58.

[0078] At 59, the user then enters the desired destination. In some embodiments, this can be done using an input device such as a touch screen of the mobile device. In other embodiments, this can be performed using some input device of the thin device (such as a microphone), through which the destination is entered by voice. Then, at 510, the navigation mini-app requests the current location from the mobile device, such as using a GPS sensor of the mobile device, and receives a location update at 511. Reference numeral 512 indicates that the update can be repeatedly performed. As shown in 530, 531, in addition, the navigation mini-app can also request updates to sensor data from sensors of the thin device (e.g., orientation sensors).

[0079] At 514 , the navigation mini app then presents a confirmation screen, which at 515 is transmitted to the thin device as a screen frame to be displayed on the thin device.

[0080] At 516, the user then begins navigating. Figure 5In the embodiment of the present invention, this is accomplished by pressing a button on the thin device. In other embodiments, this can be accomplished by other actions on the thin device, such as voice commands or actions performed using the input devices of the mobile device. At 517, the thin device then reports to the virtual operating system application that the button was pressed, and the virtual operating system application provides this to the navigation mini-app at 518. As shown in 532, 533, the navigation mini-app can also receive sensor updates (e.g., orientation sensors) from the thin device. Then, at 519, the navigation mini-app calculates the first step (the first instruction where to go or drive to), and presents the corresponding screen at 520, which is provided to the thin device for display at 521. Navigation continues in loop 522, during which the user moves, wherein the position is updated at 523, the navigation mini-app calculates the next step at 524, presents the corresponding screen at 525, and the screen is transmitted to the thin device for display at 526. Before calculating the next step at 524 , in some embodiments, the navigation mini-app may also receive sensor updates (eg, orientation sensors) from the thin device, as shown at 534 , 535 .

[0081] When navigation is complete, at some point at 527, the user closes the navigation mini-app, which then closes at 528. After this, at 529, the home screen is transferred to the thin device for display, corresponding to the home screen transferred for display at 54.

[0082] It should be noted that Figure 5 The process is only used to illustrate an example of possible communication between a mobile device having a virtual operating system application and a mini-app installed thereon and a thin device, and corresponding communication can also be performed for various other mini-apps.

Claims

1. A system for operating smart glasses, comprising: A mobile device comprising an operating system and a first communication interface, Smart glasses, the smart glasses comprising a display, a thin firmware and a second communication interface, wherein the computing power of the smart glasses is lower than that of the mobile device and does not allow installation and operation of additional applications, The mobile device further includes a virtual operating system application running under the operating system of the mobile device, wherein the virtual operating system application is configured to - managing a connection between the mobile device and the smart glasses via the first communication interface and the second communication interface, - providing screen frames to the smart glasses for display on a display of the smart glasses, - receiving and processing input from the smart glasses, wherein the input comprises at least one of user input at the smart glasses or sensor data provided by a sensor of the smart glasses, - enabling execution of an application for the smart glasses on the mobile device, The thin firmware is configured as - interacting with the virtual operating system application, using the first communication interface and the second communication interface to pair and unpair the smart glasses with the mobile device, - receiving a screen frame from the virtual operating system application to be displayed on a display of the smart glasses, and - Sending the user input or the sensor data to the virtual operating system application.

2. The system of claim 1, wherein: The virtual operating system application enables communication of these applications with components of the mobile device.

3. The system of claim 1, wherein: The virtual operating system application is configured to receive screen content from the applications and provide the screen frames to the smart glasses based on the screen content.

4. The system of claim 1, wherein: These applications include navigation applications.

5. The system of claim 1, wherein: The mobile device is a smartphone or a tablet computer.

6. The system of claim 1, wherein: The virtual operating system application is configured to communicate with other applications on the mobile device.

7. The system of claim 6, wherein: These other applications run independently of the virtual operating system application under the operating system of the mobile device.

8. A method for operating smart glasses, comprising: On the mobile device, a virtual operating system application is run for the smart glasses whose computing power is lower than that of the mobile device and which does not allow installation and running of additional applications, wherein the virtual operating system application performs the following operations: - managing a connection between the mobile device and the smart glasses via a first communication interface of the mobile device and a second communication interface of the smart glasses, - providing screen frames to the smart glasses for display on a display of the smart glasses, - receiving and processing input from the smart glasses, wherein the input comprises at least one of user input at the smart glasses or sensor data provided by a sensor of the smart glasses, and - enabling execution of an application for the smart glasses on the mobile device, The smart glasses include a thin firmware configured to - interacting with the virtual operating system application, using the first communication interface and the second communication interface to pair and unpair the smart glasses with the mobile device, - receiving a screen frame from the virtual operating system application to be displayed on the display of the smart glasses, and - Sending the user input or the sensor data to the virtual operating system application.

9. The method of claim 8, wherein: The virtual operating system application enables communication of these applications with components of the mobile device.

10. The method according to claim 8 or 9, wherein: The virtual operating system application receives screen content from the applications and provides the screen frames to the smart glasses based on the screen content.

11. The method according to claim 8 or 9, wherein: These applications include navigation applications.

12. The method according to claim 8 or 9, wherein: The virtual operating system application communicates with other applications on the mobile device.

13. A storage medium having stored thereon a computer program comprising a program code which, when executed on a processor, causes the method as claimed in any one of claims 8 to 12 to be performed.

14. A mobile device comprising the storage medium according to claim 13.

15. A kit comprising: The storage medium as claimed in claim 13, and Smart glasses that operate using a virtual operating system application of a computer program stored on the storage medium.

Images