Method for virtualizing controller on android terminal, mobile terminal, and storage medium

Abstract

Provided in the present invention are a method for virtualizing a controller on an Android terminal, a mobile terminal, and a storage medium. The method comprises the steps of: replacing an ImageView of an original lock screen wallpaper in systemui with a RelatieLayout, so as to establish a View container; in the View container, respectively adding a wallpaper interface layout and a controller operation interface layout; using a singleton pattern to realize an InputViewManager as a management class for displaying the controller operation interface layout on a lock screen, so as to externally provide two interfaces for displaying and hiding an interface, and displaying a controller operation interface by monitoring a lock screen interface click event; and establishing a screen operation monitoring event, and acquiring controller operation interface click / touch data, so as to convert the controller operation interface click / touch data into a control code of a corresponding entity controller for output. Thus, the direct startup of the virtualization of a controller on a lock screen interface can be supported.

Description

Method for virtual controller in Android terminal, mobile terminal, and storage medium Technical Field

[0001] The present invention relates to a technology for utilizing virtual keys on a terminal screen, and in particular to a method for creating a virtual controller in an Android terminal, a mobile terminal, and a storage medium. Background Art

[0002] Currently, there are many applications on Android terminals that virtualize the terminal screen into controllers such as keyboards, mice, joysticks, and control buttons to provide control input functions for devices such as computers and PADs.

[0003] However, existing virtual controller software of this type suffers from a common drawback: the opening process is cumbersome. It requires unlocking the terminal screen, finding the app on the system interface, and clicking it before it can be used. Therefore, solving this problem is key to improving the user experience.

[0004] Summary of the Invention

[0005] To this end, the main purpose of the present invention is to provide a method, a mobile terminal, and a storage medium for a virtual controller on an Android terminal to support opening a virtual controller directly on the lock screen interface.

[0006] In order to achieve the above object, according to a first aspect of the present invention, a method for creating a virtual controller in an Android terminal is provided, comprising the following steps:

[0007] Replace the ImageView of the original lock screen wallpaper in systemUI with a relative layout RelatieLayout to create a View container;

[0008] In the View container, add the wallpaper interface layout and the controller operation interface layout respectively;

[0009] Use the singleton mode to implement InputViewManager as the management class for displaying the controller operation interface layout on the lock screen, providing two interfaces for displaying and hiding the interface. By monitoring the click event of the lock screen interface, the controller operation interface is displayed;

[0010] Establish screen operation monitoring events, obtain the controller operation interface click / touch data, and convert it into the control code of the corresponding physical controller for output.

[0011] In a possible preferred embodiment, the controller operation interface layout includes several sub-layouts and is in a Framelayout layout; by monitoring the lock screen interface click event, the InputViewManager controls the switching display of the sub-layouts of each controller operation interface layout.

[0012] In a possible preferred implementation, when the screen is off, the InputViewManager controls the hiding of the controller operation interface.

[0013] In a possible preferred embodiment, the method of creating a virtual controller in an Android terminal further includes: writing a HID descriptor corresponding to the controller operation event according to the HID protocol; when the Android terminal establishes a USB HID connection with the controlled device, initializing the channel, sending the HID descriptor, and using the bulkTransfer and controlTransfer methods of the DeviceConnection class to interactively transmit the control code with the controlled device according to the HID protocol.

[0014] In a possible preferred embodiment, the method of virtualizing the controller in the Android terminal further includes: writing a HID descriptor corresponding to the controller operation event according to the HID protocol; when the Android terminal establishes a Bluetooth HID connection with the controlled device, initializing the channel, sending the HID descriptor, and using the HID Bluetooth interface provided by the Android SDK to interactively transmit the control code with the controlled device according to the HID protocol.

[0015] In a possible preferred embodiment, the controller operation interface is laid out on the upper layer of the wallpaper interface layout; the step of displaying the controller operation interface by monitoring the lock screen interface click event includes: monitoring the lock screen interface (keyguardview) click / long press event in the systemui, and when clicked a preset number of times / long pressed for a preset time, hiding the lock screen interface (keyguardview) to display the controller operation interface.

[0016] In a possible preferred embodiment, the controller operation interface layout includes at least one of a keyboard and mouse interface layout, a mouse interface layout, a keyboard interface layout, a game handle interface layout, a piano keyboard layout, and a mixing controller layout.

[0017] In a possible preferred embodiment, the sub-layouts of the controller operation interface layout include at least two of: keyboard and mouse interface layout, mouse interface layout, keyboard interface layout, game handle interface layout, piano keyboard layout, and mixing controller layout.

[0018] In order to achieve the above-mentioned purpose, according to the second aspect of the present invention, a mobile terminal is also provided, which adopts the Android operating system and includes: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method of virtual controller in the Android terminal as described above are implemented.

[0019] In order to achieve the above-mentioned purpose, according to the third aspect of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the method of creating a virtual controller in an Android terminal as described above are implemented.

[0020] The method, mobile terminal and storage medium of the Android terminal virtual controller provided by the present invention can support the direct use of the virtual controller on the lock screen interface of the Android system, so the startup steps are very few and the operation is very convenient. In addition, since this solution does not rely on APP implementation, the response speed is extremely fast, and there is no time-consuming problem of APP cold start. At the same time, there is no problem of switching between the front and back ends of the APP or being killed by the background process. Therefore, the connection is stable and not easy to disconnect, which can effectively improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0022] FIG1 is a schematic diagram of the steps of a method for implementing a virtual controller in an Android terminal according to the present invention;

[0023] FIG2 is a logical diagram of establishing a View container in the method of the Android terminal virtual controller of the present invention;

[0024] 3 to 5 are schematic diagrams of the layout of the mouse, keyboard, and keyboard-mouse interface in the method of the Android terminal virtual controller of the present invention;

[0025] 6 is a schematic diagram showing a method for creating a virtual controller in an Android terminal according to the present invention, in which a controller operation interface layout is arranged on top of a wallpaper interface layout, and sub-layouts of each controller operation interface layout are arranged in a Framelayout layout;

[0026] FIG7 is a schematic diagram of a control connection between a mobile terminal and a controlled HID device / apparatus using the method of the present invention for a virtual controller in an Android terminal under the HID protocol. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the specific technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this case are only embodiments of a part of the present invention, rather than all embodiments. It should be pointed out that, for those of ordinary skill in the art, the embodiments in this application and the features in the embodiments can be combined with each other without departing from the concept of the present invention and without conflicting with each other. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative work should fall within the disclosure and protection scope of the present invention.

[0028] In addition, the terms "first", "second", "S100", "S200", etc. in the specification, claims and drawings of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the features used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those described herein. At the same time, the terms "including" and "having" and any variations thereof in the present invention are intended to cover non-exclusive inclusions. Unless otherwise expressly specified and limited, the terms "setting", "laying out", "installing", "connecting" and "connecting" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a communication between the two elements. For ordinary technicians in this field, the specific meanings of the above terms in this case can be understood according to the specific circumstances and in combination with the existing technology.

[0029] In order to support opening a virtual controller directly on the lock screen interface of an Android terminal, as shown in Figures 1 to 6, the present invention proposes a method for opening a virtual controller on an Android terminal, the steps of which mainly include:

[0030] Step S100 replaces the ImageView of the original lock screen wallpaper in systemUI with a relative layout RelatieLayout to establish a View container.

[0031] Specifically, in the systemui module source code, the layout of the original lock screen wallpaper is BackDropView, and the view of the lock screen wallpaper is an ordinary ImageView, for example, the view id is backdrop_back, and the idea of ​​the present invention is to construct the backdrop_back of the original lock screen wallpaper as an input container input_container, and use this view as the carrier interface of the final lock screen wallpaper, which provides the possibility of carrying the controller operation interface.

[0032] Based on the above concept, this embodiment illustrates a solution for modifying the lock screen wallpaper. As shown in Figure 2, the wallpaper ImageView display interface with the id backdrop_back in BackDropView is first found, and then the ImageView control is replaced with RelatieLayout. This is equivalent to changing the view control to ViewGroup, so that a View container (i.e., input_container) can be established.

[0033] For example, in the Android system, the layout file corresponding to the lock screen wallpaper of systemUI is in superstatusbar.xml. Find the ImageView (id / backdrop_back) of the original lock screen wallpaper and change it to the relative layout RelatieLayout to create a View container (id set to input_layout).

[0034] The code example of systemui layout superstatusbar.xml is as follows:

[0035] In step S200 , a wallpaper interface layout and a controller operation interface layout are added to the View container.

[0036] Specifically, in the layout of the View container, it is preferred to add two layers of layout, namely the wallpaper interface layout (id: backdrop_back) and the controller operation interface layout (input_view_layout). The interface size is full screen, and the controller operation interface layout is preferably set on the upper layer of the wallpaper interface layout. This setting method is to facilitate the subsequent display / hiding of the controller operation interface layout without being covered by the wallpaper interface layout.

[0037] The controller referred to in the present invention covers all devices / equipment whose control panel can be virtualized through the screen, such as: mouse, keyboard, game controller, piano keyboard, mixing controller, etc., and the controller operation interface layout can be a virtual control panel layout of the device / equipment or a customized control panel layout.

[0038] Furthermore, as shown in Figures 3 to 6, considering that the controller to be simulated may have a variety of different button layout requirements, the present invention illustrates a setting scheme for adding a controller operation interface sub-layout Ui with multiple layouts to the controller operation interface layout.

[0039] For example, draw three modes of controller operation interface sub-layout Ui, respectively:

[0040] Draw the XML layout of the mouse interface layout (mouse_layout) (as shown in Figure 3, the lock screen operation interface model touchpad layout), implement the corresponding MouseView.java, MouseView.java, and get the layout View by findViewById(R.id.mouse_layout) in the View of input_view_layout.

[0041] Draw the xml layout of the keyboard interface layout (keyboard_layout) (as shown in Figure 4, the lock screen operation interface model keyboard layout), implement the corresponding KeyboardView.java, and obtain the layout View by findViewById(R.id.keyboard_layout) in the View of input_view_layout.

[0042] Draw the xml layout of the keyboard and mouse interface layout (all_mode_layout) (as shown in Figure 5, the lock screen operation interface model mouse + keyboard layout), implement the corresponding AllModeView.java, and AllModeView.java obtains the layout View by findViewById(all_mode_layout) in the View of input_view_layout.

[0043] All three layout modes use Framelayout, with width and height set to match_parent and full screen, and are initially hidden and invisible. All three views are sub-layouts of input_view_layout.

[0044] The following code example adds two layers of layout to the View container:

[0045] It should be noted here that although this example illustrates three modes of controller operation interface sub-layouts, it does not limit the number or form of sub-layouts that can be implemented by the present invention. Therefore, it can be understood that in addition to the exemplary implementation methods of the present invention, those skilled in the art can also set other sub-layouts based on actual needs without departing from the conceptual scope of the embodiment of the present invention, such as custom controller operation interface layouts such as game controllers, piano keyboards, and mixing controllers to simulate various controller layouts with control buttons / touch keys, so as to add sub-layouts or implement them separately as controller operation interface layouts. Therefore, based on the above reasons, any other variation implementation methods based on the concept of the embodiment of the present invention are within the scope of disclosure of the present invention.

[0046] Step S300 uses the singleton mode to implement InputViewManager as the management class for displaying the controller operation interface layout on the lock screen, to provide two interfaces for displaying and hiding the interface, and to display the controller operation interface by monitoring the click event of the lock screen interface.

[0047] Specifically, in order to realize the display / hide control of the controller operation interface layout, this example implements InputViewManager.java as the management class for displaying input_view_layout on the lock screen, and adopts the singleton mode to provide two interfaces, showInPutLayout() and hideInputLayout(), to respectively realize the display and hiding of the input interface.

[0048] At the same time, by monitoring the click events of the lock screen interface, the click / long press events of the keyguardview of the lock screen interface in the systemui are monitored. When the preset number of clicks / long presses are made for the preset time, the lock screen interface keyguardview is hidden to display the controller operation interface.

[0049] For example, listen to the OnClick and OnTounch events of the keyguardview in the lock screen interface of the system ui. When clicked twice, hide the keyguardview to display the controller operation interface layout (input_container). Because input_view_layout has been set on the upper layer of backdrop_back, once the keyguardview is hidden, the controller operation interface layout will be displayed directly. At this time, real-time data of interface clicks and touch data can be obtained, thereby implementing the hide and show methods to switch between display and hiding respectively.

[0050] As for the realization of the exit function, in this example, it is preferred to turn off the screen or click the exit button on the interface to control the hiding of input_container and then display the keyguradview.

[0051] At the same time, InputViewManager controls the switching display of the above three sub-layouts. For example, setMode(int mode) (mode: 0 touchpad 1, keyboard, 2, keyboard and mouse) is set to call the display and hide interfaces of the above three sub-layouts respectively to realize mode switching.

[0052] Step S400 establishes a screen operation monitoring event, obtains click / touch data of the controller operation interface, and converts it into a control code of the corresponding physical controller for output.

[0053] Specifically, after obtaining the click / touch data of the controller operation interface, all touchpads and buttons in the aforementioned controller operation interface layouts need to be converted into the codes of the corresponding buttons, such as the displacement corresponding to the sliding, the up / down information of the button, the code of the physical keyboard corresponding to each button of the keyboard, etc., so as to realize the control output function.

[0054] For example, as shown in Figure 7, when the Android terminal is connected to a device / apparatus that supports the HID protocol, the HID descriptor corresponding to the controller operation event is first written according to the HID protocol; when the Android terminal establishes a USB HID connection with the controlled device, the channel is initialized, the HID descriptor is sent, and the bulkTransfer and controlTransfer methods of the DeviceConnection class are used to transmit the control code interactively with the controlled device according to the HID protocol.

[0055] When the Android terminal establishes a Bluetooth HID connection with the controlled device, it initializes the channel, sends the HID descriptor, and uses the HID Bluetooth interface provided by the Android SDK to transmit the control code to the controlled device according to the HID protocol.

[0056] Corresponding to the above method, the present invention also provides a mobile terminal using an Android operating system, which includes: a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method of virtual controller in the Android terminal as described above.

[0057] Corresponding to the above method, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method of creating a virtual controller in an Android terminal as described above are implemented.

[0058] To sum up, the method, mobile terminal and storage medium of the virtual controller in the Android terminal provided by the present invention can support the direct use of the virtual controller in the lock screen interface of the Android system, so the startup steps are very few and the operation is very convenient. In addition, since this solution does not rely on APP implementation, on the one hand, the response speed is extremely fast, and there is no time-consuming problem of APP cold start. At the same time, there is no problem of switching between the front and back ends of the APP or being killed by the background process. Therefore, the connection is stable and not easy to disconnect, which can effectively improve the user experience.

[0059] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made based on the contents of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0060] Those skilled in the art will understand that, in addition to implementing the system, device, and various modules provided by the present invention in pure computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0061] In addition, all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a single-chip microcomputer, chip or processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0062] In addition, various implementations of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A method for creating a virtual controller on an Android terminal, comprising: Replace the ImageView of the original lock screen wallpaper in systemui with a relative layout RelatieLayout to create a View container; In the View container, add the wallpaper interface layout and the controller operation interface layout respectively; Use the singleton mode to implement InputViewManager as the management class for displaying the controller operation interface layout on the lock screen, so as to provide two interfaces for displaying and hiding the interface, and display the controller operation interface by listening to the click event of the lock screen interface; Establish screen operation monitoring events, obtain the click / touch data of the controller operation interface, and convert it into the control code of the corresponding physical controller for output.

2. According to the method of virtual controller in Android terminal in claim 1, the controller operation interface layout includes several sub-layouts and is in Framelayout layout; wherein by monitoring the click event of lock screen interface, the sub-layout switching display of each controller operation interface layout is controlled by InputViewManager.

3. The method for creating a virtual controller on an Android terminal according to claim 1, wherein the controller operation interface is hidden by InputViewManager when the screen is off.

4. The method for creating a virtual controller on an Android terminal according to claim 1, further comprising: According to the HID protocol, write the HID descriptor corresponding to the controller operation event; When the Android terminal establishes a USB HID connection with the controlled device, the channel is initialized, the HID descriptor is sent, and the bulkTransfer and controlTransfer methods of the DeviceConnection class are used to transmit the control code interactively with the controlled device according to the HID protocol.

5. The method for creating a virtual controller on an Android terminal according to claim 1, further comprising: According to the HID protocol, write the HID descriptor corresponding to the controller operation event; When the Android terminal establishes a Bluetooth HID connection with the controlled device, the channel is initialized, the HID descriptor is sent, and the HID Bluetooth interface provided by the Android SDK is used to transmit the control code interactively with the controlled device according to the HID protocol.

6. The method for virtualizing a controller on an Android terminal according to claim 1, wherein the controller operation interface is arranged on an upper layer of a wallpaper interface layout; wherein the step of displaying the controller operation interface by monitoring a click event on a lock screen interface comprises: Listen for click / long press events on the lock screen in systemui. When the preset number of clicks / long presses are reached, the lock screen is hidden to display the controller operation interface.

7. The method for creating a virtual controller on an Android terminal according to claim 1, wherein the controller operation interface layout comprises: At least one of a keyboard and mouse interface layout, a mouse interface layout, a keyboard interface layout, a game controller interface layout, a piano keyboard layout, and a mixing controller layout.

8. The method for creating a virtual controller on an Android terminal according to claim 2, wherein the sub-layout of the controller operation interface layout comprises: At least two of the following: keyboard and mouse interface layout, mouse interface layout, keyboard interface layout, game handle interface layout, piano keyboard layout, and mixing controller layout.

9. A mobile terminal using an Android operating system, comprising: A memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method for a virtual controller in an Android terminal as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method for a virtual controller in an Android terminal as claimed in any one of claims 1 to 8 are implemented.

Images