Method for performing all-gesture simulation anti-control between Android devices

Abstract

The invention provides a method for performing all-gesture simulation anti-control between Android devices. The method comprises the following steps: S1, transmitting the resolution ratio and the screen density of a mobile device end to a vehicle-mounted infotainment system device end through a communication layer; starting a process of the Shell user group permission of the mobile device end by the vehicle machine system device end through the ADB communication channel; and S2, collecting all touch gestures of a user by the in-vehicle infotainment system equipment end, and according to the resolution and screen density of the video transmitted by the mobile equipment end, achieving the purpose of reversely controlling the mobile equipment end by touching the screen of the in-vehicle infotainment system equipment end when the in-vehicle infotainment system equipment end plays the screen mirror image content of the mobile equipment end. Through close cooperation of the communication layer, the vehicle machine system equipment end and the mobile equipment end, all-gesture simulation anti-control between Android equipment is realized, so that the requirement of a user for performing complex gesture simulation anti-control between Android equipment is met, a new experience scene is brought to the user, and the user experience feeling is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of screen projection between Android devices, and more specifically, to a method for full gesture simulation reverse control between Android devices. Background Art

[0002] Against the backdrop of widespread adoption of mobile Internet and smartphones, technology for displaying mobile phone screen mirroring content on vehicle screens has gradually developed to meet users' needs for accessing and controlling mobile phone content in vehicles. Innovations and advances in wireless transmission technology, screen mirroring protocols, software support, and user interface adaptation have provided the technical foundation for achieving this goal.

[0003] The existing counter-control schemes have the following shortcomings:

[0004] Accessibility services: Through the accessibility services in the Android system, simple touch events such as single click, long press, and slide can be simulated. However, these operations are point-to-point operations and can only perform discrete touch events. They cannot simulate continuous touch events. For example, scenarios such as long press and dragging cannot be implemented through accessibility services.

[0005] Based on the above technical background and user experience requirements, current accessibility service solutions are difficult to meet the anti-control needs of complex gestures, let alone restore full gesture touch events. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method for full gesture simulation and reverse control between Android devices to meet the needs of users for complex gesture simulation and reverse control between Android devices, in response to the shortcomings of the above technical solutions.

[0007] The present invention provides a method for performing full gesture simulation reverse control between Android devices, the method comprising the following steps:

[0008] S1, transmitting the resolution and screen density of the mobile device to the vehicle system device through the communication layer; the vehicle system device starts the process of the Shell user group permission of the mobile device through the ADB communication channel;

[0009] S2, the vehicle system device collects all the user's touch gestures, and scales the x- and y-axis coordinate values ​​of the gestures according to the video resolution and screen density transmitted by the mobile device, maps and encapsulates them into a touch gesture package, and transmits the touch gesture package to the mobile device through the ADB communication channel;

[0010] S3, after receiving the touch gesture package, the mobile device triggers a gesture event through the process of Shell user group permission started by the vehicle system device on the mobile device to realize full gesture reverse control simulation.

[0011] In the method for full gesture simulation reverse control between Android devices described in the present invention, step S1 includes the following steps:

[0012] S11, the vehicle system device sends an ADB establishment request to the mobile device through the communication layer, and the mobile device responds to the ADB establishment request of the vehicle system device, establishes an ADB communication channel and sends an ADB command.

[0013] In the method for full gesture simulation reverse control between Android devices described in the present invention, step S2 includes the following steps:

[0014] S21, the vehicle system device side monitors the triggering gesture event by setting the Android view touch event listener of the user's touch intention in the mirror display interface, and collects all touch gestures of the user on the vehicle system device side.

[0015] In the method for full gesture simulation reverse control between Android devices described in the present invention, step S2 further includes the following steps:

[0016] S22, when the user triggers a valid touch gesture, the vehicle system device end scales the x- and y-axis coordinate values ​​of the touch gesture according to the video resolution and screen density scaling ratio between the mobile device end and the vehicle system device end, and encapsulates the scaled values ​​into a touch gesture package, and transmits the touch gesture package to the mobile device end through the ADB communication channel.

[0017] In the method for full gesture simulation reverse control between Android devices described in the present invention, step S2 further includes the following steps:

[0018] S23, encapsulate all touch gesture objects into a touch gesture encapsulation package in the order of one bit per 4 bytes, wherein the touch gesture encapsulation package includes the gesture action type ActionId and the X and Y coordinate axis values ​​corresponding to the finger index, and each additional finger has 8 more bytes of X and Y coordinate values.

[0019] In the method for full gesture simulation reverse control between Android devices described in the present invention, step S3 includes the following steps:

[0020] S31, the process with Shell user group permission on the mobile device parses the received touch gesture package into a specific touch event object, and triggers a gesture event through the InputEvent, int function.

[0021] In the method for full gesture simulation reverse control between Android devices described in the present invention, step S3 further includes the following steps:

[0022] S32: Parse the touch gesture package into the X and Y coordinate axis values ​​corresponding to the gesture action type ActionId and the finger index in a 4-byte one-bit order. For each additional finger, there are 8 more bytes of X and Y coordinate values. The parsed gesture action type ActionId and the X and Y coordinate axis values ​​corresponding to the finger index are encapsulated into a specific touch event object.

[0023] In the method for full gesture simulation reverse control between Android devices described in the present invention, in step S31, the process of Shell user group permissions on the mobile device side is started by the ADBServer&ADBClient tool on the vehicle system device side.

[0024] In the method for full gesture simulation reverse control between Android devices described in the present invention, in step S1, the communication layer includes a WiFi wireless channel and a USB wired channel.

[0025] The main application scenario of the method of the present invention based on full gesture simulation and reverse control between Android devices is that when the screen mirroring content of the mobile device is played on the vehicle system device, the purpose of reverse control of the mobile device can be achieved by touching the screen of the vehicle system device. Through the close cooperation of the communication layer, the vehicle system device and the mobile device, full gesture simulation and reverse control between Android devices is realized to meet the user's needs for complex gesture simulation and reverse control between Android devices, bringing new experience scenarios to users, thereby improving user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 1 is a flow chart of an embodiment of a method for full gesture simulation reverse control between Android devices according to the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] like Figure 1 , which is a flow chart of an embodiment of a method for full gesture simulation and reverse control between Android devices according to the present invention. A method for full gesture simulation and reverse control between Android devices is provided, comprising the following steps:

[0030] In step S1, the resolution and screen density of the mobile device are transmitted to the vehicle system device through the communication layer; the vehicle system device starts the process of the Shell user group permission of the mobile device through the ADB communication channel;

[0031] In step S2, the vehicle system device collects all touch gestures of the user, and scales the x- and y-axis coordinate values ​​of the gestures according to the video resolution and screen density transmitted by the mobile device, maps and encapsulates them into a touch gesture package, and transmits the touch gesture package to the mobile device through the ADB communication channel;

[0032] In step S3, after receiving the touch gesture package, the mobile device triggers a gesture event through the process of Shell user group permission started by the vehicle system device on the mobile device to realize full gesture reverse control simulation.

[0033] In this embodiment, step S1 includes the following steps:

[0034] In step S11, the vehicle system device sends an ADB establishment request to the mobile device through the communication layer, and the mobile device responds to the ADB establishment request of the vehicle system device, establishes an ADB communication channel and sends an ADB command.

[0035] In this embodiment, step S2 includes the following steps:

[0036] In step S21, the vehicle system device side monitors the triggering gesture event by setting the Android view touch event listener of the user's touch intention in the mirror display interface, and collects all the user's touch gestures on the vehicle system device side.

[0037] In this embodiment, step S2 further includes the following steps:

[0038] In step S22, when the user triggers a valid touch gesture, the vehicle system device end scales the x- and y-axis coordinate values ​​of the touch gesture according to the video resolution and screen density scaling ratio between the mobile device end and the vehicle system device end, and encapsulates the scaled values ​​into a touch gesture package, and transmits the touch gesture package to the mobile device end through the ADB communication channel.

[0039] In this embodiment, step S2 further includes the following steps:

[0040] In step S23, all touch gesture objects are encapsulated into a touch gesture encapsulation package in the order of one bit per 4 bytes, wherein the touch gesture encapsulation package includes the gesture action type ActionId and the X and Y coordinate axis values ​​corresponding to the finger index, and for each additional finger, there are 8 more bytes of X and Y coordinate values.

[0041] In this embodiment, step S3 includes the following steps:

[0042] In step S31, the process with Shell user group permission on the mobile device parses the received touch gesture package into a specific touch event object, and triggers a gesture event through the InputEvent, int function.

[0043] In this embodiment, step S3 further includes the following steps:

[0044] In step S32, the touch gesture packet is parsed into the gesture action type ActionId and the X and Y coordinate values ​​corresponding to the finger index in a 4-byte sequence. For each additional finger, 8 bytes of X and Y coordinate values ​​are added. The parsed gesture action type ActionId and the X and Y coordinate values ​​corresponding to the finger index are encapsulated into a specific touch event object. The gesture action type ActionId can be DOWN, MOVE, UP, or Cancel, etc.

[0045] In this embodiment, in step S31 , the process of Shell user group permissions on the mobile device side is started by the ADBServer&ADBClient tool on the vehicle system device side.

[0046] In this embodiment, the communication layer in step S1 includes a WiFi wireless channel and a USB wired channel.

[0047] It should be noted that the vehicle system device side of this application is of Android system type; the mobile device side is a mobile phone device of Android system type.

[0048] This application mainly consists of three parts: the communication layer, the Android Da car system device side, and the Android Phone mobile device side. The communication layer is used to transmit the resolution and screen density of the mobile device side to the car system device side; the communication layer is divided into WiFi wireless channel and USB wired channel. The difference is only the difference in physical channel, and the actual function is the same. It mainly provides data flow and transmission functions; the main functions of the Android Da car system device side include: screen mirroring video decoding, screen mirroring video display, screen touch event collection and packaging, screen mirroring internal distribution, and ADB communication channel establishment; the main functions of the Android Phone mobile device side include: screen mirroring video recording, screen mirroring video encoding, screen mirroring video stream transmission, and triggering touch events.

[0049] Through the close cooperation of the communication layer, the Android Da car system device end and the Android Phone mobile device end, full gesture simulation and reverse control between Android devices are realized to meet the user's needs for complex gesture simulation and reverse control between Android devices, bringing new experience scenarios to users and thus improving user experience.

[0050] The ADBServer and ADBClient functions are implemented on the vehicle system device. The vehicle system device communicates with the mobile device via a WiFi wireless channel or a USB wired channel, using ADBServer to establish an ADB channel with the mobile device's ADB process. Over this ADB communication channel, the vehicle system device uses ADB shell app_process to launch a specific process on the mobile device. This process has shell user group permissions and can trigger system-level gesture touch events. After completing these tasks, the vehicle system device transmits the gesture events collected by the vehicle system device to the mobile device, which then triggers system-level touch events on the shell process.

[0051] The main working process of the present invention is: the Android Da car system device end sends an ADB establishment request to the Android Phone mobile device end through the communication layer, and the Android Phone mobile device end responds to the establishment request of the Android Da car system device end to establish an ADB communication channel; the Android Da car system device end starts a process with a shell permission group on the Android Phone mobile device end through the ADB command, and distributes the touch events of the Android Da car system device end through the shell permission process to achieve the purpose of full gesture simulation reverse control.

[0052] Specifically, the ADB communication channel establishment process is as follows: the vehicle system device sends a CNXN connection request, the mobile device responds with an AUTH-type response message, the AUTH verification information is authenticated and saved, and the CNXN response is successful, and ADB communication is successfully established. Once ADB communication is successfully established, the Android DA vehicle system device can use the ADB Client to control the Android Phone mobile device through ADB commands. It should be noted that this application is based on the USBChannel channel, and all ADB communication is carried out through the USB channel, which will be referred to as USBC.

[0053] The vehicle system device generates a `CMD_CNXN` type message, carrying

[0054] The payload, `host:info; adb_version:1`, is encapsulated in ADB format using little-endian format and transmitted over USBC between Android devices. Upon receiving the `CMD_CNXN` message, the mobile device sends a `CMD_AUTH` response with a `token:radomString` payload to the vehicle via USBC. The vehicle system device encrypts the `token` sent by the mobile phone using its local private key, creates a `CMD_AUTH` message with a `singedToken; pubKeyString` formatted string as the payload, encapsulates the message, and sends it to the mobile device. After receiving the message, the mobile phone requests ADB debugging permission from the user and returns a `CMD_CNXN` message with a `adbversion:1; deviceinfo` payload to the vehicle via USBC, completing the establishment of ADB communication over USBC. This completes the establishment of the USBC-based ADB communication channel between the vehicle system device and the mobile device.

[0055] After the ADB communication channel is successfully established, we need to use this channel to send a request in the ADB Client on the Android DA vehicle system device side, and execute a process with Shell user group permissions on the Android phone mobile device side. Through this process, we can successfully complete the system-level distribution of all gesture events on the Android phone mobile device side.

[0056] The Android DA device creates an ADBClient through ADBServer, which is responsible for communicating with the ADB process on the Android mobile device. In this way, a process with Shell user group permissions started on the Android mobile device can execute high-privilege command operations on the Android mobile device. This is the most critical part of the full gesture simulation anti-control invention.

[0057] The invention consists of a process in which the vehicle system device collects touch gestures and maps and encapsulates them into touch gesture packages, transmits gesture events, parses the touch gesture packages on the mobile device, and executes a Shell user group permission process. This mechanism is used to implement full gesture simulation and reverse control between Android devices after establishing an ADB communication channel. This invention cannot be implemented without the existence of a process with the Shell user group permission.

[0058] The specific implementation process is as follows:

[0059] The vehicle system device encapsulates a `CMD_OPEN` type message, carrying a payload

[0060] `shell:CLASSPATH=apk exec app_process / system / bin xxx.class` message. This message starts an ADBClient in the ADBServer and sends it to the mobile phone through the ADBClient via USBC. After receiving the message, the ADB process on the mobile device will start the apk / xxx.class function entry specified in a shell command and start a process with shell permissions. After implementing this operation, the ADB end on the mobile device will respond with a `CMD_OK` message and respond to the ADBClient on the vehicle side through USBC in the form of no payload. At this point, the startup of the shell permission process on the mobile device is completed.

[0061] By collecting the user's full touch gestures on the vehicle system device, and scaling the x- and y-axis coordinate values ​​of the touch gestures according to the video resolution and screen density scaling ratio between the mobile device and the vehicle system device, and encapsulating them into a touch gesture package, the gesture events are executed through the process of the Shell user group permission on the mobile device to achieve the purpose of full gesture simulation reverse control.

[0062] Its main application scenario is to play the screen mirroring content of the Android Phone mobile device (or other Android devices) on the Android DA car system device, and to reversely control the Android Phone mobile device screen by touching the Android DA car system device screen; this solution mainly includes the ADBServer&ADBClient tool library on the Android DA device side, the process of the Shell user group permission on the Android Phone device side, and the full gesture collection SDK on the Android DA device side.

[0063] The full gesture parsing SDK for Android Phone mobile devices is as follows:

[0064] The ADBServer & ADBClient tool library on the Android DA vehicle-mounted system device side is used to establish an ADB communication channel with the Android mobile device side and send ADB commands. The process with Shell user group permissions on the Android Phone mobile device side is started by the ADBServer & ADBClient tools on the Android DA vehicle-mounted system device side. As a process with Shell user group permissions, its core function is to execute full gesture touch events. The full gesture collection SDK on the Android DA vehicle-mounted system device side is used to collect and encapsulate full gesture touch events performed by the user on the vehicle-mounted system device side.

[0065] The Android Phone mobile device full gesture parsing SDK is used to parse the full gesture events transmitted by the Android DA car system device and trigger them by a process with Shell user group permissions.

[0066] The beneficial effects of the method for full gesture simulation reverse control between Android devices provided by an embodiment of the present invention are at least:

[0067] 1. The present invention can collect all screen touch inputs of the user on the vehicle system device side, and simulate and reverse control all collected touch events on the mobile device side, including complex usage scenarios such as dragging, multi-finger touch, etc.

[0068] 2. Accessibility services require users to re-apply for permissions each time they enter an app. However, this solution does not require re-application after ADB is enabled and authorization is granted.

[0069] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0070] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0071] Therefore, the above is only a preferred specific embodiment of the present invention, and the scope of protection of the present invention is not limited to this. Any changes or replacements that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered within the scope of protection of the present invention. The scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for full gesture simulation reverse control between Android devices, characterized in that: The method comprises the following steps: S1, transmitting the resolution and screen density of the mobile device to the vehicle system device through the communication layer; the vehicle system device starts the process of the Shell user group permission of the mobile device through the ADB communication channel; S2, the vehicle system device collects all the user's touch gestures, and scales the x- and y-axis coordinate values ​​of the gestures according to the video resolution and screen density transmitted by the mobile device, and then maps and encapsulates them into a touch gesture package, and transmits the touch gesture package to the mobile device through the ADB communication channel; S3, after receiving the touch gesture package, the mobile device triggers a gesture event through the process of Shell user group permission started by the vehicle system device on the mobile device to realize full gesture reverse control simulation.

2. The method for full gesture simulation reverse control between Android devices according to claim 1, characterized in that: The step S1 comprises the following steps: S11, the vehicle system device sends an ADB establishment request to the mobile device through the communication layer, and the mobile device responds to the ADB establishment request of the vehicle system device, establishes an ADB communication channel and sends an ADB command.

3. The method for full gesture simulation reverse control between Android devices according to claim 1, characterized in that: The step S2 comprises the following steps: S21, the vehicle system device side monitors the triggering gesture event by setting the Android view touch event listener of the user's touch intention in the mirror display interface, and collects all touch gestures of the user on the vehicle system device side.

4. The method for full gesture simulation reverse control between Android devices according to claim 3, characterized in that: The step S2 further comprises the following steps: S22, when the user triggers a valid touch gesture, the vehicle system device end scales the x- and y-axis coordinate values ​​of the touch gesture according to the video resolution and screen density scaling ratio between the mobile device end and the vehicle system device end, and encapsulates the scaled values ​​into a touch gesture package, and transmits the touch gesture package to the mobile device end through the ADB communication channel.

5. The method for full gesture simulation reverse control between Android devices according to claim 3, characterized in that: The step S2 further comprises the following steps: S23, encapsulate all touch gesture objects into a touch gesture encapsulation package in the order of one bit per 4 bytes, wherein the touch gesture encapsulation package includes the gesture action type ActionId and the X and Y coordinate axis values ​​corresponding to the finger index, and each additional finger has 8 more bytes of X and Y coordinate values.

6. The method for full gesture simulation reverse control between Android devices according to claim 5, characterized in that: The step S3 comprises the following steps: S31, the process with Shell user group permission on the mobile device parses the received touch gesture package into a specific touch event object, and triggers a gesture event through the InputEvent, int function.

7. The method for full gesture simulation reverse control between Android devices according to claim 6, characterized in that: The step S3 further comprises the following steps: S32: Parse the touch gesture package into the X and Y coordinate axis values ​​corresponding to the gesture action type ActionId and the finger index in a 4-byte one-bit order. For each additional finger, there are 8 more bytes of X and Y coordinate values. The parsed gesture action type ActionId and the X and Y coordinate axis values ​​corresponding to the finger index are encapsulated into a specific touch event object.

8. The method for full gesture simulation reverse control between Android devices according to claim 7, characterized in that: In step S31 , the process of Shell user group permissions on the mobile device is started by the ADBServer&ADBClient tool on the vehicle system device.

9. The method for full gesture simulation reverse control between Android devices according to claim 1, characterized in that: In step S1, the communication layer includes a WiFi wireless channel and a USB wired channel.

Images