Remote touch control method and system for mobile terminal equipment
By performing trajectory thinning on the control end and introducing a jitter buffer queue and timestamp synchronization mechanism on the controlled end, the problems of operation lag and touch point offset in weak network environments during remote assistance of mobile terminal devices are solved, achieving smooth and stable remote touch control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI DROI TECH CO LTD
- Filing Date
- 2026-01-19
- Publication Date
- 2026-05-01
AI Technical Summary
Existing remote assistance solutions for mobile terminal devices are prone to out-of-order or lost touch data packets in weak network environments, resulting in operation lag and touch point misalignment. Furthermore, they rely on high-bandwidth internet connections and high-level authorization, making installation difficult and unable to achieve smooth swipe trajectory restoration.
A trajectory thinning processing method is introduced on the control end, which sends only key touch points and packages them through the IMS network signaling channel. On the controlled end, a jitter buffer queue and a relative timestamp synchronization mechanism are introduced. Combined with a curve interpolation algorithm, trajectory fitting and reconstruction are performed to ensure that the touch point sequence is executed in the trigger order and delay rhythm.
It reduces the amount of communication data, avoids network signaling channel blockage, improves the smoothness, stability and adaptability of remote touch control of mobile terminal devices, and solves the problems of operation logic errors and touch point offset in weak network environments.
Smart Images
Figure CN121967398A_ABST
Abstract
Description
A control method and system for remote touch control of a mobile terminal device Technical Field
[0001] This application relates to the field of remote touch technology for mobile terminal devices, and in particular to a control method and system for remote touch control of mobile terminal devices. Background Technology
[0002] The current remote assistance scenarios for smartphones mainly face the following technical pain points: the complex installation and authorization process of third-party software, and the risk of privacy leakage due to data being relayed through third-party servers; existing remote control solutions are usually based on real-time transmission of TCP protocol, and in weak network environments, touch data packets are prone to out-of-order or loss, resulting in problems such as "disconnection", "jumping" or "operation lag" on the controlled end, and the inability to restore smooth sliding trajectory; when the resolution of the control end and the controlled end is inconsistent, simple coordinate mapping will produce problems such as touch point offset.
[0003] Among them, the remote desktop control method based on third-party software relies on a high-bandwidth Internet connection to transmit all touch point data in real time. However, this method has poor resistance to weak networks. Once packets are lost, the trajectory will be broken. In addition, it requires high-level authorization or accessibility permissions, which makes the installation threshold high. In the ordinary native video call method, only audio and video streams are transmitted. There is a lack of control signaling channels, so interaction cannot be realized. Summary of the Invention
[0004] In view of this, the purpose of this application is to provide a control method and system for remote touch control of mobile terminal devices. By introducing a trajectory thinning processing method at the control end, only key touch points in the touch point sequence are sent, reducing the amount of communication data in the touch point sequence and avoiding the problem of dense touch data blocking the network signaling channel. The touch data is packaged and sent to the controlled end through the IMS network signaling channel. A jitter buffer queue and relative timestamp synchronization mechanism are introduced at the controlled end to ensure that the touch point sequence is executed strictly according to the triggering order and delay rhythm, preventing operational logic errors caused by network disorder. The trajectory fitting and reconstruction are performed through a curve interpolation algorithm, and a smooth transition trajectory is automatically calculated based on the key points before and after, improving the smoothness, stability and adaptability of remote touch control of mobile terminal devices.
[0005] This application provides a control method for remote touch control of a mobile terminal device. The control method is used in a control system for remote touch control of a mobile terminal device. The control system includes a control terminal and a controlled terminal, both of which are mobile terminal devices. The control method includes: the control terminal responding to an externally triggered touch event, acquiring the original touch point sequence corresponding to the touch event, and thinning the original touch point sequence to obtain a first touch point sequence; packaging and sending the first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence to the controlled terminal through a preset IMS network signaling channel; and the controlled terminal responding... Upon receiving the first touch point sequence and the time increment, the first touch point sequence is stored in a preset jitter buffer queue, and sorted in the jitter buffer queue to obtain a second touch point sequence. Based on the time increment, the second touch point sequence is reconstructed by trajectory fitting to fill in the missing touch points in the second touch point sequence, thus obtaining a target touch point sequence. The coordinates corresponding to each touch point in the target touch point sequence are normalized, and the normalized coordinates are mapped to the absolute coordinates corresponding to the local resolution of the controlled terminal. The touch effect corresponding to the absolute coordinates is then executed according to the time increment.
[0006] Furthermore, the step of obtaining the original touch point sequence corresponding to the touch event in response to an externally triggered touch event by the control terminal includes: the controlled terminal sending screen recording data corresponding to the controlled terminal to the control terminal; the control terminal receiving the screen recording data and feeding the screen recording data back to the outside; in response to an external touch operation performed on the control terminal based on the screen recording data, determining that a touch event has been triggered; and in response to the externally triggered touch event, obtaining multiple touch points corresponding to the touch operation on the control terminal to determine the original touch point sequence corresponding to the touch points.
[0007] Further, the step of thinning the trajectory of the original touch point sequence to obtain the first touch point sequence includes: determining the line connecting the starting touch point and the ending touch point based on the coordinates corresponding to the starting touch point and the ending touch point in the original touch point sequence; determining the vertical distance value from the touch point to the line for each touch point in the original touch point sequence other than the starting touch point and the ending touch point, and determining whether the vertical distance value is less than a preset distance threshold; identifying the touch points whose vertical distance value is less than the preset distance threshold as redundant touch points, and identifying the touch points whose vertical distance value is greater than or equal to the preset distance threshold as key touch points; removing the redundant touch points in the original touch point sequence, and retaining the key touch points in the original touch point sequence to obtain the first touch point sequence.
[0008] Furthermore, the step of reconstructing the trajectory of the second touch point sequence based on the time increment to fill in the missing touch points in the second touch point sequence and obtain the target touch point sequence includes: reading each touch point in the second touch point sequence at a uniform speed according to a preset buffer delay time; for each pair of adjacent key touch points in the second touch point sequence, determining the trajectory midpoint between each pair of adjacent key touch points based on the time increment corresponding to the key touch point, the coordinates corresponding to the key touch point, and the control point corresponding to the key touch point using a preset Bezier interpolation formula; adding the trajectory midpoint to the corresponding position in the second touch point sequence to obtain the target touch point sequence, thereby filling in the missing touch points in the second touch point sequence.
[0009] Furthermore, the step of normalizing the coordinates corresponding to each touch point in the target touch point sequence, mapping the normalized coordinates to absolute coordinates corresponding to the local resolution of the controlled terminal, and executing the touch effect corresponding to the absolute coordinates according to the time increment includes: obtaining the screen width and screen height values corresponding to the control terminal; determining the relative horizontal and vertical proportions corresponding to each touch point based on the coordinates corresponding to each touch point in the target touch point sequence, the screen width value, and the screen height value, so as to normalize the coordinates corresponding to each touch point; and mapping the coordinates corresponding to each touch point to absolute coordinates corresponding to the local resolution of the controlled terminal based on the relative horizontal and vertical proportions.
[0010] This application embodiment also provides a control system for remote touch control of a mobile terminal device. The control system includes a control terminal and a controlled terminal, both of which are mobile terminal devices. The control terminal is configured to, in response to an externally triggered touch event, acquire the original touch point sequence corresponding to the touch event, and perform trajectory thinning on the original touch point sequence to obtain a first touch point sequence; package and send the first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence to the controlled terminal through a preset IMS network signaling channel; the controlled terminal is configured to, in response to receiving the first touch point sequence and the... Based on the time increment, the first touch point sequence is stored in a preset jitter buffer queue, and the first touch point sequence is sorted in the jitter buffer queue to obtain a second touch point sequence. Based on the time increment, the second touch point sequence is reconstructed by trajectory fitting to fill in the missing touch points in the second touch point sequence to obtain a target touch point sequence. The coordinates corresponding to each touch point in the target touch point sequence are normalized, and the normalized coordinates are mapped to the absolute coordinates corresponding to the local resolution of the controlled terminal. The touch effect corresponding to the absolute coordinates is executed according to the time increment.
[0011] Furthermore, the control terminal includes: a trajectory thinning module, used to respond to an externally triggered touch event, obtain the original touch point sequence corresponding to the touch event, and perform trajectory thinning on the original touch point sequence to obtain a first touch point sequence; and a data sending module, used to package and send the first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence to the controlled terminal through a preset IMS network signaling channel.
[0012] Furthermore, when the trajectory thinning module is used to obtain the original touch point sequence corresponding to the touch event in response to an externally triggered touch event, the trajectory thinning module is used to: send the screen recording data corresponding to the controlled terminal to the control terminal; receive the screen recording data and feed it back to the outside; determine the triggered touch event in response to an external touch operation performed on the control terminal based on the screen recording data; and obtain multiple touch points corresponding to the touch operation on the control terminal in response to the externally triggered touch event, so as to determine the original touch point sequence corresponding to the touch points.
[0013] Furthermore, when the trajectory thinning module performs trajectory thinning on the original touch point sequence to obtain the first touch point sequence, the trajectory thinning module is used to: determine the line connecting the starting touch point and the ending touch point based on the coordinates corresponding to the starting touch point and the ending touch point in the original touch point sequence; for each touch point in the original touch point sequence other than the starting touch point and the ending touch point, determine the vertical distance value from the touch point to the line, and determine whether the vertical distance value is less than a preset distance threshold; identify the touch points whose vertical distance value is less than the preset distance threshold as redundant touch points, and identify the touch points whose vertical distance value is greater than or equal to the preset distance threshold as key touch points; remove the redundant touch points in the original touch point sequence, and retain the key touch points in the original touch point sequence to obtain the first touch point sequence.
[0014] Furthermore, the controlled terminal includes: a jitter buffer module, configured to, in response to receiving the first touch point sequence and the time increment, store the first touch point sequence into a preset jitter buffer queue, and sort the first touch point sequence in the jitter buffer queue to obtain a second touch point sequence; a trajectory reconstruction module, configured to, based on the time increment, perform trajectory fitting and reconstruction on the second touch point sequence to fill in the missing touch points in the second touch point sequence to obtain a target touch point sequence; and a coordinate mapping module, configured to, normalize the coordinates corresponding to each touch point in the target touch point sequence, map the normalized coordinates to the absolute coordinates corresponding to the local resolution of the controlled terminal, and execute the touch effect corresponding to the absolute coordinates according to the time increment.
[0015] Furthermore, when the trajectory reconstruction module is used to perform trajectory fitting and reconstruction on the second touch point sequence based on the time increment to fill in the missing touch points in the second touch point sequence and obtain the target touch point sequence, the trajectory reconstruction module is used to: read each touch point in the second touch point sequence at a uniform speed according to a preset buffer delay time; for each pair of adjacent key touch points in the second touch point sequence, based on the time increment corresponding to the key touch point, the coordinates corresponding to the key touch point, and the control point corresponding to the key touch point, determine the trajectory midpoint between each pair of adjacent key touch points using a preset Bezier interpolation formula; add the trajectory midpoint to the corresponding position in the second touch point sequence to obtain the target touch point sequence, so as to fill in the missing touch points in the second touch point sequence.
[0016] Furthermore, when the coordinate mapping module normalizes the coordinates corresponding to each touch point in the target touch point sequence and maps the normalized coordinates to absolute coordinates corresponding to the local resolution of the controlled terminal, and executes the touch effect corresponding to the absolute coordinates according to the time increment, the coordinate mapping module is used to: obtain the screen width and screen height values corresponding to the control terminal; determine the relative horizontal and vertical proportions corresponding to each touch point based on the coordinates, screen width, and screen height values in the target touch point sequence, so as to normalize the coordinates corresponding to each touch point; and map the coordinates corresponding to each touch point to absolute coordinates corresponding to the local resolution of the controlled terminal based on the relative horizontal and vertical proportions.
[0017] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the remote touch control method for mobile terminal devices described above are performed.
[0018] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the remote touch control method for a mobile terminal device as described above.
[0019] This application provides a control method and system for remote touch control of a mobile terminal device. The control method is used in a control system for remote touch control of a mobile terminal device. The control system includes a control terminal and a controlled terminal, both of which are mobile terminal devices. The control method includes: the control terminal responding to an externally triggered touch event, acquiring the original touch point sequence corresponding to the touch event, and thinning the original touch point sequence to obtain a first touch point sequence; packaging and sending the first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence to the controlled terminal through a preset IMS network signaling channel; and the controlled terminal responding to... Upon receiving the first touch point sequence and the time increment, the first touch point sequence is stored in a preset jitter buffer queue, and sorted in the jitter buffer queue to obtain a second touch point sequence. Based on the time increment, the second touch point sequence is reconstructed by trajectory fitting to fill in the missing touch points in the second touch point sequence, thus obtaining a target touch point sequence. The coordinates corresponding to each touch point in the target touch point sequence are normalized, and the normalized coordinates are mapped to the absolute coordinates corresponding to the local resolution of the controlled terminal. The touch effect corresponding to the absolute coordinates is then executed according to the time increment.
[0020] Compared to existing remote desktop control methods based on third-party software and ordinary native video calls, this method introduces a trajectory thinning process on the control end, sending only key touch points in the touch point sequence. This reduces the amount of communication data in the touch point sequence, avoiding the problem of dense touch data blocking the network signaling channel. The touch data is packaged and sent to the controlled end through the IMS network signaling channel. A jitter buffer queue and relative timestamp synchronization mechanism are introduced on the controlled end to ensure that the touch point sequence is executed strictly according to the trigger order and delay rhythm, preventing operational logic errors caused by network disorder. Furthermore, a curve interpolation algorithm is used for trajectory fitting and reconstruction, automatically calculating a smooth transition trajectory based on the key points before and after, thus improving the smoothness, stability, and adaptability of remote touch control of mobile terminal devices.
[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 is a flowchart of a remote touch control method for a mobile terminal device provided in an embodiment of this application; Figure 2 is a structural schematic diagram of a remote touch control system for a mobile terminal device provided in an embodiment of this application; Figure 3 is a structural schematic diagram of a control terminal provided in an embodiment of this application; Figure 4 is a structural schematic diagram of a controlled terminal provided in an embodiment of this application; Figure 5 is a structural schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.
[0025] Research has revealed the following key technical challenges in current smartphone remote assistance scenarios: complex installation and authorization processes for third-party software, with data relayed through third-party servers, posing a privacy risk; existing remote control solutions typically rely on real-time TCP transmission, which can lead to out-of-order or lost touch data packets in weak network environments, causing issues such as "touch interruption," "jumping," or "operation lag" on the controlled device, making it impossible to reproduce smooth swipe trajectories; and when the resolutions of the control and controlled devices are inconsistent, simple coordinate mapping can result in touch point offset issues.
[0026] Remote desktop control methods based on third-party software rely on high-bandwidth internet connections to transmit all touch point data in real time. However, this method is not robust against weak network conditions, and the data path will be interrupted if packets are lost. Furthermore, it requires high-level authorization or accessibility permissions, making installation difficult. In contrast, ordinary native video calling methods only transmit audio and video streams and lack control signaling channels, making interaction impossible.
[0027] Based on this, this application provides a remote touch control method for mobile terminal devices. By introducing a trajectory thinning processing method at the control end, only key touch points in the touch point sequence are sent, reducing the amount of communication data in the touch point sequence and avoiding the problem of dense touch data blocking the network signaling channel. The touch data is packaged and sent to the controlled end through the IMS network signaling channel. A jitter buffer queue and relative timestamp synchronization mechanism are introduced at the controlled end to ensure that the touch point sequence is executed strictly according to the triggering order and delay rhythm, preventing operational logic errors caused by network disorder. The trajectory fitting and reconstruction are performed through a curve interpolation algorithm, and a smooth transition trajectory is automatically calculated based on the key points before and after, improving the smoothness, stability and adaptability of remote touch control of mobile terminal devices.
[0028] Please refer to Figure 1, which is a flowchart of a remote touch control method for a mobile terminal device provided in an embodiment of this application. As shown in Figure 1, the remote touch control method for a mobile terminal device provided in this embodiment of the application is generally applied to a control system for remote touch control of a mobile terminal device; wherein, the control system includes a control terminal and a controlled terminal, both of which are mobile terminal devices, and the control method includes: S101, the control terminal responds to an externally triggered touch event, obtains the original touch point sequence corresponding to the touch event, and performs trajectory thinning on the original touch point sequence to obtain a first touch point sequence.
[0029] It should be noted that the control terminal is an external mobile terminal device that controls the controlled terminal via a touch screen, and the controlled terminal is an external mobile terminal device that controls the screen of the control terminal via a touch screen.
[0030] Here, mobile terminal devices refer to small electronic devices with independent data processing capabilities and human-computer interaction functions, which can access networks via wired or wireless means. They are the core carriers for touch interaction, information processing, and data transmission. Examples include handheld mobile terminals such as smartphones, tablets, handheld game consoles, and portable navigators; wearable mobile terminals such as smartwatches, smart bracelets, and smart glasses; and specialized mobile terminals such as industrial handheld terminals, medical portable terminals, and in-vehicle intelligent terminals.
[0031] In this embodiment, the touch event refers to an input signal event triggered by user touch operation, external trigger command or device sensing action, which is recognized and captured by the hardware or software layer of the control terminal (such as touch screen controller, embedded main control module), and is the core carrier for triggering touch interaction logic.
[0032] Here, the original touch point sequence is a collection of continuous touch points acquired by the control terminal in chronological order during the process of capturing touch events. It has not undergone any filtering, fitting, coordinate mapping, or noise reduction processing and is the core data carrier corresponding to the touch events.
[0033] The first touch point sequence is the touch point sequence obtained by thinning the trajectory of the original touch point sequence.
[0034] In the embodiments of this application, each touch point sequence includes the touch time and touch point coordinates corresponding to multiple touch points.
[0035] In one possible implementation of this application, in specific implementation, the step S101 in which the control terminal responds to an externally triggered touch event and obtains the original touch point sequence corresponding to the touch event may include: S1011, the controlled terminal sends the screen recording data corresponding to the controlled terminal to the control terminal.
[0036] In this step, the controlled terminal sends the screen recording data corresponding to the controlled terminal to the control terminal through a preset IMS network signaling channel.
[0037] The screen recording data refers to the digital data stream generated by the controlled terminal after real-time acquisition and encoding of its own screen display content, i.e., screen visual content.
[0038] For example, the data format of screen recording data can be divided into raw frame data (such as uncompressed image sequences in RGB / YUV format) or compressed encoded data (such as streaming media data in formats such as H.264, H.265, and MP4); the data composition can contain only screen image data, or it can synchronously contain audio data of screen operations (such as system prompts and video playback sounds); the timing feature includes timestamp information, which is used by the control terminal to realize the synchronous restoration of the picture after receiving it.
[0039] Here, the IMS network signaling channel is a logical transmission channel in the IP Multimedia Subsystem (IMS) architecture used to carry various signaling messages, realize the interaction of control commands and session management between network entities. It is the core carrier for the IMS system to complete session establishment, maintenance and release. As an IP-based multimedia communication architecture, IMS separates its signaling channel from its media channel (carrying voice, video and other service data), which conforms to the core design principle of separation of control and bearer.
[0040] S1012. The control terminal receives the screen recording data and feeds the screen recording data back to the outside.
[0041] In this step, the control terminal displays the received screen recording data on the corresponding screen on the control terminal.
[0042] S1013. In response to an external touch operation performed on the control terminal based on the screen recording data, a touch event is determined to be triggered.
[0043] In this step, the user records data on the screen of the controlled terminal and performs touch operations on the corresponding screen of the control terminal to determine the triggering of the touch event.
[0044] S1014. In response to an externally triggered touch event, obtain multiple touch points corresponding to the touch operation on the control terminal to determine the original touch point sequence corresponding to the touch points.
[0045] In this step, multiple touch points corresponding to the touch operation on the control terminal are obtained, as well as the touch time and coordinates corresponding to each touch point, in order to determine the original touch point sequence corresponding to the touch point.
[0046] For example, the control unit listens for touch events (OnTouch) to obtain the original touch point sequence with a frequency of up to "60Hz-120Hz".
[0047] In one possible implementation of this application, in specific implementation, the step of thinning the trajectory of the original touch point sequence to obtain the first touch point sequence in step S101 may include: S1015, determining the connection between the starting touch point and the ending touch point based on the coordinates corresponding to the starting touch point and the ending touch point in the original touch point sequence.
[0048] S1016. For each touch point in the original touch point sequence other than the starting touch point and the ending touch point, determine the vertical distance value from the touch point to the connecting line, and determine whether the vertical distance value is less than a preset distance threshold.
[0049] In this embodiment of the application, in order to solve the problem of signaling bandwidth congestion caused by excessively high touch sampling rate, the Douglas-Peucker thinning algorithm is adopted to retain key touch points and discard redundant touch points in order to thin the trajectory of the original touch point sequence.
[0050] Here, the preset distance threshold represents a filtering threshold (for example, the preset distance threshold can be set to 5 pixels).
[0051] S1017. The touch points whose vertical distance value is less than the preset distance threshold are determined as redundant touch points, and the touch points whose vertical distance value is greater than or equal to the preset distance threshold are determined as key touch points.
[0052] S1018. Remove the redundant touch points in the original touch point sequence and retain the key touch points in the original touch point sequence to obtain the first touch point sequence.
[0053] In this embodiment, if the vertical distance between the current touch point and the line connecting the start touch point and the end touch point is less than a preset distance threshold, the touch point is determined to be a redundant touch point on a straight line and is discarded; if the vertical distance between the current touch point and the line connecting the start touch point and the end touch point is greater than or equal to the preset distance threshold, the touch point is retained as a key touch point, thereby greatly reducing the amount of data transmitted.
[0054] S102. The first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence are packaged and sent to the controlled terminal through a preset IMS network signaling channel.
[0055] In this step, the time increment of each touch point relative to the starting touch point in the first touch point sequence is recorded, and the first touch point sequence and the time increment corresponding to each touch point are packaged into a data packet in "Protocol Buffer" format.
[0056] Furthermore, the packaged data packets are sent to the controlled terminal through a preset IMS network signaling channel.
[0057] In this embodiment, the preset IMS network signaling channel is a low-latency signaling transmission channel, which ensures the real-time responsiveness of the operation and avoids the high latency caused by the traditional "HTTP interface". Specifically, this embodiment directly embeds the operator's IMS link and provides two transmission modes to enable data transmission communication from the control end to the controlled end through either transmission mode.
[0058] The first transmission mode is the out-of-band signaling mode (SIP INFO Method), which uses the INFO message method defined in the SIP protocol standard. INFO messages are usually used to transmit additional information during a call (such as DTMF key tones). This channel is reused to carry encapsulated touch command packets, which is suitable for low-frequency, small-data-volume discrete operations such as clicks and long presses, and has high reliability.
[0059] The second transmission mode is the in-band data mode (RTP Header Extension). It utilizes the header extension field of RTP (Real-Time Transport Protocol) data packets. The video stream is transmitted via RTP packets, and the touch coordinate data is directly "piggybacked" into the header of the video data packets. It is suitable for continuous scroll / swipe operations and achieves millisecond-level synchronization. That is, the "finger swipe command" is bound to the "video screen at that moment" for transmission, ensuring that the changes in the controlled end screen are strictly synchronized with the operation command.
[0060] S103. In response to receiving the first touch point sequence and the time increment, the controlled terminal stores the first touch point sequence into a preset jitter buffer queue and sorts the first touch point sequence in the jitter buffer queue to obtain the second touch point sequence.
[0061] In this embodiment of the application, in order to solve the problem of out-of-order or inconsistent speed of instructions caused by network transmission, a jitter buffer and virtual clock synchronization mechanism are introduced, and the actual execution time of each touch point is determined by the following formula.
[0062] .
[0063] in, This indicates the actual execution time of each touch point on the controlled end; This indicates the time it takes for the instruction data packet for each touch point to arrive at the controlled terminal; This indicates the preset buffer delay time (e.g., 20ms).
[0064] In this way, all received first touch point sequences are not executed immediately, but are stored in the jitter buffer queue and rearranged in order. After a forced delay buffer time, they are output at a uniform speed to eliminate the stuttering caused by network jitter.
[0065] In this step, the second touch point sequence is obtained by sorting the first touch point sequence and the time increment of each touch point to fix the out-of-order problem caused by network transmission.
[0066] In the embodiments of this application, the jitter buffer queue is a dynamic caching and scheduling mechanism deployed on the controlled end. Its core function is to solve the problem of inconsistent data packet arrival time caused by network latency fluctuations (i.e., jitter) during the transmission of real-time data streams (such as voice, video, and screen recording streams), and to ensure smooth playback or parsing of media data.
[0067] Here, the second touch point sequence is the touch point sequence obtained by sorting the first touch point sequence in the jitter buffer queue.
[0068] S104. Based on the time increment, the second touch point sequence is reconstructed by trajectory fitting to fill in the missing touch points in the second touch point sequence, so as to obtain the target touch point sequence.
[0069] In this embodiment of the application, for the sparse key touch points in the second touch point sequence, the controlled end uses the second-order Bezier curve algorithm to automatically calculate the touch points with smooth intermediate transitions to fill in the trajectory details that are compressed by the controlled end or lost due to packet loss.
[0070] In one possible implementation of this application, step S104 may include: S1041, reading each touch point in the second touch point sequence at a constant speed according to a preset buffer delay time.
[0071] In this step, each touch point in the second touch point sequence is read at a constant speed based on the preset buffer delay time in the jitter buffer and virtual clock synchronization mechanism.
[0072] S1042. For each pair of adjacent key touch points in the second touch point sequence, based on the time increment corresponding to the key touch point, the coordinates corresponding to the key touch point, and the control point corresponding to the key touch point, the midpoint of the trajectory between each pair of adjacent key touch points is determined using a preset Bezier interpolation formula.
[0073] In this embodiment of the application, the expression of the preset Bessel interpolation (second order) formula is as follows.
[0074] .
[0075] in, This represents the midpoint of the trajectory between any two adjacent key touch points; This indicates the starting key touch point among every two adjacent key touch points; This indicates the endpoint key touch point among every two adjacent key touch points; This indicates the control point corresponding to each pair of adjacent key touch points (the control point determines the curvature of the trajectory curve corresponding to the second touch point sequence). This represents the time progress factor (from 0 to 1) corresponding to the time increment of the key touch point.
[0076] S1043. Add the midpoint of the trajectory to the corresponding position in the second touch point sequence to obtain the target touch point sequence, so as to fill the missing touch points in the second touch point sequence.
[0077] In this embodiment, the controlled terminal automatically generates a series of dense trajectory midpoints between two sparse key touch points using a second-order Bezier interpolation formula to obtain a target touch point sequence, thereby restoring a smooth touch sliding effect. This fills in the missing touch points in the second touch point sequence while repairing trajectory breaks caused by compression or packet loss.
[0078] Here, the target touch point sequence is the touch point sequence obtained by reconstructing the trajectory of the second touch point sequence.
[0079] S105. Normalize the coordinates of each touch point in the target touch point sequence, map the normalized coordinates to the absolute coordinates corresponding to the local resolution of the controlled terminal, and execute the touch effect corresponding to the absolute coordinates according to the time increment.
[0080] In this embodiment, the reconstructed target touch point sequence is mapped to the absolute coordinates of the local resolution and injected into the system layer of the controlled end in strict accordance with the time interval corresponding to the recorded time increment, so as to ensure that the sliding speed and rhythm are completely consistent with the control end.
[0081] In one possible implementation of this application, step S105 may include: S1051, obtaining the screen width value and screen height value corresponding to the control terminal.
[0082] S1052. Based on the coordinates of each touch point in the target touch point sequence, the screen width value, and the screen height value, determine the relative horizontal ratio and relative vertical ratio of each touch point to normalize the coordinates of each touch point.
[0083] Here, the coordinates of each touch point are normalized using a relative proportional mapping method to solve the touch misalignment caused by the inconsistency between the screen resolutions of the control end and the controlled end.
[0084] In this embodiment, the relative horizontal and vertical proportions of each touch point are determined by the following formula.
[0085] .
[0086] in, Indicates relative horizontal proportion; Indicates relative vertical proportion; This indicates the screen width value corresponding to the control terminal; This indicates the screen height value corresponding to the control terminal; , ) represents the coordinates of each of the touch points in the target touch point sequence.
[0087] S1053. Based on the relative horizontal ratio and the relative vertical ratio, map the coordinates corresponding to each touch point to the absolute coordinates corresponding to the local resolution of the controlled terminal.
[0088] In this step, the relative horizontal and vertical scales are multiplied by the corresponding screen width and screen height values of the controlled device to restore the absolute coordinates adapted to the local resolution of the controlled device.
[0089] The remote touch control method for mobile terminal devices provided in this application introduces a trajectory thinning processing method at the control end, sending only key touch points in the touch point sequence, reducing the amount of communication data in the touch point sequence, avoiding the problem of dense touch data blocking the network signaling channel, and packaging and sending the touch data to the controlled end through the IMS network signaling channel. At the controlled end, a jitter buffer queue and a relative timestamp synchronization mechanism are introduced to ensure that the touch point sequence is executed strictly according to the triggering order and delay rhythm, preventing operational logic errors caused by network disorder. The trajectory fitting and reconstruction is performed through a curve interpolation algorithm, and a smooth transition trajectory is automatically calculated based on the key points before and after, improving the smoothness, stability and adaptability of remote touch control of mobile terminal devices.
[0090] Please refer to Figure 2, which is a schematic diagram of the structure of a remote touch control system for a mobile terminal device provided in an embodiment of this application. As shown in Figure 2, the control system 20 includes a control terminal 210 and a controlled terminal 220, both of which are mobile terminal devices. The control terminal 210 is used to respond to an externally triggered touch event, obtain the original touch point sequence corresponding to the touch event, and perform trajectory thinning on the original touch point sequence to obtain a first touch point sequence; the control terminal 210 is used to package and send the first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence to the controlled terminal through a preset IMS network signaling channel; the controlled terminal 220 is used to respond to receiving the first touch point sequence and the... The time increment is used to store the first touch point sequence into a preset jitter buffer queue, and sort the first touch point sequence in the jitter buffer queue to obtain a second touch point sequence. Based on the time increment, the second touch point sequence is reconstructed by trajectory fitting to fill in the missing touch points in the second touch point sequence to obtain a target touch point sequence. The coordinates corresponding to each touch point in the target touch point sequence are normalized, and the normalized coordinates are mapped to the absolute coordinates corresponding to the local resolution of the controlled terminal. The touch effect corresponding to the absolute coordinates is executed according to the time increment.
[0091] Please refer to Figures 3 and 4. Figure 3 is a schematic diagram of the structure of a control terminal provided in an embodiment of this application, and Figure 4 is a schematic diagram of the structure of a controlled terminal provided in an embodiment of this application.
[0092] As shown in Figure 3, the control terminal 210 includes: a trajectory thinning module 211, used to respond to an externally triggered touch event, obtain the original touch point sequence corresponding to the touch event, and perform trajectory thinning on the original touch point sequence to obtain a first touch point sequence; and a data sending module 212, used to package and send the first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence to the controlled terminal through a preset IMS network signaling channel.
[0093] Furthermore, when the trajectory thinning module 211 is used to obtain the original touch point sequence corresponding to the touch event in response to an externally triggered touch event, the trajectory thinning module 211 is used to: send the screen recording data corresponding to the controlled terminal to the control terminal; receive the screen recording data and feed the screen recording data back to the outside; determine the triggered touch event in response to an external touch operation performed on the control terminal based on the screen recording data; and obtain multiple touch points corresponding to the touch operation on the control terminal in response to the externally triggered touch event, so as to determine the original touch point sequence corresponding to the touch points.
[0094] Furthermore, when the trajectory thinning module 211 performs trajectory thinning on the original touch point sequence to obtain the first touch point sequence, the trajectory thinning module 211 is used to: determine the line connecting the starting touch point and the ending touch point based on the coordinates corresponding to the starting touch point and the ending touch point in the original touch point sequence; for each touch point in the original touch point sequence other than the starting touch point and the ending touch point, determine the vertical distance value from the touch point to the line, and determine whether the vertical distance value is less than a preset distance threshold; determine the touch points whose vertical distance value is less than the preset distance threshold as redundant touch points, and determine the touch points whose vertical distance value is greater than or equal to the preset distance threshold as key touch points; remove the redundant touch points in the original touch point sequence, and retain the key touch points in the original touch point sequence to obtain the first touch point sequence.
[0095] Further, as shown in Figure 4, the controlled terminal 220 includes: a jitter buffer module 221, used to, in response to receiving the first touch point sequence and the time increment, store the first touch point sequence into a preset jitter buffer queue, and sort the first touch point sequence in the jitter buffer queue to obtain a second touch point sequence; a trajectory reconstruction module 222, used to, based on the time increment, perform trajectory fitting and reconstruction on the second touch point sequence to fill in the missing touch points in the second touch point sequence to obtain a target touch point sequence; and a coordinate mapping module 223, used to, normalize the coordinates corresponding to each touch point in the target touch point sequence, map the normalized coordinates to the absolute coordinates corresponding to the local resolution of the controlled terminal, and execute the touch effect corresponding to the absolute coordinates according to the time increment.
[0096] Furthermore, when the trajectory reconstruction module 222 is used to perform trajectory fitting and reconstruction on the second touch point sequence based on the time increment to fill in the missing touch points in the second touch point sequence and obtain the target touch point sequence, the trajectory reconstruction module 222 is used to: read each touch point in the second touch point sequence at a uniform speed according to a preset buffer delay time; for each pair of adjacent key touch points in the second touch point sequence, based on the time increment corresponding to the key touch point, the coordinates corresponding to the key touch point, and the control point corresponding to the key touch point, determine the trajectory midpoint between each pair of adjacent key touch points using a preset Bezier interpolation formula; add the trajectory midpoint to the corresponding position in the second touch point sequence to obtain the target touch point sequence, so as to fill in the missing touch points in the second touch point sequence.
[0097] Furthermore, when the coordinate mapping module 223 normalizes the coordinates corresponding to each touch point in the target touch point sequence and maps the normalized coordinates to absolute coordinates corresponding to the local resolution of the controlled terminal, and executes the touch effect corresponding to the absolute coordinates according to the time increment, the coordinate mapping module 223 is used to: obtain the screen width and screen height values corresponding to the control terminal; determine the relative horizontal ratio and relative vertical ratio corresponding to each touch point based on the coordinates corresponding to each touch point in the target touch point sequence, the screen width value, and the screen height value, so as to normalize the coordinates corresponding to each touch point; and map the coordinates corresponding to each touch point to absolute coordinates corresponding to the local resolution of the controlled terminal based on the relative horizontal ratio and the relative vertical ratio.
[0098] The remote touch control system for mobile terminal devices provided in this application introduces a trajectory thinning processing method at the control end, sending only key touch points in the touch point sequence, reducing the amount of communication data in the touch point sequence, avoiding the problem of dense touch data blocking the network signaling channel, and packaging and sending the touch data to the controlled end through the IMS network signaling channel. At the controlled end, a jitter buffer queue and a relative timestamp synchronization mechanism are introduced to ensure that the touch point sequence is executed strictly according to the triggering order and delay rhythm, preventing operational logic errors caused by network disorder. Furthermore, a curve interpolation algorithm is used for trajectory fitting and reconstruction, automatically calculating a smooth transition trajectory based on the key points before and after, thereby improving the smoothness, stability, and adaptability of remote touch control for mobile terminal devices.
[0099] Please refer to Figure 5, which is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 5, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.
[0100] The memory 520 stores machine-readable instructions that can be executed by the processor 510. When the electronic device 500 is running, the processor 510 and the memory 520 communicate through the bus 530. When the machine-readable instructions are executed by the processor 510, the steps of the remote touch control method for mobile terminal devices as shown in the method embodiment of Figure 1 above can be executed. For specific implementation methods, please refer to the method embodiment, which will not be repeated here.
[0101] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it can execute the steps of the remote touch control method for a mobile terminal device as shown in the method embodiment of FIG1 above. For specific implementation details, please refer to the method embodiment, which will not be repeated here.
[0102] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0103] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0105] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0106] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0107] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A remote touch control method for a mobile terminal device, characterized in that, The control method is used in a control system for remote touch control of a mobile terminal device; wherein, the control system includes a control terminal and a controlled terminal, both of which are mobile terminal devices, and the control method includes: the control terminal responding to an externally triggered touch event to obtain an original touch point sequence corresponding to the touch event, and performing trajectory thinning on the original touch point sequence to obtain a first touch point sequence; packaging and sending the first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence to the controlled terminal through a preset IMS network signaling channel; and the controlled terminal responding to receiving the first touch point sequence... Based on the time increment, the first touch point sequence is stored in a preset jitter buffer queue, and sorted in the jitter buffer queue to obtain a second touch point sequence. Based on the time increment, trajectory fitting and reconstruction are performed on the second touch point sequence to fill in the missing touch points in the second touch point sequence, resulting in a target touch point sequence. The coordinates corresponding to each touch point in the target touch point sequence are normalized, and the normalized coordinates are mapped to the absolute coordinates corresponding to the local resolution of the controlled terminal. The touch effect corresponding to the absolute coordinates is then executed according to the time increment.
2. The method according to claim 1, characterized in that, The step of obtaining the original touch point sequence corresponding to the touch event in response to an externally triggered touch event by the control terminal includes: the controlled terminal sending screen recording data corresponding to the controlled terminal to the control terminal; the control terminal receiving the screen recording data and feeding the screen recording data back to the outside; in response to an external touch operation performed on the control terminal based on the screen recording data, determining that a touch event has been triggered; and in response to the externally triggered touch event, obtaining multiple touch points corresponding to the touch operation on the control terminal to determine the original touch point sequence corresponding to the touch points.
3. The method according to claim 1, characterized in that, The step of thinning the trajectory of the original touch point sequence to obtain the first touch point sequence includes: determining the line connecting the starting touch point and the ending touch point based on the coordinates corresponding to the starting touch point and the ending touch point in the original touch point sequence; determining the vertical distance value from the touch point to the line for each touch point in the original touch point sequence other than the starting touch point and the ending touch point, and determining whether the vertical distance value is less than a preset distance threshold; identifying the touch points whose vertical distance value is less than the preset distance threshold as redundant touch points, and identifying the touch points whose vertical distance value is greater than or equal to the preset distance threshold as key touch points; removing the redundant touch points in the original touch point sequence, and retaining the key touch points in the original touch point sequence to obtain the first touch point sequence.
4. The method according to claim 1, characterized in that, The step of reconstructing the trajectory of the second touch point sequence based on the time increment to fill in the missing touch points in the second touch point sequence and obtain the target touch point sequence includes: reading each touch point in the second touch point sequence at a uniform speed according to a preset buffer delay time; for each pair of adjacent key touch points in the second touch point sequence, determining the trajectory midpoint between each pair of adjacent key touch points based on the time increment corresponding to the key touch point, the coordinates corresponding to the key touch point, and the control point corresponding to the key touch point using a preset Bezier interpolation formula; adding the trajectory midpoint to the corresponding position in the second touch point sequence to obtain the target touch point sequence to fill in the missing touch points in the second touch point sequence.
5. The method according to claim 1, characterized in that, The step of normalizing the coordinates of each touch point in the target touch point sequence and mapping the normalized coordinates to absolute coordinates corresponding to the local resolution of the controlled terminal, and executing the touch effect corresponding to the absolute coordinates according to the time increment, includes: obtaining the screen width and screen height values corresponding to the control terminal; determining the relative horizontal and vertical proportions of each touch point based on the coordinates of each touch point in the target touch point sequence, the screen width value, and the screen height value, so as to normalize the coordinates of each touch point; and mapping the coordinates of each touch point to absolute coordinates corresponding to the local resolution of the controlled terminal based on the relative horizontal and vertical proportions.
6. A control system for remote touch control of a mobile terminal device, characterized in that, The control system includes a control terminal and a controlled terminal, both of which are mobile terminal devices; the control terminal is used to respond to an externally triggered touch event, obtain the original touch point sequence corresponding to the touch event, and perform trajectory thinning on the original touch point sequence to obtain a first touch point sequence; The first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence are packaged and sent to the controlled terminal through a preset IMS network signaling channel; The controlled terminal is configured to, in response to receiving the first touch point sequence and the time increment, store the first touch point sequence into a preset jitter buffer queue, and sort the first touch point sequence in the jitter buffer queue to obtain the second touch point sequence; Based on the time increment, the second touch point sequence is reconstructed by trajectory fitting to fill in the missing touch points in the second touch point sequence, thereby obtaining the target touch point sequence; The coordinates corresponding to each touch point in the target touch point sequence are normalized, and the normalized coordinates are mapped to the absolute coordinates corresponding to the local resolution of the controlled terminal. The touch effect corresponding to the absolute coordinates is executed according to the time increment.
7. The control system according to claim 6, characterized in that, The control terminal includes: a trajectory thinning module, used to respond to an externally triggered touch event, obtain the original touch point sequence corresponding to the touch event, and perform trajectory thinning on the original touch point sequence to obtain a first touch point sequence; and a data sending module, used to package and send the first touch point sequence and the time increment corresponding to each touch point in the first touch point sequence to the controlled terminal through a preset IMS network signaling channel.
8. The control system according to claim 6, characterized in that, The controlled terminal includes: a jitter buffer module, configured to, in response to receiving the first touch point sequence and the time increment, store the first touch point sequence into a preset jitter buffer queue, and sort the first touch point sequence in the jitter buffer queue to obtain a second touch point sequence; a trajectory reconstruction module, configured to, based on the time increment, perform trajectory fitting and reconstruction on the second touch point sequence to fill in the missing touch points in the second touch point sequence to obtain a target touch point sequence; and a coordinate mapping module, configured to, normalize the coordinates corresponding to each touch point in the target touch point sequence, map the normalized coordinates to the absolute coordinates corresponding to the local resolution of the controlled terminal, and execute the touch effect corresponding to the absolute coordinates according to the time increment.
9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the remote touch control method for a mobile terminal device as described in any one of claims 1 to 5.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the remote touch control method for a mobile terminal device as described in any one of claims 1 to 5.