Operation event transmission method and device

Abstract

This application discloses an operation event transmission method and apparatus. The method includes: synchronizing the timestamps in a control terminal used to control a remote desktop with the timestamps in a controlled terminal according to a preset period to obtain synchronized timestamps; collecting operation events in the control terminal and assigning synchronized timestamps to the operation events; and sending the operation events and their corresponding timestamps to the controlled terminal, wherein the controlled terminal executes the operation events according to the order of the operation events and their corresponding timestamps. The method provided in this application at least solves the technical problem in related technologies where the use of fixed-frequency sampling leads to high execution deviations in operation events generated by the control terminal.

Description

Technical Field

[0001] This application relates to the field of remote desktop control technology, and more specifically, to an operation event transmission method and apparatus. Background Technology

[0002] In related technologies, remote desktop controller redirection solutions periodically collect controller status data on the client side and send it to the server side via a dedicated transmission channel. The server-side virtual device driver then injects the received status data into the operating system's input subsystem. At the data acquisition level, existing technologies generally employ a fixed-frequency polling mechanism. The system periodically reads complete status data of the controller device at preset fixed time intervals (typically 60Hz, i.e., every 16.67 milliseconds), including all button states, joystick positions, trigger pressure, etc., and then packages and transmits the collected status snapshots to the server side. At the dead-zone handling level, existing technologies use a hard threshold truncation method to handle joystick dead zones. When the absolute value of the joystick offset is less than a preset threshold, the system directly outputs zero; when the offset exceeds the threshold, the system directly outputs the original value. This processing method can be represented as a piecewise function: output 0 when |x|≤θ, output x when |x|>θ, where θ is the dead-zone threshold. When multiple users and multiple handles control the same controlled terminal, the existing technology has limited support for concurrent scenarios of multiple users and multiple handles. It usually adopts a simple first-come-first-served scheduling strategy and lacks precise timing synchronization and conflict arbitration mechanisms. The existing technology has achieved remote control of handles and small jitter caused by equipment aging or environmental disturbances, but the following problems still exist: (1) Time quantization error caused by fixed frequency sampling. The existing technology uses a fixed frequency polling mechanism to collect handle data, which has an unavoidable time quantization error: when the user performs handle operation, there is a time deviation between the actual time of the input event and the most recent sampling time. This deviation is evenly distributed between 0 and the sampling interval. Taking a 60Hz sampling frequency as an example, the average time error is 8.33 milliseconds and the maximum error can reach 16.67 milliseconds. Therefore, any two events that occur within the same sampling interval are identified and sent at the sampling time, which also leads to the uncontrollability and randomness of their transmission to the remote end and their occurrence order. The standard polling frequency of modern USB gamepads is 1000Hz, and theoretically the minimum interval of a single operation is only 1 millisecond. Therefore, the low-frequency polling method will lead to the omission of key gamepad operation events or timing distortion, which will seriously affect the user's operation experience and game fairness in fast-paced game scenarios. (2) Timing inconsistency problem in multi-user concurrent scenarios: In multi-user multi-gamepad concurrent scenarios, the existing technology uses independent sampling clocks for different control terminals, resulting in inconsistent time bases. Even if two users perform operations at the same time in physical time, the execution order of their events on the controlled terminal may be reversed due to the randomness of the sampling time. This timing uncertainty destroys the consistent experience of multi-player cooperative games. When multiple players try to control the same game object at the same time, unpredictable competitive results will be generated. The existing technology lacks an effective timing recovery mechanism and can only rely on the event reception order for simple scheduling, which cannot guarantee the fairness and correctness of the operation.(3) Abrupt change problem caused by dead zone offset processing: Existing technology uses hard threshold truncation to process the joystick dead zone, resulting in obvious abrupt changes in the control signal at the dead zone boundary. When the user performs fine operations, the slight offset of the joystick from the center position cannot be detected by the system until the offset exceeds the preset threshold and it suddenly activates. This non-linear response characteristic seriously affects the intuition and continuity of the control. The state sequence received by the controlled end exhibits typical step discontinuity characteristics, which manifests as the sudden start or stop of character movement in game applications, seriously destroying the naturalness of the control and the immersive experience. Summary of the Invention

[0003] This application provides an operation event transmission method and apparatus to at least solve the technical problem in the related art where the use of fixed frequency sampling leads to high deviations in the execution of operation events generated by the control terminal.

[0004] According to one aspect of the embodiments of this application, an operation event transmission method is provided, comprising: synchronizing a timestamp in a control terminal used to control a remote desktop with a timestamp in a controlled terminal according to a preset period to obtain a synchronized timestamp; collecting operation events in the control terminal and assigning the synchronized timestamp to the operation events; and sending the operation events and corresponding timestamps to the controlled terminal, wherein the controlled terminal is used to execute the operation events according to the order of the operation events and corresponding timestamps.

[0005] Optionally, the timestamp in the control terminal is synchronized with the timestamp of the controlled terminal according to a preset period to obtain a synchronized timestamp, including: sending a timestamp synchronization request to the controlled terminal and receiving the timestamp of the controlled terminal sent by the controlled terminal in response to the timestamp synchronization request; determining the clock offset between the control terminal and the controlled terminal based on the timestamp of the controlled terminal and the timestamp in the control terminal; and determining the synchronized timestamp based on the clock offset.

[0006] Optionally, determining the clock offset based on the timestamp of the controlled terminal and the timestamp in the control terminal includes: determining the round-trip delay between the controlled terminal and the control terminal based on the time when a timestamp synchronization request is sent to the controlled terminal and the time when the timestamp of the controlled terminal is sent in response to the timestamp synchronization request; obtaining multiple round-trip delays within a preset window, and determining the latest delay within the preset window based on the minimum round-trip delay among the multiple round-trip delays; obtaining multiple latest delays closest to the target time, and determining the weighted average of the multiple latest delays closest to the target time as the target value; and determining the clock offset based on the target value.

[0007] Optionally, sending the operation event and its corresponding timestamp to the controlled terminal includes: sending the operation event and its corresponding timestamp to a preset buffer in the controlled terminal, wherein the preset buffer is a sliding time window synchronized with the frame rate of the remote desktop video stream; at the end of the period corresponding to the preset buffer, reordering the operation events in the preset buffer according to the corresponding timestamp to obtain an operation event sequence, wherein the operation event sequence is used by the controlled terminal to execute the operation events sequentially according to the order of the operation events in the operation event sequence.

[0008] Optionally, the method further includes: if the time when the operation event is sent to the controlled terminal exceeds the sliding time window, obtaining the event type of the operation event; if the event type is a discrete event and the duration of the time when the operation event is sent to the controlled terminal exceeds the sliding time window is less than a preset threshold, performing the operation event as compensation; if the event type is a continuous event, directly discarding the operation event.

[0009] Optionally, the method further includes: when the joystick of the control terminal is detected to be in the dead zone of the control terminal, determining the first valid state point detected by the control terminal as the endpoint of the target curve, and determining the origin of the joystick of the control terminal as the starting point of the target curve, wherein the valid state point is used to indicate the location point where the operation event takes effect, and the target curve is used to indicate the movement path of the joystick of the control terminal from the starting point of the target curve to the endpoint of the target curve; obtaining multiple intermediate control points of the target curve, and determining the target curve based on the starting point of the target curve, the multiple intermediate control points of the target curve, and the endpoint of the target curve.

[0010] Optionally, obtaining multiple intermediate control points of the target curve includes: obtaining dead-zone-free movement data and fitting the dead-zone-free movement data to a real joystick movement trajectory; determining adjustment parameters corresponding to the multiple intermediate control points when the target curve and the real joystick movement trajectory reach minimum deviation; and determining the multiple intermediate control points based on the adjustment parameters corresponding to the multiple intermediate control points.

[0011] Optionally, the method further includes: obtaining the transition time from the starting point to the ending point of the target curve and the minimum duration interval that the control terminal can respond to; determining the number of interpolation points in the target curve based on the transition time and the minimum duration interval; determining the position of each interpolation point in the target curve based on the number of interpolation points in the target curve and a predetermined easing function; and determining the target movement trajectory of the joystick of the control terminal based on the target curve and the position of each interpolation point in the target curve.

[0012] According to another aspect of the embodiments of this application, an operation event transmission device is also provided, comprising: a synchronization module, configured to synchronize the timestamp in the control terminal used to control a remote desktop with the timestamp of the controlled terminal according to a preset period to obtain a synchronized timestamp; an allocation module, configured to collect operation events in the control terminal and allocate the synchronized timestamp to the operation events; and a sending module, configured to send the operation events and corresponding timestamps to the controlled terminal, wherein the controlled terminal is configured to execute the operation events according to the order of the operation events and corresponding timestamps.

[0013] According to another aspect of the embodiments of this application, a computer device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described operation event transmission method.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described operation event transmission method.

[0015] In this embodiment, the timestamps in the control terminal used to control the remote desktop are synchronized with the timestamps of the controlled terminal according to a preset period to obtain synchronized timestamps; operation events in the control terminal are collected, and the synchronized timestamps are assigned to the operation events; the operation events and their corresponding timestamps are sent to the controlled terminal, wherein the controlled terminal is used to execute the operation events according to the order of the operation events and their corresponding timestamps. By using an instant collection method triggered by input operation events, the polling collection method in the traditional method is avoided, and the purpose of recording the timestamps in a timely manner when the operation events occur is achieved, thereby achieving the technical effect of accurately executing the operation events. This solves the technical problem in related technologies where the use of fixed-frequency sampling leads to high execution deviations of operation events generated by the control terminal. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing an operation event transmission method according to an embodiment of this application;

[0018] Figure 2 This is a flowchart of an operation event transmission method according to an embodiment of this application;

[0019] Figure 3This is a schematic diagram of an RTT calculation process according to an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of a preset buffer according to an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of a control terminal dead zone according to an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of a target curve according to an embodiment of this application;

[0023] Figure 7 This is a structural diagram of an operation event transmission device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0027] To address the problems existing in related technologies, embodiments of this application provide an operation event transmission method, which can run in... Figure 1 The computer terminal shown is explained below.

[0028] The operation event transmission method embodiments provided in this application can be executed in a mobile terminal, computer terminal or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing an operation event transmission method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the operation event transmission method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned operation event transmission method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0031] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0032] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0033] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0034] In the above operating environment, this application provides an embodiment of an operation event transmission method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0035] Figure 2 This is a flowchart of an operation event transmission method according to an embodiment of this application, such as... Figure 2As shown, the method includes the following steps:

[0036] Step S202: Synchronize the timestamp in the control terminal used to control the remote desktop with the timestamp of the controlled terminal according to a preset period to obtain the synchronized timestamp.

[0037] Step S204: Collect operation events from the control terminal and assign the synchronized timestamp to the operation events;

[0038] Step S206: Send the operation event and its corresponding timestamp to the controlled terminal, wherein the controlled terminal is used to execute the operation event according to the sorting of the operation event and its corresponding timestamp.

[0039] Through steps S202 to S206 above, the timestamps in the control terminal used to control the remote desktop are synchronized with the timestamps of the controlled terminal according to a preset period to obtain synchronized timestamps; operation events in the control terminal are collected, and the synchronized timestamps are assigned to the operation events; the operation events and their corresponding timestamps are sent to the controlled terminal, wherein the controlled terminal executes the operation events according to the order of the operation events and their corresponding timestamps. This real-time acquisition method based on input operation events avoids the polling acquisition method of traditional methods, achieving the goal of timely recording of timestamps when operation events occur, thereby achieving the technical effect of accurate execution of operation events. This solves the technical problem in related technologies where fixed-frequency sampling leads to high execution deviations in operation events generated by the control terminal. A detailed explanation follows.

[0040] In some embodiments of this application, the specific steps for synchronizing the timestamp in the control terminal with the timestamp in the controlled terminal according to a preset period to obtain the synchronized timestamp are as follows: sending a timestamp synchronization request to the controlled terminal and receiving the timestamp of the controlled terminal sent by the controlled terminal in response to the timestamp synchronization request; determining the clock offset between the control terminal and the controlled terminal based on the timestamp of the controlled terminal and the timestamp in the control terminal; and determining the synchronized timestamp based on the clock offset.

[0041] The specific steps for determining the clock offset based on the timestamp of the controlled terminal and the timestamp of the control terminal are as follows: The round-trip delay between the controlled terminal and the control terminal is determined based on the time when a timestamp synchronization request is sent to the controlled terminal and the time when the timestamp of the controlled terminal is received in response to the timestamp synchronization request; multiple round-trip delays within a preset window are obtained, and the minimum round-trip delay among these multiple delays is determined as the latest delay within the preset window; multiple latest delays closest to the target time are obtained, and the weighted average of these multiple latest delays closest to the target time is determined as the target value; the clock offset is determined based on the target value.

[0042] Specifically, addressing the issues of inconsistent event occurrence order caused by fixed-frequency sampling in traditional technologies and high deviations in operation event execution due to timing inconsistencies in multi-user concurrent scenarios, this application employs an event-driven timestamp-based precise replay optimization strategy. The core idea of ​​this solution is to abandon the fixed-frequency sampling mode on the control end and instead adopt an input event-triggered real-time acquisition and transmission mechanism. High-precision timestamps record the exact occurrence time of each operation, and clock-synchronized precise timestamp replay is implemented on the controlled end. The event-driven solution achieves real-time response by acquiring state change events from the underlying controller API: when a user presses a button, moves a joystick, or adjusts a trigger, the system immediately captures the event, records its precise timestamp, and sends it immediately. This mechanism fundamentally eliminates the quantization error of fixed sampling, improving time accuracy to the microsecond level, and avoids inconsistent event occurrence order at remote locations through timestamp recording.

[0043] In events involving timestamp-based replay, clock synchronization is a prerequisite for accurate replay. Since the control and controlled ends operate on different physical devices, their system clocks inevitably deviate. This deviation includes not only static clock differences but also dynamic clock drift. To address this issue, this application designs a minimum-delay clock synchronization algorithm for a star topology. The algorithm's basic principle is to use the controlled end as the sole time reference source, synchronizing all control ends with the controlled end (rather than synchronizing with each other) to ensure a unified time reference. Furthermore, based on the favorable network conditions of the game scenario, the round-trip time (RTT) between the control and controlled ends is measured multiple times, and... As a benchmark for unidirectional delay estimation, the unidirectional network delay is estimated, and then the clock offset at both ends is calculated.

[0044] For example, the control unit periodically sends multiple timestamp synchronization requests to the controlled unit, and the controlled unit immediately replies with its current local timestamp upon receiving each request. The control unit calculates the clock offset using the symmetric network assumption based on the sending and receiving timestamps and the controlled unit's timestamp. Simultaneously, considering the randomness of network latency, the algorithm removes outliers (too large or too small) from the multiple RTT values ​​calculated each time, and selects the smallest RTT value within the normal value window as the latest RTT value for this maintenance. Furthermore, by maintaining the weighted average of the most recent N (usually N=10) RTT measurements, the algorithm effectively suppresses the interference of network jitter on clock synchronization accuracy. The weight allocation for the most recent N RTT measurements uses a time decay mode, with newer measurements having higher weights, ensuring the algorithm's rapid response to changes in network conditions. To address clock drift during system operation, the algorithm implements a periodic synchronization maintenance strategy. The system automatically performs clock resynchronization every 30 seconds, updating the clock offset parameters. When a significant change in network conditions is detected (such as an RTT mutation exceeding a threshold), the system also triggers additional synchronization operations to ensure continuous maintenance of clock accuracy.

[0045] like Figure 3 As shown, the RTT process can be represented as follows: The client sends a message to the server at time t1, the server receives the message at time t2, the server sends an acknowledgment message to the client at time t3, and the client receives the acknowledgment message from the server at time t4. Here, t2-t1=offset+delay, t4-t3=delay-offset, offset represents clock offset, and delay represents network delay.

[0046] In some embodiments of this application, the specific steps for sending the operation event and its corresponding timestamp to the controlled terminal are as follows: The operation event and its corresponding timestamp are sent to a preset buffer in the controlled terminal, wherein the preset buffer is a sliding time window synchronized with the frame rate of the remote desktop video stream; at the end of the period corresponding to the preset buffer, the operation events in the preset buffer are reordered according to the corresponding timestamps to obtain an operation event sequence, wherein the operation event sequence is used by the controlled terminal to execute the operation events sequentially according to the order of the operation events in the operation event sequence. If the time when the operation event is sent to the controlled terminal exceeds the sliding time window, the event type of the operation event is obtained; if the event type is a discrete event and the duration of the time when the operation event is sent to the controlled terminal exceeding the sliding time window is less than a preset threshold, the operation event is executed compensatorily; if the event type is a continuous event, the operation event is directly discarded.

[0047] like Figure 4 As shown, the event replay module on the controlled end bears the crucial task of converting received timestamp events into accurate operations. Considering the instability of network transmission and the synchronization problem of event occurrence among multiple users, an efficient event sequencing mechanism needs to be designed. Based on the aforementioned analysis, this study adopts an event buffering strategy separated by data type to smooth transmission latency and achieve arbitration of the event occurrence order among multiple users. Therefore, an event buffer window mechanism based on frame synchronization is introduced. The controlled end establishes a sliding time window synchronized with the frame rate of the remote desktop video stream (e.g., set to a 13ms window, which is the fastest speed at which humans process visual stimuli). All arriving events first enter the buffer instead of being executed immediately to ensure the event occurrence order under the premise of almost no impact. At the end of the window period, the system reorders all events collected within the window according to their original timestamps, restores the true occurrence order of events, and then schedules them for unified execution. For example: Figure 4 For events corresponding to timestamps t1 and t3, the system employs differentiated processing strategies based on the event type and current state for delayed events arriving beyond the window time limit (as shown in the case of t2 arriving after the window ends). For discrete events like button presses, compensation is performed if the delay is within acceptable limits; otherwise, the event is discarded and logged. For continuous data such as joystick input, due to its time-sensitive nature, expired data is directly discarded to ensure the system always reflects the latest state. This mechanism achieves an optimal balance between delay and accuracy by introducing a controllable buffer delay to ensure the correctness of event timing and effectively reduce latency fluctuations caused by unstable network transmission.

[0048] To address the problem of abrupt changes in dead zones, this application proposes an intelligent path reconstruction algorithm based on cubic Bézier curves. The core idea of ​​this algorithm is based on the physical intuition of motion continuity: any real joystick operation necessarily involves a continuous movement from a stationary state to the target position, a process that can be precisely described mathematically using a smooth parametric curve. Bézier curves, as a mature curve representation method in computer graphics, possess good numerical stability and intuitive geometric meaning. If the connection point between two different curves can be smooth, then at least a cubic curve is required. Curves higher than cubic are prone to jitter and may fail to reflect the shape and regularity of actual objects.

[0049] For example: when the joystick of the control terminal is detected to be in the dead zone of the control terminal, the first valid state point detected by the control terminal is determined as the end point of the target curve, and the origin of the joystick of the control terminal is determined as the start point of the target curve. The valid state point represents the location where the operation event takes effect, and the target curve represents the movement path of the joystick of the control terminal from the start point to the end point of the target curve. Multiple intermediate control points of the target curve are obtained, and the target curve is determined based on the start point, the multiple intermediate control points, and the end point of the target curve. Figure 5 As shown, the dead zone is the area indicated by the dashed circle. Figure 6 This is a schematic diagram of the target curve. Therefore, a cubic Bézier curve is used to reconstruct the path, achieving self-elimination of dead zone offset. This ensures the continuity of joystick operation, avoids abrupt changes or disconnections caused by dead zone issues in game scenarios, and improves the gaming experience of the remote controller.

[0050] With the origin as The intermediate control points are as follows: , The destination is For example, the parametric equation of the target curve is shown below:

[0051]

[0052] In the formula, t represents time.

[0053] in addition, , The constraints are shown in the following equation:

[0054]

[0055] In the formula, Let be the unit vector in the target direction. The direction of the tangent and the initial offset of the curve at the starting point are determined, and its steepness is controlled by α. It will be closer to the effective state point, which determines the steepness of the curve as it approaches the effective state point. The effective state point is represented by α, while the parameters α and β are determined by minimizing the deviation from the actual joystick movement trajectory. Specifically, in some embodiments of this application, obtaining multiple intermediate control points of the target curve includes: obtaining dead-zone-free movement data and fitting the dead-zone-free movement data to the actual joystick movement trajectory; determining the adjustment parameters corresponding to the multiple intermediate control points when the target curve and the actual joystick movement trajectory reach the minimum deviation; determining the multiple intermediate control points based on the adjustment parameters corresponding to the multiple intermediate control points, as shown in the following formula:

[0056]

[0057] In the formula, This represents the actual joystick movement trajectory fitted with dead-zone-free data (fitted with user data without dead zones). and This represents the final confirmed values ​​for adjusting parameters α and β. This represents the target curve to be fitted. This represents the i-th time point.

[0058] This optimization strategy ensures that the reconstructed path meets both mathematical smoothness requirements and closely resembles the operating habits of real users. Since the controlled end needs to begin the reconstruction process immediately upon detecting a sudden change in the dead zone, the density and distribution of interpolation points directly affect the quality of the reconstruction and computational efficiency. Overly dense interpolation increases unnecessary computational overhead, while overly sparse interpolation may result in an insufficiently smooth reconstructed path. Therefore, to balance this trade-off, this application employs an adaptive interpolation density control mechanism based on the target state amplitude. The formula for calculating the number of interpolation points is:

[0059]

[0060] In the formula, This represents the estimated state transition time. This represents the minimum time interval that the system can process.

[0061] The calculation formula is as follows:

[0062]

[0063] In the formula, This represents the average speed at which the joystick moves. This represents the variance of the joystick movement speed among different users.

[0064] The specific confirmation process of the target curve is as follows: obtain the transition time from the starting point to the ending point of the target curve and the minimum time interval that the control terminal can respond to; determine the number of interpolation points in the target curve based on the transition time and the minimum time interval; determine the position of each interpolation point in the target curve based on the number of interpolation points in the target curve and a pre-determined easing function; determine the target movement trajectory of the joystick of the control terminal based on the target curve and the position of each interpolation point in the target curve.

[0065] After determining the number of interpolation points, it is necessary to further clarify the specific location of each interpolation point on the Bézier curve. The most direct method is to uniformly sample the parameter t, that is, let (where i = 0,1,…n). However, due to the parametric nature of Bézier curves, a uniform distribution of parameters does not guarantee a uniform distribution of the arc length of the interpolation points on the curve, which may lead to uneven motion speed of the reconstructed trajectory.

[0066] To more realistically simulate user joystick operation habits, this embodiment employs an ease-in-out nonlinear parameter mapping strategy. This strategy is based on ergonomic observations: real finger operation typically exhibits a movement pattern of "initial acceleration - constant speed in the middle - deceleration near the target." The easing function is defined as:

[0067]

[0068] Applying this easing function to the parameter mapping of the Bézier curve, the position of the i-th interpolation point is calculated as follows: This nonlinear mapping results in a denser distribution of interpolation points near the start and end points of the trajectory, while they become relatively sparse in the middle region. This achieves a smooth start, a fluid middle section, and a smooth finish. Compared to the arc length parameterization method, this strategy maintains the naturalness of the motion while having lower computational complexity, making it suitable for real-time system applications.

[0069] Understandably, the operation event transmission method provided in this application eliminates the time quantization error caused by fixed sampling, thus resolving the problem of out-of-order event occurrence. It adopts an event-driven input capture strategy, abandoning the traditional fixed low-frequency polling mechanism. When the user performs a gamepad operation, the system immediately captures and transmits the data, fundamentally eliminating the time quantization error caused by fixed sampling and the resulting out-of-order event occurrence. Traditional solutions, such as 60Hz, have an average sampling delay of 8.33 milliseconds and a maximum delay of 16.67 milliseconds. This application records the occurrence time in real time, approaching the native polling accuracy of the gamepad hardware, making the remote control experience close to the effect of a local direct connection. Ensuring the consistency and fairness of event occurrence order in multi-user scenarios: The timestamp reordering mechanism designed in this application's embodiments achieves clock alignment between multiple control terminals and controlled terminals through RTT measurement and periodic synchronization maintenance. Furthermore, combined with the event buffer window mechanism, the system can reorder events from different control terminals according to their original timestamps, restoring the true occurrence order of events, ensuring fair scheduling and correct execution under multi-user operation, and effectively solving the timing conflict problem in multi-player cooperative games. Achieve smooth and continuous dead zone transitions, improving the precision of control.

[0070] Meanwhile, the operation event transmission method provided in this embodiment employs an intelligent path reconstruction algorithm based on cubic Bézier curves, replacing the traditional hard threshold truncation method. When the joystick is detected moving from the dead zone to the effective area, the system automatically generates a smooth transition path that conforms to ergonomics, eliminating the abrupt change at the dead zone boundary. Users can obtain continuous and natural response feedback when performing precise operations, significantly improving the smoothness and consistency of operation in game scenarios.

[0071] In practical applications, users remotely control games via terminal devices. The operation event transmission method provided in this application significantly reduces control latency in cloud gaming scenarios, providing a control experience close to that of local games, particularly suitable for action, racing, and fighting games with high real-time requirements. In LAN collaboration and esports competitions, multiple players may need to use their respective controllers connected to the same game console or computer. This application's multi-user timing arbitration mechanism ensures fair scheduling and correct execution of inputs from multiple controllers, avoiding a decline in game experience due to timing conflicts and guaranteeing fairness in multiplayer games. Especially for professional esports players and hardcore gamers, millisecond-level differences in event occurrence time can directly affect the outcome of a match. This application achieves consistency and fairness in the order of event occurrence without increasing event latency and eliminates control abrupt changes at dead zone boundaries, meeting the stringent requirements of professional esports scenarios for extreme response speed and precise, continuous control.

[0072] Figure 7 An operation event transmission device is shown, the device comprising:

[0073] The synchronization module 50 is used to synchronize the timestamp in the control terminal used to control the remote desktop with the timestamp of the controlled terminal according to a preset period to obtain the synchronized timestamp.

[0074] Allocation module 52 is used to collect operation events from the control terminal and allocate the synchronized timestamp to the operation events;

[0075] The sending module 54 is used to send the operation event and the corresponding timestamp to the controlled terminal, wherein the controlled terminal is used to execute the operation event according to the sorting of the operation event and the corresponding timestamp.

[0076] The aforementioned operation event transmission device synchronizes the timestamps in the control terminal and the controlled terminal at a preset period to obtain synchronized timestamps; it collects operation events in the control terminal and assigns the synchronized timestamps to the operation events; and it sends the operation events and their corresponding timestamps to the controlled terminal. The controlled terminal executes the operation events according to the order of the operation events and their corresponding timestamps. This real-time acquisition method, triggered by input operation events, avoids the polling acquisition method used in traditional methods, achieving the goal of timely recording the timestamps when operation events occur. This results in accurate execution of operation events and solves the technical problem in related technologies where fixed-frequency sampling leads to high execution deviations in operation events generated by the control terminal.

[0077] The synchronization module 50 includes a synchronization submodule, used to synchronize the timestamp in the control terminal with the timestamp of the controlled terminal according to a preset period to obtain a synchronized timestamp, including: sending a timestamp synchronization request to the controlled terminal and receiving the timestamp of the controlled terminal sent by the controlled terminal in response to the timestamp synchronization request; determining the clock offset between the control terminal and the controlled terminal based on the timestamp of the controlled terminal and the timestamp in the control terminal; and determining the synchronized timestamp based on the clock offset.

[0078] The synchronization submodule includes an offset unit, used to determine a clock offset based on the timestamp of the controlled terminal and the timestamp of the control terminal, including: determining the round-trip delay between the controlled terminal and the control terminal based on the time when a timestamp synchronization request is sent to the controlled terminal and the time when the timestamp of the controlled terminal is sent in response to the timestamp synchronization request; acquiring multiple round-trip delays within a preset window, and determining the latest delay within the preset window based on the minimum round-trip delay among the multiple round-trip delays; acquiring multiple latest delays closest to the target time, and determining the weighted average of the multiple latest delays closest to the target time as the target value; and determining the clock offset based on the target value.

[0079] The sending module 54 includes a sending submodule for sending the operation event and its corresponding timestamp to the controlled terminal, including: sending the operation event and its corresponding timestamp to a preset buffer in the controlled terminal, wherein the preset buffer is a sliding time window synchronized with the frame rate of the remote desktop video stream; at the end of the period corresponding to the preset buffer, reordering the operation events in the preset buffer according to the corresponding timestamp to obtain an operation event sequence, wherein the operation event sequence is used by the controlled terminal to execute the operation events sequentially according to the order of the operation events in the operation event sequence.

[0080] The sending submodule includes an execution unit, configured to: obtain the event type of the operation event when the time when the operation event is sent to the controlled end exceeds the sliding time window; compensate for the operation event when the event type is a discrete event and the duration of the time when the operation event is sent to the controlled end exceeding the sliding time window is less than a preset threshold; and directly discard the operation event when the event type is a continuous event.

[0081] The sending module 54 includes a curve submodule, used to determine the first valid state point detected by the control terminal as the endpoint of the target curve and the origin of the joystick of the control terminal as the starting point of the target curve when the joystick of the control terminal is detected to be in the dead zone of the control terminal. The valid state point is used to indicate the position point where the operation event takes effect, and the target curve is used to indicate the movement path of the joystick of the control terminal from the starting point of the target curve to the endpoint of the target curve. The module also acquires multiple intermediate control points of the target curve and determines the target curve based on the starting point of the target curve, the multiple intermediate control points of the target curve, and the endpoint of the target curve.

[0082] The curve submodule includes a curve unit for acquiring multiple intermediate control points of the target curve, including: acquiring dead-zone-free movement data and fitting the dead-zone-free movement data to a real joystick movement trajectory; determining adjustment parameters corresponding to the multiple intermediate control points when the target curve and the real joystick movement trajectory reach minimum deviation; and determining the multiple intermediate control points based on the adjustment parameters corresponding to the multiple intermediate control points.

[0083] The curve unit includes: a determination subunit, used to obtain the transition time from the start point to the end point of the target curve and the minimum time interval that the control terminal can respond to; determine the number of interpolation points in the target curve based on the transition time and the minimum time interval; determine the position of each interpolation point in the target curve based on the number of interpolation points in the target curve and a pre-determined easing function; and determine the target movement trajectory of the joystick of the control terminal based on the target curve and the position of each interpolation point in the target curve.

[0084] It should be noted that, Figure 7 The operation event transmission device shown is used to perform Figure 2 The operation event transmission method shown above also applies to the operation event transmission device, and will not be repeated here.

[0085] This application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described operation event transmission method.

[0086] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the operation event transmission method in this application.

[0087] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0088] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0092] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0093] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for transmitting operation events, characterized in that, include: The timestamps in the control terminal used to control the remote desktop are synchronized with the timestamps of the controlled terminal according to a preset period to obtain synchronized timestamps. Collect operation events from the control terminal and assign the synchronized timestamp to the operation events; The operation event and its corresponding timestamp are sent to the controlled terminal, wherein the controlled terminal is used to execute the operation event according to the sorting of the operation event and its corresponding timestamp.

2. The method according to claim 1, characterized in that, The timestamps in the control terminal and the controlled terminal are synchronized according to a preset period to obtain synchronized timestamps, including: Send a timestamp synchronization request to the controlled terminal, and receive the timestamp of the controlled terminal in response to the timestamp synchronization request; The clock offset between the control terminal and the controlled terminal is determined based on the timestamp of the controlled terminal and the timestamp of the control terminal. The synchronized timestamp is determined based on the clock offset.

3. The method according to claim 2, characterized in that, Determining the clock offset based on the timestamp of the controlled terminal and the timestamp of the control terminal includes: The round-trip time between the controlled terminal and the control terminal is determined based on the time when the controlled terminal sends the timestamp synchronization request to the controlled terminal and the time when the controlled terminal sends the timestamp of the controlled terminal in response to the timestamp synchronization request. The round-trip delays within a preset window are obtained, and the minimum round-trip delay among the multiple round-trip delays is determined as the latest delay within the preset window. Obtain the multiple latest delays closest to the target time, and determine the weighted average of the multiple latest delays closest to the target time as the target value; The clock offset is determined based on the target value.

4. The method according to claim 1, characterized in that, Sending the operation event and its corresponding timestamp to the controlled terminal includes: The operation event and its corresponding timestamp are sent to a preset buffer in the controlled terminal, wherein the preset buffer is a sliding time window synchronized with the frame rate of the remote desktop video stream; At the end of the period corresponding to the preset buffer, the operation events in the preset buffer are reordered according to the corresponding timestamps to obtain an operation event sequence, wherein the operation event sequence is used by the controlled terminal to execute the operation events in the order of the operation event sequence.

5. The method according to claim 4, characterized in that, The method further includes: If the time when the operation event is sent to the controlled terminal is outside the sliding time window, the event type of the operation event is obtained; If the event type is a discrete event and the time when the operation event is sent to the controlled terminal exceeds the duration of the sliding time window by less than a preset threshold, the operation event is executed as compensation. If the event type is a continuous event, the operation event is discarded directly.

6. The method according to claim 1, characterized in that, The method further includes: When the joystick of the control terminal is detected to be in the dead zone of the control terminal, the first valid state point detected by the control terminal is determined as the end point of the target curve, and the origin of the joystick of the control terminal is determined as the starting point of the target curve. The valid state point is used to indicate the location point where the operation event takes effect, and the target curve is used to indicate the movement path of the joystick of the control terminal from the starting point of the target curve to the end point of the target curve. Obtain multiple intermediate control points of the target curve, and determine the target curve based on the starting point of the target curve, the multiple intermediate control points of the target curve, and the ending point of the target curve.

7. The method according to claim 6, characterized in that, Obtaining multiple intermediate control points of the target curve includes: Acquire dead-zone-free movement data and fit the dead-zone-free movement data into the actual joystick movement trajectory; When the target curve and the actual joystick movement trajectory reach the minimum deviation, the adjustment parameters corresponding to the plurality of intermediate control points are determined; The plurality of intermediate control points are determined based on the adjustment parameters corresponding to the plurality of intermediate control points.

8. The method according to claim 7, characterized in that, The method further includes: Obtain the transition time from the start point to the end point of the target curve by the control terminal and the minimum time interval at which the control terminal can respond; The number of interpolation points in the target curve is determined based on the transition duration and the minimum duration interval; The position of each interpolation point in the target curve is determined based on the number of interpolation points in the target curve and a pre-determined easing function; The target movement trajectory of the joystick at the control end is determined based on the target curve and the position of each interpolation point in the target curve.

9. An operation event transmission device, characterized in that, include: The synchronization module is used to synchronize the timestamp in the control terminal used to control the remote desktop with the timestamp of the controlled terminal according to a preset period, so as to obtain the synchronized timestamp. The allocation module is used to collect operation events from the control terminal and allocate the synchronized timestamp to the operation events. A sending module is used to send the operation event and its corresponding timestamp to the controlled terminal, wherein the controlled terminal is used to execute the operation event according to the sorting of the operation event and its corresponding timestamp.

10. A computer device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; The processor, connected to the memory, is used to execute the operation event transmission method according to any one of claims 1 to 8.

Images