Software development kit-based toy remote interaction delay alignment method and system

Abstract

The application discloses a toy remote interaction delay alignment method and system based on a software development kit, relates to the technical field of remote interaction control, and comprises the following steps: establishing an inter-user media session among a plurality of control terminals and establishing a control session between the control terminal and the toy; the software development kit collects a control instruction sending timestamp and a toy execution timestamp to calculate a control link delay, and acquires a media link delay of the inter-user media session; a compensation amount is calculated according to a delay difference value, a virtual time axis and a control instruction buffer queue are established, and the sending is dispatched according to the generation time plus the compensation amount; in a multi-user scenario, a collaborative time track containing the expected effective time of each control behavior is constructed, and a recommended collaborative control time window is given. The fine-grained alignment of the control link and the media link is realized, the delay fluctuation is inhibited, and the synchronicity and the sense of immersion of the multi-user remote interaction toy are enhanced.

Description

Technical Field

[0001] This invention relates to the field of remote interactive control technology, specifically to a method and system for remote interactive delay alignment of toys based on a software development kit. Background Technology

[0002] With the development of mobile internet, real-time audio and video communication, and smart toys, toys connect to user terminals via Bluetooth or WiFi, enabling remote control and audio-visual interaction between users through the network. In a typical scenario, control commands are transmitted to the toy for execution via a control link, while the audio and video of the toy's environment are transmitted back to the remote user via a media link. Due to differences in transmission paths, protocol stacks, and buffering strategies between the control link and the media link, the time when the toy's actions take effect often differs from the time when the visual and audio presentation occurs. Existing technologies mostly use fixed delays or simple buffering methods for coarse alignment, making it difficult to achieve fine-grained compensation for different control sessions and different network states. This easily leads to users perceiving a lack of synchronization between operation and feedback.

[0003] In remote interactive scenarios, there is a need for multiple control terminals to control the same or multiple toys in turn or in parallel. Existing solutions typically only mark control rights on the server side, lacking fine-grained management of the current control session and candidate control sessions. This results in control commands that have not yet been sent or are en route by the original controller being executed during the control handover process, causing conflicts with the operation of the new controller. At the same time, candidate control sessions do not pre-establish compensation parameters that match their own link characteristics before gaining control, and need to reconverge after the handover, which can easily lead to obvious desynchronization and sudden changes in experience during the handover phase.

[0004] Furthermore, existing remote toy interaction systems mostly implement latency processing logic at the application layer, lacking unified latency measurement, compensation strategies, and command scheduling capabilities for control and media sessions at the software development kit (SDK) level. This results in high development costs and inconsistent behavior for third-party applications when integrating with multi-toy, multi-user interaction scenarios. For multi-user collaborative control of multiple toys, existing technologies generally only provide independent control interfaces for each user, failing to build a cross-user time collaboration view based on latency and compensation information for each control session, and also unable to provide users with recommended collaborative control time windows, making it difficult to meet the requirements of high synchronization and multi-user collaborative interaction. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a method and system for remote interactive delay alignment of toys based on a software development kit, so as to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a toy remote interaction delay alignment method based on a software development kit, comprising: Establish media sessions between users among multiple control terminals, and establish control sessions between control terminals and toys to send control commands; The delay processing module, which includes a software development kit integrated into the control terminal, calculates the control link delay based on the timestamp of the control terminal sending control commands and the timestamp of the toy executing control commands; and obtains the media link delay of the media session between users. The compensation amount is calculated for the control session based on the control link delay and the media link delay. A virtual time axis and a control command buffer queue are established, and the transmission time is determined by adding the compensation amount to the generation time. For the same toy, maintain the current control session and candidate control sessions, and cache the compensation amount for the candidate control session. When control is switched, the original current control session freezes the buffer queue and only responds to the instructions of the new current control session. The new current control session initializes the virtual timeline with the cache compensation amount.

[0007] The present invention is further configured such that the control session includes: a network control link established between the remote control terminal and the control terminal to which the toy belongs, and a wireless connection established between the control terminal to which the toy belongs and the toy via Bluetooth or WiFi, wherein the control link latency includes the transmission latency of the network control link and the transmission latency of the local wireless connection.

[0008] The present invention is further configured such that calculating the compensation amount for the control session based on the control link delay and the media link delay includes: setting the compensation amount as a monotonic function of the difference between the media link delay and the control link delay; determining the alignment error based on the deviation between the current compensation amount and the difference; updating the compensation amount for the control session only when the absolute value of the alignment error exceeds a preset error threshold; and keeping the compensation amount unchanged when the absolute value of the alignment error is less than or equal to the error threshold.

[0009] The present invention is further configured such that establishing a virtual timeline and control command buffer queue includes: establishing an independent control command buffer queue for each control session; the software development kit in the control terminal calculates the planned transmission time in each buffer queue based on the generation time of the control command and the corresponding compensation amount; scanning each buffer queue at the control session level; and retrieving the control command from the corresponding buffer queue and transmitting it only when the current system time reaches the planned transmission time after being mapped by the virtual timeline, in order to avoid mutual interference between control commands of different control sessions.

[0010] The present invention is further configured such that the compensation amount for the candidate control session cache includes: periodically issuing test control instructions to the control terminal corresponding to the candidate control session through control signaling, obtaining the estimated control link delay and estimated media link delay of the candidate control session by the execution timestamp returned by the toy and combined with the test audio and video frame timestamps in the media session between users, calculating the candidate compensation amount, storing the candidate compensation amount in the candidate cache, and using it to initialize the virtual time axis when the candidate control session obtains control.

[0011] The present invention is further configured such that, when control is switched, the original current control session freezes the buffer queue by: setting the control instruction buffer queue of the control terminal corresponding to the original current control session to a frozen state, prohibiting new control instructions from being enqueued, and marking the control instructions that have not yet been sent in the queue as invalid. When the toy receives a control instruction, it executes only the control instructions sent by the control terminal corresponding to the new current control session according to the control session identifier.

[0012] The present invention is further configured to include: calculating the interaction timing coherence potential for each control session within a preset time window based on the alignment error, the magnitude of the compensation amount change, and the frequency of control command transmission; limiting the adjustment step size of the control session compensation amount when the interaction timing coherence potential is higher than a preset coherence threshold; and allowing the compensation amount to be updated with an adjustment step size greater than the limit step size when the interaction timing coherence potential is lower than the coherence threshold.

[0013] The present invention is further configured such that, during the protection period after the new current control session is switched, the delayed alignment logic calculates the alignment error and interaction timing coherence potential of the new current control session in multiple consecutive time windows. Only when the alignment error is within the preset error tolerance range and the interaction timing coherence potential is higher than the coherence threshold, the restriction on the adjustment step size of the compensation amount of the new current control session is lifted.

[0014] The present invention is further configured such that the multiple control terminals include a first control terminal and a second control terminal. The first control terminal is used to: simultaneously display on its display interface the control behavior of a first user on a first toy via the first control terminal and the control behavior of a second user on a second toy via the second control terminal; wherein, the first control terminal calculates the expected effective time of the first user's control behavior being perceived by the second user based on the control link delay, media link delay, and compensation amount of the control session to which the first toy belongs; calculates the expected effective time of the second user's control behavior being perceived by the first user based on the control link delay, media link delay, and compensation amount of the control session to which the second toy belongs; and constructs a collaborative time trajectory including the expected effective time; identifies the time overlap area of ​​the expected effective time of the first toy and the second toy's actions in the collaborative time trajectory as a recommended collaborative control time window, and prompts the user for the recommended collaborative control time window on the display interface; when the first user triggers a control operation within the recommended collaborative control time window, the corresponding control instruction is marked as a collaborative control instruction and prioritized for sending in the control instruction buffer queue.

[0015] This invention also provides a toy remote interaction delay alignment system based on a software development kit (SDK), used to implement the aforementioned toy remote interaction delay alignment method based on the SSD, comprising: Session establishment module: Establishes media sessions between users among multiple control terminals, and establishes control sessions between the control terminal and the toy for sending control commands; Latency Calculation Module: Using a latency processing module that includes a software development kit integrated into the control terminal, the module calculates the control link latency based on the timestamp of the control terminal sending control commands and the timestamp of the toy executing the control commands; it also obtains the media link latency of the media session between users. Compensation calculation module: Calculates the compensation amount for the control session based on the control link delay and media link delay, establishes a virtual time axis and control command buffer queue, and determines the transmission time by adding the compensation amount to the generation time; Control switching module: Maintains the current control session and candidate control sessions for the same toy, and caches the compensation amount for the candidate control sessions. When control is switched, the original current control session freezes the buffer queue and only responds to the instructions of the new current control session. The new current control session initializes the virtual time with the cache compensation amount.

[0016] This invention provides a method and system for delay alignment in remote interaction with toys based on a software development kit (SDK). The method establishes media sessions between multiple control terminals and control sessions between the control terminals and the toy for sending control commands. It calculates the control link delay using a delay processing module integrated into the control terminal, based on the timestamps of the control commands sent by the control terminal and the execution timestamps of the control commands executed by the toy. It also obtains the media link delay of the media sessions between users; calculates compensation for the control sessions based on the control link delay and media link delay; establishes a virtual timeline and a control command buffer queue; and determines the sending time by adding the compensation amount to the generation time. For the same toy, it maintains a current control session and candidate control sessions, caching the compensation amount for candidate control sessions. During control switching, the original current control session freezes the buffer queue and only responds to commands from the new current control session. The new current control session initializes the virtual timeline with the cached compensation amount. The beneficial effects include: By collecting control command transmission timestamps and toy execution timestamps using a software development kit on the control terminal side, the control link latency is calculated. The media link latency is then calculated using the timestamps collected from the peer end and the playback timestamps on the local end. Compensation amounts are calculated at the control session level based on this, introducing a virtual timeline and a control command buffer queue. The transmission time is determined by adding the compensation amount to the generation time. Therefore, instead of relying on fixed delays or simple buffers, dynamic, fine-grained latency compensation is performed for each control session, making the toy's action activation time closer to the audio and video presentation time, significantly reducing the perceived asynchrony between operation and feedback in remote interaction. When calculating the compensation amount, the compensation amount is set as a monotonic function of the difference between the media link delay and the control link delay. Alignment error is introduced, and the compensation amount is updated only when the alignment error exceeds a preset threshold. Furthermore, within a preset time window, the interaction timing coherence potential is calculated by comprehensively considering the alignment error, the magnitude of the compensation amount change, and the frequency of control command transmission. When the interaction timing coherence potential is high, the adjustment step size of the compensation amount is limited, and when the interaction timing coherence potential is low, the adjustment step size is widened. During the protection period after the control switch, the step size limit is lifted only when both the alignment error and the interaction timing coherence potential meet the conditions. This ensures alignment accuracy while suppressing frequent and drastic changes in the compensation amount, reducing the jitter and flicker in audiovisual and motion perception, and improving the stability and robustness of the system in complex network environments. In multi-user bidirectional or multidirectional control scenarios, the first control terminal can simultaneously display the control behaviors of its own user on the first toy and the control behaviors of the other user on the second toy on the display interface. It calculates the estimated effective time of each control behavior perceived by the other end based on the control link latency, media link latency, and compensation amount of the two control sessions, constructing a collaborative time trajectory. The overlapping area of ​​the estimated effective times of different toy actions is identified in the collaborative time trajectory as a recommended collaborative control time window, and a prompt is displayed on the interface. Control operations triggered within this time window are marked as collaborative control commands and prioritized for dispatch. Through this time-based collaborative guidance mechanism, multiple users can more intuitively grasp the synchronization interval between the actions perceived by the other end, enhancing rhythm and coordination, and significantly improving the immersion and controllability of multi-user collaborative gameplay.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 The flowchart illustrates a toy remote interactive delay alignment method based on a software development kit, as an exemplary embodiment of the present invention. Detailed Implementation

[0019] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention. Example 1:

[0022] A method for delay alignment of remote interactive toys based on software development kits, such as Figure 1 As shown, it includes: Establish media sessions between users among multiple control terminals, and establish control sessions between control terminals and toys to send control commands; The delay processing module, which includes a software development kit integrated into the control terminal, calculates the control link delay based on the timestamp of the control terminal sending control commands and the timestamp of the toy executing control commands; and obtains the media link delay of the media session between users. The compensation amount is calculated for the control session based on the control link delay and the media link delay. A virtual time axis and a control command buffer queue are established, and the transmission time is determined by adding the compensation amount to the generation time. For the same toy, maintain the current control session and candidate control sessions, and cache the compensation amount for the candidate control session. When control is switched, the original current control session freezes the buffer queue and only responds to the instructions of the new current control session. The new current control session initializes the virtual timeline with the cache compensation amount.

[0023] The present invention is further configured such that the control session includes: a network control link established between the remote control terminal and the control terminal to which the toy belongs, and a wireless connection established between the control terminal to which the toy belongs and the toy via Bluetooth or WiFi, wherein the control link latency includes the transmission latency of the network control link and the transmission latency of the local wireless connection.

[0024] Specifically, the remote interactive system includes multiple control terminals, a control terminal belonging to the toy, the toy itself, and media and control signaling services deployed in the cloud. The multiple control terminals are held by different users. The toy establishes a local connection with its control terminal via Bluetooth or WiFi, and the control terminal then accesses the cloud service via the internet. The system needs to simultaneously meet the needs of audio and video communication between users and the control of the remote toy. A media session is established between multiple control terminals. Specifically, when the first user initiates a remote interaction request on the first control terminal, the first control terminal sends a session establishment request to the media service. The media service assigns media session identifiers to the first and second control terminals and completes session negotiation, establishing audio and video transmission and reception channels between the two ends respectively. Subsequently, the video and audio streams captured by the first and second control terminals through cameras and microphones are forwarded and played back to each other by the media service, thus forming a stable media session between users. This session carries the real-time images and sounds of both parties, facilitating remote observation of the toy's status and the other party's interactive feedback. Building upon the established media session, a control session is established between the control terminal and the toy to enable remote control of the toy. Specifically, after the first user gains control of the target toy, the first control terminal establishes a network control link with the toy's control terminal via a control signaling service. This network control link carries control commands and execution result feedback from the first control terminal to the toy's control terminal. Simultaneously, the toy's control terminal establishes a wireless connection with the toy via Bluetooth or WiFi. This wireless connection carries the issuance of specific control commands from the toy's control terminal to the toy and status feedback from the toy. Thus, the control session logically consists of two interconnected links: the "network control link between the remote control terminal and the toy's control terminal" and the "wireless connection between the toy's control terminal and the toy." During the transmission of control commands, each control command sent by the first control terminal first reaches the toy's control terminal via the internet, incurring a transmission delay in the network control link along this path. After receiving the command, the toy's control terminal sends the command to the toy via Bluetooth or WiFi. The toy decodes and executes the corresponding action, incurring a transmission delay in the local wireless connection along this path. Since control commands must pass through both the network control link and the local wireless connection sequentially to reach the toy and be executed, the total control link delay seen from the control terminal's perspective can be understood as the sum of the network control link transmission delay and the local wireless connection transmission delay. In practical implementation, the first control terminal appends a transmission timestamp to the control command it sends. The toy's control terminal records the execution timestamp when it detects that the toy has completed an action or returned an execution confirmation, and feeds it back to the first control terminal through the control signaling service. The first control terminal estimates the total delay of the control link based on the difference between the transmission timestamp and the execution timestamp. The total delay equivalently reflects the combined transmission characteristics of the network control link and the local wireless connection. By dividing the control session structure into two parts, the network control link and the local wireless connection, and explicitly defining that the control link delay consists of the delays of these two parts; Meanwhile, since the media sessions between users are established independently of the control sessions, the media sessions adopt audio and video-oriented transmission protocols and buffering strategies, while the control sessions employ lightweight control signaling mechanisms. The two are decoupled in terms of protocol stack and scheduling strategies. This is achieved by establishing media sessions between users across multiple control terminals and constructing a control session between the control terminals and the toy, consisting of a network control link and a local wireless connection.

[0025] Regarding the control link, when the first user performs a control operation (such as clicking a button or sliding a joystick) on the first control terminal, the software development kit (SDK) generates the corresponding control command and simultaneously obtains the transmission timestamp of the command from the local time source, writes it into the metadata of the control command, and forwards it to the control terminal to which the toy belongs via the control signaling service. Upon receiving the control command, the control terminal to which the toy belongs sends a control command to the toy via Bluetooth or WiFi. When it detects that the toy has completed the corresponding action or receives an execution confirmation event from the toy, the control terminal to which the toy belongs records the execution timestamp of the toy and feeds it back to the first control terminal via the control signaling service. After receiving the execution timestamp, the SSD in the first control terminal performs a difference calculation with the previously recorded transmission timestamp, using this time difference as the control link delay corresponding to the control command, thus reflecting the combined delay of the network control link and the local wireless connection during the process from the issuance of the control command to the actual execution by the toy. By repeating the above process for multiple control commands and performing statistical analysis within a sliding time window, a relatively stable estimate of the control link delay under the current control session is obtained.

[0026] Regarding the media link, the second control terminal is responsible for collecting audio and video data of the toy's environment and transmitting it back to the first control terminal via a media service. Specifically, when the second control terminal sends video frames captured by the camera or audio frames captured by the microphone into the encoding module, it assigns a capture timestamp to each frame. This timestamp can be based on the second control terminal's local time base or a unified timeline of the media session. The captured audio and video frames are forwarded by the media service to the first control terminal and enter the playback buffer. When the video rendering module or audio playback module of the first control terminal actually outputs the frame's image or sound, the software development kit reads the corresponding playback timestamp. The first control terminal calculates the time difference between the playback timestamp and the capture timestamp, and considers this difference as the transmission and buffering delay of the audio and video frame on the user-to-user media link. By performing the same processing on the acquisition timestamps and playback timestamps of multiple frames of audio and video data over a period of time, and removing outliers and then performing statistical processing on the remaining samples, we can obtain the media link latency estimate under the current media session. This estimate can be used in conjunction with the aforementioned control link latency to further calculate the compensation amount and adjust the virtual time axis, thereby aligning the toy's action feedback time with the audio and video presentation time. Furthermore, in another implementation, the media service or real-time audio / video service (RTC) estimates the link latency of the current user-to-user media session on the server side based on information such as the underlying transmission status, round-trip time measurement results, and playback buffer queuing time. It then periodically or on demand sends the media link latency parameter, representing this latency, to the first control terminal and / or the remote interactive platform server via an interface. Upon receiving the media link latency parameter provided by the media service or real-time audio / video service, the software development kit in the first control terminal can directly use it as a media link latency parameter in the compensation calculation, without needing to repeatedly perform the difference statistics between the acquisition timestamp and the playback timestamp locally on the terminal. In practical applications, media link latency parameters can be obtained through the aforementioned local timestamp calculation method, latency estimation methods provided by media services or real-time audio and video services, or a combination of both. Regardless of the source of the media link latency parameter, it serves as a unified parameter characterizing the latency of media session links between users. It is input into the compensation calculation logic along with the control link latency. By adjusting the sending time of the virtual timeline and the control command buffer queue, the synchronous presentation of remote toy actions in the user's visuals and audio is achieved.

[0027] The present invention is further configured such that calculating the compensation amount for the control session based on the control link delay and the media link delay includes: setting the compensation amount as a monotonic function of the difference between the media link delay and the control link delay; determining the alignment error based on the deviation between the current compensation amount and the difference; updating the compensation amount of the control session only when the absolute value of the alignment error exceeds a preset error threshold; and keeping the compensation amount unchanged when the absolute value of the alignment error is less than or equal to the error threshold. Specifically, the software development kit (SDK) in the control terminal maintains a set of compensation parameters for each control session, used to uniformly offset control commands in time. The SSD first calculates the difference between the media link delay and the control link delay, inputs this difference into a preset monotonic mapping relationship, and obtains the target compensation amount for that control session. The monotonic mapping means that when the media link delay is relatively large compared to the control link delay, the target compensation amount increases with the difference; when the difference is small or even negative, the target compensation amount no longer increases or is limited to a preset range, thus avoiding overcompensation. The specific function of the monotonic mapping is not restricted here, but is limited according to the application scenario. After obtaining the target compensation amount, the software development kit compares the current compensation amount with the target compensation amount to obtain the deviation between the two, which is used as a measure of alignment error. When the absolute value of the alignment error is greater than a preset error threshold, it is determined that the current compensation amount does not match the changes in the link state, and an update operation needs to be performed. During the update, a gradual approximation method is used to adjust the current compensation amount towards the target compensation amount by a certain margin, so that the compensation amount slowly converges with the changes in the link state. When the absolute value of the alignment error is less than or equal to the error threshold, it is considered that the current compensation amount is sufficient to cover the difference between the control link latency and the media link latency. At this time, the compensation amount is kept unchanged to avoid frequent adjustments to the compensation parameters due to slight jitter and to reduce system oscillations in latency fluctuation scenarios. After the compensation amount is determined, a virtual timeline and a control command buffer queue are established at the control session level. An independent control command buffer queue is created for each control session. The software development kit (SDK) in the control terminal calculates the planned transmission time in each buffer queue based on the generation time of the control command and the corresponding compensation amount. Each buffer queue is scanned at the control session level, and the control command is retrieved and transmitted only when the current system time, mapped through the virtual timeline, reaches the planned transmission time. This prevents interference between control commands from different control sessions. The virtual timeline maps the current system time of the control terminal to the interactive reference time within that control session. When a control command is generated, the SSD records the generation time and calculates the planned transmission time of the command by adding the compensation amount to the generation time. The SSD writes the control command along with the planned transmission time into the control command buffer queue of the corresponding control session and periodically compares the current time with the planned transmission time of each command on the virtual timeline. Only when the current time reaches or exceeds the planned transmission time is the command retrieved from the buffer queue and transmitted to the toy through the control session.

[0028] The present invention is further configured such that the compensation amount for the candidate control session cache includes: periodically sending test control instructions to the control terminal corresponding to the candidate control session through control signaling, obtaining the estimated control link delay and estimated media link delay of the candidate control session by the execution timestamp returned by the toy and combined with the test audio and video frame timestamps in the media session between users, calculating the candidate compensation amount, storing the candidate compensation amount in the candidate cache, and using it to initialize the virtual time axis when the candidate control session obtains control rights; Specifically, consider a scenario where a current control session and a candidate control session coexist during remote interaction with the same toy. Assume the first control terminal currently holds control of the target toy, and the corresponding control session is the current control session; the second control terminal is in a waiting-to-takeover or queued state, and the corresponding control session is recorded as a candidate control session. To avoid experiencing a lengthy convergence process again after a control switch, a compensation buffer matching the actual link characteristics is pre-built for the candidate control session while it has not yet gained control. Specifically, through the control signaling service, test control commands are periodically issued to the second control terminal corresponding to the candidate control session. The content of the test control commands can be calibration commands that have little impact on the toy or do not produce obvious actions, such as status queries or virtual commands that are intercepted by the toy's internal logic and not executed. After receiving the test control command, the candidate control terminal sends the corresponding control command to the toy via a local Bluetooth or WiFi link. After the toy completes parsing and execution, its own control terminal records the execution timestamp and feeds it back to the second control terminal via the control signaling link. The integrated software development kit collects the timestamp of the test command under the candidate control session. Combined with the time of test command transmission, the second control terminal can estimate the estimated control link latency of the candidate control session. Simultaneously, test segments for calibration are embedded in the media sessions between users. When a test control command is issued, the acquisition of a test scenario related to the test command is triggered on the second control terminal side, so that the test segment is captured by the second control terminal and sent to the media link. When the first control terminal receives the test segment, the software development kit records the acquisition timestamp and playback timestamp of the test segment, and the transmission and buffering characteristics of the media link within the test period are obtained based on the difference between the two. By statistically analyzing the time differences of audio and video frames within multiple test periods, the estimated media link latency under the media path conditions corresponding to the candidate control session can be obtained. After obtaining the estimated control link delay and estimated media link delay, the software development kit in the second control terminal, following the same compensation calculation logic as in the main control session, sets the compensation amount as a monotonic function of the difference between the two types of delays, and calculates a candidate compensation amount that matches the current network condition and local wireless link condition of the candidate control session. This candidate compensation amount is not directly applied to the toy control, but is stored in a candidate cache corresponding one-to-one with the candidate control session and associated with the candidate control session identifier. When the system determines that the candidate control session should take over control according to the business rules, it reads the corresponding candidate compensation amount from the candidate cache to initialize the virtual timeline and control command buffer queue of the candidate control session, so that the new current control session has time compensation parameters that match its actual link conditions from the beginning.

[0029] The present invention is further configured such that, when control is switched, the original current control session freezes the buffer queue, which includes: the software development kit in the control terminal corresponding to the original current control session sets its control instruction buffer queue to a frozen state, prohibits new control instructions from being enqueued, and marks the control instructions that have not yet been sent in the queue as invalid. When the toy receives a control instruction, it only executes the control instructions sent by the control terminal corresponding to the new current control session according to the control session identifier. Specifically, assuming that the same toy is initially controlled by the first control terminal during remote interaction, the corresponding control session is recorded as the current control session; the second control terminal is in a candidate state, and the corresponding control session is recorded as the candidate control session. The cloud-based control signaling service maintains the mapping relationship between the toy identifier and the current control session identifier. When it is determined, based on business rules (such as queuing rules, owner transfer, timeout release, etc.), that control needs to be switched from the first control terminal to the second control terminal, the control signaling service sends a control switch notification to both ends. The notification includes the toy identifier, the old and new control session identifiers, and the effective time of the switch. Upon receiving the control handover notification, the first control terminal updates the status of the control instruction buffer queue corresponding to the current control session using the integrated software development kit (SDK). Specifically, the SSD first marks the buffer queue as frozen, prohibiting new control instructions from being written to the queue. Then, it iterates through all unsent control instructions in the queue, marking them as invalid and skipping them during subsequent scheduling. For instructions already issued but not yet executed by the toy, filtering is performed on the toy side via the handover notification issued by the control signaling service to prevent them from continuing execution after the handover. Through this process, the original current control session logically no longer generates new valid control outputs, and its historical residual control instructions will not be mistakenly sent or executed after the handover. On the toy side, each control command carries a control session identifier in the control protocol. The toy's control terminal or the toy itself maintains the currently effective control session identifier. When a control switch notification arrives at the toy's control terminal, the control terminal updates the currently effective control session identifier to the identifier of the new current control session and synchronizes this identifier to the toy itself. Afterward, when the toy receives any control command, it first parses the control session identifier carried in the command and compares it with the locally recorded currently effective control session identifier. Only if they match is the control command sent to the execution queue; otherwise, the command is discarded or marked as invalid. Because the original current control session is frozen in the buffer queue on the control terminal side and is marked as invalid for not sending commands, even if a very small number of commands arrive on the toy side due to network latency, their control session identifiers will be inconsistent with the updated currently effective control session identifier, and thus be filtered out by the toy side. By freezing and invalidating the buffer queue on the control terminal side, and using the instruction filtering mechanism on the toy side based on the control session identifier, this embodiment achieves complete isolation of residual control instructions from the original current control session during the control handover process. This ensures that the toy only executes control instructions sent by the control terminal corresponding to the new current control session, thus avoiding instruction conflicts and abnormal actions between different controllers.

[0030] The present invention is further configured to include: calculating the interaction timing coherence potential for each control session within a preset time window based on the alignment error, the magnitude of the compensation amount change, and the frequency of control command transmission; limiting the adjustment step size of the control session compensation amount when the interaction timing coherence potential is higher than a preset coherence threshold; and allowing the compensation amount to be updated with an adjustment step size greater than the limit step size when the interaction timing coherence potential is lower than the coherence threshold. Specifically, in order to suppress drastic fluctuations in compensation while ensuring alignment accuracy and improving the perceived stability of the remote interaction process, a calculation and application mechanism for the interaction timing coherence potential is introduced for each control session. The software development kit in the control terminal maintains a sliding time window for each control session, such as a number of seconds or a number of rounds of control commands. Within this time window, multi-dimensional data related to time delay alignment is continuously recorded to characterize the recent timing stability of the control session. Specifically, within a preset time window of each control session, the software development kit continuously records three types of quantities: first, the evolution of alignment error, i.e., the magnitude and trend of the deviation between the current compensation amount and the target compensation amount obtained based on the difference between control link delay and media link delay, used to reflect the current alignment level of the control session and whether there is a continuous deviation; second, the magnitude of compensation amount changes, statistically analyzing the difference and direction of change before and after the most recent compensation amount updates within the window, used to characterize whether the compensation parameters are being adjusted frequently and significantly; and third, the frequency of control command transmission, recording the number of control command triggers and their time distribution within the window, used to reflect the user's actual operational activity and continuity in the control session. These three types of information together constitute a multi-dimensional description of the recent temporal behavior of the control session. After collecting the above data, the software development kit (SDK) comprehensively evaluates the alignment error, the magnitude of the compensation change, and the frequency of control command transmission at the end of each time window or when the window slides to a new position. This evaluation maps these values ​​to a comprehensive quantity characterizing the timing stability and interaction continuity of the control session—the interaction timing coherence potential. In principle, when the alignment error is relatively small and does not fluctuate drastically within the window, the compensation update magnitude is small and the adjustment direction is relatively stable, and the frequency of control command transmission is relatively continuous and balanced, the timing performance of the control session is considered to be relatively coherent, and the corresponding interaction timing coherence potential takes a higher value. Conversely, when the alignment error remains large or fluctuates significantly within the window, the compensation changes frequently with large amplitudes in both directions, and the distribution of control commands exhibits a sudden or extremely sparse state, the timing state of the control session is considered to be unstable or in a transitional phase, and the corresponding interaction timing coherence potential takes a lower value. After obtaining the interaction timing coherence potential, the adjustment step size of the compensation amount update is controlled by the coherence threshold. When the interaction timing coherence potential of a certain control session in the current window is higher than the preset coherence threshold, it is determined that the control session is in a stable state with good alignment and high user operation continuity. At this time, the compensation amount update adopts a limited adjustment step size, allowing only small and gradual fine-tuning to avoid large fluctuations in the compensation amount due to short-term link jitter or individual abnormal samples, thereby avoiding obvious jumps or flashes in the screen and actions from the user experience perspective. When the interaction timing coherence potential of a control session within the current window is lower than the coherence threshold, it is determined that the control session is currently in a stage of large alignment deviation or rapid state change. At this time, the update of the compensation amount is allowed to adopt an adjustment range greater than the aforementioned limit step size, so that the compensation amount can quickly converge with the new link characteristics in a short period of time, shorten the duration of the misalignment state, and reduce the cumulative impact of long-term poor alignment on the interactive experience.

[0031] The present invention is further configured such that, during the protection period after the new current control session after the control switch, the delayed alignment logic calculates the alignment error and interaction timing coherence potential of the new current control session in multiple consecutive time windows, and only when the alignment error is within the preset error tolerance range and the interaction timing coherence potential is higher than the coherence threshold, the restriction on the adjustment step size of the compensation amount of the new current control session is lifted. Specifically, when control of the same toy is switched from the current control session to a candidate control session, to avoid drastic fluctuations in compensation adjustments due to the unstable link state during the initial switching phase, a protection period is set for the new current control session, and constraints are imposed on the compensation adjustment strategy during this period. Specifically, when a candidate control session takes over control of the toy according to business rules and marks it as the new current control session, the software development kit in the control terminal sets a protection flag for that control session and initializes a continuous time window counter for calculating alignment errors and interaction timing coherence potential. This limits the compensation adjustment step size of the control session to a preset small range, or adopts a strict gradual adjustment strategy. During the protection period, the software development kit (SDK) continuously acquires the alignment error and interaction timing coherence potential of the new current control session in each sliding time window, following the mechanism of the aforementioned embodiments. The alignment error measures the deviation between the current compensation amount and the target compensation amount determined by the difference between control link delay and media link delay, reflecting the alignment accuracy. The interaction timing coherence potential comprehensively reflects the timing stability and interaction continuity of the control session within the window. For each sliding time window, it is first determined whether the alignment error within the window is within a preset error tolerance range, i.e., whether the alignment deviation has converged to an allowable accuracy range. Based on this, it is then determined whether the interaction timing coherence potential corresponding to the window is higher than a preset coherence threshold, i.e., whether the recent timing performance has sufficient coherence and stability. To avoid the randomness of a single window influencing decision-making, multiple consecutive time windows are used for joint judgment. The software development kit maintains a count of continuously satisfying windows during the protection period. When both alignment error is within a preset error tolerance range and interaction timing coherence is above a coherence threshold within a given time window, the continuous satisfaction count is incremented. If any window fails to meet these conditions, the count is reset to zero or decayed according to preset rules. Only when the continuous satisfaction count reaches the set window count threshold is the system determined that the new current control session has achieved both high alignment accuracy and high timing coherence within a certain period, indicating that the link status and compensation parameters of the control session have entered a stable operating range. After the above-mentioned conditions are met, the software development kit removes the restriction on the adjustment step size of the compensation amount for the new current control session, switching the control session from the protection state to the normal adaptive adjustment state. In the normal state, the update of the compensation amount can adopt a more flexible adjustment strategy based on a combination of alignment error and interactive timing coherence potential. For example, a larger step size can be allowed for rapid convergence during link abrupt changes, while a smaller step size can be maintained for fine-tuning during the stable phase. By introducing a multi-window statistical criterion with alignment error and interactive timing coherence potential as joint constraints during the protection period after control handover, the drastic adjustment of the compensation amount and timing jitter that occur in the early stages of the new current control session due to unstable link conditions can be effectively avoided. Only after the alignment accuracy and timing coherence have stabilized to the expected level are the step size restrictions lifted, allowing the compensation amount adjustment process to achieve a more reasonable balance between responding to link changes and suppressing oscillations, thus improving the overall interactive stability and user-perceived continuity after control handover.

[0032] The present invention is further configured such that the multiple control terminals include a first control terminal and a second control terminal. The first control terminal is used to: simultaneously display on its display interface the control behavior of a first user on a first toy via the first control terminal and the control behavior of a second user on a second toy via the second control terminal; wherein, the first control terminal calculates the expected effective time of the first user's control behavior being perceived by the second user based on the control link delay, media link delay, and compensation amount of the control session to which the first toy belongs; calculates the expected effective time of the second user's control behavior being perceived by the first user based on the control link delay, media link delay, and compensation amount of the control session to which the second toy belongs; constructs a collaborative time trajectory including the expected effective time; identifies the time overlap area of ​​the expected effective time of the first toy and the second toy's actions in the collaborative time trajectory as a recommended collaborative control time window; and prompts the user for the recommended collaborative control time window on the display interface; when the first user triggers a control operation within the recommended collaborative control time window, the corresponding control instruction is marked as a collaborative control instruction and prioritized for sending in the control instruction buffer queue; Specifically, in this embodiment, a two-person collaborative remote interaction scenario is considered, including a first control terminal, a second control terminal, a first toy, and a second toy. The first toy is seen by the second user through a media session in the environment where the first control terminal is located, and the second toy is seen by the first user through a media session in the environment where the second control terminal is located. A media session between the first and second control terminals has been established, and corresponding control sessions have been established with their respective toys. The control link latency, media link latency, and compensation amount for each control session are obtained according to the aforementioned embodiment. Based on this, the software development kit (SDK) in the first control terminal first provides a unified display of the control behaviors of both the local and remote users. Specifically, when the first user issues a control operation to the first toy on the first control terminal, the SSD abstracts this control operation into a control behavior record, including the behavior type, generation time, and target toy identifier. Simultaneously, it obtains the second user's control behavior record for the second toy from the second control terminal through media sessions and control signaling between users, forming a time sequence. The first control terminal displays the first user's control behavior and the second user's control behavior for the second toy simultaneously in a timeline or list format on its local display interface, allowing the first user to intuitively view the temporal distribution of both user actions. Based on the control link latency, media link latency, and compensation amount of the control session to which the first toy belongs, the first control terminal calculates the estimated effective time for each control action of the first user to be perceived by the second user. In principle, the software development kit (SDK) uses the control action generation time as a reference, offsets the planned transmission time of the control command by the compensation amount of the current control session, and, combined with the control link latency and media link latency, estimates the time when the first toy completes the corresponding action and is presented on the second user's display interface through the media session. This time is taken as the estimated effective time of the first user's control action on the second user's side. Similarly, the first control terminal also performs the same processing on the second user's control actions obtained from the second control terminal, based on the control link latency, media link latency, and compensation amount of the control session to which the second toy belongs, estimating the estimated effective time for these actions to be perceived by the first user. Through the above processing, the first control terminal maps the control actions of both parties onto a common time reference axis, obtaining a collaborative time trajectory that includes the estimated effective times of both parties' actions. After constructing the collaborative time trajectory, the software development kit analyzes the distribution of the expected effective times of the first and second toy actions on the timeline. For each pair of similar expected effective times or time periods, it identifies intervals where the two toy actions overlap at the user's perception level and uses these overlapping intervals as recommended collaborative control time windows. These recommended collaborative control time windows reflect that if both parties initiate specific types of control operations within these time windows, the toy actions seen by both users on each other's terminals will be more synchronized in time, which is beneficial for forming a collaborative rhythm and interactive effect. The system graphically identifies these time windows on the first control terminal display interface, for example, by highlighting the corresponding intervals on the timeline or providing collaborative suggestion prompts in the control interface, thereby guiding the first user to operate within the recommended time windows. When the first user triggers a control operation on the first toy within the recommended collaborative control time window, the software development kit (SDK) marks the corresponding control command as a collaborative control command within the local control session and assigns it a higher scheduling priority in the control command buffer queue of that control session. During subsequent command sending scheduling, when the current time on the virtual timeline approaches or reaches the planned sending time of multiple control commands, collaborative control commands are preferentially retrieved from the buffer queue for sending, ensuring that these commands arrive at the toy and complete execution as close as possible to the recommended collaborative time window within the allowable range of the link.

[0033] In another embodiment, the present invention can also employ a cloud-based architecture for handling latency. Specifically, in addition to integrating software development kits on each control terminal side, a remote interactive platform server is also set up as a centralized computing node for latency processing and alignment. The remote interactive platform server can be implemented as an independent toy control cloud service, or it can be deployed in conjunction with the control service module in an existing business server or real-time audio and video platform to uniformly receive timing data from each control terminal and the control terminal to which the toy belongs; In this embodiment, when each control terminal generates a control command locally, the software development kit (SDK) adds a transmission timestamp, a control session identifier, and a media session identifier to the control command, and reports this information to the remote interactive platform server via control signaling. When the control terminal to which the toy belongs detects that the toy has completed the action corresponding to a certain control command or receives an execution confirmation from the toy, it records the toy's execution timestamp, also carrying the control session identifier, and reports the execution timestamp and the media session identifier associated with the control session to the remote interactive platform server. The remote interactive platform server can simultaneously obtain the control command timestamps and media session identifiers uploaded by each control terminal, as well as the toy execution timestamp uploaded by the toy's control terminal, at the session level. This allows for the reconstruction of the complete timing link from the issuance to the execution of the control command in the cloud, providing a unified data foundation for subsequent calculation of control link latency, media link latency parameters, and compensation amounts on the server side. This enables a cloud-based latency processing solution that can replace or cooperate with the local terminal calculation method. Example 2

[0034] This exemplary toy remote interaction delay alignment system based on a software development kit (SDK) is used to implement the aforementioned toy remote interaction delay alignment method based on an SSD, including: Session establishment module: Establishes media sessions between users among multiple control terminals, and establishes control sessions between the control terminal and the toy for sending control commands; Latency Calculation Module: Using a latency processing module that includes a software development kit integrated into the control terminal, the module calculates the control link latency based on the timestamp of the control terminal sending control commands and the timestamp of the toy executing the control commands; it also obtains the media link latency of the media session between users. Compensation calculation module: Calculates the compensation amount for the control session based on the control link delay and media link delay, establishes a virtual time axis and control command buffer queue, and determines the transmission time by adding the compensation amount to the generation time; Control switching module: Maintains the current control session and candidate control sessions for the same toy, and caches the compensation amount for the candidate control sessions. When control is switched, the original current control session freezes the buffer queue and only responds to the instructions of the new current control session. The new current control session initializes the virtual time with the cache compensation amount.

[0035] It should be noted that the toy remote interactive delay alignment system based on a software development kit (SDK) provided in the above embodiments and the toy remote interactive delay alignment method based on an SSD provided in the above embodiments belong to the same concept. The specific methods of operation of each module and unit have been described in detail in the method embodiments and will not be repeated here. In practical applications, the toy remote interactive delay alignment system based on an SSD provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0036] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for delay alignment of remote interaction in toys based on a software development kit, characterized in that, include: Establish media sessions between users among multiple control terminals, and establish control sessions between control terminals and toys to send control commands; The delay processing module, which includes a software development kit integrated into the control terminal, calculates the control link delay based on the timestamp of the control terminal sending control commands and the timestamp of the toy executing control commands; and obtains the media link delay of the media session between users. The compensation amount is calculated for the control session based on the control link delay and the media link delay. A virtual time axis and a control command buffer queue are established, and the transmission time is determined by adding the compensation amount to the generation time. For the same toy, maintain the current control session and candidate control sessions, and cache the compensation amount for the candidate control session. When control is switched, the original current control session freezes the buffer queue and only responds to the instructions of the new current control session. The new current control session initializes the virtual timeline with the cache compensation amount.

2. The toy remote interaction delay alignment method based on a software development kit according to claim 1, characterized in that, The control session includes: a network control link established between the remote control terminal and the control terminal of the toy, and a wireless connection established between the control terminal of the toy and the toy via Bluetooth or WiFi. The control link latency includes the transmission latency of the network control link and the transmission latency of the local wireless connection.

3. The toy remote interactive delay alignment method based on a software development kit according to claim 1, characterized in that, The calculation of compensation for the control session based on the control link delay and media link delay includes: setting the compensation amount as a monotonic function of the difference between the media link delay and the control link delay; determining the alignment error based on the deviation between the current compensation amount and the difference; updating the compensation amount of the control session only when the absolute value of the alignment error exceeds a preset error threshold; and keeping the compensation amount unchanged when the absolute value of the alignment error is less than or equal to the error threshold.

4. The toy remote interactive delay alignment method based on a software development kit according to claim 1, characterized in that, The establishment of a virtual timeline and control command buffer queue includes: establishing an independent control command buffer queue for each control session; the software development kit in the control terminal calculates the planned transmission time in each buffer queue based on the generation time of the control command and the corresponding compensation amount; scanning each buffer queue at the control session level; and retrieving the control command from the corresponding buffer queue and transmitting it only when the current system time reaches the planned transmission time after being mapped by the virtual timeline, in order to avoid mutual interference between control commands of different control sessions.

5. The toy remote interactive delay alignment method based on a software development kit according to claim 1, characterized in that, The compensation amount for the candidate control session cache includes: periodically sending test control commands to the control terminal corresponding to the candidate control session through control signaling, obtaining the estimated control link delay and estimated media link delay of the candidate control session by combining the execution timestamp returned by the toy with the test audio and video frame timestamps in the media session between users, calculating the candidate compensation amount, storing the candidate compensation amount in the candidate cache, and using it to initialize the virtual timeline when the candidate control session gains control.

6. The toy remote interactive delay alignment method based on a software development kit according to claim 1, characterized in that, When control is switched, the original current control session freezes the buffer queue, which includes: the software development kit in the control terminal corresponding to the original current control session sets its control instruction buffer queue to a frozen state, prohibits new control instructions from being enqueued, and marks the control instructions that have not yet been sent in the queue as invalid. When the toy receives control instructions, it will only execute the control instructions sent by the control terminal corresponding to the new current control session according to the control session identifier.

7. The toy remote interaction delay alignment method based on a software development kit according to claim 3, characterized in that, Also includes: For each control session, the interaction timing coherence potential is calculated within a preset time window based on the alignment error, the magnitude of the compensation change, and the frequency of control command transmission. When the interaction timing coherence potential is higher than the preset coherence threshold, the adjustment step size of the control session compensation is limited. When the interaction timing coherence potential is lower than the coherence threshold, the compensation is allowed to be updated with an adjustment step size greater than the limit step size.

8. The toy remote interaction delay alignment method based on a software development kit according to claim 7, characterized in that, During the protection period after the switch of control, the delayed alignment logic calculates the alignment error and interaction timing coherence potential of the new current control session in multiple consecutive time windows. Only when the alignment error is within the preset error tolerance range and the interaction timing coherence potential is higher than the coherence threshold, the restriction on the adjustment step size of the compensation amount of the new current control session is lifted.

9. The toy remote interactive delay alignment method based on a software development kit according to claim 1, characterized in that, Multiple control terminals include a first control terminal and a second control terminal. The first control terminal is used to: simultaneously display on its display interface the control behavior of a first user on a first toy via the first control terminal and the control behavior of a second user on a second toy via the second control terminal; wherein, the first control terminal calculates the estimated effective time when the first user's control behavior is perceived by the second user based on the control link delay, media link delay, and compensation amount of the control session to which the first toy belongs; calculates the estimated effective time when the second user's control behavior is perceived by the first user based on the control link delay, media link delay, and compensation amount of the control session to which the second toy belongs; and constructs a collaborative time trajectory including the estimated effective time; identifies the time overlap area of ​​the estimated effective time of the first toy and the second toy's actions in the collaborative time trajectory as a recommended collaborative control time window, and prompts the user for the recommended collaborative control time window on the display interface; when the first user triggers a control operation within the recommended collaborative control time window, the corresponding control instruction is marked as a collaborative control instruction and prioritized for sending in the control instruction buffer queue.

10. A toy remote interaction delay alignment system based on a software development kit, used to implement the toy remote interaction delay alignment method based on a software development kit as described in any one of claims 1-9, characterized in that, include: Session establishment module: Establishes media sessions between users among multiple control terminals, and establishes control sessions between the control terminal and the toy for sending control commands; Latency Calculation Module: Using a latency processing module that includes a software development kit integrated into the control terminal, the module calculates the control link latency based on the timestamp of the control terminal sending control commands and the timestamp of the toy executing the control commands; it also obtains the media link latency of the media session between users. Compensation calculation module: Calculates the compensation amount for the control session based on the control link delay and media link delay, establishes a virtual time axis and control command buffer queue, and determines the transmission time by adding the compensation amount to the generation time; Control switching module: Maintains the current control session and candidate control sessions for the same toy, and caches the compensation amount for the candidate control sessions. When control is switched, the original current control session freezes the buffer queue and only responds to the instructions of the new current control session. The new current control session initializes the virtual timeline with the cache compensation amount.

Images