Audio-video synchronous playing method and XR device

Abstract

The application relates to the computer technical field and discloses a sound-picture synchronous playing method and an XR device. In the method, a slave device and a master device establish a unified time reference, receive and play an audio-video file carrying a time stamp broadcast by the master device; the slave device receives a global playing position broadcast by the master device, calculates a deviation value of a local playing position and the global playing position; and according to the deviation value and a synchronization threshold interval, an adjustment strategy of a playing progress of the audio-video file is determined, wherein the local playing position is determined according to the time stamp, and the synchronization threshold interval is determined based on device performance and network performance of the slave device. Through the sound-picture synchronous playing method, the unified time reference is established between the master device and the slave device, the adaptive synchronization control strategy is designed, and the multi-XR device audio-video content synchronous playing is realized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for synchronized audio and video playback and an XR device. Background Technology

[0002] In the fields of extended reality (XR), encompassing virtual reality (VR), augmented reality (AR), and mixed reality (MR), multi-XR device collaboration has been widely applied in entertainment and industrial collaboration scenarios. Its core requirement is to achieve synchronized audio and video playback across multiple devices and consistent results of multi-user interaction. The accuracy of audio-visual synchronization directly determines the immersive experience and collaborative effect.

[0003] Existing multi-XR device synchronization technologies have significant drawbacks: local timestamp-based solutions lack a globally unified time base and are affected by network latency differences and device clock drift, making it impossible to resolve dynamic playback offsets across multiple devices and resulting in low synchronization accuracy; frame rate matching-based solutions, by adjusting playback speed or frame drop alignment progress, are prone to audio distortion and video stuttering, and have poor adaptability to network jitter, leading to insufficient system stability. Summary of the Invention

[0004] The purpose of this application is to provide a method for synchronized audio and video playback and an XR device, which enables the establishment of a unified high-precision time base between the master device and the slave device, and to design an adaptive synchronization control strategy to achieve synchronized playback of audio and video content from multiple XR devices.

[0005] To address the aforementioned technical problems, this application provides a method for synchronized audio and video playback, applied to a slave device. The method includes: establishing a unified time reference with a master device; receiving and playing an audio and video file carrying a timestamp broadcast by the master device; receiving a global playback position broadcast by the master device and calculating a deviation value between the local playback position and the global playback position; and determining an adjustment strategy for the playback progress of the audio and video file based on the deviation value and a synchronization threshold range, wherein the local playback position is determined based on the timestamp, and the synchronization threshold range is determined based on the device performance and network performance of the slave device.

[0006] This application also provides a method for synchronized audio and video playback, applied to a master device. The method includes: broadcasting a time synchronization signal to at least one slave device and receiving a response signal sent by the at least one slave device; determining a clock offset based on the response signal and broadcasting the clock offset to the slave device corresponding to the response signal, so that the at least one slave device can establish a unified time reference based on the clock offset; generating an audio and video file carrying a timestamp and sending the audio and video file to the at least one slave device; broadcasting a global playback position at preset time intervals, so that the at least one slave device can calculate the deviation value between the local playback position and the global playback position, and determine an adjustment strategy for the playback progress of the audio and video file based on the deviation value and a synchronization threshold range; wherein, the global playback position includes a timestamp and a version identifier of the audio and video file, and the synchronization threshold range is determined based on the device performance and network performance of the slave device.

[0007] This application also provides an XR device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described audio-visual synchronization playback method.

[0008] Compared to related technologies, this application designs an adaptive synchronization control strategy for multi-XR device collaboration scenarios such as entertainment, industrial collaboration, and virtual simulation, ultimately achieving synchronized playback of audio and video content across multiple XR devices. Specifically, the master device and slave devices establish a unified time base. The master device then broadcasts the global playback position at time intervals. The slave devices calculate the deviation value based on the global playback position and their own local playback position, and determine the adjustment strategy for the playback progress of the video file based on the deviation value and the synchronization threshold range, thus achieving unified playback progress. Furthermore, the synchronization threshold range is determined based on the network performance and device performance of the slave devices; therefore, the synchronization threshold range differs for different devices, allowing for the determination of the optimal adjustment method for each device.

[0009] Furthermore, the strategy for determining the playback progress adjustment of the audio and video file based on the deviation value and the synchronization threshold range includes: if the deviation value is less than the minimum value of the synchronization threshold range, setting the playback rate according to the deviation value, playing the audio and video file at the playback rate, and performing image smoothing on the audio and video file; if the deviation value is within the synchronization threshold range, adjusting the playback progress to the global playback position and expanding the file cache space; if the deviation value is greater than the maximum value of the synchronization threshold range, pausing playback, adjusting the playback progress to the global playback position and re-caching the file, and re-playing when the cached playback progress is greater than the global playback position.

[0010] In addition, the synchronization threshold range is determined by the following methods: the minimum value of the synchronization threshold range is determined based on a preset deviation benchmark value and a network jitter coefficient, wherein the network jitter coefficient is determined based on the standard deviation of network jitter within a preset time period; the maximum value of the synchronization threshold range is determined based on a preset cache security threshold, a maximum network round-trip latency, and a device rendering load coefficient.

[0011] In addition, establishing a unified time reference with the master device includes: receiving a time synchronization signal sent by the master device, sending a response signal to the master device, allowing the master device to determine the clock offset based on the response signal, and broadcasting the clock offset to the slave device; and adjusting the local time based on the clock offset broadcast by the master device.

[0012] Additionally, receiving the time synchronization signal sent by the master device and sending a response signal to the master device includes: receiving and parsing the time synchronization signal, obtaining a first timestamp carried by the time synchronization signal, and determining a second timestamp of the received time synchronization signal, wherein the first timestamp is used to indicate the time when the master device sent the time synchronization signal; generating a response signal carrying the first timestamp, the second timestamp, and a third timestamp, wherein the third timestamp is used to indicate the time when the slave device sent the response signal; and sending the response signal to the master device.

[0013] In addition, the method further includes: in response to a user's interactive operation on the audio / video file, recording an interaction timestamp and generating interactive event information carrying the interaction timestamp, wherein the interactive event information includes an interaction operation type, operation parameters, and an interaction timestamp; sending the interactive event information to the master device for the master device to broadcast to other slave devices, wherein the interaction timestamp is used to determine the transmission delay, and the interaction operation type and the operation parameters are used to adjust the screen content of the audio / video file.

[0014] Additionally, determining the clock offset based on the response signal includes: determining a fourth timestamp when the response signal is received, and parsing the response signal to obtain a first timestamp, a second timestamp, and a third timestamp, wherein the first timestamp indicates the time when the master device sends the time synchronization signal, the second timestamp indicates the time when the slave device receives the time synchronization signal, and the third timestamp indicates the time when the slave device sends the response signal; and calculating the clock offset based on the first timestamp, the second timestamp, the third timestamp, and the fourth timestamp.

[0015] In addition, generating the audio and video file carrying a timestamp includes: adding a timestamp to the original audio and video file; for each slave device, determining the file block size according to the network bandwidth and caching capacity of the slave device; splitting the original audio and video file into multiple sub-files according to the file block size; and encoding and compressing the multiple sub-files according to a preset encoding and compression algorithm to obtain the audio and video file carrying the timestamp. Attached Figure Description

[0016] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0017] Figure 1 This is a master-slave architecture topology diagram of multiple XR devices involved in some embodiments of this application; Figure 2 This is a system architecture diagram of the slave device and master device involved in some embodiments of this application; Figure 3 This is an exemplary flowchart illustrating a method for synchronized audio-visual playback applied to a slave device, as shown in some embodiments of this application; Figure 4 This is a logic diagram of adaptive calculation of synchronization threshold intervals involved in some embodiments of this application; Figure 5 This is a flowchart of the adjustment strategy determination method involved in some embodiments of this application; Figure 6 This is a timing diagram of the time synchronization algorithm involved in some embodiments of this application; Figure 7 This is a flowchart illustrating the synchronous processing of interactive events in some embodiments of this application; Figure 8 This is an exemplary flowchart of a method for synchronized audio and video playback applied to a main device, as shown in some embodiments of this application; Figure 9 This is a flowchart illustrating the audio and video content preprocessing process involved in some embodiments of this application; Figure 10 This is a structural block diagram of an XR device shown in some embodiments of this application. Detailed Implementation

[0018] To more clearly illustrate the technical solutions of the embodiments in this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this specification. For those skilled in the art, without creative effort, the technical solutions or means disclosed in this specification can be applied to other scenarios based on this technical content.

[0019] It should be understood that the terms "system," "device," "unit," and / or "module" used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other words can achieve the same purpose, they may be replaced by other expressions.

[0020] Unless otherwise specified, the technical terms used to describe components, elements, etc. in this specification are not singular but may include plural. Generally speaking, terms such as "comprising" or "including" only indicate that explicitly identified steps, elements, or components are included, and these steps, elements, and components do not constitute an exclusive list, as the described method or apparatus may also include other steps or components.

[0021] This specification uses flowcharts to illustrate the operational steps performed by the apparatus or system of related embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the order of these steps based on the knowledge and information conveyed by the embodiments in this specification. Adjustments include, but are not limited to, reversing the order of steps, merging multiple steps, and splitting a step.

[0022] Figure 1 This is a master-slave architecture topology diagram of multiple XR devices involved in some embodiments of this application. For example... Figure 1 As shown, this master-slave architecture includes one master device and at least one slave device. The master device serves as the time reference source, playback control center, and interactive event synchronization center, responsible for broadcasting the global playback position, clock offset, and interactive events. Slave devices follow the master device for time synchronization, playback control, and interactive event responses, receiving instructions from the master device and dynamically adjusting their local playback state. During audio and video file playback, the master device sends the audio and video files to each slave device via the network, ensuring that the playback of the audio and video files on the master device and slave devices is identical. Furthermore, users can directly operate the slave devices; all operations and viewing content are synchronized through the master device, ensuring a consistent user experience across multiple users.

[0023] It should be noted that, Figure 1The master-slave architecture topology diagram of multiple XR devices shown is merely an example. The master-slave architecture described in this specification is for the purpose of more clearly illustrating the technical solutions of the embodiments in this specification, and does not constitute a limitation on the technical solutions provided in the embodiments of this specification. For example, Figure 1 The number of master and slave devices in this specification is merely illustrative and is not intended to limit the scope of patent protection of this application. Depending on the actual situation, any number of master and slave devices may be used. As those skilled in the art will understand, with the development of synchronized playback and the emergence of new business scenarios, the technical solutions provided in the embodiments of this specification are also applicable to similar technical problems.

[0024] Figure 2 This is a system architecture diagram of the slave device and master device involved in some embodiments of this application, such as... Figure 2 As shown, the slave device includes a synchronization control module, a time synchronization module, a buffer, and an interactive event handling module. The master device includes a synchronization control module, a content preprocessing module, a time synchronization module, a content distribution module, and an interactive event handling module. Specifically, the master device acts as the global collaboration core. The time synchronization module is used to determine a unified time base and send time signals; the content preprocessing module performs standardization processing on the raw audio and video, including segmentation, adding timestamps, and file compression encoding; the content distribution module broadcasts the preprocessed audio and video files to all slave devices, providing a data source for synchronized playback in the slave device buffers; the synchronization control module broadcasts the globally timestamped playback position at an adaptive frequency; and the interactive event handling module receives interactive event information uploaded by the slave devices and then broadcasts the interactive event information to other slave devices. The slave device acts as the execution terminal. Its time synchronization module receives the time synchronization signal from the master device and returns a response signal. It also receives the clock offset broadcast by the master device and adjusts the local time to ensure consistency with the globally unified time base. The buffer stores the audio and video files distributed by the master device. The synchronization control module receives the global playback position broadcast by the master device, compares it with the local playback position, and determines the dynamic adjustment of the playback progress based on the audio and video timestamps. The interactive event processing module captures the interactive operations of the local user, including gesture commands, location feedback, etc., generates interactive event information carrying interactive timestamps, and receives interactive event information broadcast by the master device. It adjusts the screen content of the audio and video files according to the interactive event information.

[0025] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units are absent in this embodiment.

[0026] In some related embodiments, there are two mainstream technical solutions in the field of audio-visual synchronization for XR devices, but both have significant drawbacks. One type is a synchronization solution based on local timestamps. Its core logic is to add a device-local timestamp to the audio and video frames, and achieve synchronization by comparing the timestamps to sort and match the frames. The fundamental problem with this solution is the lack of a globally unified time reference benchmark: the local clocks of different XR devices are affected by crystal oscillator drift, naturally resulting in time deviations. Furthermore, the latency differences between devices during network transmission further exacerbate this problem, making it impossible to effectively correct the dynamic offset of playback progress between devices. Ultimately, this leads to low synchronization accuracy and fails to meet the basic requirements of multi-user collaborative scenarios. The other type is a synchronization solution based on frame rate matching, which forces the playback progress to align by dynamically adjusting the audio playback rate or video frame rate (such as speeding up or slowing down the playback speed). The shortcomings of this type of solution stem from its core logic of "sacrificing experience for synchronization": adjusting the speed or frame rate directly leads to audio pitch distortion (becoming sharper or coarser), and video stuttering or skipping, severely damaging the immersive experience of XR devices. Simultaneously, this solution is extremely unresponsive to network jitter; when network conditions fluctuate drastically, frequent frame rate or speed adjustments are required, increasing system overhead and reducing system stability, making it difficult to support long-term collaborative operation of multiple devices. Furthermore, neither type of solution designs an effective synchronization mechanism for multi-user interaction scenarios; the spatiotemporal deviation of interactive operations cannot be compensated, further leading to inconsistent interaction results across devices and hindering the expansion of multi-XR device collaborative applications.

[0027] In view of this, some embodiments of this specification provide a method for synchronized audio and video playback, designing an adaptive synchronization control strategy to ultimately achieve synchronized playback of audio and video content across multiple XR devices. Specifically, the master device and slave devices of this application establish a unified time base. Then, the master device broadcasts the global playback position at time intervals. The slave device calculates the deviation value based on the global playback position and its own local playback position, and determines the adjustment strategy for the playback progress of the video file based on the deviation value and the synchronization threshold range, thereby achieving unified playback progress. Furthermore, the synchronization threshold range is determined based on the network performance and device performance of the slave device. Therefore, the synchronization threshold range is different for different devices, allowing for the determination of the optimal adjustment method for each device.

[0028] Figure 3 This is an exemplary flowchart illustrating a method for synchronized audio-visual playback applied to a slave device, as shown in some embodiments of this application. In some embodiments, Figure 3 The process shown can be specifically executed by each module of the slave device. For example... Figure 3 As shown, in some embodiments, the process may include the following steps.

[0029] Step 300: Establish a unified time reference between the slave device and the master device.

[0030] Step 310: Receive and play the timestamped audio and video files broadcast by the master device from the device.

[0031] In step 310, the audio and video files received from the slave device carry timestamps based on a unified time base. These timestamps are determined by the master device based on this unified time base, and a precise timestamp is added to each frame of each audio and video file to ensure that all audio and video frames from all devices are measured based on the same time dimension. This time base is established using either the Network Time Protocol (NTP) or the Precision Time Protocol (PTP), achieving sub-millisecond or millisecond precision. After receiving the timestamped audio and video files from the slave device, they are transmitted to the audio and video stream buffer via the content distribution module. During playback, the local playback position is determined based on the frame timestamps. For example, if the timestamp of the currently playing frame is 1000 milliseconds, the local playback position is 1000 milliseconds. In other words, the local playback position of the audio and video played by the slave device does not depend on its own local clock, but rather on the timestamps uniformly calibrated by both the master and slave devices.

[0032] Step 320: Receive the global playback position broadcast by the master device from the device and calculate the deviation between the local playback position and the global playback position.

[0033] In step 320, the master device, acting as the playback control center, broadcasts the global playback position to all slave devices at an adaptive frequency (default 80 milliseconds / time to 120 milliseconds / time, adjusting the broadcast frequency lower when network jitter is low and higher when network jitter is high). The broadcast information carries a timestamp and version identifier to avoid interference from old data. As mentioned earlier, the local playback position of a slave device is determined by the unified time base timestamp of the current playback frame. For example, if the master device is currently playing a frame with timestamp 1500 milliseconds, it broadcasts 1500 milliseconds as the global playback position. If a slave device is currently playing a frame with timestamp 1480 milliseconds, its local playback position is 1480 milliseconds. After receiving the global playback position, the slave device directly calculates the deviation value: =Global playback position - Local playback position; that is Since both the global playback position and the local playback position are determined based on a unified time base, the deviation value... It can accurately reflect the synchronization deviation between the slave device and the master device, rather than the playback deviation caused by device clock drift or different time references.

[0034] Step 330: The device determines the adjustment strategy for the playback progress of the audio and video files based on the deviation value and the synchronization threshold range.

[0035] In step 330, the synchronization threshold range is the criterion for dynamically adapting to the device and network status. It includes the minimum value of the synchronization threshold range (i.e., the maximum deviation value that is imperceptible to the user) and the maximum value of the synchronization threshold range (i.e., the minimum deviation value that triggers a pause and reset). Both the minimum and maximum values ​​of the synchronization threshold range are dynamically calculated values. Specifically, they are dynamically adjusted by real-time collection of device rendering load, which represents device performance, and network jitter or round-trip time (RTT), which represents network performance, to balance synchronization accuracy and system stability.

[0036] Please see Figure 4 , Figure 4 This is an adaptive calculation logic diagram of the synchronization threshold interval involved in some embodiments of this application. In some embodiments, the minimum value of the synchronization threshold interval is determined based on a preset deviation benchmark value and a network jitter coefficient, wherein the network jitter coefficient is determined based on the standard deviation of network jitter within a preset time period; the maximum value of the synchronization threshold interval is determined based on a preset cache security threshold, a maximum network round-trip latency, and a device rendering load coefficient.

[0037] In one example, the device initializes a synchronization threshold range, for instance, initializing the minimum (lower limit) of the synchronization threshold range to 20 to 40 milliseconds (the maximum deviation baseline imperceptible to the user); and the maximum (upper limit) of the synchronization threshold range to 130 to 170 milliseconds (the buffer safety threshold baseline). Then, the threshold range is periodically synchronized every 4 to 6 seconds, and dynamically adjusted based on device rendering load, network jitter, or RTT.

[0038] In one example, the minimum value of the synchronization threshold range is the maximum critical value at which the user cannot perceive the playback deviation. Its calculation core is based on the user's perception benchmark, dynamically adapting to the intensity of network jitter to avoid frequent adjustments due to minor network fluctuations. The calculation method for this minimum value can be: Minimum value of synchronization threshold range = Preset deviation benchmark value × (1 + Network jitter coefficient).

[0039] The preset deviation benchmark value is the user-perceptible deviation benchmark value. This benchmark value is determined based on the human eye and ear's perception threshold of audio-visual synchronization deviation. For example, when the deviation value is ≤40 milliseconds, the user has no obvious perception. Therefore, setting this benchmark range is the core anchor point for the lower limit calculation. In this embodiment, the user-perceptible deviation benchmark value is set in the range of 20 milliseconds to 40 milliseconds. The network jitter coefficient is determined based on the standard deviation of network jitter within a preset time period. The specific method for determining the network jitter coefficient can be network jitter standard deviation ÷ normalized benchmark, where the network jitter standard deviation represents the fluctuation of network transmission delay in the last 4 to 6 seconds. The larger the standard deviation, the more unstable the network. The normalized benchmark is used to avoid the network jitter coefficient being too large or too small, and its value range can be 40 milliseconds to 60 milliseconds. The upper limit of the network jitter coefficient is between 1.2 and 1.8 to prevent the lower limit from being excessively amplified when there is extreme network jitter. For example, when the jitter is extremely strong, the lower limit is amplified by a maximum of 1.8 times to avoid excessively low synchronization accuracy.

[0040] To facilitate understanding, a specific example of calculating the minimum value of the synchronization threshold interval is given here. Assuming the baseline deviation is 30 milliseconds; the standard deviation of network jitter over the past 4 seconds is 20 milliseconds, and the normalized denominator is 50 milliseconds, then the network jitter coefficient is 0.4; the minimum value of the synchronization threshold interval = 30 milliseconds × (1 + 0.4) = 42 milliseconds.

[0041] In one example, the maximum value of the synchronization threshold range is the minimum critical value at which synchronization can only be restored through pausing, jumping, or re-caching. It is based on the worst-case network conditions and device load capacity to prevent synchronization from completely failing due to network latency or device lag. The maximum value can be calculated as follows: Maximum value of the synchronization threshold range = Preset cache safety threshold + Maximum network round-trip latency + Device rendering load coefficient.

[0042] The preset cache safety threshold is the minimum safe cache duration from the device's cache. Even if the network suddenly interrupts, the cache can still support continuous playback for a period of time, reserving time for jumps and resynchronization to avoid playback interruptions. The cache safety threshold can range from 130 milliseconds to 170 milliseconds. Network round-trip latency is the network round-trip latency within a preset time period. In this embodiment, the maximum value of the network round-trip latency within the most recent 4 to 6 seconds is obtained. This application uses the maximum value instead of the average or fluctuation value to determine the operating status in order to cope with extreme network latency (such as occasional high latency peaks), ensuring that the upper limit can cover the worst-case scenario and avoiding situations where sudden high latency causes the deviation value to exceed the upper limit without triggering a reset. The device rendering load factor is used to represent the hardware rendering performance of the slave device. The value of the rendering load factor is determined based on the current rendering load rate. For example, if the device is rendering complex scenes and processing interactions at the same time, the rendering speed is slow and the rendering load rate is high (such as 75% to 85%), which may cause local playback progress to lag. In this case, the value of the rendering load factor is between 40 milliseconds and 60 milliseconds. If the device rendering load rate is low (such as 45% to 55%), the synchronization accuracy can be improved. In this case, the value of the rendering load factor is between -35 milliseconds and -25 milliseconds, so as to avoid unnecessary caching wait.

[0043] To facilitate understanding, a specific example of calculating the maximum value of the synchronization threshold range is given here. Assume the preset cache safety threshold is 150 milliseconds; the maximum network round-trip latency is 60 milliseconds; the current rendering load rate of the slave device is 80%, and the load factor is 50 milliseconds. Then, the maximum value of the synchronization threshold range = 150 milliseconds + 60 milliseconds + 50 milliseconds = 260 milliseconds.

[0044] Furthermore, neither the minimum nor the maximum value of the synchronization threshold range is a fixed value. It needs to be updated at a fixed period to adapt to network fluctuations and changes in device performance in real time, and to maintain a continuous balance between synchronization accuracy and system stability. This ensures that the accuracy is not insufficient when the network is good due to fixed parameters, nor that frequent resets are triggered when the network is poor.

[0045] In some embodiments, the slave device determines an adjustment strategy for the playback progress of an audio / video file based on the deviation value and a synchronization threshold range: if the deviation value is less than the minimum value of the synchronization threshold range, the playback rate is set according to the deviation value, the audio / video file is played at the playback rate, and the audio / video file is smoothed; if the deviation value is within the synchronization threshold range, the playback progress is adjusted to the global playback position, and the file cache space is expanded; if the deviation value is greater than the maximum value of the synchronization threshold range, playback is paused, the playback progress is adjusted to the global playback position, and the file is re-cached; when the cached playback progress is greater than the global playback position, playback resumes.

[0046] Please see Figure 5 , Figure 5 This is a flowchart illustrating the adjustment strategy determination method according to some embodiments of this application. In one example, the global playback position is received and parsed from the device, and then the deviation between the local playback position and the global playback position is calculated. In conjunction with a synchronization threshold range that is dynamically updated at preset time intervals (4 to 6 seconds), the audio and video playback progress is adjusted through three differentiated hierarchical strategies to ultimately achieve precise synchronization with the master device. The following details the operational logic, parameter basis, and design purpose of each adjustment strategy.

[0047] If the deviation value is less than the minimum value of the synchronization threshold range, the tiered adjustment strategy is in fine-tuning mode. The device only adjusts the playback rate proportionally to the deviation value, with the adjustment range strictly controlled within ±0.5% to ±1.5%. For example... =10 milliseconds, the device slightly increases the playback rate (e.g., by 0.3%) to slowly catch up with the overall progress, rather than jumping abruptly, to ensure a seamless experience and avoid audio distortion. A limiting and smoothing control algorithm is used to avoid pitch changes and image jitter, achieving smooth playback rate adjustment and imperceptible alignment. The deviation value is... It is the deviation between the global playback position broadcast by the master device and the local playback position of the slave device. The ratio of the deviation value is the current deviation value. The ratio of the deviation to the minimum value of the synchronization threshold range; this deviation ratio directly determines the playback rate, and the calculation method can be: In the formula, For playback speed, To the maximum video playback speed, This represents the minimum value within the synchronization threshold range.

[0048] If the deviation value is within the synchronization threshold range, the tiered adjustment strategy is in catch-up mode, directly executing a seek operation to skip the intermediate deviation value and jump to the global playback position broadcast by the main device. Before jumping, the buffer status of the content 0.8 to 1.2 seconds before and after the target playback position needs to be determined to avoid stuttering due to insufficient buffer after jumping. Furthermore, buffer pre-verification is performed to prevent secondary deviations, i.e., the buffer space is expanded before jumping, for example, adjusted to 1.8 to 2.2 seconds before the playback position and 0.8 to 1.2 seconds after the playback position. After jumping, a hysteresis suppression strategy is activated, prohibiting secondary jumps for a period of time (400 to 600 milliseconds). In this way, moderate deviation values ​​are quickly eliminated, and by expanding the buffer range and limiting secondary jumps, repeated deviation values ​​due to network jitter are avoided, ensuring playback stability.

[0049] If the deviation value exceeds the maximum value of the synchronization threshold range, the tiered adjustment strategy is reset. A strategy of pausing, jumping, buffering, and resuming playback is adopted, controlled by a state machine. Specifically, the current audio / video playback is first stopped to prevent the deviation value from continuing to increase (without pausing, audio-visual misalignment and inaccurate interaction may occur). Then, the playback progress is adjusted to the global playback position, and the audio / video files are re-cached. The cache duration needs to be extended by 1.8 to 2.2 seconds from the target position to reserve sufficient cache space to cope with possible network fluctuations or device rendering delays, preventing the deviation value from reappearing after playback resumes. If the cached content progress covers the global playback position and meets the extended cache duration requirement, playback is resumed via state machine control to avoid screen tearing or audio popping caused by sudden restarts.

[0050] In some embodiments, in step 300, the slave device and the master device establish a unified time base to achieve time synchronization in the following manner: the slave device receives a time synchronization signal sent by the master device, sends a response signal to the master device, and the master device determines the clock offset based on the response signal and broadcasts the clock offset to the slave device; the slave device adjusts its local time based on the clock offset broadcast by the master device. Specifically, the slave device generates and returns a time synchronization signal to the master device in the following manner: the slave device receives and parses the time synchronization signal, obtains the first timestamp carried by the time synchronization signal, and determines the second timestamp of the received time synchronization signal, wherein the first timestamp is used to indicate the time when the master device broadcasts the time synchronization signal, and the second timestamp is used to indicate the time when the time synchronization signal is received; generates a response signal carrying the first timestamp, the second timestamp, and a third timestamp, wherein the third timestamp is used to indicate the time when the response signal is sent; and sends the response signal to the master device, so that the master device determines the clock offset based on the response signal.

[0051] Please see Figure 6 , Figure 6 This is a timing diagram of the time synchronization algorithm involved in some embodiments of this application. In one example, this application employs a unified time base establishment mechanism (NTP, PTP, or other equivalent protocols) to establish a unified time base between the master device and the slave device. First, the slave device receives a time synchronization signal sent by the master device, which carries the sending time of the master device. In this process, the master device initiates the transmission at an adaptive interval (which can be 0.5 to 2 seconds and shortened when the network is unstable), rather than sending it all at once, to ensure the long-term stability of the time base. During this process, the master device records the transmission time. Then, when receiving the time synchronization signal from the device, the local received time is recorded. At this time, the device's local time has not yet been calibrated and there is a deviation; recording time Then, a response signal is generated, which must contain three sets of key timestamps, including the master device's sending timestamp. The timestamp from the time synchronization signal received by the device. And the time it takes for the device to send a response signal. The sending time This reflects the local time when the response signal leaves the slave device. After the slave device sends the response signal, the master device records the local reception timestamp when it receives the response signal. Next, the master device calculates the clock offset. In addition, the master device also calculates the network round-trip time (RTT) to eliminate the interference of network latency on time calibration. After receiving the clock offset broadcast by the master device, the slave device subtracts the offset from its own local time to adjust its local time. After adjustment, the slave device's local time is perfectly aligned with the master device's time base.

[0052] Additionally, if the master device fails to receive a response from the slave device within the preset timeout period, or if the packet loss rate exceeds 10%, a synchronization request retransmission is triggered. The preferred value for the preset timeout period is 400 milliseconds to 600 milliseconds to avoid calibration failure due to network interruption or packet loss, ensuring successful signal exchange. The master device initiates calibration at a fixed interval (preferably a base interval of 0.5 seconds to 2 seconds). When a clock offset exceeding the threshold of 3 to 8 milliseconds is detected, the calibration interval is shortened to 400 to 600 milliseconds to ensure time base stability.

[0053] In this way, the master device sends a synchronization signal, the slave device sends back three timestamps, the master device calculates the clock offset, and the slave device adjusts its local time to achieve time calibration. At the same time, combined with the retransmission mechanism and the periodic calibration mechanism, a unified time reference is established between the master and slave devices, which effectively ensures the accuracy of subsequent audio and video timestamp parsing and playback deviation value calculation, and is also a prerequisite for achieving audio and video synchronization of multiple devices.

[0054] In some embodiments, the slave device also responds to the user's interactive operation on the audio and video file by recording an interaction timestamp and generating interaction event information carrying the interaction timestamp, wherein the interaction event information includes the interaction operation type, operation parameters, and interaction timestamp; and sends the interaction event information to the master device for the master device to broadcast to other slave devices; wherein the interaction timestamp is used to determine the transmission delay, and the interaction operation type and operation parameters are used to adjust the content of the audio and video file.

[0055] Please see Figure 7 , Figure 7This is a flowchart illustrating the synchronous processing of interactive events in some embodiments of this application. In one example, user A (the interaction triggering end) triggers an interactive operation on a slave device (the triggering end), including but not limited to gesture commands (such as waving or clicking), location feedback (such as device movement), and function operation commands (such as switching scenes); the slave device (the triggering end) captures the interaction in real time and records the interaction timestamp. (Based on a unified time base, ensuring consistency with the time dimension of other devices); then, interactive event information carrying an interaction timestamp is generated from the device (trigger end). This interactive event information includes the interaction operation type, operation parameters, and interaction timestamp. The interaction operation type specifies the category of the user's operation (e.g., gesture click, fast movement, or button trigger), the operation parameters are the specific data for screen adjustments (e.g., click coordinates, movement speed, and scene switching command codes), and the interaction timestamp... Used to indicate the global moment when the interaction occurs.

[0056] After generating the interaction event information, the slave device (trigger) uploads the complete interaction event information to the master device. The master device verifies the device's legitimacy and the completeness of the parameters within 8 to 12 milliseconds. Upon successful verification, the master device, acting as the interaction event synchronization center, broadcasts the event information to all other slave devices (receivers) in real time. After receiving the interaction event broadcast by the master device, other slave devices (receivers) first obtain their own current local time. According to the device's local time and interaction timestamps Calculate transmission delay ,because and All are based on a unified time base, and the calculated delay is the actual time elapsed, without any error caused by inconsistent time bases.

[0057] The receiving device performs time compensation based on the interaction type and presents the compensated image to user B (the interaction receiver). Specifically: If the interaction type is low-speed (such as button clicks and simple gestures), linear interpolation compensation is used to correct the interaction response timing based on transmission delay, ensuring that the image adjustment is synchronized with the actual operation of the triggering end. If the interaction type is high-speed (such as fast movement and dynamic gestures), non-linear prediction compensation based on historical trajectories is used. By analyzing the historical trajectory of the operation (such as the device's movement direction and speed), the actual position of the object after the delay is predicted, and the image display coordinates are corrected (e.g., if the triggering device moves quickly, the receiving end uses prediction compensation to avoid the image display lagging behind the actual operation). If the current video image is a high-precision scenario, Kalman filtering compensation is used to further improve the prediction accuracy and adapt to high-precision interaction scenarios such as industrial XR. In this way, the image adjustment results of all receiving ends are consistent with the original interaction effect of the triggering end, and users have no perceptible delay difference, ensuring an immersive experience for multi-device collaboration.

[0058] The screen synchronization mechanism of this application achieves interactive consistency, solving the problems of asynchronous interaction times between multiple devices and screen deviation caused by transmission delays, ensuring that all users see completely consistent interaction results. Secondly, it guarantees real-time performance, with the master device verification time controlled within the range of 8 to 12 milliseconds, and no additional delay in the broadcast distribution process. Combined with the application of a hierarchical compensation algorithm, it can effectively guarantee the real-time performance of interactive responses. Furthermore, it has good adaptability, distinguishing between low-speed and high-speed interactions for different interaction types and adopting corresponding compensation strategies. It ensures compensation accuracy while taking into account computational efficiency, and can adapt to different scenario requirements such as entertainment and industrial collaboration. Finally, this mechanism can improve security. Relying on the legality verification step set by the master device, it can effectively prevent interactions initiated by unauthorized devices from interfering with global synchronization, thereby improving system security.

[0059] A method for synchronized audio-visual playback is implemented according to one or more of the above embodiments. This method establishes a unified high-precision time base and designs an adaptive synchronization control strategy to achieve synchronized playback of audio and video content on multiple XR devices. Specifically, addressing the problem that existing technologies typically rely on local timestamps and lack a globally unified time reference, resulting in low synchronization accuracy when multiple devices are affected by network latency and clock drift, this application adopts a method where the master device sends a time synchronization request to the slave devices, and the slave devices return response signals. The master device calculates the clock offset of each slave device based on the response signal, and then each slave device adjusts its local time according to this offset, so that all devices ultimately share the same time measurement dimension. Based on this, the master device broadcasts the current global playback position at fixed time intervals. Each slave device then calculates the deviation value by combining the global playback position with its own current local playback position. Based on the deviation value and the preset synchronization threshold range, the playback progress of the audio and video files is dynamically adjusted to achieve a consistent playback rhythm. It is worth mentioning that the synchronization threshold range used by each slave device is determined individually based on its own network performance and device performance. Therefore, different devices have different synchronization threshold ranges, which enables the matching of the optimal playback adjustment strategy for each device.

[0060] Figure 8 This is an exemplary flowchart illustrating a method for synchronized audio-visual playback applied to a main device, as shown in some embodiments of this application. In some embodiments, Figure 8 The illustrated process can be specifically executed by individual modules within the device. For example... Figure 8 As shown, in some embodiments, the process may include the following steps.

[0061] Step 810: The master device broadcasts a time synchronization signal to at least one slave device and receives a response signal from at least one slave device.

[0062] Step 820: Determine the clock offset based on the response signal and broadcast the clock offset to the slave device corresponding to the response signal, so that at least one slave device can establish a unified time reference based on the clock offset.

[0063] Step 830: Generate an audio / video file carrying a timestamp and send the audio / video file to at least one slave device.

[0064] Step 840: Broadcast the global playback position at preset time intervals, so that at least one slave device can calculate the deviation value between the local playback position and the global playback position, and determine the adjustment strategy for the playback progress of the audio and video files based on the deviation value and the synchronization threshold range.

[0065] The global playback position includes a timestamp and the version identifier of the audio and video files, and the synchronization threshold range is determined based on the device performance and network performance of the slave device.

[0066] Steps 810, 820 and 840 have been described in detail above and will not be repeated here.

[0067] In step 830, the master device generates an audio / video file carrying a timestamp in the following manner, including the following steps: adding a timestamp to the original audio / video file; for each slave device, determining the file block size based on the slave device's network bandwidth and caching capacity; splitting the original audio / video file into multiple sub-files according to the file block size; and encoding and compressing the multiple sub-files according to a preset encoding and compression algorithm to obtain an audio / video file carrying a timestamp.

[0068] Please see Figure 9 , Figure 9This is a flowchart illustrating the audio and video content preprocessing process involved in some embodiments of this application. In one example, the master device first determines the original audio and video content and verifies the legality of the audio and video content format. Then, based on an established unified time reference, it adds a precise timestamp to each audio and video frame. After adding the timestamp, it checks whether the timestamp deviation between adjacent frames meets the frame rate requirements (e.g., the deviation between adjacent frames in a 30 fps video is approximately 33 milliseconds) to avoid timestamp errors. In this way, after all slave devices receive the file, they can locate the globally unified playback time through the timestamp (e.g., if a frame has a timestamp of 1000 milliseconds, all slave devices must play it at the 1000 millisecond moment of the unified time reference), providing a synchronization reference point for subsequent position alignment. Next, the master device adapts to the network bandwidth and file caching capabilities of each slave device, using 70% to 90% of the average bandwidth of the slave devices over the past 20 to 40 seconds as the core reference for segmentation (to avoid excessive bandwidth usage leading to transmission stuttering). Specifically, the master device segments the audio and video files into 3- to 12-second segments based on the slave device's bandwidth: when the slave device's network bandwidth is ≤2Mbps (low bandwidth), the optimal file block size is 3 to 5 seconds (reducing the amount of data in a single block and lowering transmission pressure); when the slave device's network bandwidth is ≥8Mbps (high bandwidth), the optimal file block size is 8 to 12 seconds (increasing the amount of data in a single block, reducing the number of blocks, and improving playback smoothness). Simultaneously, the block size must match the slave device's cache capacity; for slave devices with smaller caches, the block size is reduced accordingly to avoid cache overflow causing playback interruptions. In this way, different block sizes are set for different slave devices with varying hardware and network characteristics, avoiding problems such as transmission stuttering and insufficient cache on some devices, ensuring that all slave devices can stably receive files. The main device splits the original audio and video file into multiple sub-files based on the file block size, and then compresses and encodes these sub-files. For example, video files use H.265 and equivalent high-efficiency encoding formats, while audio files use Advanced Audio Coding (AAC) and equivalent adapted formats. After encoding, the main device also needs to verify the integrity of the encoding and double-check the consistency of the timestamps. Sub-files that pass the verification are combined into the final audio and video file carrying timestamps; those that fail are returned for reprocessing. Through encoding compression, the file size is minimized while ensuring image and sound quality, improving network transmission efficiency. The final verification ensures that the files are synchronized and without anomalies, avoiding playback or synchronization failures due to encoding errors.

[0069] In the aforementioned method for synchronized audio and video playback applied to the master device, a unified high-precision time base is established, and an adaptive synchronization control strategy is designed to achieve synchronized playback of audio and video content across multiple XR devices. Specifically, addressing the problem that existing technologies typically rely on local timestamps and lack a globally unified time reference, resulting in low synchronization accuracy when multiple devices are affected by network latency and clock drift, this application employs a method where the master device sends a time synchronization request to the slave devices, and the slave devices return response signals. The master device calculates the clock offset of each slave device based on the response signal, and then each slave device adjusts its local time according to this offset, so that all devices ultimately share the same time measurement dimension. Based on this, the master device broadcasts the current global playback position at fixed time intervals. Each slave device then calculates the deviation value by combining the global playback position with its own current local playback position. Based on the deviation value and the preset synchronization threshold range, the playback progress of the audio and video files is dynamically adjusted to achieve a consistent playback rhythm. It is worth mentioning that the synchronization threshold range used by each slave device is determined individually based on its own network performance and device performance. Therefore, different devices have different synchronization threshold ranges, which enables the matching of the optimal playback adjustment strategy for each device.

[0070] The steps of the various methods described above are only for clarity. In implementation, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0071] Another embodiment of this application relates to an XR device, such as... Figure 10 As shown, it includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described audio-visual synchronization playback method.

[0072] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0073] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0074] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application.

Claims

1. A method for synchronized audio-visual playback, characterized in that, Applied to a slave device, the method includes: Establish a unified time reference with the main equipment; Receive and play the timestamped audio and video files broadcast by the main device; Receive the global playback position broadcast by the master device, and calculate the deviation between the local playback position and the global playback position; Based on the deviation value and the synchronization threshold range, an adjustment strategy for the playback progress of the audio and video files is determined, wherein the local playback position is determined based on the timestamp, and the synchronization threshold range is determined based on the device performance and network performance of the slave device; wherein... Based on the deviation value and the synchronization threshold range, a strategy for adjusting the playback progress of the audio and video files is determined, including: If the deviation value is less than the minimum value of the synchronization threshold range, the playback rate is set according to the deviation value, the audio and video file is played at the playback rate, and the audio and video file is smoothed. The playback rate is determined based on the maximum video playback rate, the minimum value of the synchronization threshold range, and the deviation value. If the deviation value is within the synchronization threshold range, the playback progress is adjusted to the global playback position, and the file cache space is expanded; If the deviation value is greater than the maximum value of the synchronization threshold range, playback is paused, the playback progress is adjusted to the global playback position, and the file is re-cached. When the cached playback progress is greater than the global playback position, playback resumes.

2. The audio-visual synchronized playback method according to claim 1, characterized in that, The synchronization threshold range is determined by the following methods: The minimum value of the synchronization threshold interval is determined based on a preset deviation benchmark value and a network jitter coefficient, wherein the network jitter coefficient is determined based on the standard deviation of network jitter within a preset time period. The maximum value of the synchronization threshold range is determined based on the preset cache security threshold, the maximum network round-trip latency, and the device rendering load coefficient.

3. The audio-visual synchronized playback method according to claim 1, characterized in that, The establishment of a unified time base with the master device includes: The device receives a time synchronization signal sent by the master device, sends a response signal to the master device, and the master device determines the clock offset based on the response signal and broadcasts the clock offset to the slave device. Adjust the local time according to the clock offset broadcast by the master device.

4. The audio-visual synchronized playback method according to claim 3, characterized in that, The step of receiving the time synchronization signal sent by the master device and sending a response signal to the master device includes: The system receives and parses the time synchronization signal, obtains a first timestamp carried by the time synchronization signal, and determines a second timestamp of the received time synchronization signal, wherein the first timestamp is used to indicate the time when the master device sends the time synchronization signal; Generate a response signal carrying the first timestamp, the second timestamp, and the third timestamp, wherein the third timestamp is used to indicate the time when the slave device sends the response signal; The response signal is sent to the master device.

5. The audio-visual synchronized playback method according to claim 1, characterized in that, The method further includes: In response to the user's on-screen interaction with the audio and video file, the interaction timestamp is recorded and interaction event information carrying the interaction timestamp is generated, wherein the interaction event information includes the interaction operation type, operation parameters and interaction timestamp; The interactive event information is sent to the master device, which then broadcasts it to other slave devices. The interaction timestamp is used to determine the transmission delay, and the interaction operation type and operation parameters are used to adjust the picture content of the audio and video file.

6. A method for synchronized audio and video playback, characterized in that, Applied to a master device, the method includes: Broadcast a time synchronization signal to at least one slave device and receive a response signal from the at least one slave device; The clock offset is determined based on the response signal, and the clock offset is broadcast to the slave device corresponding to the response signal, so that the at least one slave device can establish a unified time reference based on the clock offset; Generate an audio / video file carrying a timestamp, and send the audio / video file to the at least one slave device; The global playback position is broadcast at preset time intervals, allowing at least one slave device to calculate the deviation between its local playback position and the global playback position. Based on the deviation and a synchronization threshold range, an adjustment strategy for the playback progress of the audio / video file is determined. The global playback position includes a timestamp and the version identifier of the audio / video file, and the synchronization threshold range is determined based on the device performance and network performance of the slave devices. Based on the deviation value and the synchronization threshold range, a strategy for adjusting the playback progress of the audio and video files is determined, including: If the deviation value is less than the minimum value of the synchronization threshold range, the playback rate is set according to the deviation value, the audio and video file is played at the playback rate, and the audio and video file is smoothed. The playback rate is determined based on the maximum video playback rate, the minimum value of the synchronization threshold range, and the deviation value. If the deviation value is within the synchronization threshold range, the playback progress is adjusted to the global playback position, and the file cache space is expanded; If the deviation value is greater than the maximum value of the synchronization threshold range, playback is paused, the playback progress is adjusted to the global playback position, and the file is re-cached. When the cached playback progress is greater than the global playback position, playback resumes.

7. The audio-visual synchronized playback method according to claim 6, characterized in that, Determining the clock offset based on the response signal includes: A fourth timestamp is determined when the response signal is received, and the response signal is parsed to obtain a first timestamp, a second timestamp, and a third timestamp, wherein the first timestamp is used to indicate the time when the master device sends the time synchronization signal, the second timestamp is used to indicate the time when the slave device receives the time synchronization signal, and the third timestamp is used to indicate the time when the slave device sends the response signal; The clock offset is calculated based on the first timestamp, the second timestamp, the third timestamp, and the fourth timestamp.

8. The audio-visual synchronized playback method according to claim 6, characterized in that, The generation of audio and video files carrying timestamps includes: Add timestamps to the original audio and video files; For each slave device, the file block size is determined based on the slave device's network bandwidth and caching capacity. Based on the file block size, the original audio and video file is split into multiple sub-files; The multiple sub-files are encoded and compressed according to a preset encoding and compression algorithm to obtain the audio and video file carrying a timestamp.

9. An XR device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the audio-visual synchronization playback method as described in any one of claims 1 to 5, or the audio-visual synchronization playback method as described in any one of claims 6 to 8.

Images