Audio and picture synchronous playing method and XR equipment

Abstract

The invention relates to the technical field of computers, and discloses a sound and picture synchronous playing method and XR equipment. In the method, a unified time reference is established between slave equipment and master equipment, and audio and video files which are broadcast by the master equipment and carry timestamps are received and played; the slave device receives the global playing position broadcasted by the master device, and calculates a deviation value between the local playing position and the global playing position; according to the deviation value and a synchronization threshold interval, an adjustment strategy of the playing progress of the audio and video file is determined, the local playing position is determined according to the timestamp, and the synchronization threshold interval is determined based on the equipment performance and the network performance of the slave equipment. Through the audio and picture synchronous playing method, a unified time reference is established between the master device and the slave device, and an adaptive synchronous control strategy is designed, so that synchronous playing of audio and video contents of multiple XR devices is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and particularly relates to a sound-picture synchronous playing method and an XR device. BACKGROUND

[0002] In the technical field of extended reality (XR) covering virtual reality (VR), augmented reality (AR) and mixed reality (MR), multi-XR device collaboration has been widely applied in entertainment and industrial collaboration scenes, and the core requirement is to realize multi-device audio-video synchronous playing and consistent multi-user interaction results, and the sound-picture synchronous precision directly determines the immersive experience and collaboration effect.

[0003] The existing multi-XR device synchronization technology has obvious defects: the scheme based on local time stamp lacks a global unified time reference, is affected by network delay difference and device clock drift, cannot solve multi-device dynamic playing offset, and has low synchronization precision; the scheme based on frame rate matching adjusts playing rate or frame dropping to align progress, easily causes audio distortion and picture lag, and has poor network jitter adaptability and insufficient system stability. SUMMARY

[0004] The purpose of the present application is to provide a sound-picture synchronous playing method and an XR device, so as to establish a unified high-precision time reference between a master device and a slave device, and design an adaptive synchronization control strategy, thereby realizing multi-XR device audio-video content synchronous playing.

[0005] To solve the above technical problems, the present application provides a sound-picture synchronous playing method applied to a slave device, and the method comprises the following steps: establishing a unified time reference with a master device; receiving and playing an audio-video file carrying a time stamp broadcast by the master device; receiving a global playing position broadcast by the master device, and calculating a deviation value of a local playing position and the global playing position; determining an adjustment strategy of a playing progress of the audio-video file according to the deviation value and a synchronization threshold interval, wherein the local playing position is determined according to the time stamp, and the synchronization threshold interval is determined based on device performance and network performance of the slave device.

[0006] The application also provides an audio-video synchronous playing method applied to a master device, the method comprising: broadcasting a time synchronization signal to at least one slave device and receiving a response signal sent by the at least one slave device; determining a clock offset based on the response signal and broadcasting the clock offset to a slave device corresponding to the response signal for the at least one slave device to establish a unified time reference based on the clock offset; generating an audio-video file carrying a timestamp and sending the audio-video file to the at least one slave device; broadcasting a global playing position at a preset time interval for the at least one slave device to calculate a deviation value of a local playing position from the global playing position and determine an adjustment strategy of a playing progress of the audio-video file according to the deviation value and a synchronization threshold interval; wherein the global playing position comprises a timestamp and a version identifier of the audio-video file, and the synchronization threshold interval is determined based on device performance and network performance of the slave device.

[0007] The application also provides an XR device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the audio-video synchronous playing method described above.

[0008] The application, relative to the related art, designs an adaptive synchronization control strategy for multi-XR device collaborative scenarios such as entertainment, industrial collaboration and virtual simulation, and finally realizes synchronous playing of audio-video content of multiple XR devices. Specifically, the master device and the slave device establish a unified time reference, then the master device broadcasts a global playing position at a time interval, the slave device calculates a deviation value according to the global playing position and its own local playing position, and determines an adjustment strategy of a playing progress of a video file according to the deviation value and a synchronization threshold interval, to realize the unity of the playing progress; in addition, the synchronization threshold interval is determined according to the network performance and device performance of the slave device, so that the synchronization threshold intervals of different devices are different, so that the optimal adjustment mode is determined for each device.

[0009] In addition, the adjustment strategy of the playing progress of the audio-video file is determined according to the deviation value and the synchronization threshold interval, comprising: if the deviation value is less than the minimum value of the synchronization threshold interval, setting a playing rate according to the deviation value, playing the audio-video file at the playing rate, and performing picture smoothing processing on the audio-video file; if the deviation value is within the synchronization threshold interval, adjusting the playing progress to the global playing position and expanding the file cache space; if the deviation value is greater than the maximum value of the synchronization threshold interval, pausing playing, adjusting the playing progress to the global playing position and re-caching the file, and re-playing when the cached playing progress is greater than the global playing position.

[0010] In addition, the synchronization threshold interval is determined by: a minimum value of the synchronization threshold interval, determined according to a preset deviation reference value and a network jitter coefficient, wherein the network jitter coefficient is determined according to a network jitter standard deviation in a preset time period; and a maximum value of the synchronization threshold interval, determined according to a preset cache security threshold, a network round-trip delay maximum value, and a device rendering load coefficient.

[0011] In addition, the establishment of the unified time reference with the master device comprises: receiving a time synchronization signal sent by the master device, sending a response signal to the master device, so that the master device determines a clock offset according to the response signal, and broadcasts the clock offset to the slave device; and adjusting a local time according to the clock offset broadcast by the master device.

[0012] In addition, the receiving of the time synchronization signal sent by the master device and the sending of the response signal to the master device comprise: receiving and analyzing the time synchronization signal, obtaining a first timestamp carried by the time synchronization signal, and determining a second timestamp of receiving the time synchronization signal, wherein the first timestamp is used to indicate a time of sending the time synchronization signal by the master device; generating a response signal carrying the first timestamp, the second timestamp, and a third timestamp, wherein the third timestamp is used to indicate a time of sending the response signal by the slave device; and sending the response signal to the master device.

[0013] In addition, the method further comprises: in response to a picture interaction operation of a user on the audio and video file, recording an interaction timestamp and generating interaction event information carrying the interaction timestamp, wherein the interaction event information comprises an interaction operation type, an operation parameter, and the interaction timestamp; and sending the interaction event information to the master device for broadcasting to other slave devices, wherein the interaction timestamp is used to determine a transmission delay, and the interaction operation type and the operation parameter are used to adjust picture content of the audio and video file.

[0014] In addition, the determination of the clock offset according to the response signal comprises: determining a fourth timestamp of receiving the response signal, and analyzing the response signal to obtain a first timestamp, a second timestamp, and a third timestamp, wherein the first timestamp is used to indicate a time of sending the time synchronization signal by the master device, the second timestamp is used to indicate a time of receiving the time synchronization signal by the slave device, and the third timestamp is used to indicate a time of sending the response signal by the slave device; and calculating the clock offset based on the first timestamp, the second timestamp, the third timestamp, and the fourth timestamp.

[0015] In addition, the generating the audio-video file carrying the timestamp comprises: adding a timestamp to an original audio-video file; determining a file block size according to a network bandwidth and a cache capability of each slave device; splitting the original audio-video file into a plurality of sub-files according to the file block size; and performing encoding compression on the plurality of sub-files according to a preset encoding compression algorithm to obtain the audio-video file carrying the timestamp. BRIEF DESCRIPTION OF DRAWINGS

[0016] One or more embodiments are illustrated by way of example in the figures that are part of this document and which illustrate key principles of the embodiments. The drawings disclose like reference characters and numbers to disclose like elements throughout the several views of the drawings. The figures are not necessarily to scale and the emphasis is on the functional operation of the embodiments.

[0017] Figure 1 is a master-slave architecture topology diagram of a multi-XR device related to some embodiments of the present application; Figure 2 is a system architecture diagram of a slave device and a master device related to some embodiments of the present application; Figure 3 is an exemplary flowchart of an audio-video synchronous playback method applied to a slave device according to some embodiments of the present application; Figure 4 is a logic diagram of adaptive calculation of a synchronization threshold interval related to some embodiments of the present application; Figure 5 is a flowchart of an adjustment strategy determination method related to some embodiments of the present application; Figure 6 is a timing diagram of a time synchronization algorithm related to some embodiments of the present application; Figure 7 is a flowchart of interactive event synchronization processing related to some embodiments of the present application; Figure 8 is an exemplary flowchart of an audio-video synchronous playback method applied to a master device according to some embodiments of the present application; Figure 9 is a flowchart of audio-video content preprocessing related to some embodiments of the present application; Figure 10 is a structural block diagram of an XR device according to some embodiments of the present application. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present specification, the embodiments will be described in detail below with reference to the drawings. Obviously, the following description is some examples or embodiments of the present specification, and for those skilled in the art, the technical solutions or means disclosed in the present specification can also be applied to other scenarios without creative labor.

[0019] It should be understood that the "system", "device", "unit" and / or "module" used in the present specification is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0020] Unless otherwise specified, the technical terms of components, elements, etc. described in the present specification do not refer to a single number, but can also include a plurality. Generally, the terms "include", "contain" and the like only indicate the inclusion of the steps, elements or components explicitly identified, and these steps, elements and components do not constitute an exclusive list, and the method or device described can also include other steps or components.

[0021] Flowcharts are used in the present specification to illustrate the operation steps performed by the device or system of the related embodiments, but the order used to describe these steps should not be understood as a limitation on the execution order of the steps, unless otherwise specified. Those skilled in the art can adjust the execution order of these steps according to the knowledge information conveyed by the embodiments of the present specification, and the adjustment includes but is not limited to reversing the sequence, merging multiple steps, and splitting a step.

[0022] Figure 1 is a master-slave architecture topology diagram of a multi-XR device related to some embodiments of the present application. As shown in Figure 1 The master-slave architecture includes one master device and at least one slave device, wherein the master device is a device that serves as a time reference source, a play control center and an interactive event synchronization center, responsible for broadcasting the global play position, clock offset and interactive event; the slave device is a device that follows the master device for time synchronization, play control and interactive event response, receives instructions from the master device and dynamically adjusts the local play state. During the audio and video file playback process, the master device sends the audio and video file to each slave device through the network, and the pictures of the audio and video files played by the master device and the slave device should be the same. In addition, the user can directly operate the slave device, and all operations and viewing content are synchronized through the master device to ensure consistent multi-user experience.

[0023] It should be noted that Figure 1The master-slave architecture topology diagram of the multi-XR device shown is only an example, and the master-slave architecture is described in the embodiments of the present specification to more clearly illustrate the technical solutions of the embodiments of the present specification, and does not constitute a limitation on the technical solutions provided by the embodiments of the present specification. For example, Figure 1 The number of master devices and slave devices in the above is only illustrative, and is not used to limit the patent protection scope of the present application. According to actual conditions, there can be any number of master devices and slave devices, and those skilled in the art can know that, with the development of synchronous playback and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present specification are also applicable to similar technical problems.

[0024] Figure 2 is a system architecture diagram of the slave device and the master device involved in some embodiments of the present application, as shown in Figure 2 The slave device includes a synchronization control module, a time synchronization module, a buffer area, and an interactive event processing module. The master device includes a synchronization control module, a content preprocessing module, a time synchronization module, a content distribution module, and an interactive event processing module. Specifically, the master device serves as a global coordination core, the time synchronization module is used to determine a unified time reference and send a time signal; the content preprocessing module is used to perform standardized processing on original audio and video, including segmentation, adding a timestamp, and compression and encoding processing of a file; the content distribution module broadcasts the preprocessed audio and video file to all slave devices to provide a data source for synchronous playback of the buffer area of the slave device; the synchronization control module broadcasts a global playback position with a timestamp at an adaptive frequency; and the interactive event processing module receives interactive event information uploaded by the slave device, and then broadcasts the interactive event information to other slave devices. The slave device serves as an execution terminal, the time synchronization module of the slave device returns a response signal after receiving the time synchronization signal of the master device, and adjusts the local time after receiving the clock offset broadcast by the master device, to ensure consistency with the global unified time reference; the buffer area stores the audio and video file distributed by the master device; the synchronization control module is used to receive the global playback position broadcast by the master device, compare the local playback position, and determine a dynamic adjustment of the playback progress based on the audio and video timestamp; and the interactive event processing module captures the interactive operation of the local user, including gesture instructions, position feedback, and other operations, generates interactive event information carrying an interactive timestamp, and receives the interactive event information broadcast by the master device, and adjusts the picture content of the audio and video file according to the interactive event information.

[0025] It is worth mentioning that each module involved in the present embodiment is a logical module. In actual application, one logical unit can be one physical unit, or a part of one physical unit, or realized by a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed in the present application are not introduced in the present embodiment, but this does not mean that there are no other units in the present embodiment.

[0026] In some related embodiments, there are two mainstream technical solutions in the field of XR device audio-visual synchronization, but both have significant defects. One is a synchronization scheme based on local timestamps, the core logic of which is to add device local timestamps to audio and video frames, to achieve frame sorting and matching by comparing timestamps, and thus achieve synchronization. The fundamental problem of this scheme is the lack of a globally unified time reference: the local clocks of different XR devices are affected by crystal drift, naturally have time deviation values, and the delay differences of each device in the network transmission process further exacerbate this problem, resulting in dynamic shifts in the playback progress between devices that cannot be effectively corrected, ultimately leading to low synchronization accuracy, which cannot meet the basic needs of multi-user collaborative scenarios. The other is a synchronization scheme based on frame rate matching, which forcibly aligns the playback progress by dynamically adjusting the audio playback rate or video frame rate (such as speeding up or slowing down the playback speed). The defect of this scheme lies in its core logic of "sacrificing experience for synchronization": adjusting the rate or frame rate will directly cause audio pitch distortion (sharp or thick), video frame freezing or jumping, and severely damage the immersive experience of XR devices; at the same time, this scheme has poor adaptability to network jitter, and when the network conditions fluctuate dramatically, frequent frame rate or rate adjustments are required, which not only increases system overhead but also leads to decreased system stability, making it difficult to support long-term collaborative operation of multiple devices. In addition, neither of the two schemes has designed an effective synchronization mechanism for multi-user interactive scenarios, and the time and space deviation values of interactive operations cannot be compensated, further leading to inconsistent interactive results on each device, restricting the expansion of multi-XR device collaborative applications.

[0027] Therefore, some embodiments of the present specification provide an audio-visual synchronization playback method, which designs an adaptive synchronization control strategy to ultimately achieve synchronized playback of audio and video content on multiple XR devices. Specifically, the master device and the slave device establish a unified time reference, then the master device broadcasts the global playback position at a time interval, the slave device calculates the deviation value based on the global playback position and its own local playback position, and determines the adjustment strategy of the video file playback progress according to the deviation value and the synchronization threshold interval, to achieve the unification of the playback progress; in addition, the synchronization threshold interval is determined according to the network performance and device performance of the slave device, so the synchronization threshold intervals of different devices are different, making it possible to determine the optimal adjustment mode for each device.

[0028] Figure 3 is an exemplary flowchart of an audio-visual synchronization playback method applied to a slave device according to some embodiments of the present application. In some embodiments, Figure 3 The flowchart shown in can be specifically executed by modules of the slave device. As Figure 3 shown in some embodiments, the flowchart can include the following steps.

[0029] Step 300, the slave device and the master device establish a unified time reference.

[0030] Step 310, receiving and playing the audio and video file broadcasted by the master device from the slave device.

[0031] In step 310, the audio and video file received by the slave device carries a timestamp of a unified time reference, which is determined by the master device based on the unified time reference and adds an accurate timestamp to each frame of each audio and video file, ensuring that the audio and video frames of all devices are measured based on the same time dimension; wherein the time reference is established through the Network Time Protocol (NTP) or Precision Time Protocol (PTP) protocol, with an accuracy of sub-millisecond or millisecond level. After the slave device receives the audio and video file with the timestamp, it is transmitted to the cache area of the audio and video stream through the content distribution module, and when the audio and video file is played, the local playback position is determined according to the timestamp of the frame, such as the timestamp of the current playback frame being 1000 milliseconds, i.e. the local playback position is 1000 milliseconds. That is, the local playback position of the audio and video played by the slave device does not depend on its own local clock, but is based on the timestamp after the master-slave device unified calibration.

[0032] Step 320, the slave device receives the global playback position broadcasted by the master device, and calculates the deviation value of the local playback position and the global playback position.

[0033] In step 320, the master device as the playback control center will broadcast the global playback position to all slave devices at an adaptive frequency (default 80 milliseconds / time to 120 milliseconds / time, the broadcast frequency becomes lower when the network jitter is small, and the broadcast frequency becomes higher when the network jitter is large), and the broadcast information carries a timestamp and a version identifier to avoid interference by old data. As mentioned earlier, the local playback position of the slave device is determined by the unified time reference timestamp of the current playback frame. For example, the master device currently plays to the frame with a timestamp of 1500 milliseconds, and broadcasts 1500 milliseconds as the global playback position. The slave device currently plays to the frame with a timestamp of 1480 milliseconds, and the local playback position is 1480 milliseconds; after the slave device receives the global playback position, it directly calculates the deviation value: = global playback position - local playback position; that is, Since the global playback position and the local playback position are both determined based on the unified time reference, the deviation value can truly reflect the synchronization deviation value of the slave device and the master device, rather than the playback deviation caused by device clock drift or different time references.

[0034] Step 330, the slave device determines the adjustment strategy of the playback progress of the audio and video file according to the deviation value and the synchronization threshold interval.

[0035] In step 330, the synchronization threshold interval is a judgment criterion dynamically adapted to the device and network states, which includes a minimum value of the synchronization threshold interval (i.e., a maximum deviation value that is not perceived by a user) and a maximum value of the synchronization threshold interval (i.e., a minimum deviation value that triggers a pause reset). Both the minimum value and the maximum value of the synchronization threshold interval are dynamically calculated values, and are specifically dynamically adjusted by real-time collection of a device rendering load used to represent a device performance and collection of network jitter or Round-Trip Time (RTT) used to represent a network performance, so as to balance synchronization accuracy and system stability.

[0036] Referring to Figure 4 , Figure 4 is a synchronization threshold interval self-adaptive calculation logic diagram involved in some embodiments of the present application. In some embodiments, the minimum value of the synchronization threshold interval is determined according to a preset deviation reference value and a network jitter coefficient, wherein the network jitter coefficient is determined according to a network jitter standard deviation in a preset time period; and the maximum value of the synchronization threshold interval is determined according to a preset buffer safety threshold, a network RTT maximum value and a device rendering load coefficient.

[0037] In one example, a synchronization threshold interval is initialized by a device, for example, the minimum value (lower limit) of the synchronization threshold interval is initialized to 20 milliseconds to 40 milliseconds (a maximum deviation reference value that is not perceived by a user); and the maximum value (upper limit) of the synchronization threshold interval is initialized to 130 milliseconds to 170 milliseconds (a buffer safety threshold reference value). Then, the synchronization threshold interval is periodically synchronized at a time interval of every 4 seconds to 6 seconds, and is dynamically adjusted according to a device rendering load, network jitter or RTT.

[0038] In one example, the minimum value of the synchronization threshold interval is a maximum critical value of a playback deviation value that is not perceived by a user, and a calculation core thereof is based on a user perception reference value and dynamically adapted to network jitter intensity, so as to avoid frequent adjustment caused by slight network fluctuations. The calculation method of the minimum value can be: the minimum value of the synchronization threshold interval = preset deviation reference value × (1 + network jitter coefficient).

[0039] The preset deviation reference value is a user-inperceptible deviation reference value, which is determined based on a perception threshold of a human eye and a human ear to an audio-video synchronization deviation value. For example, when the deviation value is less than or equal to 40 milliseconds, the user has no obvious perception, and therefore, the reference range is set as the core anchor point for lower limit calculation. In this embodiment, the user-inperceptible deviation reference value is set in a range of 20 milliseconds to 40 milliseconds. The network jitter coefficient is determined according to a network jitter standard deviation in a preset time period. The specific determination method of the network jitter coefficient can be network jitter standard deviation ÷ normalization reference, wherein the network jitter standard deviation represents a fluctuation degree of network transmission delay in the last 4 seconds to 6 seconds, and the greater the standard deviation, the more unstable the network. The normalization reference is used to avoid that the network jitter coefficient is too large or too small, and the value range of the normalization reference can be 40 milliseconds to 60 milliseconds. The upper limit of the network jitter coefficient is between 1.2 and 1.8, so as to prevent the lower limit from being excessively enlarged when the network is extremely jittered. For example, when the jitter is extremely strong, the lower limit is enlarged by at most 1.8 times, to avoid that the synchronization accuracy is too low.

[0040] For ease of understanding, a specific calculation example of the minimum value of the synchronization threshold interval is given here. It is assumed that the deviation reference value is 30 milliseconds, the network jitter standard deviation in the last 4 seconds is 20 milliseconds, and the normalization denominator is 50 milliseconds. Therefore, the network jitter coefficient is 0.4, and the minimum value of the synchronization threshold interval is 30 milliseconds x (1+0.4) = 42 milliseconds.

[0041] In one example, the maximum value of the synchronization threshold interval is the minimum critical value that can recover synchronization by pausing, jumping or rebuffering, which is obtained based on the worst case of the network and the load capacity of the device, to avoid that the synchronization is completely invalid due to network delay or device lag. The calculation method of the maximum value can be: the maximum value of the synchronization threshold interval = preset buffer safety threshold + network round-trip delay maximum value + device rendering load coefficient.

[0042] The preset cache safety threshold is the minimum safe cache duration of the device cache area, so that the cache area can support continuous playing for a period of time even if the network is interrupted, time is reserved for jumping and resynchronization to avoid playing interruption, and the value range of the cache safety threshold can be 130 milliseconds to 170 milliseconds. The network round-trip delay is the network round-trip delay in a preset time period. In this embodiment, the maximum value of the network round-trip delay in the last 4 seconds to 6 seconds is obtained, and the maximum value instead of the average value or fluctuation value is used to determine the running state, so as to cope with extreme delay of the network (for example, a high delay peak occasionally occurs), ensure that the upper limit can cover the worst case, and avoid that the deviation value exceeds the upper limit without triggering reset due to sudden high delay. The device rendering load coefficient is used to represent the hardware rendering performance of the slave device, and the value of the rendering load coefficient is determined according to the current rendering load rate. For example, if the device simultaneously renders a complex scene and processes interaction, at this time, the rendering speed is slow, the rendering load rate is high (for example, 75% to 85%), and the local playing progress may lag, so that the value of the rendering load coefficient ranges from 40 milliseconds to 60 milliseconds. If the device rendering load rate is low (for example, 45% to 55%), the synchronization accuracy can be improved, so that the value of the rendering load coefficient ranges from -35 milliseconds to -25 milliseconds, to avoid unnecessary cache waiting.

[0043] To facilitate understanding, a specific calculation example of the maximum value of the synchronization threshold interval is given here. It is assumed that the preset cache safety threshold is 150 milliseconds, the maximum value of the network round-trip delay is 60 milliseconds, the current rendering load rate of the slave device is 80%, and the load coefficient is 50 milliseconds. Then the maximum value of the synchronization threshold interval = 150 milliseconds + 60 milliseconds + 50 milliseconds = 260 milliseconds.

[0044] In addition, neither the minimum value nor the maximum value of the synchronization threshold interval is a fixed value, and needs to be updated at a fixed period, so as to adapt to network fluctuations and device performance changes in real time, continuously balance between synchronization accuracy and system stability, and neither cause insufficient accuracy when the network is good due to fixed parameters, nor frequently trigger reset when the network is poor.

[0045] In some embodiments, the slave device determines an adjustment strategy of the playing progress of the audio and video file according to the deviation value and the synchronization threshold interval: if the deviation value is less than the minimum value of the synchronization threshold interval, the playing rate is set according to the deviation value, the audio and video file is played at the playing rate, and the picture is smoothed; if the deviation value is in the synchronization threshold interval, the playing progress is adjusted to the global playing position, and the file cache space is expanded; and if the deviation value is greater than the maximum value of the synchronization threshold interval, the playing is paused, the playing progress is adjusted to the global playing position and the file is rebuffered, and the playing is restarted when the buffered playing progress is greater than the global playing position.

[0046] Please refer to Figure 5 ,Figure 5 is a flowchart of the adjustment policy determination method involved in some embodiments of the present application. In one example, the global play position is received from the device and parsed, and then the deviation value of the local play position from the global play position is calculated , combined with the synchronization threshold interval dynamically updated at preset time intervals (4-6 seconds), the audio and video play progress is adjusted through three differentiated hierarchical adjustment strategies, and finally the precise synchronization with the master device is achieved. The operation logic, parameter basis and design purpose of each adjustment strategy are explained in detail below.

[0047] If the deviation value is less than the minimum value of the synchronization threshold interval, the hierarchical adjustment strategy is fine-tuning mode, and the slave device only fine-tunes the play rate by the deviation value proportion, and the adjustment amplitude is strictly controlled within ±0.5% to ±1.5%; for example =10 milliseconds, the slave device slightly increases the play rate (e.g. by 0.3%), and slowly catches up with the global progress, rather than instantaneously jumping to ensure no perception and no audio distortion, using amplitude limiting and smoothing control algorithm to avoid tone changes and picture jitter, to achieve smooth adjustment of the play rate, no perception alignment. Wherein, the deviation value is the deviation value of the global play position broadcast by the master device and the local play position of the slave device, and the deviation value proportion is the ratio of the current deviation value to the minimum value of the synchronization threshold interval; the deviation value proportion directly determines the play rate, and the calculation method can be: , wherein, is the play rate, is the maximum video play rate, is the minimum value of the synchronization threshold interval.

[0048] If the deviation value is within the synchronization threshold interval, the hierarchical adjustment strategy is the catch-up mode, which directly executes the Seek operation to skip the intermediate deviation value part and jump to the global play position broadcast by the master device; before jumping, it also needs to determine the cache status of the content 0.8-1.2 seconds before and after the target play position to avoid stuttering due to insufficient cache after jumping; and perform buffer pre-verification to prevent secondary deviation, i.e. expand the cache space before jumping, such as adjusting to 1.8-2.2 seconds before the play position and 0.8-1.2 seconds after the play position, and after jumping, the lag suppression strategy is started, i.e. secondary jumping is prohibited within a certain period of time (400-600 milliseconds) in the future. In this way, moderate deviation values are quickly eliminated, and by expanding the cache range and limiting secondary jumping, the deviation value is prevented from repeatedly fluctuating due to network jitter, ensuring the stability of the play.

[0049] If the deviation value is greater than the maximum value of the synchronization threshold interval, the hierarchical adjustment strategy is reset mode. The strategy of heavy pause→jump→buffer→resume playback is adopted, which is controlled by the state machine. Specifically, the current audio and video playback is stopped first to avoid the continuous expansion of the deviation value (if not paused, problems such as audio and video disorder and interaction misalignment may occur); then the playback progress is adjusted to the global playback position, and the audio and video file is rebuffered, and the buffer time needs to be extended by 1.8 seconds to 2.2 seconds based on the target position, so as to reserve sufficient buffer space to cope with subsequent possible network fluctuations or device rendering delays, and avoid the deviation value after resuming playback. If the buffered content progress covers the global playback position and meets the extended buffer time requirement, the playback is resumed through the state machine control to avoid screen tearing or audio popping caused by sudden start.

[0050] In some embodiments, the uniform time reference is established between the slave device and the master device in step 300, and the time synchronization is achieved by the following manner: the slave device receives the time synchronization signal sent by the master device, sends a response signal to the master device, so that the master device determines the clock offset according to the response signal, and broadcasts the clock offset to the slave device; the slave device adjusts the local time according to the clock offset broadcasted by the master device. The slave device generates and returns the time synchronization signal to the master device by the following manner. The slave device receives and analyzes the time synchronization signal, obtains the first timestamp carried by the time synchronization signal, and determines the second timestamp of receiving the time synchronization signal, wherein the first timestamp is used to indicate the time of the master device broadcasting the time synchronization signal, and the second timestamp is used to indicate the time of receiving the time synchronization signal; a response signal carrying the first timestamp, the second timestamp, and the third timestamp is generated, wherein the third timestamp is used to indicate the time of sending the response signal; the response signal is sent to the master device, so that the master device determines the clock offset based on the response signal.

[0051] Please refer to Figure 6 , Figure 6 is the timing diagram of the time synchronization algorithm involved in some embodiments of the present application. In one example, the present application adopts a uniform time reference establishment mechanism (NTP, PTP or other equivalent protocols) to establish a uniform time reference between the master device and the slave device. First, the slave device receives the time synchronization signal sent by the master device, which carries the sending time of the master device, wherein the master device initiates (the period can be 0.5 seconds to 2 seconds, which is shortened when the network is unstable) the sending of the time synchronization signal at an adaptive period, rather than sending it once, to ensure that the time reference is stable for a long time, and at this time the master device records the sending time . Then, when the slave device receives the time synchronization signal, it records the local receiving time , at which time the local time of the slave device has not been calibrated and there is a deviation value; the time Afterwards, a response signal is generated, which must contain three key time stamps, the sending time stamp of the master device , the time stamp of the slave device receiving the time synchronization signal , and the sending time of the slave device sending the response signal , which reflects the local time of the slave device when the response signal leaves the slave device . After the slave device sends the response signal, the master device records a local receiving time stamp when it receives the response signal ; then the master device calculates the clock offset ; in addition, the master device also calculates the network round-trip time (RTT) to exclude the interference of network delay on time calibration . After the slave device receives the clock offset (ClockOffset) broadcast by the master device, it adjusts the local time by subtracting the offset from the local time, so that the local time of the slave device is completely aligned with the time reference of the master device.

[0052] In addition, when the master device does not receive the response of the slave device within the preset timeout time, or the packet loss rate exceeds 10%, the synchronization request retransmission is triggered, wherein the preferred value range of the preset timeout time is 400 milliseconds to 600 milliseconds, so as to avoid calibration failure caused by network interruption or packet loss and ensure the success of signal interaction. The master device initiates calibration at a fixed period (preferably a basic period of 0.5 seconds to 2 seconds), and when it is detected that the clock offset exceeds the threshold value in the range of 3 milliseconds to 8 milliseconds, the calibration period is shortened to 400 milliseconds to 600 milliseconds to ensure the stability of the time reference.

[0053] In this way, the time calibration is realized by the master device sending the synchronization signal, the slave device returning the three time stamps, the master device calculating the clock offset, and the slave device adjusting the local time. At the same time, the retransmission mechanism and the periodic calibration mechanism are combined to establish a unified time reference for the master and slave devices, effectively ensuring the accuracy of subsequent audio and video timestamp analysis and playback deviation value calculation, and also being the prerequisite for realizing multi-device audio and video synchronization.

[0054] In some embodiments, the slave device also records an interaction time stamp and generates interaction event information carrying the interaction time stamp in response to the user's picture interaction operation on the audio and video file, wherein the interaction event information includes the interaction operation type, the operation parameter and the interaction time stamp; the interaction event information is sent to the master device for broadcasting to other slave devices; wherein the interaction time stamp is used to determine the transmission delay, and the interaction operation type and the operation parameter are used to adjust the picture content of the audio and video file.

[0055] Please refer to Figure 7 , Figure 7is the flow chart of the interactive event synchronization process involved in some embodiments of the present application. In one example, user A (the trigger end) triggers an interactive operation on a slave device (the trigger end), including but not limited to gesture instructions (such as waving hands, clicking), position feedback (such as device movement), and function operation instructions (such as switching scenes); the slave device (the trigger end) captures the interaction in real time, records the interaction timestamp (based on a unified time reference, ensuring consistency with other devices in the time dimension); then the slave device (the trigger end) generates interactive event information carrying the interaction timestamp, which includes the interaction operation type, the operation parameter, and the interaction timestamp, wherein the interaction operation type is used to specify the operation category performed by the user (such as gesture clicking, fast movement, or button triggering), the operation parameter is the specific data of the picture adjustment (such as click coordinates, movement speed, and scene switching instruction code, etc.), and the interaction timestamp is used to indicate the global time when the interaction occurs.

[0056] After the generation of the interactive event information, the slave device (the trigger end) uploads the complete interactive event information to the master device, and the master device verifies the device legality and the parameter integrity within 8-12 milliseconds. After the verification, the master device, as the interactive event synchronization center, broadcasts the event information to all other slave devices (the receiving end) in real time. After receiving the interactive event broadcast by the master device, the other slave devices (the receiving end) first obtain their current device local time , and then calculate the transmission delay according to the device local time and the interaction timestamp , because and are based on a unified time reference, the calculated delay is the real time consumption, and there is no error caused by the inconsistency of the time reference.

[0057] The slave device (receiving end) performs time compensation according to the interaction type, and presents the compensated picture to user B (interaction receiving end), as follows: if the interaction type is low-speed interaction (such as button click and simple gesture), linear interpolation compensation is adopted, the interaction response timing is corrected based on transmission delay, and the picture adjustment is synchronized with the actual operation time of the trigger end; if the interaction type is high-speed interaction (such as fast movement and dynamic gesture), nonlinear prediction compensation based on historical trajectory is adopted, the actual position of the object after delay is predicted by analyzing the historical trajectory of the operation (such as device moving direction and speed), and the picture display coordinates are corrected (such as trigger end device fast movement, receiving end avoids picture display lagging behind actual operation through prediction compensation); if the current video picture is a high-precision requirement scene, Kalman filter compensation is adopted to further improve the prediction accuracy and adapt to high-precision interaction scenes such as industrial XR. In this way, the picture adjustment results of all receiving ends are consistent with the original interaction effect of the trigger end, users have no perceptual delay difference, and the immersive experience of multi-device collaboration is ensured.

[0058] The picture synchronization mechanism of the present application realizes interaction consistency, can solve the problem of different interaction time synchronization between multiple devices and transmission delay caused picture deviation, and ensure that all users see the same interaction result; secondly, the real-time performance is ensured, the time consumption of the main device verification is controlled in the range of 8 to 12 milliseconds, and the process of broadcast distribution does not have additional delay, combined with the application of hierarchical compensation algorithm, the real-time performance of interaction response can be ensured; thirdly, it has good adaptability, different interaction types are distinguished into low-speed interaction and high-speed interaction and corresponding compensation strategies are adopted, the compensation accuracy is ensured while the calculation efficiency is considered, and it can adapt to different scene requirements such as entertainment and industrial collaboration; finally, the mechanism can improve the security, relying on the legality verification link set by the main device, the interaction initiated by illegal devices can effectively avoid interfering with global synchronization, and the security of the system is improved.

[0059] According to one or more of the above embodiments, an audio-video synchronous playback method is implemented. The method establishes a unified high-precision time reference and designs an adaptive synchronization control strategy, thereby realizing the synchronous playback of multiple XR devices on audio-video content. Specifically, the prior art generally relies on local timestamps, lacks a globally unified time reference, and results in low synchronization accuracy of multiple devices when affected by network delays and clock drift. The present application sends a time synchronization request from the master device to the slave device, and returns a response signal from the slave device. The master device calculates the clock offset of each slave device according to the response signal, and then each slave device adjusts its local time according to the offset, so that all devices ultimately share the same time measurement dimension. On this basis, the master device broadcasts the current global playback position at fixed time intervals, each slave device calculates the deviation value based on the global playback position and its current local playback position, and adjusts the playback progress of the audio-video file according to the deviation value and the preset synchronization threshold interval, to achieve consistency in playback rhythm. It is worth mentioning that the synchronization threshold interval used by each slave device is determined individually according to its own network performance and device performance, so different devices have different synchronization threshold intervals, thereby being able to match the optimal playback adjustment strategy for each device.

[0060] Figure 8 is an exemplary flowchart of an audio-video synchronous playback method applied to a master device according to some embodiments of the present application. In some embodiments, Figure 8 The flowchart shown can be specifically executed by modules in the slave device. As Figure 8 shown, in some embodiments, the flowchart can include the following steps.

[0061] Step 810, the master device broadcasts a time synchronization signal to at least one slave device, and receives a response signal sent by the at least one slave device.

[0062] Step 820, determine the clock offset based on the response signal, and broadcast the clock offset to the slave device corresponding to the response signal, for the at least one slave device to establish a unified time reference based on the clock offset.

[0063] Step 830, generate an audio-video file carrying a timestamp, and send the audio-video file to at least one slave device.

[0064] Step 840, broadcast the global playback position at a preset time interval, for the at least one slave device to calculate the deviation value of the local playback position and the global playback position, and determine the adjustment strategy of the playback progress of the audio-video file according to the deviation value and the synchronization threshold interval.

[0065] Wherein, the global playback position includes a timestamp and a version identifier of the audio-video file, and the synchronization threshold interval is determined based on the device performance and network performance of the slave device.

[0066] The steps 810, 820 and 840 have been described in detail above, and will not be repeated here.

[0067] In step 830, the master device generates the audio and video file carrying the time stamp by adding a time stamp to the original audio and video file, determining the file block size according to the network bandwidth and the cache capacity of each slave device, splitting the original audio and video file into multiple sub-files according to the file block size, and encoding and compressing the multiple sub-files according to a preset encoding and compression algorithm to obtain the audio and video file carrying the time stamp.

[0068] Please refer to Figure 9 , Figure 9is a flowchart of audio and video content preprocessing involved in some embodiments of the present application. In one example, the master device first determines the original audio and video content, and checks the legality of the audio and video content format, and then marks each audio frame and video frame based on the established unified time reference to add accurate timestamps; after adding, check whether the timestamp deviation value of adjacent frames meets the frame rate requirement (such as the adjacent frame deviation value of 30 frames / second video is about 33 milliseconds), to avoid timestamp disorder. In this way, after all the slave devices get the file, they can locate the global unified playback time (such as a frame timestamp of 1000 milliseconds, all slave devices need to play at the 1000 millisecond time point of the unified time reference), to provide a synchronization reference point for subsequent position alignment. Then, the master device adapts the network bandwidth and the file caching capability of each slave device, and takes the average bandwidth of the slave device within 20 to 40 seconds as the core reference for segmentation (to avoid high bandwidth occupation leading to transmission lag); among them, the master device cuts the audio and video file into 3 to 12 second small pieces according to the bandwidth of the slave device: when the network bandwidth of the slave device is less than or equal to 2 Mbps (low bandwidth), the file block size is preferably 3 to 5 seconds (to reduce the amount of single block data and reduce transmission pressure); when the network bandwidth of the slave device is greater than or equal to 8 Mbps (high bandwidth): the file block size is preferably 8 to 12 seconds (to increase the amount of single block data and reduce the number of blocks, to improve the smoothness of playback); at the same time, the block size also needs to match the cache capacity of the slave device, for the slave device with small cache, the block size is also reduced synchronously to avoid cache overflow leading to playback interruption. In this way, different block sizes are set for different slave devices according to their hardware and network differences, to avoid transmission lag and insufficient cache of some devices, and to ensure that all slave devices can stably receive the file. The master device splits the original audio and video file into multiple sub-files according to the file block size, and compresses and encodes the multiple sub-files, for example, the video file is encoded in H.265 and equivalent high-efficiency encoding format, and the audio file is encoded in Advanced Audio Coding (AAC) and equivalent adaptive format. After encoding, the master device also needs to check the encoding integrity, and check the timestamp consistency again, and the qualified sub-files are combined into the final audio and video file with timestamps; the unqualified ones are returned for reprocessing. Through encoding and compression, the file size is minimized under the premise of ensuring the quality of the picture and the sound, the network transmission efficiency is improved, and through the final check, it is ensured that the file can be synchronized and has no abnormality, to avoid playback or synchronization failure caused by encoding errors.

[0069] In the aforementioned method for synchronized audio and video playback applied to the master device, a unified high-precision time base is established, and an adaptive synchronization control strategy is designed to achieve synchronized playback of audio and video content across multiple XR devices. Specifically, addressing the problem that existing technologies typically rely on local timestamps and lack a globally unified time reference, resulting in low synchronization accuracy when multiple devices are affected by network latency and clock drift, this application employs a method where the master device sends a time synchronization request to the slave devices, and the slave devices return response signals. The master device calculates the clock offset of each slave device based on the response signal, and then each slave device adjusts its local time according to this offset, so that all devices ultimately share the same time measurement dimension. Based on this, the master device broadcasts the current global playback position at fixed time intervals. Each slave device then calculates the deviation value by combining the global playback position with its own current local playback position. Based on the deviation value and the preset synchronization threshold range, the playback progress of the audio and video files is dynamically adjusted to achieve a consistent playback rhythm. It is worth mentioning that the synchronization threshold range used by each slave device is determined individually based on its own network performance and device performance. Therefore, different devices have different synchronization threshold ranges, which enables the matching of the optimal playback adjustment strategy for each device.

[0070] The steps of the various methods described above are only for clarity. In implementation, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0071] Another embodiment of this application relates to an XR device, such as... Figure 10 As shown, it includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described audio-visual synchronization playback method.

[0072] The memory and the processor are connected via a bus. The bus can include any number of interconnecting buses and bridges depending on the specific application of the mobile terminal. The bus connects the various circuits of the memory and the processor together and mediates data communication among the various circuits. Since the bus is known, it will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single device or a plurality of devices, such as a plurality of receivers and transmitters, which are configured to transmit and receive data over a transmission medium. The data processed by the processor is transmitted over the wireless medium via the antenna. Further, the antenna also receives data and transmits the data to the processor.

[0073] The processor is responsible for managing the bus and general processing, which can include the execution of software stored in the memory. The processor can be implemented with one or more general purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein.

[0074] As those skilled in the art will readily appreciate, the above described various embodiments can be modified in various ways without departing from the spirit or scope of the present application.

Claims

1. A method for synchronized audio-visual playback, characterized in that, Applied to a slave device, the method includes: Establish a unified time reference with the main equipment; Receive and play the timestamped audio and video files broadcast by the main device; Receive the global playback position broadcast by the master device, and calculate the deviation between the local playback position and the global playback position; Based on the deviation value and the synchronization threshold range, an adjustment strategy for the playback progress of the audio and video files is determined, wherein the local playback position is determined based on the timestamp, and the synchronization threshold range is determined based on the device performance and network performance of the slave device; wherein... Based on the deviation value and the synchronization threshold range, a strategy for adjusting the playback progress of the audio and video files is determined, including: If the deviation value is less than the minimum value of the synchronization threshold range, the playback rate is set according to the deviation value, the audio and video file is played at the playback rate, and the audio and video file is smoothed. The playback rate is determined based on the maximum video playback rate, the minimum value of the synchronization threshold range, and the deviation value. If the deviation value is within the synchronization threshold range, the playback progress is adjusted to the global playback position, and the file cache space is expanded; If the deviation value is greater than the maximum value of the synchronization threshold range, playback is paused, the playback progress is adjusted to the global playback position, and the file is re-cached. When the cached playback progress is greater than the global playback position, playback resumes.

2. The audio-visual synchronized playback method according to claim 1, characterized in that, The synchronization threshold range is determined by the following methods: The minimum value of the synchronization threshold interval is determined based on a preset deviation benchmark value and a network jitter coefficient, wherein the network jitter coefficient is determined based on the standard deviation of network jitter within a preset time period. The maximum value of the synchronization threshold range is determined based on the preset cache security threshold, the maximum network round-trip latency, and the device rendering load coefficient.

3. The audio-visual synchronized playback method according to claim 1, characterized in that, The establishment of a unified time base with the master device includes: The device receives a time synchronization signal sent by the master device, sends a response signal to the master device, and the master device determines the clock offset based on the response signal and broadcasts the clock offset to the slave device. Adjust the local time according to the clock offset broadcast by the master device.

4. The audio-visual synchronized playback method according to claim 3, characterized in that, The step of receiving the time synchronization signal sent by the master device and sending a response signal to the master device includes: The system receives and parses the time synchronization signal, obtains a first timestamp carried by the time synchronization signal, and determines a second timestamp of the received time synchronization signal, wherein the first timestamp is used to indicate the time when the master device sends the time synchronization signal; Generate a response signal carrying the first timestamp, the second timestamp, and the third timestamp, wherein the third timestamp is used to indicate the time when the slave device sends the response signal; The response signal is sent to the master device.

5. The audio-visual synchronized playback method according to claim 1, characterized in that, The method further includes: In response to the user's on-screen interaction with the audio and video file, the interaction timestamp is recorded and interaction event information carrying the interaction timestamp is generated, wherein the interaction event information includes the interaction operation type, operation parameters and interaction timestamp; The interactive event information is sent to the master device, which then broadcasts it to other slave devices. The interaction timestamp is used to determine the transmission delay, and the interaction operation type and operation parameters are used to adjust the picture content of the audio and video file.

6. A method for synchronized audio and video playback, characterized in that, Applied to a master device, the method includes: Broadcast a time synchronization signal to at least one slave device and receive a response signal from the at least one slave device; The clock offset is determined based on the response signal, and the clock offset is broadcast to the slave device corresponding to the response signal, so that the at least one slave device can establish a unified time reference based on the clock offset; Generate an audio / video file carrying a timestamp, and send the audio / video file to the at least one slave device; The global playback position is broadcast at preset time intervals, allowing at least one slave device to calculate the deviation between its local playback position and the global playback position. Based on the deviation and a synchronization threshold range, the device determines an adjustment strategy for the playback progress of the audio / video file. The global playback position includes a timestamp and the version identifier of the audio / video file, and the synchronization threshold range is determined based on the device performance and network performance of the slave device.

7. The audio-visual synchronized playback method according to claim 6, characterized in that, Determining the clock offset based on the response signal includes: A fourth timestamp is determined when the response signal is received, and the response signal is parsed to obtain a first timestamp, a second timestamp, and a third timestamp, wherein the first timestamp is used to indicate the time when the master device sends the time synchronization signal, the second timestamp is used to indicate the time when the slave device receives the time synchronization signal, and the third timestamp is used to indicate the time when the slave device sends the response signal; The clock offset is calculated based on the first timestamp, the second timestamp, the third timestamp, and the fourth timestamp.

8. The audio-visual synchronized playback method according to claim 6, characterized in that, The generation of audio and video files carrying timestamps includes: Add timestamps to the original audio and video files; For each slave device, the file block size is determined based on the slave device's network bandwidth and caching capacity. Based on the file block size, the original audio and video file is split into multiple sub-files; The multiple sub-files are encoded and compressed according to a preset encoding and compression algorithm to obtain the audio and video file carrying a timestamp.

9. An XR device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the audio-visual synchronization playback method as described in any one of claims 1 to 5, or the audio-visual synchronization playback method as described in any one of claims 6 to 8.

Images