A distributed audio and video synchronization method based on multi-field coupling

Abstract

This application discloses a distributed audio and video synchronization method based on multi-field coupling, relating to the field of distributed audio and video synchronization technology. It obtains the time progress field, rhythm field, and semantic field through multi-source node state data broadcast by neighboring nodes, and obtains the time progress error, rhythm error, and semantic error based on these fields. The optimal playback speed is determined based on the time progress error, rhythm error, and semantic error, and the synchronization mode is determined based on the semantic error. Finally, an audio and video synchronization control strategy is generated based on the optimal playback speed and synchronization mode. This strategy is executed, and the process enters the next synchronization cycle until the convergence condition is met, completing synchronization. Through multi-field coupling, synchronization from time alignment to perceptual consistency is achieved, balancing playback continuity, rhythm consistency, and semantic consistency, thus improving the audio and video synchronization effect and user experience in a distributed environment.

Description

Technical Field

[0001] This application relates to the field of distributed audio and video synchronization technology, specifically to a distributed audio and video synchronization method based on multi-field coupling. Background Technology

[0002] With the development of smart terminal devices, application scenarios for multi-device collaborative audio and video playback are increasing, such as multi-screen home viewing, in-vehicle entertainment systems, simultaneous live streaming across multiple terminals, and immersive XR experiences. In these scenarios, it is necessary to ensure that audio and video playback remains synchronized across multiple devices to achieve a consistent user experience.

[0003] Currently, multi-terminal audio and video synchronization technologies mainly include the following categories: 1. Time-synchronization-based schemes, which achieve synchronization by establishing a unified time reference. 2. Master-slave control-based schemes, which set a master node as a reference node. 3. Buffering and latency compensation-based schemes, which achieve synchronization by adjusting the buffer area. Schemes based on audio or media feature alignment, some of which utilize audio features for synchronization.

[0004] Although the above methods can achieve audio and video synchronization to a certain extent, the following problems still exist: 1. The synchronization goal is limited to time consistency; 2. There is a lack of playback continuity control mechanism; 3. The influence of audio rhythm structure is not considered; 4. There is a lack of semantic synchronization capability; 5. There is a strong dependence on the global clock or central node; 6. It is difficult to adapt to heterogeneous device environments. Summary of the Invention

[0005] The purpose of this application is to provide a distributed audio and video synchronization method based on multi-field coupling, which solves the problems of existing technologies, such as synchronization objectives being limited to time consistency, lack of playback continuity control mechanisms, failure to consider the influence of audio rhythm structure, lack of semantic-level synchronization capabilities, strong dependence on global clocks or central nodes, and / or difficulty in adapting to heterogeneous device environments. This application can shift from time alignment to perceptual consistency in a distributed environment, while simultaneously taking into account playback continuity, rhythm consistency, and semantic consistency, thereby improving the overall synchronization effect and user experience.

[0006] This application is achieved through the following technical solution:

[0007] A distributed audio-video synchronization method based on multi-field coupling includes:

[0008] For any distributed audio and video playback device, the status data of the multi-source nodes corresponding to the distributed audio and video playback device is collected, and the multi-source node status data of the distributed audio and video playback device is broadcast to its neighboring nodes; the neighboring node refers to a one-hop neighboring device of the distributed audio and video playback device.

[0009] After the distributed audio and video playback device receives the multi-source node status data broadcast by the neighboring node, it obtains the time progress field, rhythm field and semantic field according to the multi-source node status data, and obtains the time progress error, rhythm error and semantic error according to the time progress field, rhythm field and semantic field respectively.

[0010] The optimal playback speed is determined based on the time progress error, rhythm error, and semantic error, and the synchronization mode is determined based on the semantic error. The audio and video synchronization control strategy within the current synchronization cycle is obtained based on the optimal playback speed and the synchronization mode.

[0011] The distributed audio and video playback device executes the audio and video synchronization control strategy and enters the next synchronization cycle for distributed audio and video synchronization until the time progress error, rhythm error and semantic error meet the convergence termination condition, thus completing the distributed audio and video synchronization based on multi-field coupling.

[0012] In one possible implementation, collecting the status data of the multi-source nodes corresponding to the distributed audio and video playback device includes:

[0013] Collect the playback progress, audio signal, and video frames corresponding to the distributed audio and video playback device;

[0014] The beat frequency, rhythm phase, and audio energy are extracted from the audio signal to obtain audio time structure data;

[0015] Scene tags and action events are extracted from the video frames to obtain video content state data;

[0016] The playback progress, audio time structure data, and video content status data are collectively used as multi-source node status data.

[0017] In one possible implementation, the time progress field, rhythm field, and semantic field are obtained respectively based on the multi-source node state data, including:

[0018] For any distributed audio and video playback device, determine the smoothed delay estimate of the arrival time of neighboring nodes to the distributed audio and video playback device, and obtain the influence weight of neighboring nodes on the distributed audio and video playback device based on the smoothed delay estimate;

[0019] The time progress field corresponding to the distributed audio and video playback device is obtained by weighted summation based on the playback progress and corresponding influence weight in the multi-source node status data corresponding to the neighbor nodes of the distributed audio and video playback device.

[0020] The rhythm field corresponding to the distributed audio and video playback device is obtained by weighted summation of the audio time structure data and the corresponding influence weights in the multi-source node state data corresponding to the neighbor nodes of the distributed audio and video playback device; the rhythm field includes the neighborhood beat frequency reference, the neighborhood rhythm phase reference, and the neighborhood audio energy reference.

[0021] Based on the video content status and corresponding influence weights in the multi-source node status data corresponding to the neighbor nodes of the distributed audio and video playback device, the semantics of the neighbor nodes are weighted and summed using a maximum value calculation function to obtain a semantic field; the semantic field includes neighborhood scene label references and neighborhood action event references.

[0022] In one possible implementation, the time progress error, rhythm error, and semantic error are obtained based on the time progress field, rhythm field, and semantic field, respectively, including:

[0023] For any distributed audio and video playback device, determine the difference between the corresponding time progress field and its playback progress to obtain the time progress error;

[0024] Based on the neighborhood beat frequency reference, neighborhood rhythm phase reference, and neighborhood audio energy reference in the rhythm error, the frequency error, phase error, and energy error are obtained respectively. The frequency error, phase error, and energy error are then weighted, summed, and the square root is taken to obtain the rhythm error.

[0025] Conditional assignments are performed based on the neighborhood scene label references and neighborhood action event references in the semantic field. The values ​​obtained from the two conditional assignments are then weighted and summed to obtain the semantic error.

[0026] In one possible implementation, determining the optimal playback speed based on the timing error, rhythm error, and semantic error includes:

[0027] The predicted time error is obtained based on the time progress error, the predicted rhythm error is obtained based on the rhythm error, and the predicted semantic error is obtained based on the semantic error.

[0028] Based on the predicted time error, predicted rhythm error, and predicted semantic error, an objective function with playback speed multiplier as the variable is constructed.

[0029] With the objective function as the goal, the playback speed multiplier is solved to obtain the optimal playback speed.

[0030] In one possible implementation, the objective function is:

[0031] ;

[0032] in, Let be the objective function corresponding to the i-th distributed audio / video playback device. As the weight for time schedule error, For rhythm error weights, For semantic error weights, Weights are limited to account for the degree of deviation. Weights are limited to account for the degree of change. Let be the prediction time error corresponding to the i-th distributed audio / video playback device. Let be the prediction rhythm error corresponding to the i-th distributed audio / video playback device. Let be the predicted semantic error corresponding to the i-th distributed audio / video playback device. For the current moment, To control the cycle, For a moment Playback speed multiplier For a moment The playback speed multiplier.

[0033] In one possible implementation, determining the synchronization mode based on the semantic error includes:

[0034] If the semantic error is less than the first preset semantic threshold, then the synchronization mode is determined to be the normal synchronization mode.

[0035] If the semantic error is greater than a first preset semantic threshold but less than a second preset semantic threshold, then the synchronization mode is determined to be an accelerated synchronization mode; if the second preset semantic threshold is greater than the first preset semantic threshold, the accelerated synchronization mode increases the weight of the time progress error and / or the weight of the semantic error compared to the normal synchronization mode.

[0036] If the semantic error is greater than the second preset semantic threshold, the synchronization mode is determined to be a forced synchronization mode; the forced synchronization mode includes at least one of keyframe alignment, brief rebuffering, pause and wait, or jump to the nearest semantically consistent keyframe.

[0037] In one possible implementation, the method further includes:

[0038] When the synchronization mode is the forced synchronization mode, the synchronization execution action corresponding to the forced synchronization mode and the optimal playback speed are used together as the audio and video synchronization control strategy in the current synchronization cycle.

[0039] In one possible implementation, the audio and video synchronization control strategy is executed through the distributed audio and video playback device, including:

[0040] When the synchronization mode is normal synchronization mode or accelerated synchronization mode, the distributed audio and video playback device adjusts its own actual playback speed multiplier by using the optimal playback speed to obtain the adjusted actual playback speed multiplier, and adjusts the playback progress according to the adjusted actual playback speed multiplier.

[0041] When the synchronization mode is forced synchronization mode, the synchronization execution action corresponding to the forced synchronization mode is executed.

[0042] In one possible implementation, the distributed audio / video playback device adjusts its own actual playback speed multiplier using the optimal playback speed, including:

[0043] Based on a preset playback speed smoothing coefficient, the distributed audio and video playback device performs a weighted sum of its actual playback speed multiplier and the optimal playback speed to obtain the adjusted actual playback speed multiplier.

[0044] Compared with the prior art, this application has the following advantages and beneficial effects:

[0045] This application discloses a distributed audio and video synchronization method based on multi-field coupling. It obtains the time progress field, rhythm field, and semantic field through multi-source node state data broadcast by neighboring nodes, and then obtains the time progress error, rhythm error, and semantic error based on these fields. The optimal playback speed is determined based on the time progress error, rhythm error, and semantic error, and the synchronization mode is determined based on the semantic error. Finally, an audio and video synchronization control strategy is generated based on the optimal playback speed and synchronization mode. This strategy is executed, and the process enters the next synchronization cycle until the convergence condition is met, completing synchronization. Through multi-field coupling, synchronization from time alignment to perceptual consistency is achieved, balancing playback continuity, rhythm consistency, and semantic consistency, thus improving the audio and video synchronization effect and user experience in a distributed environment. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the exemplary embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0047] Figure 1 A flowchart illustrating a distributed audio-video synchronization method based on multi-field coupling, provided as an embodiment of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this application are only for explaining this application and are not intended to limit this application.

[0049] like Figure 1 As shown, this application provides a distributed audio and video synchronization method based on multi-field coupling, including:

[0050] S101. For any distributed audio and video playback device, collect the multi-source node status data corresponding to the distributed audio and video playback device, and broadcast its own multi-source node status data to its neighboring nodes through the distributed audio and video playback device; the neighboring node refers to a one-hop neighboring device of the distributed audio and video playback device.

[0051] Neighboring nodes are typically located in the same physical network as the distributed audio and video playback device (such as the same WiFi environment or the same wired network under the same switch), and the status data packets sent by the distributed audio and video playback device can directly reach the neighboring nodes without needing to be forwarded through a central server or other intermediate nodes.

[0052] S102. After the distributed audio and video playback device receives the multi-source node status data broadcast by the neighboring node, it obtains the time progress field, rhythm field and semantic field according to the multi-source node status data, and obtains the time progress error, rhythm error and semantic error according to the time progress field, rhythm field and semantic field respectively.

[0053] S103. Determine the optimal playback speed based on the time progress error, rhythm error and semantic error, and determine the synchronization mode based on the semantic error, and obtain the audio and video synchronization control strategy within the current synchronization cycle based on the optimal playback speed and the synchronization mode.

[0054] S104. The distributed audio and video playback device executes the audio and video synchronization control strategy and enters the next synchronization cycle to perform distributed audio and video synchronization until the time progress error, rhythm error and semantic error meet the convergence termination condition, thus completing the distributed audio and video synchronization based on multi-field coupling.

[0055] This application establishes a closed-loop iterative process of state acquisition, environmental field construction, error calculation, control decision-making, control execution, convergence judgment, and state feedback, enabling each node to continuously maintain a consistent, rhythmic, and semantically consistent perceptual synchronization state in a dynamic distributed environment.

[0056] In one possible implementation, collecting the status data of the multi-source nodes corresponding to the distributed audio and video playback device includes:

[0057] Collect the playback progress, audio signal, and video frames corresponding to the distributed audio and video playback device;

[0058] The beat frequency, rhythm phase, and audio energy are extracted from the audio signal to obtain audio time structure data;

[0059] Scene tags and action events are extracted from the video frames to obtain video content state data;

[0060] The playback progress, audio time structure data, and video content status data are collectively used as multi-source node status data.

[0061] For example, for the i-th distributed audio / video playback device, the playback progress at time t can be obtained. Audio signals and video frames Beat frequencies can be extracted from audio signals. Rhythm and Phase and audio energy Obtain audio time structure data Scene tags can be extracted from video frames. and action events Obtain video content status data .

[0062] More specifically, it can be seen from audio signals Short-time energy is extracted from the decoded PCM audio data as follows:

[0063]

[0064] in, is the short-term energy, used to characterize the intensity of the current short segment of sound; n is the time window position (i.e., the current moment) at which the energy is being calculated, or the frame number. For example: with a sampling rate of 48kHz, n = the 48000th sample point. m is the offset index within the window, used to characterize the amount of forward shift within the current time window; This represents the sum of the squares of the absolute values ​​of the audio amplitudes taken from the current moment backwards through M sampling points; Let be the audio amplitude at time nm; short-time energy is the cumulative square of the audio amplitude within this window, used to describe the sound intensity at that moment. Therefore, the audio energy can be obtained through the above method of obtaining short-time energy. .

[0065] The beat period is obtained by performing a short-time Fourier transform on the audio signal to obtain its time-frequency representation, constructing a beat detection function based on the spectral amplitude changes between adjacent frames, and then performing autocorrelation analysis on the beat detection function. Let the position of the maximum peak of the autocorrelation within the candidate range be... Then the beat period of the i-th distributed audio and video playback device is:

[0066] First, perform a short-time Fourier transform on the audio signal to obtain its time-frequency representation:

[0067] ;

[0068] in, For the i-th distributed audio / video playback device, the audio discrete sampling signal is used. For analysis window functions; This is the current time position of the analysis window (i.e., the frame index). The current frame rate variable; For the i-th distributed audio and video playback device at time... The spectrum on the spectrum, where J represents the imaginary unit.

[0069] Based on time-frequency representation, the spectral flow or energy change of each frame is calculated, and a beat detection function is constructed, which is represented by the spectral amplitude difference between adjacent frames:

[0070]

[0071] in, For beat detection function, For the current frame at frequency Spectral amplitude at that location, This represents the spectral amplitude at frequency ω in the previous analysis frame.

[0072] The amplitude difference between adjacent frames reflects the intensity of abrupt changes in audio over time; drum beats and downbeats typically cause this. A peak was observed.

[0073] For the beat detection function Perform autocorrelation calculation:

[0074]

[0075] in, This represents the autocorrelation value when the hysteresis is k; k is the frame-level delay. For a moment The value of the beat detection function on the audio. The peak value of the autocorrelation function corresponds to the repeated beat intervals in the audio.

[0076] The corresponding beat period is:

[0077]

[0078]

[0079] in, This represents the position of the maximum peak value of the autocorrelation within the candidate range; The time interval between adjacent analysis frames; Let be the beat cycle of the i-th distributed audio / video playback device. This is the minimum value of the frame-level latency. This represents the maximum value of the frame-level latency. The search range for the beat cycle corresponds to a preset beat speed range, such as 40 BPM to 240 BPM.

[0080] Based on the beat period, the beat frequency can be determined as follows:

[0081]

[0082] in: Let be the beat frequency of the i-th distributed audio and video playback device, in BPM; The time interval is measured in seconds.

[0083] Based on the beat period, the beat phase can be determined as follows:

[0084]

[0085] in, This represents the time corresponding to the most recent detection of a beat peak (or replay moment) by the i-th distributed audio / video playback device.

[0086] The scene category of the current video can be obtained through the VIT model (Vision Transformer, a computer vision model based on the Transformer architecture), thus obtaining the scene label. It's possible to determine whether a key action event has occurred at the current time using a time-based detection model pre-trained with the VIT model or other existing object detection models. Examples include: firing a gun / exploding / running / hitting / door opening. However, it's worth noting that pre-defined scene labels and key action events can also achieve distributed audio-visual synchronization.

[0087] In one possible implementation, the time progress field, rhythm field, and semantic field are obtained respectively based on the multi-source node state data, including:

[0088] S102.1 For any distributed audio and video playback device, determine the smoothed delay estimate of the arrival time of neighboring nodes to the distributed audio and video playback device, and obtain the influence weight of neighboring nodes on the distributed audio and video playback device based on the smoothed delay estimate.

[0089] For example, the spatial weight function is dynamically determined based on the node communication delay, which is calculated from the data packet sending and receiving times. To avoid weight fluctuations caused by network jitter, the communication delay is time-smoothed, and an exponentially weighted average method is used to calculate the delay estimate.

[0090] We can first initialize the smoothing time constant as follows:

[0091] ;

[0092] in, The initial smoothing time constant characterizes how quickly the system expects to adapt to changing timescales. In subsequent updates to the smoothing time constant, it can be determined based on inter-node communication delays; the greater the delay, the larger the time constant. This is a preset default value, which can be preset according to the system scenario, such as 0.5s.

[0093] The smoothing time constant is updated as follows:

[0094]

[0095]

[0096] in, Let be the smoothing time constant corresponding to the i-th distributed audio / video playback device at time t. Let i be the set of neighbor nodes of the i-th distributed audio / video playback device. Let be the total number of neighboring nodes of the i-th distributed audio / video playback device. This is a smoothed delay estimate between the i-th distributed audio / video playback device and its j-th neighbor node at time t-1. Let be the smoothing coefficient corresponding to the i-th distributed audio and video playback device at time t. The faster the data updates, the larger the smoothing coefficient. The smoothing coefficient can be dynamically adjusted according to the data update time interval. The higher the update frequency, the larger the smoothing coefficient. Let be the time interval between the current data and the previous data for the i-th distributed audio / video playback device. Let be the smoothing time constant corresponding to the i-th distributed audio and video playback device at time t-1.

[0097] To determine the influence of neighboring nodes on the current node, the link delay from the j-th neighboring node to the i-th distributed audio / video playback device is defined as:

[0098]

[0099] in, The timestamp at which the multi-source node state data sent by the j-th neighbor node is received by the i-th distributed audio / video playback device at time t. The timestamp for the j-th neighbor node sending multi-source node status data at time t.

[0100] To reduce the impact of network jitter, link delay is smoothed, and the smoothed delay estimate is obtained as follows:

[0101] ;

[0102] in, Let be the link latency from the j-th neighbor node to the i-th distributed audio / video playback device at time t. This is a smoothed latency estimate from the j-th neighbor node to the i-th distributed audio / video playback device at time t-1. This is a smooth delay estimate for the link.

[0103] Therefore, the original weights can be obtained based on the smoothing delay estimation:

[0104]

[0105] in, Let be the original weights from the j-th neighbor node to the i-th distributed audio / video playback device at time t. The offset used for calculating the stable weights is determined based on the minimum system delay to avoid numerical divergence and improve system robustness.

[0106] The offset can be set to:

[0107]

[0108] in, This is a preset minimum link delay constant (or minimum communication delay threshold) used in the system to introduce an offset in weight calculation to avoid instability caused by a denominator that is zero or too small. The offset serves to prevent division by zero and to prevent weight explosion. ; It is the smoothed link delay estimate between the i-th distributed audio / video playback device and its neighboring nodes (i.e., the smoothed link delay or the filtered delay estimate).

[0109] By normalizing the original weights, we can obtain the influence weights of neighboring nodes on the distributed audio and video playback device:

[0110]

[0111] in, Let be the influence weight of the j-th neighbor node to the i-th distributed audio / video playback device at time t.

[0112] S102.2. The playback progress and corresponding influence weight in the multi-source node status data corresponding to the neighbor nodes of the distributed audio and video playback device are weighted and summed to obtain the time progress field corresponding to the distributed audio and video playback device.

[0113] For example, the weighted summation based on the playback progress and corresponding influence weights in the multi-source node status data corresponding to the neighbor nodes of the distributed audio and video playback device is as follows:

[0114] ;

[0115] in, Let j be the playback progress of the j-th neighboring node. Let be the time progress field corresponding to the i-th distributed audio and video playback device; the influence weights are not only used for the weighted aggregation of neighborhood states, but also participate in the node synchronization control process. By influencing the degree of response of nodes to neighborhood differences, the convergence rate is adaptively adjusted, thereby forming a dynamic convergence mechanism based on weight modulation.

[0116] S102.3. The rhythm field corresponding to the distributed audio and video playback device is obtained by weighted summation based on the audio time structure data and corresponding influence weights in the multi-source node state data corresponding to the neighboring nodes of the distributed audio and video playback device; the rhythm field includes the neighborhood beat frequency reference, the neighborhood rhythm phase reference, and the neighborhood audio energy reference.

[0117] For example, a weighted sum is performed based on the audio time structure data and corresponding influence weights in the multi-source node state data corresponding to the neighbor nodes of the distributed audio and video playback device, including:

[0118]

[0119]

[0120]

[0121] in, This serves as the neighborhood beat frequency reference for the i-th distributed audio / video playback device. This serves as the neighborhood rhythm phase reference for the i-th distributed audio / video playback device. Let i be the neighborhood audio energy reference corresponding to the i-th distributed audio / video playback device. This refers to the beat frequency in the audio time structure data. This refers to the rhythmic phase in the audio time structure data. Audio energy in audio temporal structure data;

[0122] S102.4. Based on the video content status and corresponding influence weight in the multi-source node status data corresponding to the neighbor nodes of the distributed audio and video playback device, the semantics of the neighbor nodes are weighted and summed using a maximum value calculation function to obtain a semantic field; the semantic field includes neighborhood scene label reference and neighborhood action event reference.

[0123] For example, based on the video content status and corresponding influence weights in the multi-source node status data corresponding to the neighbor nodes of the distributed audio and video playback device, a maximum value calculation function is used to perform a weighted summation of the semantics of the neighbor nodes, including:

[0124]

[0125]

[0126] in, This serves as a reference for the neighborhood scene labels corresponding to the i-th distributed audio / video playback device. Let argmax be the neighborhood action event reference for the i-th distributed audio / video playback device, and let argmax represent the function for finding the maximum value. This represents the scene label corresponding to the j-th neighbor node. When the value is 'c', the value is assigned to 1; 'c' represents one of the tags. This represents the action event corresponding to the j-th neighbor node. for If , then the value is assigned to 1; It is one of all actions.

[0127] In one possible implementation, the time progress error, rhythm error, and semantic error are obtained based on the time progress field, rhythm field, and semantic field, respectively, including:

[0128] S102.5 For any distributed audio and video playback device, determine the difference between the corresponding time progress field and its playback progress to obtain the time progress error;

[0129] For example, the difference between the corresponding time progress field and its playback progress is determined, and the time progress error is obtained as follows:

[0130]

[0131] in, Let be the time progress error corresponding to the i-th distributed audio / video playback device. This represents the playback progress corresponding to the i-th distributed audio / video playback device.

[0132] S102.6. Based on the neighborhood beat frequency reference, neighborhood rhythm phase reference, and neighborhood audio energy reference in the rhythm error, obtain the frequency error, phase error, and energy error respectively, and then sum the frequency error, phase error, and energy error by weight and take the square root to obtain the rhythm error.

[0133] For example, based on the neighborhood beat frequency reference, neighborhood rhythm phase reference, and neighborhood audio energy reference in the rhythm error, the frequency error, phase error, and energy error are obtained respectively as follows:

[0134]

[0135]

[0136]

[0137] in, Let be the frequency error corresponding to the i-th distributed audio / video playback device. Let be the phase error corresponding to the i-th distributed audio / video playback device. Let i be the energy error corresponding to the i-th distributed audio / video playback device. Let be the beat frequency corresponding to the i-th distributed audio / video playback device. Let be the rhythm phase corresponding to the i-th distributed audio / video playback device. Let be the audio energy corresponding to the i-th distributed audio / video playback device. This is a phase wrap function used to fold the input phase difference within a preset period range, keeping the phase error within a continuously measurable interval. ; For the phase wrapping function, Pi For finding the remainder function.

[0138] The frequency error, phase error, and energy error are weighted, summed, and their square roots are taken to obtain the rhythm error:

[0139] ;

[0140]

[0141] in, Let be the rhythm error corresponding to the i-th distributed audio / video playback device. Weights are applied to the preset frequency errors. Weights are applied to the preset phase error. The preset energy error weighting is applied.

[0142] S102.7. Assign conditional values ​​based on the neighborhood scene label reference and the neighborhood action event reference in the semantic field, and sum the values ​​obtained from the two conditional assignments by weighting to obtain the semantic error.

[0143] ;

[0144] in, Let i be the semantic error corresponding to the i-th distributed audio / video playback device. This indicates that when the scene label corresponding to the i-th distributed audio and video playback device is different from the neighboring scene label reference, the value is assigned to 1; This indicates that when the action event corresponding to the i-th distributed audio / video playback device is different from the neighboring action event reference, the value is assigned to 1; The scene weight represents the degree to which scene inconsistency affects synchronization control; The event weight represents the degree of importance of inconsistency between action events to synchronization control; and the scene weight and event weight should generally satisfy the following: Semantic differences are represented in a weighted form to reflect the importance of different semantic factors.

[0145] Action events are usually more sensitive than scene changes; for example, "firing a gun" or "explosion" are more likely to be perceived as asynchronous by users than "switching between indoor and outdoor environments".

[0146] Action events can be further subdivided, for example, into sets of event categories. This can include ordinary actions, obvious actions (e.g., high-impact actions), and key events. The event weight can then be expressed as:

[0147] ;

[0148] in, This is a mapping function from event categories to weights. Indicates an action event The weighting coefficient is determined based on the action type: when When it is a normal action, =0.3; when When it is an obvious action, =0.7; when When it is a critical action event, =1.0.

[0149] Scene weights can be represented as:

[0150]

[0151] in, A function representing the mapping from scene labels to weights; Representing scene tags The weighting coefficients are determined based on the type of scenario change; when When it is a normal scene switch, =0.2; when When the scene changes rapidly, =0.5; when When it is a critical scene transition, =0.7.

[0152] For action events, they can be divided into ordinary actions, obvious actions, and key action events according to the event type, and each can be assigned a different weight; for scene semantics, corresponding weights can be assigned according to the degree of scene change or the importance of scene switching.

[0153] In one possible implementation, determining the optimal playback speed based on the timing error, rhythm error, and semantic error includes:

[0154] S103.1 Obtain the predicted time error based on the time progress error, obtain the predicted rhythm error based on the rhythm error, and obtain the predicted semantic error based on the semantic error.

[0155] For example, the i-th distributed audio and video playback device, in each synchronization cycle, adjusts the time progress based on the obtained time progress error. Rhythm error and semantic error A unified objective function is constructed and control decisions for the current synchronization cycle are generated.

[0156] Time error is represented in signed form: when When, it indicates that the i-th distributed audio / video playback device lags behind the neighborhood reference progress; when When, it indicates that the i-th distributed audio and video playback device is ahead of the neighborhood reference progress.

[0157] Let the playback speed multiplier of the i-th distributed audio and video playback device be 1 / 2. The default playback speed multiplier is 1. (During the control cycle...) Within, candidate playback speeds are used. The predicted playback progress is as follows:

[0158] ;

[0159] The neighborhood reference playback progress prediction is as follows:

[0160] ;

[0161] Therefore, the prediction time error is:

[0162] ;

[0163] in, To predict the playback progress, For neighborhood reference playback progress prediction, This refers to the prediction time error.

[0164] The prediction rhythm error can then be determined as:

[0165] ;

[0166] in, To predict rhythm error, For the i-th distributed audio and video playback device, the playback speed multiplier is... Under its influence, the predicted equivalent cycle time in the evaluation of the next control cycle is affected; The neighborhood reference beat period is predicted from the beat period of neighboring nodes or the beat frequency of the neighborhood for distributed audio and video playback devices and is used for rhythm synchronization comparison.

[0167] The neighborhood reference beat can be represented as:

[0168] ;

[0169] ;

[0170] in, For the j-th distributed audio and video playback device at time... The beat cycle, The original influence weight from the j-th neighbor node to the i-th device.

[0171] The cycle period can be expressed as:

[0172] ;

[0173] ;

[0174] in, For the i-th distributed audio and video playback device at time... The beat cycle, This represents the location of the maximum peak value of the autocorrelation within the candidate range. To control the cycle.

[0175] Methods for obtaining semantic prediction error may include:

[0176] The scene feature vector is obtained as follows:

[0177] ;

[0178] in, This represents the actual media data played by the i-th node at the current moment. Here is a scene feature extraction function, which is used to extract video frames or video segments currently being played from the i-th distributed audio and video playback device. In the process, feature vectors that can represent "what scene the current image belongs to" are extracted.

[0179] ;

[0180] ;

[0181] in, This is an HSV color histogram. For Gabor texture feature functions, Layout features: Sobel edge operator functions A pre-defined scene semantic library is used to store multiple candidate scene categories and their corresponding standard scene feature prototype vectors; A pre-defined action semantic library is used to store multiple candidate action event categories and their corresponding standard action feature prototype vectors; Pre-defined action semantic library The k-th candidate action event category in the data represents a discrete semantic category, such as walking, running, jumping, etc. Let k be the standard feature prototype vector of the k-th candidate action event category. The k-th standard offline scene tag in the preset scene semantic library, such as living room, street, highway, shopping mall, stadium, battle, disaster, etc. This is the prototype vector of the standard scene features corresponding to the k-th standard offline scene label.

[0182] For the scene label of the current i-th distributed audio / video playback device, find the most similar category in the semantic database as follows:

[0183] ;

[0184] in, Let be the scene label identified by the i-th distributed audio / video playback device at the current moment. For the k-th candidate scene category in the semantic library, Let be the video scene feature vector corresponding to the scene label of the i-th distributed audio and video playback device;

[0185] The action feature vector is obtained as follows:

[0186] ;

[0187] ;

[0188] in, Let be the standard feature prototype vector corresponding to the action category of the i-th distributed audio / video playback device. This is an action feature extraction function, used to extract action features from the currently playing video frame or short video clip of the i-th distributed audio / video playback device. In the process, extract motion feature vectors that can represent "whether an action has occurred in the current frame, the strength of the action, the direction of the movement, and the range of the movement"; The optical flow amplitude characteristic function is used to calculate the optical flow for adjacent frames, and then the mean, maximum or quantile of the optical flow amplitude is statistically analyzed. The larger the value, the more obvious the overall motion of the picture. The motion region proportion feature function is defined by inter-frame difference and thresholding, which represents the proportion of the area of ​​the motion region in the current video segment to the total area of ​​the entire frame. Let be the motion energy characteristic function, and let the mean of the squared differences between frames represent the overall visual change intensity of the current video segment.

[0189] From the action semantic database, find the standard action category that is most similar to the current action features:

[0190] ;

[0191] ;

[0192] ;

[0193] ;

[0194] in, To predict semantic errors, For the i-th distributed audio and video playback device, at the playback speed multiplier... Then, the predicted scene label for the next control cycle; The expected media position to be played by the i-th distributed audio / video playback device. The video scene feature vector at that location, For the i-th distributed audio and video playback device, at the playback speed multiplier... Then, predict the action event category corresponding to the next control cycle; The expected media position to be played by the i-th distributed audio / video playback device. Video action feature vector at the location; for The corresponding standard scene feature prototype vector; for The corresponding standard action feature prototype vector; The neighborhood reference vit scene label is obtained by weighted voting based on the scene labels predicted by neighboring nodes; This refers to the neighborhood reference action event category obtained by weighted voting based on the predicted action events from neighboring nodes. As scene label error weights; This sets the error weight for action events. Action events are typically more easily perceived by users than scene labels. For example, asynchronous actions like explosions, shooting, and hitting will show more noticeable differences than in normal scenes. You can set the following: ;and ,default .

[0195] S103.2 Based on the predicted time error, predicted rhythm error, and predicted semantic error, construct an objective function with the playback speed multiplier as the variable;

[0196] In one possible implementation, the objective function is:

[0197] ;

[0198] in, Let be the objective function corresponding to the i-th distributed audio / video playback device. As the weight for time schedule error, For rhythm error weights, For semantic error weights, Weights are limited to account for the degree of deviation. Weights are limited to account for the degree of change. Let be the prediction time error corresponding to the i-th distributed audio / video playback device. Let be the prediction rhythm error corresponding to the i-th distributed audio / video playback device. Let be the predicted semantic error corresponding to the i-th distributed audio / video playback device. For the current moment, To control the cycle, For a moment Playback speed multiplier For a moment The playback speed multiplier.

[0199] The value can be determined by combining preset initial values ​​with dynamic adjustments based on the operating status. These correspond to time progress error, rhythm error, and semantic error, respectively. Their initial values ​​can be preset according to the application scenario and based on the semantic error. Time progress error and rhythm error Dynamic adjustment. and For playback continuity constraint weights, Used to limit the degree to which the playback speed deviates from the default playback speed. This is used to limit the range of playback speed changes between adjacent control cycles. Both can be preset according to the system's requirements for playback naturalness, or dynamically adjusted based on device processing power, buffer length, and network latency. Therefore, the objective function can both drive rapid node convergence and avoid stuttering or unnatural playback caused by sudden changes in playback speed.

[0200] S103.3. With minimizing the objective function as the objective, solve for the playback speed multiplier to obtain the optimal playback speed.

[0201] For example, the optimal playback speed for the current synchronization period can be obtained by minimizing the objective function as follows:

[0202] ;

[0203] in, Let be the optimal playback speed for the i-th distributed audio / video playback device. argmin This represents a function that seeks the minimum value. This is the preset minimum value for playback speed multiplier. This is the preset maximum value for the playback speed multiplier; when the i-th distributed audio / video playback device lags behind the neighboring reference progress, then... >1, playback can be sped up by increasing the actual playback speed multiplier; when the i-th distributed audio / video playback device is ahead of the neighboring reference progress, then <1, playback can be slowed down by reducing the actual playback speed multiplier; when the i-th distributed audio / video playback device is basically consistent with the neighboring reference state, then A value close to 1 can maintain a normal playback speed.

[0204] In one possible implementation, determining the synchronization mode based on the semantic error includes:

[0205] S103.4 If the semantic error is less than the first preset semantic threshold, then the synchronization mode is determined to be the normal synchronization mode.

[0206] S103.5 If the semantic error is greater than the first preset semantic threshold and less than the second preset semantic threshold, then the synchronization mode is determined to be the accelerated synchronization mode; the second preset semantic threshold is greater than the first preset semantic threshold, and the accelerated synchronization mode increases the time progress error weight and / or semantic error weight compared with the normal synchronization mode.

[0207] S103.6 If the semantic error is greater than the second preset semantic threshold, then the synchronization mode is determined to be the forced synchronization mode; the forced synchronization mode includes at least one of keyframe alignment, brief rebuffering, pause and wait, or jump to the nearest semantically consistent keyframe.

[0208] In one possible implementation, the method further includes:

[0209] When the synchronization mode is the forced synchronization mode, the synchronization execution action corresponding to the forced synchronization mode and the optimal playback speed are used together as the audio and video synchronization control strategy in the current synchronization cycle.

[0210] When the semantic error is small, the system uses normal synchronization weights. When the semantic error reaches the first threshold, the weights for timing error and semantic error are increased to accelerate the convergence of the distributed audio and video playback devices to the neighboring reference state. When the semantic error reaches the second semantic threshold or a key semantic event is detected as out of sync, the distributed audio and video playback devices prioritize semantic consistency and generate a forced synchronization action. The forced synchronization execution action includes at least one of keyframe alignment, brief rebuffering, pause waiting, or semantically consistent frame jump.

[0211] For example, it can be based on semantic error And the synchronization mode is determined by the detection results of key semantic events. Let the first semantic threshold be... And the second semantic threshold is And satisfy: ;

[0212] when When that happens, the synchronization mode is determined. In normal synchronization mode, normal weight and playback speed adjustment range can be used to smoothly change the playback speed multiplier within a small range. For example, it can be preset. Use the preset Synchronize.

[0213] when When that happens, the synchronization mode is determined. To accelerate the synchronization mode, the weight of the timing error can be increased. and semantic error weights It also expands the range of adjustable playback speed multipliers, enabling distributed audio and video playback devices to converge to the neighboring reference state more quickly.

[0214] The baseline weight in normal synchronization mode is defined as follows: .in, The time progress error weight in normal synchronization mode; The rhythm difference weights in normal synchronization mode; This represents the semantic difference weights in normal mode.

[0215] To emphasize smooth synchronization in normal synchronization mode, you can set: ;default: Playback speed multiplier It can be limited to a normal adjustment range: ,and , This is the maximum speed offset in normal synchronization mode. (Default) ,but:

[0216] ;

[0217] The above formula indicates that the speed can be reduced to a maximum of 0.97 times and accelerated to a maximum of 1.03 times. The playback speed is only slightly adjusted, and users are unlikely to perceive the sudden change.

[0218] when When that happens, the synchronization mode is determined. In the forced synchronization mode, semantic consistency can be prioritized as a constraint to generate forced synchronization control decisions. These decisions include at least one of keyframe alignment, brief rebuffering, pause and wait, or jumping to the semantically consistent nearest keyframe. When the target keyframe is close and the buffer is sufficient, keyframe alignment is selected; when a node lags significantly, jumping to a semantically consistent frame or brief rebuffering is selected; when a node is significantly ahead, pause and wait or reduce playback speed is selected; when the buffer is insufficient or the target frame is unavailable, brief rebuffering is selected. This ensures that key semantic events remain consistent across multiple nodes.

[0219] Keyframe distance (which must be calculated from playback progress + keyframe position) is defined as:

[0220] ;

[0221] in: This is the current playback progress. This represents the playback progress corresponding to the most recent semantic keyframe.

[0222] Buffer duration (how long data can be played) is defined as:

[0223] ;

[0224] in: The current amount of data in the buffer queue (in seconds or frames). This represents the current playback rate.

[0225] when and At that time, perform keyframe alignment playback;

[0226] when When the keyframe is in use, jump to the semantically consistent keyframe;

[0227] When playback control (lead / lag) is ahead Reduce playback speed or pause for 100–300 ms;

[0228] behind Speed ​​up playback or briefly skip frames.

[0229] When buffer is insufficient Execution: Brief rebuffering (200–500ms).

[0230] Therefore, the audio and video synchronization control strategy for the current synchronization cycle is output. ;

[0231] in, For optimal playback speed, In synchronous mode, This refers to the corresponding synchronization action in forced synchronization mode. When the synchronization mode is normal synchronization mode or accelerated synchronization mode, the synchronization action is executed accordingly. It can be left blank, mainly based on the optimal playback speed. Playback speed control; when the synchronization mode is in forced synchronization mode, actions are performed according to synchronization. Perform operations such as keyframe alignment, brief rebuffering, or pause and wait.

[0232] Therefore, the objective function serves as the decision-making basis for the synchronization controller, and is used to convert the differences between multiple scenes into actual playback speed control quantities.

[0233] In one possible implementation, the audio and video synchronization control strategy is executed through the distributed audio and video playback device, including:

[0234] S104.1 When the synchronization mode is normal synchronization mode or accelerated synchronization mode, the distributed audio and video playback device adjusts its own actual playback speed multiplier by using the optimal playback speed to obtain the adjusted actual playback speed multiplier, and adjusts the playback progress according to the adjusted actual playback speed multiplier.

[0235] S104.2. When the synchronization mode is the forced synchronization mode, execute the synchronization action corresponding to the forced synchronization mode.

[0236] In one possible implementation, the distributed audio / video playback device adjusts its own actual playback speed multiplier using the optimal playback speed, including:

[0237] Based on a preset playback speed smoothing coefficient, the distributed audio and video playback device performs a weighted sum of its actual playback speed multiplier and the optimal playback speed to obtain the adjusted actual playback speed multiplier.

[0238] For example, in synchronization mode In normal or accelerated synchronization mode, the local playback speed is not abruptly changed. Instead, it uses a smooth update method to generate the actual playback speed multiplier used in the current synchronization cycle:

[0239]

[0240] in, Let be the playback speed multiplier actually used by the i-th distributed audio / video playback device in the current synchronization cycle. This is the playback speed multiplier of the previous synchronization cycle. The playback speed smoothing coefficient, and satisfies... A larger playback speed smoothing factor results in smoother playback speed changes; a smaller factor results in a faster response to synchronization errors. In normal synchronization mode, the playback speed smoothing factor can be set to a larger value to ensure natural playback; in accelerated synchronization mode, the factor can be appropriately reduced to improve convergence speed.

[0241] Normal synchronous mode aims for natural playback, so the playback speed smoothing factor should be relatively large, for example: Preferred .

[0242] Subsequently, the playback progress is updated according to the actual playback speed multiplier as follows:

[0243]

[0244] when At times, playback is sped up to catch up with the neighboring reference progress; when When, slow down playback to wait for other neighboring nodes to approach; when At the same time, maintain normal playback speed.

[0245] When in synchronization mode In forced synchronization mode, synchronous execution actions are performed first. Instead of relying solely on continuous adjustment of playback speed, synchronous actions include at least one of the following:

[0246] Keyframe alignment: Adjust the local playback position to the keyframe position corresponding to the semantic state of the neighboring reference;

[0247] Brief rebuffering: Pauses playback and rebuffers, realigning the local playback state with the neighboring reference state;

[0248] Pause and wait: When there is a large lead, pause briefly or slow down the playback speed to wait for other neighboring nodes to enter the same semantic state;

[0249] Semantically consistent frame jump: Jump to the nearest video frame or keyframe that is consistent with the semantic events of the neighborhood reference.

[0250] Synchronous execution of actions After execution is complete, the next synchronization cycle begins to perform distributed audio and video synchronization.

[0251] After the i-th distributed audio and video playback device performs synchronization control, its playback progress, audio time structure data, and video content status data are updated. Entering the next synchronization cycle, state acquisition, environmental field construction, and error calculation are re-executed, and the time progress error, rhythm error, and semantic error are re-obtained based on the updated state.

[0252] The convergence condition is determined based on the three types of errors obtained from the recalculation. Specifically, when the timing error satisfies: Rhythm differences satisfy: And the semantic differences satisfy: When the i-th distributed audio / video playback device meets the convergence condition within the current synchronization period, it is considered to have achieved convergence. This is the time progress convergence threshold. The rhythm convergence threshold, This is the semantic convergence threshold. The convergence threshold is used to determine whether the distributed audio and video playback devices have reached a synchronized and stable state, and the semantic convergence threshold, the first semantic threshold, and the second semantic threshold satisfy the following: .

[0253] To avoid misjudgments caused by network jitter, instantaneous playback deviations, or semantic recognition errors, the i-th distributed audio / video playback device is only considered to have completed iterative convergence and entered a stable synchronization state when all three convergence conditions—time progress, rhythm, and semantics—are met within M consecutive synchronization cycles. Here, M is a preset number of stable cycles, which can be set according to the synchronization cycle length, network stability, and application scenario requirements. i = 1, 2, ..., NP, where NP represents the total number of distributed audio / video playback devices.

[0254] When all distributed audio and video playback devices in the system, or a predetermined proportion of such devices, enter a synchronized and stable state, the system is considered to have entered a synchronized and stable state. After entering this state, synchronization control does not cease; instead, it continues to periodically perform state checks and error calculations. If, subsequently, changes in network latency, device processing capabilities, buffer status, or asynchrony of key semantic events cause any error to exceed the corresponding convergence threshold again, the distributed audio and video playback devices re-enter the control decision-making and synchronization execution process.

[0255] Thus, this application forms a closed-loop iterative process of state acquisition, environmental field construction, error calculation, control decision, control execution, convergence judgment, and state feedback, enabling each distributed audio and video playback device to continuously maintain a perceptual synchronization state of time consistency, rhythm consistency, and semantic consistency in a dynamic distributed environment.

[0256] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0257] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0258] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0259] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0260] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0261] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A distributed audio-video synchronization method based on multi-field coupling, characterized in that, include: For any distributed audio and video playback device, the status data of the multi-source nodes corresponding to the distributed audio and video playback device is collected, and the multi-source node status data of the distributed audio and video playback device is broadcast to its neighboring nodes; the neighboring node refers to a one-hop neighboring device of the distributed audio and video playback device. After the distributed audio and video playback device receives the multi-source node status data broadcast by the neighboring node, it obtains the time progress field, rhythm field and semantic field according to the multi-source node status data, and obtains the time progress error, rhythm error and semantic error according to the time progress field, rhythm field and semantic field respectively. The optimal playback speed is determined based on the time progress error, rhythm error, and semantic error, and the synchronization mode is determined based on the semantic error. The audio and video synchronization control strategy within the current synchronization cycle is obtained based on the optimal playback speed and the synchronization mode. The distributed audio and video playback device executes the audio and video synchronization control strategy and enters the next synchronization cycle for distributed audio and video synchronization until the time progress error, rhythm error and semantic error meet the convergence termination condition, thus completing the distributed audio and video synchronization based on multi-field coupling.

2. The distributed audio-video synchronization method based on multi-field coupling according to claim 1, characterized in that, Collect the status data of the multi-source nodes corresponding to the distributed audio and video playback device, including: Collect the playback progress, audio signal, and video frames corresponding to the distributed audio and video playback device; The beat frequency, rhythm phase, and audio energy are extracted from the audio signal to obtain audio time structure data; Scene tags and action events are extracted from the video frames to obtain video content state data; The playback progress, audio time structure data, and video content status data are collectively used as multi-source node status data.

3. The distributed audio-video synchronization method based on multi-field coupling according to claim 1, characterized in that, Based on the multi-source node state data, the time progress field, rhythm field, and semantic field are obtained respectively, including: For any distributed audio and video playback device, determine the smoothed delay estimate of the arrival time of neighboring nodes to the distributed audio and video playback device, and obtain the influence weight of neighboring nodes on the distributed audio and video playback device based on the smoothed delay estimate; The time progress field corresponding to the distributed audio and video playback device is obtained by weighted summation based on the playback progress and corresponding influence weight in the multi-source node status data corresponding to the neighbor nodes of the distributed audio and video playback device. The rhythm field corresponding to the distributed audio and video playback device is obtained by weighted summation of the audio time structure data and the corresponding influence weights in the multi-source node state data corresponding to the neighbor nodes of the distributed audio and video playback device; the rhythm field includes the neighborhood beat frequency reference, the neighborhood rhythm phase reference, and the neighborhood audio energy reference. Based on the video content status and corresponding influence weights in the multi-source node status data corresponding to the neighbor nodes of the distributed audio and video playback device, the semantics of the neighbor nodes are weighted and summed using a maximum value calculation function to obtain a semantic field; the semantic field includes neighborhood scene label references and neighborhood action event references.

4. The distributed audio-video synchronization method based on multi-field coupling according to claim 3, characterized in that, Based on the time progress field, rhythm field, and semantic field, the time progress error, rhythm error, and semantic error are obtained respectively, including: For any distributed audio and video playback device, determine the difference between the corresponding time progress field and its playback progress to obtain the time progress error; Based on the neighborhood beat frequency reference, neighborhood rhythm phase reference, and neighborhood audio energy reference in the rhythm error, the frequency error, phase error, and energy error are obtained respectively. The frequency error, phase error, and energy error are then weighted, summed, and the square root is taken to obtain the rhythm error. Conditional assignments are performed based on the neighborhood scene label references and neighborhood action event references in the semantic field. The values ​​obtained from the two conditional assignments are then weighted and summed to obtain the semantic error.

5. The distributed audio-video synchronization method based on multi-field coupling according to claim 1, characterized in that, Determining the optimal playback speed based on the aforementioned time progress error, rhythm error, and semantic error includes: The predicted time error is obtained based on the time progress error, the predicted rhythm error is obtained based on the rhythm error, and the predicted semantic error is obtained based on the semantic error. Based on the predicted time error, predicted rhythm error, and predicted semantic error, an objective function with playback speed multiplier as the variable is constructed. With the objective function as the goal, the playback speed multiplier is solved to obtain the optimal playback speed.

6. The distributed audio and video synchronization method based on multi-field coupling according to claim 5, characterized in that, The objective function is: ； in, Let be the objective function corresponding to the i-th distributed audio / video playback device. As the weight for time schedule error, For rhythm error weights, For semantic error weights, Weights are limited to account for the degree of deviation. Weights are limited to account for the degree of change. Let be the prediction time error corresponding to the i-th distributed audio / video playback device. Let be the prediction rhythm error corresponding to the i-th distributed audio / video playback device. Let be the predicted semantic error corresponding to the i-th distributed audio / video playback device. For the current moment, To control the cycle, For a moment Playback speed multiplier For a moment The playback speed multiplier.

7. The distributed audio-video synchronization method based on multi-field coupling according to claim 6, characterized in that, Determining the synchronization mode based on the semantic error includes: If the semantic error is less than the first preset semantic threshold, then the synchronization mode is determined to be the normal synchronization mode. If the semantic error is greater than a first preset semantic threshold but less than a second preset semantic threshold, then the synchronization mode is determined to be an accelerated synchronization mode; if the second preset semantic threshold is greater than the first preset semantic threshold, the accelerated synchronization mode increases the weight of the time progress error and / or the weight of the semantic error compared to the normal synchronization mode. If the semantic error is greater than the second preset semantic threshold, the synchronization mode is determined to be a forced synchronization mode; the forced synchronization mode includes at least one of keyframe alignment, brief rebuffering, pause and wait, or jump to the nearest semantically consistent keyframe.

8. The distributed audio-video synchronization method based on multi-field coupling according to claim 7, characterized in that, Also includes: When the synchronization mode is the forced synchronization mode, the synchronization execution action corresponding to the forced synchronization mode and the optimal playback speed are used together as the audio and video synchronization control strategy in the current synchronization cycle.

9. The distributed audio-video synchronization method based on multi-field coupling according to claim 7, characterized in that, Executing an audio-video synchronization control strategy through the distributed audio-video playback device includes: When the synchronization mode is normal synchronization mode or accelerated synchronization mode, the distributed audio and video playback device adjusts its own actual playback speed multiplier by using the optimal playback speed to obtain the adjusted actual playback speed multiplier, and adjusts the playback progress according to the adjusted actual playback speed multiplier. When the synchronization mode is forced synchronization mode, the synchronization execution action corresponding to the forced synchronization mode is executed.

10. The distributed audio-video synchronization method based on multi-field coupling according to claim 9, characterized in that, The distributed audio and video playback device adjusts its actual playback speed multiplier using the optimal playback speed, including: Based on a preset playback speed smoothing coefficient, the distributed audio and video playback device performs a weighted sum of its actual playback speed multiplier and the optimal playback speed to obtain the adjusted actual playback speed multiplier.

Images