Adaptive synchronization playing method and device of audio and video and medium

Abstract

This disclosure provides an adaptive synchronous playback method, device, and medium for audio and video, relating to the field of audio and video processing technology. The method is applied to an electronic device and includes: acquiring current audio playback characteristic values ​​of an audio playback device playing audio data, current video display characteristic values ​​of a video display device displaying video data, and current transmission characteristic values ​​of a communication channel; the video display device transmitting audio data to the audio playback device through the communication channel; determining a current audio and video synchronization quantization value based on the current audio playback characteristic value, current video display characteristic value, and current transmission characteristic value; determining a target synchronization sub-strategy from an audio and video synchronization strategy that matches the current audio and video synchronization quantization value, the audio and video synchronization strategy reflecting the mapping relationship between different audio and video synchronization quantization value ranges and corresponding synchronization sub-strategies; and controlling at least one of the audio playback device, video display device, and communication channel according to the target synchronization sub-strategy.

Description

Technical Field

[0001] This disclosure relates to the field of audio and video processing technology, and more specifically, to an adaptive synchronous playback method, device, and medium for audio and video. Background Technology

[0002] With the widespread application of Virtual Reality (VR) devices in in-vehicle entertainment, immersive navigation, and multimedia interactive scenarios, the integration of VR devices with in-vehicle speakers to achieve immersive viewing has emerged. In this process, the VR device is responsible for displaying the immersive video, while the in-vehicle speakers are responsible for receiving and playing the audio sent by the VR device.

[0003] However, because VR devices can directly display video, while car speakers need to receive audio data sent by the VR device via Bluetooth or WiFi before they can play audio, there is a certain delay between audio playback and video display, resulting in audio-video desynchronization. This creates a disconnect for users when watching videos, reducing their immersive experience. Summary of the Invention

[0004] One object of this disclosure is to provide a new technical solution for adaptive synchronized playback of audio and video.

[0005] According to a first aspect of this disclosure, an adaptive synchronous playback method for audio and video is provided, applied to an electronic device, comprising: The current audio playback characteristic value of the audio playback device when playing audio data, the current video display characteristic value of the video display device when displaying video data, and the current transmission characteristic value of the communication channel are obtained respectively. The video display device transmits the audio data to the audio playback device through the communication channel. The current audio-video synchronization quantization value is determined based on the current audio playback feature value, the current video display feature value, and the current transmission feature value. A target synchronization sub-strategy that matches the current audio-video synchronization quantization value is determined from the audio-video synchronization strategy, wherein the audio-video synchronization strategy reflects the mapping relationship between different audio-video synchronization quantization value ranges and the corresponding synchronization sub-strategies; Control at least one of the audio playback device, the video display device, and the communication channel according to the target synchronization sub-strategy.

[0006] Optionally, determining the current audio-video synchronization quantization value based on the current audio playback feature value, the current video display feature value, and the current transmission feature value includes: The current audio-video synchronization feature value is determined based on the current audio playback feature value, the current video display feature value, and the current transmission feature value. Based on the current audio and video synchronization feature value, determine the current audio and video synchronization quantization value.

[0007] Optionally, determining the current audio-video synchronization quantization value based on the current audio playback feature value, the current video display feature value, and the current transmission feature value includes: Based on the current audio playback feature value, the current video display feature value, and the current transmission feature value, determine the current initial audio and video synchronization quantization value; The current initial audio and video synchronization quantization value is smoothed based on the previous historical audio and video synchronization quantization value to obtain the current audio and video synchronization quantization value.

[0008] Optionally, the current audio playback feature value includes: the audio timestamp of the currently playing audio frame, the current decoding duration corresponding to the currently playing audio frame, and the current load of the audio playback device; And / or, the current video display feature values ​​include: the video timestamp of the currently displayed video frame, the encoding duration corresponding to the currently played audio frame, the rendering duration of the currently displayed video frame, and the current load of the video display device; And / or, the current transmission characteristics include: the current packet loss rate of the audio data, the current transmission delay of the audio data, and the current transmission delay jitter of the audio data.

[0009] Optionally, determining the target synchronization sub-strategy that matches the current audio-video synchronization quantization value from the audio-video synchronization strategy includes: If the current audio-video synchronization quantization value is greater than or equal to the first threshold, the target synchronization sub-strategy is determined to be: improve the audio playback quality of the audio data and / or improve the video display quality of the video data.

[0010] Optionally, determining the target synchronization sub-strategy that matches the current audio-video synchronization quantization value from the audio-video synchronization strategy includes: When the current audio-video synchronization quantization value is greater than or equal to the second threshold and less than the first threshold, the current audio-video change trend is determined based on the current audio-video synchronization quantization value and the previous historical audio-video synchronization quantization value. When the current audio and video trend is deteriorating, the target synchronization sub-strategy is determined to be: increase the amount of backup audio data in the audio buffer of the audio playback device; If the current audio and video trend is improving, the target synchronization sub-strategy is determined to be: reduce the amount of backup audio data in the audio buffer of the audio playback device.

[0011] Optionally, determining the target synchronization sub-strategy that matches the current audio-video synchronization quantization value from the audio-video synchronization strategy includes: If the current audio-video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold, determine the duration for which the current audio-video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold. Based on the duration, the target synchronization sub-strategy is determined to be: executing at least one of switching the communication channel to a high-reliability communication channel, reducing the audio playback quality of the audio data, and reducing the video display quality of the video data. The longer the duration, the more items among switching the communication channel to a high-reliability communication channel, reducing the audio playback quality of the audio data, and reducing the video display quality of the video data are executed.

[0012] Optionally, determining the target synchronization sub-strategy that matches the current audio-video synchronization quantization value from the audio-video synchronization strategy includes: If the current audio-video synchronization quantization value is less than the third threshold, the target synchronization sub-strategy is determined to be: re-execute the clock synchronization operation of the audio playback device and the video display device, output a prompt message, and re-establish the communication channel at least one of the following:

[0013] According to a second aspect of this disclosure, an electronic device is provided, the electronic device comprising: a memory and a processor, the memory being configured to store computer instructions, and the processor being configured to invoke the computer instructions from the memory to perform a method as described in any one of the first aspects.

[0014] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the first aspects.

[0015] This disclosure provides an adaptive synchronous playback method for audio and video, applied to an electronic device, comprising: acquiring the current audio playback characteristic value of an audio playback device when playing audio data, the current video display characteristic value of a video display device when displaying video data, and the current transmission characteristic value of a communication channel, wherein the video display device transmits audio data to the audio playback device through the communication channel; determining a current audio-video synchronization quantization value based on the current audio playback characteristic value, the current video display characteristic value, and the current transmission characteristic value; determining a target synchronization sub-strategy matching the current audio-video synchronization quantization value from an audio-video synchronization strategy, wherein the audio-video synchronization strategy reflects the mapping relationship between different audio-video synchronization quantization value ranges and corresponding synchronization sub-strategies; and controlling at least one of the audio playback device, the video display device, and the communication channel according to the target synchronization sub-strategy. This method can adjust the audio-video synchronization adjustment mode to match the degree of audio-video synchronization imbalance reflected by the current audio-video synchronization quantization value, thereby achieving adaptive audio-video synchronization adjustment and ensuring adaptive synchronous playback of audio and video.

[0016] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.

[0018] Figure 1 This is a flowchart illustrating an audio-visual adaptive synchronous playback method provided in this disclosure; Figure 2 This is a schematic diagram of the structure of an audio and video adaptive synchronous playback device provided in this disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device provided in this disclosure. Detailed Implementation

[0019] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0020] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0021] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0022] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0024] This disclosure provides an adaptive synchronous playback method for audio and video, applied to an electronic device, for achieving synchronous playback of audio from an audio playback device and video from a video display device. The audio playback device is used to play audio data; an example of an audio playback device is a car audio system. The video display device is used to display video data (containing only image data); an example of a video display device is a virtual display device.

[0025] In one embodiment of this disclosure, the electronic device is specifically a device independent of audio playback devices and video display devices, such as a vehicle head unit. In another embodiment of this disclosure, the electronic device may also be a video display device.

[0026] like Figure 1 As shown, the audio and video adaptive synchronous playback method provided in this disclosure includes the following steps S1100 to S1400.

[0027] In step S1100, the current audio playback characteristic value of the audio playback device when playing audio data, the current video display characteristic value of the video display device when displaying video data, and the current transmission characteristic value of the communication channel are obtained respectively. The video display device transmits audio data to the audio playback device through the communication channel.

[0028] In this embodiment, the video display device pre-stores video data and audio data to be played. The video data includes multiple video frames. The audio data includes multiple audio frames, which can be understood as audio segments. These multiple audio frames constitute a time-continuous audio data set. During audio and video playback, the video display device encodes the audio data while displaying the video data. Further, the video display device transmits the encoded audio data to the audio playback device via a communication channel. The audio playback device decodes the received audio data and plays it. It should be noted that the video display device and the audio playback device have already synchronized their clocks before executing the adaptive synchronous playback method for audio and video provided in this disclosure.

[0029] The current audio playback feature value specifically refers to the feature value belonging to the audio playback device that currently affects the playback progress of the audio data. In one embodiment of this disclosure, the current audio playback feature value includes: the audio timestamp of the currently playing audio frame, the current decoding duration corresponding to the currently playing audio frame, and the current load of the audio playback device. The audio timestamp of the currently playing audio frame refers to the timestamp of the currently playing audio frame in the audio data. The current load of the audio playback device refers to the current load of the processor (e.g., microcontroller unit) involved in audio playback within the audio playback device.

[0030] Furthermore, the current video display feature value specifically refers to the feature value belonging to the video display device that currently affects the display progress of the video data. In one embodiment of this disclosure, the current video display feature value includes: the video timestamp of the currently displayed video frame, the encoding duration corresponding to the currently played audio frame, the rendering duration of the currently displayed video frame, and the current load of the video display device. The video timestamp of the currently displayed video frame refers to the timestamp of the currently played video frame in the video data. The current load of the video display device refers to the current load of the processors (e.g., central processing unit and / or graphics processing unit) involved in video display within the video display device.

[0031] Furthermore, the current transmission characteristic value specifically refers to the characteristic value that currently affects the transmission progress of audio data and belongs to the characteristics of the communication channel. In one embodiment of this disclosure, the current transmission characteristic value includes: the current packet loss rate of the audio data packet, the current transmission delay of the audio data packet, and the current transmission delay jitter of the audio data packet.

[0032] It should be noted that the specific methods for obtaining the aforementioned current audio playback feature value, current video display feature value, and current transmission feature value can be obtained using traditional methods, and will not be elaborated here.

[0033] Step S1200: Determine the current audio and video synchronization quantization value based on the current audio and video playback feature value, the current video display feature value, and the current transmission feature value.

[0034] The current audio / video synchronization quantization value is used to quantize the degree of synchronization in the current audio / video synchronization state. The larger the current audio / video synchronization quantization value, the better the current audio / video synchronization state, and vice versa.

[0035] In one embodiment of this disclosure, step S1200 can be implemented using deep learning. Specifically: a training sample set is obtained; a preset neural network is trained based on the training sample set to obtain an audio-video synchronization quantization model; the current audio-video playback feature value, the current video display feature value, and the current transmission feature value are input into the audio-video synchronization quantization model to obtain the audio-video synchronization quantization value output by the audio-video synchronization quantization model, which is denoted as the current audio-video synchronization quantization value. The training sample set includes multiple sets of training samples. Each training sample includes audio-video playback feature values, video display feature values, and transmission feature values ​​as samples, and audio-video synchronization quantization values ​​as labels for the aforementioned samples. The training samples can be obtained through experience and / or experimentation.

[0036] In another embodiment of this disclosure, step S1200 is specifically implemented by the following steps S1210 and S1211.

[0037] Step S1210: Determine the current audio and video synchronization feature value based on the current audio playback feature value, the current video display feature value, and the current transmission feature value.

[0038] In one embodiment of this disclosure, taking the following as an example, the current audio playback feature values ​​include: the audio timestamp of the currently played audio frame, the current decoding duration corresponding to the currently played audio frame, and the current load of the audio playback device; the current video display feature values ​​include: the video timestamp of the currently displayed video frame, the encoding duration corresponding to the currently played audio frame, the rendering duration of the currently displayed video frame, and the current load of the video display device; and the current transmission feature values ​​include: the current packet loss rate of the audio data, the current transmission delay of the audio data, and the current transmission delay jitter of the audio data. The specific implementation of the above step S1210 is as follows.

[0039] The difference between the audio timestamp of the currently playing audio frame and the video timestamp of the currently displayed video frame is used to determine the current audio-video delay error. Furthermore, in order to unify the units, the current audio-video delay error is mapped to the current normalized audio-video delay error using the following formula.

[0040] (Formula 1) in, The difference between the audio timestamp of the currently playing audio frame and the video timestamp of the currently displayed video frame; This is the maximum difference between the audio timestamp of the played audio frame and the video timestamp of the displayed video frame when audio and video are played synchronously. This is the video timestamp of the currently normalized display video frame.

[0041] The current overall load is determined based on the current load of the video display device and the current load of the audio playback device. Furthermore, to standardize the units of measurement, the current overall load is mapped to the current normalized overall load using the following formula (Formula 2). The current overall load can be exemplified as the weighted sum of the current loads of the video display device and the audio playback device. A higher current overall load indicates longer video rendering and audio encoding times for the video display device, longer audio decoding times for the audio playback device, and poorer audio-video synchronization.

[0042] Formula 2) Where R is the current overall load; k is the exponential coefficient used to perform a nonlinear mapping on R, adjusting the degree of influence of R on the audio and video synchronization state, which can be determined based on experience and / or experiments.

[0043] Based on the current packet loss rate of the audio data, the current packet loss rate of the audio data is mapped to the current normalized packet loss rate using the following formula 3.

[0044] (Formula 3) in, This represents the current packet loss rate. This represents the maximum packet loss rate of audio data under synchronized audio and video playback conditions. This represents the current normalized packet loss rate.

[0045] The current overall latency is determined based on the current transmission latency of the audio data, the encoding duration of the currently played audio frame, the rendering duration of the currently displayed video frame, and the current decoding duration of the currently played audio frame. Furthermore, to standardize the units of measurement, the current overall latency is mapped to the current normalized overall latency using the following formula (Formula 4). The current overall latency can be exemplarily defined as the weighted sum of the current transmission latency of the audio data, the encoding duration of the currently played audio frame, the rendering duration of the currently displayed video frame, and the current decoding duration of the currently played audio frame. A higher current overall latency results in longer video rendering and audio encoding times for the video display device, longer audio decoding times for the audio playback device, and poorer audio-video synchronization.

[0046] (Formula 4) Based on the current transmission delay jitter of the audio data, the current transmission delay jitter of the audio data is mapped to the current normalized transmission delay jitter using the following formula five.

[0047] (Formula 5) in, This refers to the current transmission delay jitter. This represents the maximum value of transmission delay jitter when audio and video are played synchronously. This refers to the current normalized transmission delay jitter.

[0048] After obtaining the current normalized audio / video delay error, current normalized overall load, current normalized packet loss rate, current normalized overall delay, and current normalized transmission delay jitter, these parameters are used as the current audio / video synchronization feature values.

[0049] Step S1211: Determine the current audio and video synchronization quantization value based on the current audio and video synchronization feature value.

[0050] In this embodiment of the disclosure, step S1211 can be specifically implemented by the following formula six.

[0051] (Formula 6) Where S is the current audio / video synchronization quantization value. , , , and The weights for the corresponding current audio-video synchronization feature values ​​can be determined based on experience and / or experiments, and , , , and It satisfies the following formula seven.

[0052] (Formula 7) Based on Formula 6 above, we know that the value of S ranges from 0 to 100. The larger the value of S, the better the audio and video synchronization, and vice versa.

[0053] Step S1300: Determine the target synchronization sub-strategy that matches the current audio and video synchronization quantization value from the audio and video synchronization strategy. The audio and video synchronization strategy reflects the mapping relationship between different audio and video synchronization quantization value ranges and the corresponding synchronization sub-strategies.

[0054] In this embodiment, an audio-visual synchronization strategy reflecting the correspondence between different audio-visual synchronization quantization value ranges and their corresponding synchronization sub-strategies is pre-maintained based on experience and / or experiments and stored in the electronic device. Different audio-visual synchronization quantization value ranges correspond to different degrees of audio-visual synchronization imbalance. Different degrees of audio-visual synchronization imbalance are corrected by different synchronization sub-strategies. Each synchronization sub-strategy is an audio-visual synchronization adjustment method, and the adjustment object corresponding to the synchronization sub-strategy is at least one of the audio playback device, video display device, and communication channel. The higher the degree of audio-visual synchronization imbalance corresponding to the audio-visual synchronization quantization value range, the greater the adjustment force of the corresponding synchronization sub-strategy's audio-visual synchronization adjustment.

[0055] For example, the correspondence between the range of audio and video synchronization quantization values ​​and the degree of audio and video synchronization imbalance is shown in Table 1 below.

[0056] Table 1

[0057] For step S1300 above, firstly, the audio / video synchronization quantization value range to which the current audio / video synchronization quantization value belongs is determined from the audio / video synchronization strategy. Further, the synchronization sub-strategy corresponding to the audio / video synchronization quantization value range to which the current audio / video synchronization quantization value belongs is determined as the target synchronization sub-strategy that matches the current audio / video synchronization quantization value. It can be understood that when the audio / video synchronization quantization value range to which the current audio / video synchronization quantization value belongs is different, the target synchronization sub-strategy matching the current audio / video synchronization quantization value will be different.

[0058] Step S1400: Control at least one of the audio playback device, video display device, and communication channel according to the target synchronization sub-strategy.

[0059] Through the above step S1400, the audio and video synchronization adjustment method can be adjusted to match the degree of audio and video synchronization imbalance reflected by the current audio and video synchronization quantization value. This achieves adaptive audio and video synchronization adjustment and ensures adaptive synchronous playback of audio and video.

[0060] Based on the above, this disclosure provides an adaptive synchronous playback method for audio and video, applied to electronic devices. The method includes: acquiring the current audio playback characteristic value of an audio playback device playing audio data, the current video display characteristic value of a video display device displaying video data, and the current transmission characteristic value of a communication channel; the video display device transmitting audio data to the audio playback device through the communication channel; determining a current audio-video synchronization quantization value based on the current audio playback characteristic value, the current video display characteristic value, and the current transmission characteristic value; determining a target synchronization sub-strategy matching the current audio-video synchronization quantization value from an audio-video synchronization strategy, wherein the audio-video synchronization strategy reflects the mapping relationship between different audio-video synchronization quantization value ranges and corresponding synchronization sub-strategies; and controlling at least one of the audio playback device, the video display device, and the communication channel according to the target synchronization sub-strategy. This method can adjust the audio-video synchronization adjustment mode to match the degree of audio-video synchronization imbalance reflected by the current audio-video synchronization quantization value, thus achieving adaptive audio-video synchronization adjustment and ensuring adaptive synchronous playback of audio and video.

[0061] In one embodiment of this disclosure, in order to avoid a jump in the current synchronization quantization value compared to the previous historical audio-visual synchronization quantization value (the current synchronization quantization value calculated previously), which would cause a sudden change in the target synchronization sub-strategy corresponding to the current synchronization quantization value compared to the audio-visual synchronization adjustment method corresponding to the target synchronization sub-strategy corresponding to the previous historical audio-visual synchronization quantization value, thereby causing problems such as reduced audio-visual synchronization adjustment efficiency and instability, the above step S1200 in the adaptive synchronous playback method of audio and video provided in this disclosure is specifically implemented through the following steps S1220 and S1221.

[0062] Step S1220: Determine the current initial audio and video synchronization quantization value based on the current audio playback feature value, the current video display feature value, and the current transmission feature value.

[0063] Step S1221: Perform smoothing processing on the current initial audio and video synchronization quantization value based on the previous historical audio and video synchronization quantization value to obtain the current audio and video synchronization quantization value.

[0064] In this embodiment of the disclosure, S calculated by Formula Six above is denoted as the current initial audio-video synchronization quantization value. Furthermore, in one embodiment of the disclosure, step S1221 can be specifically implemented using Formula Eight below.

[0065] (Formula 8) in, This is the current initial audio / video synchronization quantization value. This is the current audio / video synchronization quantization value. This is the previous historical audio and video synchronization quantization value. It is the smoothing coefficient, and The value range is greater than 0 and less than 1. It can be set up based on experiments and / or experience.

[0066] In one embodiment of this disclosure, the above step S1400 is specifically implemented as follows: step S1410.

[0067] Step S1410: If the current audio-video synchronization quantization value is greater than or equal to the first threshold, the target synchronization sub-strategy is determined to be: improve the audio playback quality of the audio data and / or improve the video display quality of the video data.

[0068] In this embodiment of the disclosure, the first threshold is the minimum value of the audio-video synchronization quantization value when the audio-video synchronization imbalance is balanced. Taking Table 1 as an example, the first threshold can be exemplarily 85.

[0069] Improving the audio playback quality of audio data can be achieved by increasing the sampling rate of the audio data. Improving the video display quality of video data can be achieved by increasing the rendering complexity of the video data. It should be noted that this disclosure does not limit the specific implementation methods for improving the audio playback quality of audio data and the video display quality of video data.

[0070] If the current audio-video synchronization quantization value is greater than or equal to the first threshold, it indicates that the audio-video synchronization imbalance is balanced, i.e., audio-video synchronization. At this point, the target synchronization sub-strategy can be determined as improving the audio playback quality of the audio data and / or improving the video display quality of the video data. Thus, after executing the above step S1400, the adaptive synchronized playback method for audio and video provided in this disclosure can improve the audio playback quality of the audio data played by the audio playback device and / or improve the video display quality of the video data displayed by the video display device while ensuring audio-video synchronization.

[0071] In one embodiment of this disclosure, the above step S1400 is specifically implemented as follows: steps S1420 to S1422.

[0072] Step S1420: If the current audio and video synchronization quantization value is greater than or equal to the second threshold and less than the first threshold, determine the current audio and video change trend based on the current audio and video synchronization quantization value and the previous historical audio and video synchronization quantization value.

[0073] In this embodiment of the disclosure, the second threshold is the minimum value of the audio-video synchronization quantization value when the audio-video synchronization imbalance is mildly balanced. Taking Table 1 as an example, the second threshold can be exemplarily 60. If the current audio-video synchronization quantization value is greater than or equal to the second threshold and less than the first threshold, it indicates that the audio-video synchronization imbalance is mild. At this time, the current audio-video change trend is determined according to Formula Nine below.

[0074] (Formula 9) Among them, If the value is greater than 0, the current audio and video trend is determined to be improving; If the value is less than 0, the current audio / video trend is determined to be deterioration; If the value is 0, the current audio and video trend is determined to be unchanged.

[0075] Step S1421: If the current audio and video trend is deteriorating, determine the target synchronization sub-strategy as: increase the amount of backup audio data in the audio buffer of the audio playback device.

[0076] It should be noted that, in step S1421 above, increasing the amount of backup audio data in the audio buffer of the audio playback device specifically refers to increasing the amount of backup audio data in the audio buffer according to a set step size, based on the current amount of backup audio data in the audio buffer.

[0077] When the current audio and video trend is deteriorating, after executing the above step S1400, the amount of backup audio data in the audio buffer of the audio playback device can be increased. This allows the audio buffer of the audio playback device to buffer sufficient audio frames, which can withstand the interference of instantaneous transmission anomalies caused by short-term audio frame loss and communication channel jitter on the playback effect of audio data, and helps with audio and video synchronization.

[0078] Step S1422: If the current audio and video trend is improving, determine the target synchronization sub-strategy as: reduce the amount of backup audio data in the audio buffer of the audio playback device.

[0079] It should be noted that the reduction of the amount of backup audio data in the audio buffer of the audio playback device in step S1422 above specifically refers to reducing the amount of backup audio data in the audio buffer according to a set step size, based on the current amount of backup audio data in the audio buffer.

[0080] Corresponding to step S1421 above, if the current audio and video trend is improving, after executing step S1400 above, the amount of backup audio data in the audio buffer of the audio playback device can be reduced, thereby reducing the occupancy of the audio buffer of the audio playback device.

[0081] Corresponding to steps S1421 and S1422 above, if the current audio and video change trend remains unchanged, the target synchronization sub-strategy is determined to keep the amount of backup audio data in the current audio buffer unchanged.

[0082] In one embodiment of this disclosure, the above step S1400 is specifically implemented as follows: steps S1430 to S1431.

[0083] Step S1430: If the current audio and video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold, determine the duration during which the current audio and video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold.

[0084] The third threshold is the minimum value of the audio-video synchronization quantization value when the audio-video synchronization imbalance is moderately balanced. Taking Table 1 as an example, the third threshold can be 40. If the current audio-video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold, it indicates that the audio-video synchronization imbalance is moderate. In this case, the duration for which the current audio-video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold is recorded.

[0085] Step S1431: Based on the duration, determine the target synchronization sub-strategy as follows: execute at least one of the following: switch the communication channel to a high-reliability communication channel, reduce the audio playback quality of the audio data, and reduce the video display quality of the video data. The longer the duration, the more items among the following will be executed: switch the communication channel to a high-reliability communication channel, reduce the audio playback quality of the audio data, and reduce the video display quality of the video data.

[0086] In one embodiment of this disclosure, the communication channels include a Bluetooth communication channel, a Wi-Fi communication channel, and a wired communication channel. Therefore, the Wi-Fi communication channel has higher transmission reliability than the Bluetooth communication channel, and the wired communication channel has higher transmission reliability than the Wi-Fi communication channel. Specifically, switching the communication channel to a high-reliability communication channel involves switching from a Wi-Fi communication channel to a wired communication channel when the current communication channel is a Wi-Fi communication channel, and switching from a Bluetooth communication channel to a Wi-Fi communication channel when the current communication channel is a Bluetooth communication channel.

[0087] Furthermore, reducing the audio playback quality of audio data can be achieved by lowering the audio data sampling rate. Similarly, reducing the video display quality of video data can be achieved by reducing the video data rendering complexity. It should be noted that this disclosure does not limit the specific implementation methods for reducing the audio playback quality of audio data or the video display quality of video data.

[0088] The longer the duration, the more items in step S1400 above will be executed, such as switching the communication channel to a high-reliability communication channel, reducing the audio playback quality of the audio data, and reducing the video display quality of the video data, thus achieving a greater degree of audio-video synchronization adjustment.

[0089] In one embodiment of this disclosure, the above step S1400 is specifically implemented as follows: step S1440.

[0090] Step S1440: If the current audio and video synchronization quantization value is less than the third threshold, determine the target synchronization sub-strategy as: re-execute the clock synchronization operation of the audio playback device and the video display device, output prompt information, and re-establish the communication channel at least one of the following.

[0091] If the current audio-video synchronization quantization value is less than the third threshold, it indicates that the audio-video synchronization imbalance is severe. In this case, the target synchronization sub-strategy is determined to be: re-execute the clock synchronization operation of the audio playback device and the video display device, output a prompt message, and re-establish the communication channel at least one of the following: after executing the above step S1400, audio-video synchronization can be restarted.

[0092] This disclosure also provides an adaptive synchronous playback device 200 for audio and video, applied to electronic devices, such as... Figure 2 As shown, it includes: The acquisition module 210 is used to acquire the current audio playback feature value of the audio playback device when playing audio data, the current video display feature value of the video display device when displaying video data, and the current transmission feature value of the communication channel, respectively. The video display device transmits the audio data to the audio playback device through the communication channel. The first determining module 220 is used to determine the current audio and video synchronization quantization value based on the current audio playback feature value, the current video display feature value, and the current transmission feature value; The second determining module 230 is used to determine a target synchronization sub-strategy that matches the current audio-video synchronization quantization value from the audio-video synchronization strategy, wherein the audio-video synchronization strategy reflects the mapping relationship between different audio-video synchronization quantization value ranges and corresponding synchronization sub-strategies. The control module 240 is used to control at least one of the audio playback device, the video display device, and the communication channel according to the target synchronization sub-strategy.

[0093] In one embodiment of this disclosure, the first determining module 220 is specifically used for: The current audio-video synchronization feature value is determined based on the current audio playback feature value, the current video display feature value, and the current transmission feature value. Based on the current audio and video synchronization feature value, determine the current audio and video synchronization quantization value.

[0094] In one embodiment of this disclosure, the first determining module 220 is specifically used for: Based on the current audio and video synchronization feature value, determine the current initial audio and video synchronization quantization value; The current initial audio and video synchronization quantization value is smoothed based on the previous historical audio and video synchronization quantization value to obtain the current audio and video synchronization quantization value.

[0095] In one embodiment of this disclosure, the current audio playback feature value includes: the audio timestamp of the currently played audio frame, the current decoding duration corresponding to the currently played audio frame, and the current load of the audio playback device; And / or, the current video display feature values ​​include: the video timestamp of the currently displayed video frame, the encoding duration corresponding to the currently played audio frame, the rendering duration of the currently displayed video frame, and the current load of the video display device; And / or, the current transmission characteristics include: the current packet loss rate of the audio data, the current transmission delay of the audio data, and the current transmission delay jitter of the audio data.

[0096] In one embodiment of this disclosure, the second determining module 230 is specifically used for: If the current audio-video synchronization quantization value is greater than or equal to the first threshold, the target synchronization sub-strategy is determined to be: improve the audio playback quality of the audio data and / or improve the video display quality of the video data.

[0097] In one embodiment of this disclosure, the second determining module 230 is specifically used for: When the current audio-video synchronization quantization value is greater than or equal to the second threshold and less than the first threshold, the current audio-video change trend is determined based on the current audio-video synchronization quantization value and the previous historical audio-video synchronization quantization value. When the current audio and video trend is deteriorating, the target synchronization sub-strategy is determined to be: increase the amount of backup audio data in the audio buffer of the audio playback device; If the current audio and video trend is improving, the target synchronization sub-strategy is determined to be: reduce the amount of backup audio data in the audio buffer of the audio playback device.

[0098] In one embodiment of this disclosure, the second determining module 230 is specifically used for: If the current audio-video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold, determine the duration for which the current audio-video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold. Based on the duration, the target synchronization sub-strategy is determined to be: executing at least one of switching the communication channel to a high-reliability communication channel, reducing the audio playback quality of the audio data, and reducing the video display quality of the video data. The longer the duration, the more items among switching the communication channel to a high-reliability communication channel, reducing the audio playback quality of the audio data, and reducing the video display quality of the video data are executed.

[0099] In one embodiment of this disclosure, the second determining module 230 is specifically used for: If the current audio-video synchronization quantization value is less than the third threshold, the target synchronization sub-strategy is determined to be: re-execute the clock synchronization operation of the audio playback device and the video display device, output a prompt message, and re-establish the communication channel at least one of the following:

[0100] This disclosure also provides an electronic device, which includes the audio and video adaptive synchronous playback device 200 provided in any of the above-described device embodiments.

[0101] This disclosure also provides another electronic device, such as Figure 3 As shown, the electronic device 300 includes a memory 310 and a processor 320. The memory 310 is used to store computer instructions, and the processor 320 is used to call the computer instructions from the memory 310 to execute any of the adaptive synchronous playback methods for audio and video provided in the above method embodiments.

[0102] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the audio and video adaptive synchronous playback methods provided in the above-described method embodiments.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0104] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.

[0105] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0106] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include one or more of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.

[0107] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object programs written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as the "C" language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, can execute computer-readable program instructions to implement various aspects of the embodiments of this disclosure by utilizing state information from the computer-readable program instructions.

[0108] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.

[0112] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.

Claims

1. An adaptive synchronous playback method for audio and video, characterized in that, Applied to electronic devices, including: The current audio playback characteristic value of the audio playback device when playing audio data, the current video display characteristic value of the video display device when displaying video data, and the current transmission characteristic value of the communication channel are obtained respectively. The video display device transmits the audio data to the audio playback device through the communication channel. The current audio-video synchronization quantization value is determined based on the current audio playback feature value, the current video display feature value, and the current transmission feature value. A target synchronization sub-strategy that matches the current audio-video synchronization quantization value is determined from the audio-video synchronization strategy, wherein the audio-video synchronization strategy reflects the mapping relationship between different audio-video synchronization quantization value ranges and the corresponding synchronization sub-strategies; Control at least one of the audio playback device, the video display device, and the communication channel according to the target synchronization sub-strategy.

2. The method according to claim 1, characterized in that, The step of determining the current audio-video synchronization quantization value based on the current audio playback feature value, the current video display feature value, and the current transmission feature value includes: The current audio-video synchronization feature value is determined based on the current audio playback feature value, the current video display feature value, and the current transmission feature value. Based on the current audio and video synchronization feature value, determine the current audio and video synchronization quantization value.

3. The method according to claim 1, characterized in that, The step of determining the current audio-video synchronization quantization value based on the current audio playback feature value, the current video display feature value, and the current transmission feature value includes: Based on the current audio playback feature value, the current video display feature value, and the current transmission feature value, determine the current initial audio and video synchronization quantization value; The current initial audio and video synchronization quantization value is smoothed based on the previous historical audio and video synchronization quantization value to obtain the current audio and video synchronization quantization value.

4. The method according to any one of claims 1-3, characterized in that, The current audio playback feature values ​​include: the audio timestamp of the currently playing audio frame, the current decoding duration corresponding to the currently playing audio frame, and the current load of the audio playback device; And / or, the current video display feature values ​​include: the video timestamp of the currently displayed video frame, the encoding duration corresponding to the currently played audio frame, the rendering duration of the currently displayed video frame, and the current load of the video display device; And / or, the current transmission characteristics include: the current packet loss rate of the audio data, the current transmission delay of the audio data, and the current transmission delay jitter of the audio data.

5. The method according to claim 1, characterized in that, The step of determining a target synchronization sub-strategy that matches the current audio-video synchronization quantization value from the audio-video synchronization strategy includes: If the current audio-video synchronization quantization value is greater than or equal to the first threshold, the target synchronization sub-strategy is determined to be: improve the audio playback quality of the audio data and / or improve the video display quality of the video data.

6. The method according to claim 1, characterized in that, The step of determining a target synchronization sub-strategy that matches the current audio-video synchronization quantization value from the audio-video synchronization strategy includes: When the current audio-video synchronization quantization value is greater than or equal to the second threshold and less than the first threshold, the current audio-video change trend is determined based on the current audio-video synchronization quantization value and the previous historical audio-video synchronization quantization value. When the current audio and video trend is deteriorating, the target synchronization sub-strategy is determined to be: increase the amount of backup audio data in the audio buffer of the audio playback device; If the current audio and video trend is improving, the target synchronization sub-strategy is determined to be: reduce the amount of backup audio data in the audio buffer of the audio playback device.

7. The method according to claim 1, characterized in that, The step of determining a target synchronization sub-strategy that matches the current audio-video synchronization quantization value from the audio-video synchronization strategy includes: If the current audio-video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold, determine the duration for which the current audio-video synchronization quantization value is greater than or equal to the third threshold and less than the second threshold. Based on the duration, the target synchronization sub-strategy is determined to be: executing at least one of switching the communication channel to a high-reliability communication channel, reducing the audio playback quality of the audio data, and reducing the video display quality of the video data. The longer the duration, the more items among switching the communication channel to a high-reliability communication channel, reducing the audio playback quality of the audio data, and reducing the video display quality of the video data are executed.

8. The method according to claim 1, characterized in that, The step of determining a target synchronization sub-strategy that matches the current audio-video synchronization quantization value from the audio-video synchronization strategy includes: If the current audio-video synchronization quantization value is less than the third threshold, the target synchronization sub-strategy is determined to be: re-execute the clock synchronization operation of the audio playback device and the video display device, output a prompt message, and re-establish the communication channel at least one of the following:

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to retrieve the computer instructions from the memory to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images