Audio and video synchronization method and apparatus, electronic device and storage medium

By obtaining the audio and video synchronization detection results in the cloud computer and adjusting the frame rate, the problems of jitter and out-synchronization during audio and video playback in the cloud computer are solved, and the user experience is improved.

WO2025001916A1PCT designated stage expired Publication Date: 2025-01-02ZTE CORP

Patent Information

Application Number
PCT/CN2024/099792
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-29
Filing Date
2024-06-18
Publication Date
2025-01-02

AI Technical Summary

Technical Problem

In cloud computers, due to unstable network transmission and different decoding performances of different terminals during audio and video playback, the sound and picture are prone to jitter and out-synchronization, affecting the user experience.

Method used

By obtaining the audio and video synchronization detection results during audio and video playback, the decoded frame rate of the client or the encoded frame rate of the server is adjusted based on the detection results to ensure that the client caches video frames and realizes audio and video synchronization.

Benefits of technology

It effectively improves the out-synchronization of audio and video playback due to unstable network transmission or different decoding performance of different terminals, and improves user viewing experience and satisfaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024099792_02012025_PF_FP_ABST
    Figure CN2024099792_02012025_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure provides an audio and video synchronization method and apparatus, an electronic device and a storage medium. The method comprises: in an audio and video playing process, obtaining an audio and video synchronization detection result; on the basis of the audio and video synchronization detection result, executing an audio and video synchronization strategy corresponding to the audio and video synchronization detection result, wherein the audio and video synchronization strategy is used for adjusting the decoding frame frequency of a client or the coding frame frequency of a server, such that the client caches video frames.
Need to check novelty before this filing date? Find Prior Art

Description

Audio and video synchronization method, device, electronic device and storage medium

[0001] This disclosure claims priority to Chinese patent application No. 202310786876.2, filed on June 29, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of information processing technology, and in particular to an audio and video synchronization method, device, electronic device, and storage medium. Background Art

[0003] When playing online videos and local videos on a cloud computer, the transmission process of sound and picture includes multiple links. For example, the sound card captures the original audio, the graphics card captures the changes in the picture, and separately generates audio streams and video streams for transmission to the terminal. After being transmitted to the terminal, the audio and video are decoded and presented on the terminal for auditory and visual perception.

[0004] Summary of the Invention

[0005] In one aspect, a method for synchronizing audio and video is provided, the method comprising:

[0006] During audio and video playback, obtain audio and video synchronization detection results;

[0007] Based on the audio and video synchronization detection result, an audio and video synchronization strategy corresponding to the audio and video synchronization detection result is executed. The audio and video synchronization strategy is used to adjust the decoding frame rate of the client or the encoding frame rate of the server so that the client caches the video frames.

[0008] In another aspect, an audio and video synchronization device is provided, the device comprising:

[0009] The communication module is used to obtain the audio and video synchronization detection results during the audio and video playback process;

[0010] The processing module is used to execute the audio and video synchronization strategy corresponding to the audio and video synchronization detection result based on the audio and video synchronization detection result. The audio and video synchronization strategy is used to adjust the video decoding frame rate of the client or the video encoding frame rate of the server so that the client caches the video frames.

[0011] On the other hand, an electronic device is provided, comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program instructions executable by the processor; and when the processor executes the computer program instructions, the audio and video synchronization method described in any one of the above aspects or embodiments is implemented.

[0012] On the other hand, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed on an electronic device, the audio and video synchronization method described in any one of the above aspects or embodiments is implemented.

[0013] On the other hand, a computer program product is provided, which includes computer program instructions, and when the computer program instructions are executed, the audio and video synchronization method according to any one of the above aspects or embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a schematic diagram of the architecture of an audio and video playback system provided by an embodiment of the present disclosure.

[0015] FIG2 is a flow chart of an audio and video synchronization method provided by an embodiment of the present disclosure.

[0016] FIG3 is a schematic structural diagram of an audio and video synchronization device provided by an embodiment of the present disclosure.

[0017] FIG4 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0019] In the description of this disclosure, unless otherwise specified, " / " means "or." For example, A / B can mean A or B. "And / or" herein is merely a description of an association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can mean: only A, only B, and A and B. Furthermore, "at least one" means one or more, and "a plurality" means two or more. Words such as "first" and "second" do not limit the quantity or order of execution, and words such as "first" and "second" do not necessarily mean different.

[0020] It should be noted that, in this disclosure, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in this disclosure as "exemplarily" or "for example" should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, an embodiment or design using "exemplarily" or "for example" should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in some way.

[0021] Cloud computing is a technology that provides computing resources and services to users through the Internet. It concentrates computing, storage, and network resources in data centers. Users can access these resources through the Internet without having to purchase and maintain physical computing devices and servers.

[0022] Cloud computing is an application of cloud computing technology in the client sector. It leverages cloud computing to centralize computing and storage capabilities in the cloud, allowing users to access cloud applications and data through their terminals. Compared to traditional local computers, cloud computing offers powerful computing and storage capabilities, as well as secure and reliable data management and backup.

[0023] With the advancement of computer and communications technologies, more stable networks and greater bandwidth have enabled the widespread application of cloud computing in various fields. Cloud computing has become an integral part of everyday life, penetrating into office, education, and entertainment, raising the bar for audio and video quality. The quality of audio and video presented on cloud computing platforms is also constantly improving. The quality of audio and video on cloud computing depends on multiple factors, including the player, network transmission quality, codec performance, and terminal performance.

[0024] In cloud computing, the server side refers to the cloud's cluster of computing and storage devices. It uses virtualization technology to run multiple virtual machines on the same physical machine, enabling resource sharing and load balancing. The server side also includes the cloud's network transmission equipment and management system, ensuring the stability and reliability of cloud computing services.

[0025] The client refers to the terminal device used by the user, including PCs (personal computers), mobile phones, tablets, etc. The client is primarily responsible for sending commands and data requests to the server, receiving data and responses from the server, and decoding and displaying the data. The client's terminal performance and network transmission quality have a significant impact on the audio and video experience and quality of the cloud computer.

[0026] For example, Figure 1 provides a system architecture for audio and video playback, which is applied to scenarios where online and local videos are played on a cloud computer. In this system architecture, the video application is responsible for video decoding and rendering, while the audio application is responsible for audio decoding and playback. The graphics card driver captures drawing commands and passes them to the server for encoding; the sound card driver captures audio data and passes it to the server for encoding. After encoding, the server transmits the audio and video streams to the client.

[0027] Currently, after receiving audio and video data, the server encodes it using a specific compression algorithm. Generally, audio is encoded using an audio codec, most commonly AAC (Advanced Audio Coding), and video is encoded using a video codec, such as H.264 or H.265. The encoding process compresses audio and video data into the smallest possible bitstream for easier transmission and storage.

[0028] Continuing with Figure 1, the client decodes, plays, and displays the received audio and video streams through thread scheduling. The decoded audio stream is played on the sound card, while the decoded video stream is converted into soft / hard video streams, which are then rendered on the screen. As decoding and rendering proceed, the audio and video data is gradually restored to the original video and audio signals, allowing users to enjoy a high-quality audio and video experience, such as listening to music and watching movies.

[0029] Hard on-screen refers to transferring the hard video stream directly to the graphics card for processing and display through hardware acceleration. Soft on-screen refers to transferring the soft video stream to the central processing unit (CPU) for processing through software, which then displays the processed results on the screen.

[0030] During the long processing of acquisition, encoding, network transmission, decoding, and presentation, differences in terminal decoding performance and network transmission quality can easily cause jitter and asynchrony in sound and picture playback, affecting the user experience.

[0031] The three links (a), (b), and (c) in Figure 1 are the main links that introduce audio and video freezes and audio and video asynchrony. Link (a) is mainly caused by the decoding performance and functional differences of the player software in the virtual machine, while links (b) and (c) are the main directions for optimizing audio and video freezes and asynchrony on the cloud desktop protocol server and client.

[0032] In current cloud computing solutions, the client does not perform additional buffering after receiving the audio stream sent by the server, but directly decodes and transmits it to the local sound card device. This results in a certain amount of noise at the 50Hz frequency when network quality problems occur. This noise problem may reduce audio quality and affect the user experience.

[0033] At the same time, when the client receives and processes the video stream, it only performs a certain amount of buffering during the initial in-band formation. As local decoding consumes resources, the buffer queue is quickly exhausted, and the video stream will also become stuck due to network jitter. This problem is especially obvious in high-quality and poor network environments because the bandwidth of the video stream is large.

[0034] In addition, in step (c) of Figure 1, due to the differences in terminal resource consumption for audio decoding and video decoding, if hard decoding and hard display are not used on clients with poor decoding performance, certain audio and video asynchrony will occur.

[0035] In the existing technology, there are roughly three types of audio and video synchronization solutions. The first is to synchronize the audio and video based on the time information contained in the data packet in the Real-time Transport Protocol (RTP) or Network Time Protocol (NTP) in the transmission protocol, and align the time of the receiving end with the time of the sending end; the second is to cache the audio data and video data at the sending end, merge the audio and video data, and send them to the receiving end in a unified manner. The third is to discard the compensation data processing. When the audio frame is lost, the voice silence frame is compensated. When the video frame is lost, the corresponding audio frame is discarded.

[0036] However, with the rapid development of cloud computing and cloud services, these solutions have limitations in terms of applicability and certain shortcomings. The first solution mainly relies on RTP or NTP for synchronization, which is not applicable to the Transmission Control Protocol (TCP) of cloud computers, and therefore has certain limitations in practical applications. The second solution has high requirements for both sending end resources and network stability. When the network is unstable, the real-time performance of audio and video is also affected to a certain extent. The third solution achieves synchronization by discarding data. Although it can ensure audio and video synchronization, it also directly reduces the quality of the audio and video itself, affecting the user experience.

[0037] In view of this, the present disclosure provides an audio and video synchronization method for the audio and video playback in a cloud computer scenario, which is caused by problems such as unstable network transmission and differences in decoding performance of different terminals, resulting in jitter and asynchrony in sound and picture playback. The method obtains the audio and video synchronization detection results during the audio and video playback; based on the audio and video synchronization detection results, executes the audio and video synchronization strategy corresponding to the audio and video synchronization detection results. The audio and video synchronization strategy is used to adjust the decoding frame rate of the client or the encoding frame rate of the server so that the client caches the video frames. In this way, the cache accumulation of the client video frames is guaranteed, the situation of asynchrony in audio and video playback caused by unstable network transmission or different decoding performance of different terminals is improved, and the user's viewing experience and satisfaction are improved.

[0038] The present disclosure provides an audio and video synchronization method, which is applied to the client in Figure 1. As shown in Figure 2, the method includes the following steps:

[0039] S101. During audio and video playback, obtain audio and video synchronization detection results.

[0040] As a possible implementation method, during the audio and video playback process, the timestamp of the currently played audio frame and the timestamp of the currently played video frame are obtained; based on the timestamp of the currently played audio frame and the timestamp of the currently played video frame, the audio and video synchronization detection result is determined.

[0041] In some embodiments, before the audio and video playback duration reaches a first preset duration, a second indication message sent by the server is periodically received. The second indication message carries the server's encoding frame rate. Based on the second indication message, the client's decoding frame rate is adjusted to equalize the server's encoding frame rate. In this way, when audio and video playback is unstable, the client's decoding frame rate is adjusted in real time to equalize the client's decoding frame rate with the server's encoding frame rate, avoiding problems such as audio and video playback asynchrony or screen freezes caused by deviations between the server's encoding frame rate and the client's decoding frame rate.

[0042] For example, before the audio and video playback time reaches 5 seconds, the encoding frame rate of the server has a certain jitter. The client can receive the second indication information sent by the server once per second, and adjust the decoding frame rate of the client according to the second indication information to make the decoding frame rate of the client equal to the encoding frame rate of the server.

[0043] In some embodiments, after the audio and video playback duration reaches a first preset duration, audio and video synchronization detection results are periodically obtained. For example, after the audio and video playback duration reaches 5 seconds, audio and video synchronization detection results are obtained every 5 seconds. This allows the client's decoding frame rate or the server's encoding frame rate to be adjusted in real time based on the detection results, ensuring a cached accumulation of client video frames.

[0044] In some embodiments, after the audio and video playback duration reaches a first preset duration, the period for obtaining audio and video synchronization detection results is determined based on at least one of terminal performance, network quality, and user needs. For example, for users with high synchronization requirements, the period can be shortened to provide more accurate synchronization; while for users with less stringent synchronization requirements, the period can be appropriately extended to reduce the terminal load.

[0045] It's understandable that on higher-performance terminals, a shorter period can be used to obtain synchronization detection results to achieve more accurate synchronization. For lower-performance terminals, the period can be appropriately extended to reduce terminal load and energy consumption. In poor network conditions, packet loss and lag are more likely to occur, so increasing the period is necessary to minimize these issues. In better network conditions, the period can be appropriately shortened to improve the accuracy and real-time nature of detection results.

[0046] In addition to the above factors, other factors, such as the player type, bit rate, and resolution, can also be considered to determine an appropriate period for obtaining audio and video synchronization detection results, thereby ensuring the quality and effect of audio and video playback. In some embodiments, before the video is played, the audio data corresponding to the video is buffered for a second preset duration. This ensures that the audio frames do not advance too much ahead of the video frames at the beginning.

[0047] In some embodiments, the second preset duration is determined based on network quality and the size of the client's cache space.

[0048] In some embodiments, the smaller the cache space of the client, the shorter the second preset duration, so as to prevent the cache space from being filled up and affecting the playback effect.

[0049] In some embodiments, before a video is played, the server receives scene recognition results from the server and adjusts decoding parameters based on the scene recognition results. Decoding parameters include video clarity, decoding frame rate, and color saturation. This allows the client to use different decoding parameters in different scenarios based on the scene recognition results, providing users with a better viewing experience.

[0050] For example, for scenes that require high definition, the client can choose a higher video definition to provide a clearer and more delicate video effect; for some low-bitrate environments or occasions of watching videos while moving, the client can lower the video definition accordingly to reduce network traffic usage and obtain a smoother experience.

[0051] S102: Based on the audio and video synchronization detection result, execute an audio and video synchronization strategy corresponding to the audio and video synchronization detection result.

[0052] The audio and video synchronization policy is used to adjust the client's decoding frame rate or the server's encoding frame rate so that the client can cache video frames.

[0053] The server's encoding frame rate refers to the number of video frames encoded per second during the video encoding process. The server's encoding frame rate is related to the video sampling rate and frame rate. A high encoding frame rate ensures smoothness and clarity, but also increases bandwidth and storage requirements.

[0054] The client's decoding frame rate refers to the number of frames per second that the client reads and decodes from the video stream during video playback. The client's decoding frame rate is typically related to the video's frame rate. Different video frame rates require different decoding frame rates for smooth playback. If the client's decoding frame rate is too low, the video may freeze or play too fast, affecting the viewing experience.

[0055] In some embodiments, when the audio and video synchronization detection result is that the timestamp of the currently played audio frame is less than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than a preset threshold, the audio and video synchronization strategy is to reduce the decoding frame rate of the client so that the decoding frame rate of the client is less than the encoding frame rate of the server, and cache the video frames.

[0056] In some embodiments, the greater the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame, the greater the reduction in the decoding frame rate of the client.

[0057] In some embodiments, while reducing the decoding frame rate of the client, the difference between the timestamp of the currently playing video frame and the timestamp of the currently playing audio frame is obtained and calculated in real time; when the difference between the timestamp of the currently playing video frame and the timestamp of the currently playing audio frame is within a preset range, the reduction of the decoding frame rate of the client is stopped.

[0058] For example, assuming the preset threshold is 1 second, after the audio and video playback duration reaches 5 seconds, the timestamp of the currently playing audio frame is obtained to be 8 minutes and 5 seconds, and the timestamp of the currently playing video frame is obtained to be 8 minutes and 7 seconds. The timestamp of the currently playing audio frame is less than the timestamp of the currently playing video frame, and the difference between the timestamp of the currently playing video frame and the timestamp of the currently playing audio frame is greater than 1 second. In this case, the audio and video synchronization strategy is to reduce the client's decoding frame rate so that the client's decoding frame rate is less than the server's encoding frame rate.

[0059] Assuming that the encoding frame rate of the server in the previous example is 25 frames per second and the decoding frame rate of the client is 30 frames per second, the decoding frame rate of the client is reduced to 20 frames per second so that the decoding frame rate of the client is less than the encoding frame rate of the server.

[0060] In other embodiments, when the audio and video synchronization detection result shows that the timestamp of the currently playing audio frame is greater than the timestamp of the currently playing video frame, and the difference between the timestamp of the currently playing audio frame and the timestamp of the currently playing video frame is greater than a preset threshold, the audio and video synchronization strategy is to send a first indication message to the server, and the first indication message is used to instruct the server to reduce the encoding frame rate so that the client caches the video frames. In this case, it indicates that the client's decoding performance is poor, and some video data is discarded, resulting in a lag in viewing experience. Therefore, the client sends the first indication message to the server, instructing it to reduce the encoding frame rate of the server, thereby ensuring the continuity and integrity of the encoded video data and achieving audio and video synchronization.

[0061] In some embodiments, the greater the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame, the greater the reduction in the encoding frame rate of the server.

[0062] In some embodiments, while reducing the encoding frame rate of the server, the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is obtained and calculated in real time; when the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is within a preset range, the reduction of the encoding frame rate of the server is stopped.

[0063] For example, assuming the preset threshold is 1 second, after the audio and video playback duration reaches 5 seconds, the timestamp of the currently playing audio frame is obtained to be 6 minutes and 5 seconds, and the timestamp of the currently playing video frame is obtained to be 6 minutes and 2 seconds. The timestamp of the currently playing audio frame is greater than the timestamp of the currently playing video frame, and the difference between the timestamp of the currently playing video frame and the timestamp of the currently playing audio frame is greater than 1 second. Therefore, the client sends a first instruction message to the server, instructing it to reduce the encoding frame rate of the server, thereby ensuring the continuity and integrity of the encoded video data and achieving audio and video synchronization.

[0064] In some embodiments, the preset threshold can be customized or determined based on the client's decoding frame rate and the server's encoding frame rate. A higher decoding frame rate corresponds to a smaller preset threshold to ensure the effect of synchronized audio and video playback.

[0065] In other embodiments, when the audio and video synchronization detection result is that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range, the audio and video synchronization strategy is to adjust the decoding frame rate of the client and the encoding frame rate of the server to be equal.

[0066] Exemplarily, when the audio and video synchronization detection result is that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range, if the decoding frame rate of the client is greater than the encoding frame rate of the server, the decoding frame rate of the client is reduced so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0067] As another example, when the audio and video synchronization detection result is that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range, if the decoding frame rate of the client is less than the encoding frame rate of the server, the decoding frame rate of the client is increased so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0068] As another example, when the audio and video synchronization detection result is that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range, if the decoding frame rate of the client is greater than the encoding frame rate of the server, the server is instructed to increase the encoding frame rate so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0069] As another example, when the audio and video synchronization detection result is that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range, if the decoding frame rate of the client is lower than the encoding frame rate of the server, the server is instructed to lower the encoding frame rate so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0070] For example, after adjusting the decoding frame rate of the client or the encoding frame rate of the server, if the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is less than 1 second, and the decoding frame rate of the client is 20 frames / second and the encoding frame rate of the server is 25 frames / second, then the decoding frame rate of the client is adjusted to 25 frames / second so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0071] Based on this, during the audio and video playback process, the client uses the audio data as a benchmark, and determines the client's decoding frame rate and the server's encoding frame rate based on whether the audio data is ahead or behind, to ensure sufficient caching of video frames. At the start of video playback, the audio frames are cached first to ensure that the audio frames are not too ahead of the video frames at the start. After the video playback stabilizes, if the sound lags behind the video, the client's decoding frame rate is controlled to achieve cache accumulation of video frames; if the sound is ahead of the video, the server's encoding frame rate is controlled to achieve cache accumulation of video frames, which improves the situation of audio and video playback asynchrony caused by unstable network transmission or different decoding performance of different terminals, and enhances the user's viewing experience and satisfaction.

[0072] As an example, in a cloud computer scenario, the audio and video synchronization method may include the following steps:

[0073] Step 1: The server identifies the playback scene of the audio and video and obtains the scene recognition result.

[0074] Step 2: The server sends the scene recognition results to the client.

[0075] Step 3: The server recognizes that the audio and video have started playing, notifies the client that the audio and video playback has started, and sends the encoded sound data and video data to it.

[0076] Step 4: Before playing the video, the client buffers 3 seconds of audio data corresponding to the video.

[0077] Step 5: During video playback, the server records the timestamps corresponding to the audio and video data when capturing sound and images, and sends them to the client together with the audio and video streams after encoding.

[0078] Step 6: Before the audio and video playback duration reaches 5 seconds, the client periodically receives the second indication information sent by the server, and the second indication information carries the encoding frame rate of the server; the client adjusts the decoding frame rate of the client according to the second indication information so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0079] Step 7: After the audio and video playback duration reaches 5 seconds, periodically obtain the audio and video synchronization detection result; wherein the audio and video synchronization detection result is determined based on the timestamp of the currently played audio frame and the timestamp of the currently played video frame.

[0080] Step 8: Based on the audio and video synchronization detection result, the client executes the audio and video synchronization strategy corresponding to the audio and video synchronization detection result.

[0081] (1) When the audio and video synchronization detection result shows that the timestamp of the currently played audio frame is less than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than a preset threshold, the audio and video synchronization strategy is to reduce the decoding frame rate of the client so that the decoding frame rate of the client is less than the encoding frame rate of the server.

[0082] (2) When the audio and video synchronization detection result is that the timestamp of the currently played audio frame is greater than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is greater than a preset threshold, the audio and video synchronization strategy is to send a first indication message to the server, and the first indication message is used to instruct the server to reduce the encoding frame rate so that the client caches the video frame.

[0083] (3) When the audio and video synchronization detection result shows that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range, the audio and video synchronization strategy is to adjust the decoding frame rate of the client and the encoding frame rate of the server to be equal.

[0084] The above mainly introduces the solution of the embodiment of the present disclosure from the perspective of method. An audio and video synchronization device is also shown below, which is used to execute the audio and video synchronization method in any of the above embodiments and possible implementation methods. It can be understood that in order to implement the audio and video synchronization method, the audio and video synchronization device includes hardware structures and / or software modules corresponding to the execution of each function; those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiment of the present disclosure, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0085] The embodiment of the present disclosure can divide the audio and video synchronization device into functional modules according to the above-mentioned method embodiment. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one functional module. The above-mentioned integrated module can be implemented in the form of hardware or software. It should be noted that the division of modules in the embodiment of the present disclosure is schematic and is only a logical function division. There may be other division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0086] FIG3 is a diagram of an audio and video synchronization device provided by an embodiment of the present disclosure. The audio and video synchronization device 20 includes: a communication module 21 and a processing module 22 .

[0087] The communication module 21 is used to obtain the audio and video synchronization detection results during the audio and video playback process;

[0088] The processing module 22 is used to execute an audio and video synchronization strategy corresponding to the audio and video synchronization detection result based on the audio and video synchronization detection result. The audio and video synchronization strategy is used to adjust the decoding frame rate of the client or the encoding frame rate of the server so that the client caches the video frames.

[0089] In some embodiments, the communication module 21 is used to obtain the timestamp of the currently played audio frame and the timestamp of the currently played video frame during the audio and video playback process; and determine the audio and video synchronization detection result based on the timestamp of the currently played audio frame and the timestamp of the currently played video frame.

[0090] In some embodiments, the processing module 22 is used to implement an audio and video synchronization strategy of reducing the decoding frame rate of the client so that the decoding frame rate of the client is less than the encoding frame rate of the server, and caching the video frames when the audio and video synchronization detection result is that the timestamp of the currently played audio frame is less than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than a preset threshold.

[0091] In some embodiments, the processing module 22 is used to, when the audio and video synchronization detection result is that the timestamp of the currently played audio frame is greater than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is greater than a preset threshold, the audio and video synchronization strategy is to send a first indication information to the server, and the first indication information is used to instruct the server to reduce the encoding frame rate so that the client caches the video frame.

[0092] In some embodiments, the processing module 22 is used to adjust the audio and video synchronization strategy to be equal between the decoding frame rate of the client and the encoding frame rate of the server when the audio and video synchronization detection result is that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range.

[0093] In some embodiments, the communication module 21 is further configured to periodically obtain an audio and video synchronization detection result after the audio and video playback duration reaches a first preset duration.

[0094] In some embodiments, the communication module 21 is used to periodically receive a second indication message sent by the server before the audio and video playback duration reaches a first preset duration, where the second indication message carries the encoding frame rate of the server; and according to the second indication message, adjust the decoding frame rate of the client so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0095] In some embodiments, the processing module 22 is configured to buffer audio data corresponding to the video for a second preset duration before the video is played.

[0096] In the case of implementing the functions of the above-mentioned integrated modules in hardware, the embodiments of the present disclosure also provide a possible structure of an electronic device for performing the audio and video synchronization method provided in the embodiments of the present disclosure. As shown in Figure 4, the electronic device 300 includes: a communication interface 303, a processor 302, and a bus 304. In some embodiments, the electronic device may also include a memory 301.

[0097] The processor 302 can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present disclosure. The processor 302 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof, and can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present disclosure. The processor 302 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP (Digital Signal Processor) and a microprocessor, and the like.

[0098] The communication interface 303 is used to connect to other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0099] The memory 301 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0100] As a possible implementation, the memory 301 can exist independently of the processor 302. The memory 301 can be connected to the processor 302 via a bus 304 to store instructions or program codes. When the processor 302 calls and executes the instructions or program codes stored in the memory 301, the audio and video synchronization method provided in the embodiment of the present disclosure can be implemented.

[0101] In another possible implementation, the memory 301 may also be integrated with the processor 302 .

[0102] Bus 304 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 304 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG4 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0103] Some embodiments of the present disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), which stores computer program instructions. When the computer program instructions are executed on a computer, the computer executes the audio and video synchronization method as described in any of the above embodiments.

[0104] In an exemplary embodiment, the computer may be the aforementioned audio and video synchronization device, and the present disclosure does not limit the specific form of the computer.

[0105] In some examples, the computer-readable storage media described above may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memories (EPROMs), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this disclosure may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0106] An embodiment of the present disclosure provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is enabled to execute the audio and video synchronization method described in any one of the above embodiments.

[0107] The above is only a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or replacements within the technical scope disclosed in the present disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. An audio and video synchronization method, comprising: During the audio and video playback process, obtain the audio and video synchronization detection result; Based on the audio and video synchronization detection result, an audio and video synchronization strategy corresponding to the audio and video synchronization detection result is executed, and the audio and video synchronization strategy is used to adjust the decoding frame rate of the client or the encoding frame rate of the server so that the client caches video frames.

2. The method according to claim 1, wherein: During the audio and video playback process, obtaining the audio and video synchronization detection result includes: During the audio and video playback process, obtaining the timestamp of the currently played audio frame and the timestamp of the currently played video frame; The audio and video synchronization detection result is determined based on the timestamp of the currently played audio frame and the timestamp of the currently played video frame.

3. The method according to claim 2, wherein: When the audio and video synchronization detection result is that the timestamp of the currently played audio frame is less than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than a preset threshold, the audio and video synchronization strategy is to reduce the decoding frame rate of the client so that the decoding frame rate of the client is less than the encoding frame rate of the server, and cache the video frames.

4. The method according to claim 2, wherein: When the audio and video synchronization detection result is that the timestamp of the currently played audio frame is greater than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is greater than a preset threshold, the audio and video synchronization strategy is to send a first indication message to the server, and the first indication message is used to instruct to reduce the encoding frame rate of the server so that the client caches video frames.

5. The method according to claim 2, wherein: When the audio and video synchronization detection result is that the difference between the timestamp of the currently playing audio frame and the timestamp of the currently playing video frame is within a preset range, the audio and video synchronization strategy is to adjust the decoding frame rate of the client and the encoding frame rate of the server to be equal.

6. The method according to claim 1, wherein: During the audio and video playback process, obtaining the audio and video synchronization detection result includes: After the audio and video playback duration reaches a first preset duration, the audio and video synchronization detection result is periodically obtained.

7. The method according to claim 6, further comprising: Before the audio and video playback duration reaches the first preset duration, periodically receiving second indication information sent by the server, where the second indication information carries the encoding frame rate of the server; According to the second indication information, the decoding frame rate of the client is adjusted so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

8. The method according to claim 1, further comprising: Before the video is played, the audio data corresponding to the video is buffered for a second preset time length.

9. An electronic device, comprising: a processor and a memory for storing instructions executable by the processor; The processor is configured to execute the instruction so that the electronic device performs the audio and video synchronization method according to any one of claims 1-8.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the audio and video synchronization method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Corrected streaming media audio and video synchronizing method and apparatus

    CN106603543A

  • Audio and video synchronization method, audio and video synchronization apparatus and audio and video synchronization device

    CN107801080A

  • HEVC video playing method and device, terminal and storage medium

    CN109819315A

  • Frame loss processing method, video playing terminal and computer readable storage medium

    CN112073823A

  • Method and device for synchronously processing audio and video, and storage medium

    WO2018120557A1

Cited By

  • Animation scene real-time rendering and audio synchronization system

    CN120711215A

  • Image compensation method, system and device and computer readable storage medium

    CN121585827A