Audio and video synchronization method and device, electronic equipment and storage medium

Abstract

The invention provides an audio and video synchronization method and device, electronic equipment and a storage medium, relates to the technical field of information processing, and is used for reducing the situation of asynchronization in audio and video playing. The method comprises the following steps: acquiring an audio and video synchronization detection result in an audio and video playing process; based on the audio and video synchronization detection result, an audio and video synchronization strategy corresponding to the audio and video synchronization detection result is executed, and the audio and video synchronization strategy is used for adjusting the video decoding frame frequency of the client or the video coding frame frequency of the server, so that the client caches the video frame.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to an audio and video synchronization method, device, electronic device and storage medium. Background Art

[0002] In the process of playing online videos and local videos in the cloud computer, the transmission process of sound and picture includes multiple links, such as the sound card capturing the original audio and the graphics card capturing the changes in the picture, and transmitting them to the terminal as separate audio streams and video streams, respectively. After being transmitted to the terminal, the audio and video are decoded and presented on the terminal for auditory and visual perception. In the long process of acquisition, encoding, network transmission, decoding, and presentation, the differences in terminal decoding performance and network transmission quality can easily cause jitter and asynchrony in sound and picture playback, affecting the user experience. Summary of the invention

[0003] Embodiments of the present application provide an audio and video synchronization method, device, electronic device, and storage medium for reducing the occurrence of asynchronism in audio and video playback.

[0004] On the one hand, a method for synchronizing audio and video is provided, the method comprising:

[0005] During the audio and video playback process, obtain the audio and video synchronization detection result;

[0006] Based on the audio and video synchronization detection result, an audio and video synchronization strategy corresponding to the audio and video synchronization detection result is executed. The audio and video synchronization strategy is used to adjust the decoding frame rate of the client or the encoding frame rate of the server so that the client caches the video frames.

[0007] On the other hand, there is provided an audio and video synchronization device, the device comprising:

[0008] The communication module is used to obtain the audio and video synchronization detection results during the audio and video playback process;

[0009] The processing module is used to execute the audio and video synchronization strategy corresponding to the audio and video synchronization detection result based on the audio and video synchronization detection result. The audio and video synchronization strategy is used to adjust the client video decoding frame rate or the server video encoding frame rate so that the client caches the video frame.

[0010] On the other hand, an electronic device is provided, comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program instructions executable by the processor; and the processor implements the audio and video synchronization method of any of the above embodiments when executing the computer program instructions.

[0011] On the other hand, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed on an electronic device, the audio and video synchronization method of any of the above embodiments is implemented.

[0012] On the other hand, a computer program product is provided, which includes computer program instructions, and when the computer program instructions are executed, the audio and video synchronization method of any of the above embodiments is implemented.

[0013] The technical solution provided by the embodiment of the present application implements the audio and video synchronization strategy corresponding to the audio and video synchronization detection result during the audio and video playback process, adjusts the decoding frame rate of the client or the encoding frame rate of the server, so that the client caches the video frame. In this way, the cache accumulation of the client video frame is guaranteed, the situation of audio and video playback asynchronization caused by unstable network transmission or different decoding performance of different terminals is improved, and the user's viewing experience and satisfaction are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of the architecture of an audio and video playback system provided in an embodiment of the present application;

[0015] Figure 2 A flowchart of an audio and video synchronization method provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of the structure of an audio and video synchronization device provided in an embodiment of the present application;

[0017] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different.

[0020] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of "exemplary" or "for example" should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0021] Cloud computing is a technology that provides computing resources and services to users through the Internet. It concentrates computing, storage, network and other resources in data centers. Users can access these resources through the Internet without purchasing and maintaining physical computing devices and servers.

[0022] Cloud computing is an application of cloud computing technology in the client field. It uses cloud computing technology to concentrate computing and storage processing capabilities in the cloud, so that users can access cloud applications and data through terminals. Compared with traditional local computers, the advantages of cloud computers lie in their powerful computing and storage capabilities, as well as secure and reliable data management and backup.

[0023] With the development of computer technology and communication technology, more stable networks and larger bandwidths have made cloud computing increasingly widely used in various fields. Cloud computers have also entered people's lives, infiltrating into office, education, entertainment and other aspects, and have put forward higher requirements for the audio and video viewing experience of cloud computers. The quality of audio and video presentation in cloud computers is also constantly improving. The quality of audio and video presentation in cloud computers depends on many aspects, such as players, network transmission quality, codec performance, terminal performance, etc.

[0024] In cloud computing, the server refers to the computing and storage device cluster in the cloud. It uses virtualization technology to run multiple virtual machines on the same physical machine to achieve resource sharing and load balancing. The server also includes the network transmission equipment and management system in the cloud to ensure the stability and reliability of cloud computing services.

[0025] The client refers to the terminal device used by the user, including PCs, mobile phones, tablets, etc. The client is mainly responsible for sending instructions and data requests to the server, receiving data and responses returned by the server, and decoding and displaying the data. The terminal performance and network transmission quality of the client have a great impact on the audio and video perception and quality of the cloud computer.

[0026] For example, Figure 1 A system architecture for audio and video playback is provided, which is applied to the scenario of playing online videos and local videos in a cloud computer. In the system architecture, the video application is used to decode the video and render, and the audio application is used to decode the audio and play the sound. The graphics card driver captures the drawing instructions and passes them to the server for encoding; the sound card driver captures the audio data and passes it to the server for encoding. After encoding, the server transmits the audio stream and the video stream to the client.

[0027] At present, after receiving the audio and video data, the server encodes it according to a specific compression algorithm. Generally speaking, audio encoding uses an audio encoder, the most commonly used is AAC encoding, and video encoding uses a video encoder, such as H.264 or H.265 encoding. The encoding process compresses the audio and video data into the smallest possible bit stream for easy transmission and storage.

[0028] Continue to refer Figure 1 , the client decodes, plays and displays the received audio and video streams through thread scheduling. After being decoded, the audio stream is played on the sound card device, and the video stream is decoded to obtain the soft / hard video stream and then rendered on the screen soft / hard. As decoding and rendering proceed, the audio and video data is gradually restored to the original video and audio signals, and users can finally enjoy high-quality audio and video experience by listening to music, watching movies, etc.

[0029] Among them, hard on-screen means that the hard video stream is directly transferred to the graphics card for processing and display through hardware acceleration. Soft on-screen means that the soft video stream is transferred to the central processing unit (CPU) for processing through software, and then the CPU displays the processing results on the screen.

[0030] Figure 1 The three links (a), (b), and (c) are the main links that introduce audio and video freezes and audio and video asynchrony. Link (a) is mainly caused by the decoding performance and functional differences of the player software in the virtual machine, while links (b) and (c) are the main directions for optimizing audio and video freezes and asynchrony on the cloud desktop protocol server and client.

[0031] In the current cloud computing solution, the client does not perform additional buffering after receiving the audio code stream sent by the server, but directly decodes and transmits it to the local sound card device. This results in a certain amount of noise at the 50HZ frequency when there is a quality problem with the network. This noise problem may reduce the audio quality and affect the user experience.

[0032] At the same time, when the client receives and processes the video code stream, it only does a certain amount of buffering when it is initially formed in the band. As the local decoding is consumed, the buffer queue is quickly exhausted, and the video stream will also become stuck due to network jitter. This stuck problem will be more obvious, especially in a high-quality network environment, because the bandwidth of the video stream is larger.

[0033] In addition, Figure 1 In the (c) step, due to the difference in terminal resource consumption of audio decoding and video decoding, if hard decoding and hard display are not used on the client with poor decoding, a certain audio and video asynchrony will occur.

[0034] In the prior art, there are roughly three solutions for audio and video synchronization. The first is to synchronize and calibrate the audio and video based on the time information in the data packet in the Real-time Transport Protocol (RTP) or Network Time Protocol (NTP) in the transmission protocol, and align the time of the receiving end with the time of the sending end; the second is to cache the audio data and video data at the sending end, merge the audio and video data, and send them to the receiving end. The third is to discard the compensation data processing. When the audio frame is lost, the voice silence frame is compensated, and when the video frame is lost, the corresponding audio frame is discarded.

[0035] However, with the rapid development of cloud computing and cloud services, these solutions have their applicability and certain shortcomings. The first solution mainly relies on RTP or NTP for synchronization, which is not applicable to the Transmission Control Protocol (TCP) of cloud computers, so there are certain limitations in practical applications. The second solution has high requirements on the sending end resources and network stability. When the network is unstable, the real-time performance of audio and video is also affected to a certain extent. The third solution achieves synchronization by discarding data. Although it can ensure audio and video synchronization, it also directly reduces the quality of the audio and video itself, affecting the user experience.

[0036] In view of this, the present invention provides an audio and video synchronization method for the jitter and asynchronism of sound and picture playback caused by unstable network transmission, different decoding performance differences of different terminals, etc. during audio and video playback in a cloud computer scenario. The method obtains the audio and video synchronization detection result during the audio and video playback; based on the audio and video synchronization detection result, an audio and video synchronization strategy corresponding to the audio and video synchronization detection result is executed, and the audio and video synchronization strategy is used to adjust the decoding frame rate of the client or the encoding frame rate of the server so that the client caches the video frame. In this way, the cache accumulation of the client video frame is guaranteed, the situation of asynchronism of audio and video playback caused by unstable network transmission or different decoding performance of different terminals is improved, and the user's viewing experience and satisfaction are improved.

[0037] The embodiment of the present invention provides an audio and video synchronization method, which is applied to Figure 1 Client. Figure 2 As shown, the method comprises the following steps:

[0038] S101. During audio and video playback, obtain audio and video synchronization detection results.

[0039] As a possible implementation method, during the audio and video playback process, the timestamp of the currently played audio frame and the timestamp of the currently played video frame are obtained; based on the timestamp of the currently played audio frame and the timestamp of the currently played video frame, the audio and video synchronization detection result is determined.

[0040] In some embodiments, before the audio and video playback duration reaches the first preset duration, the second indication information sent by the server is periodically received, and the second indication information carries the encoding frame rate of the server; according to the second indication information, the decoding frame rate of the client is adjusted so that the decoding frame rate of the client is equal to the encoding frame rate of the server. In this way, when the audio and video playback is unstable, the decoding frame rate of the client is adjusted in real time so that the decoding frame rate of the client is equal to the encoding frame rate of the server, avoiding problems such as audio and video playback being out of sync or screen freezes due to deviations between the encoding frame rate of the server and the decoding frame rate of the client.

[0041] For example, before the audio and video playback duration reaches 5 seconds, the encoding frame rate of the server has a certain jitter. The client can receive the second indication information sent by the server once per second, and adjust the decoding frame rate of the client according to the second indication information to make the decoding frame rate of the client equal to the encoding frame rate of the server.

[0042] In some embodiments, after the audio and video playback duration reaches a first preset duration, the audio and video synchronization detection result is periodically obtained. For example, after the audio and video playback duration reaches 5 seconds, the audio and video synchronization detection result is obtained every 5 seconds. In this way, the decoding frame rate of the client or the encoding frame rate of the server can be adjusted in real time according to the detection result to ensure the cache accumulation of the client video frame.

[0043] In some embodiments, after the audio and video playback duration reaches a first preset duration, the period for obtaining the audio and video synchronization detection result is determined according to at least one of the terminal performance, network quality, and user requirements. For example, for users with high synchronization requirements, the period can be shortened to provide a more accurate synchronization effect; and for users with low synchronization requirements, the period can be appropriately extended to reduce the load on the terminal.

[0044] It is understandable that on a terminal with higher performance, a shorter cycle can be used to obtain synchronization detection results to achieve more accurate synchronization. For a terminal with lower performance, the cycle can be appropriately extended to reduce the load and energy consumption of the terminal. In the case of poor network quality, problems such as packet loss and freezing are prone to occur, so the cycle needs to be increased to avoid these problems as much as possible. In the case of good network quality, the cycle can be appropriately shortened to improve the accuracy and real-time nature of the detection results.

[0045] In addition to the above factors, other factors may be considered, such as the type of player, bit rate, resolution, etc., to determine a suitable period for obtaining the audio and video synchronization detection result, thereby ensuring the quality and effect of audio and video playback. In some embodiments, before the video is played, the audio data corresponding to the video of the second preset duration is buffered. This ensures that the audio frame does not lead the video frame too much in the initial stage.

[0046] In some embodiments, the second preset duration is determined based on network quality and a cache space size of the client.

[0047] In some embodiments, the smaller the cache space of the client, the shorter the second preset duration, so as to avoid the cache space being filled up and affecting the playback effect.

[0048] In some embodiments, before the video is played, the scene recognition result sent by the server is received; according to the scene recognition result, the decoding parameters are adjusted. The decoding parameters include the clarity, decoding frame rate and color saturation of the video. In this way, the client can use different decoding parameters in different scenes according to the scene recognition result to provide users with better viewing effects.

[0049] For example, for scenes that require high definition, the client can choose a higher video definition to provide a clearer and more delicate video effect; for some low-bitrate environments or occasions where videos are watched on the move, the client can lower the video definition accordingly to reduce network traffic usage and obtain a smoother experience.

[0050] S102: Based on the audio and video synchronization detection result, execute an audio and video synchronization strategy corresponding to the audio and video synchronization detection result.

[0051] The audio and video synchronization strategy is used to adjust the decoding frame rate of the client or the encoding frame rate of the server so that the client caches the video frames.

[0052] The encoding frame rate of the server refers to the number of video frames encoded by the server per second during the video encoding process. The encoding frame rate of the server is related to the video sampling rate and the video frame rate. A high encoding frame rate can ensure the smoothness and clarity of the video, but it will also increase the bandwidth and storage requirements of the video.

[0053] The client's decoding frame rate refers to the number of video frames that the client reads and decodes from the video stream per second when playing the video. The client's decoding frame rate is usually related to the video's frame rate. Different video frame rates require different decoding frame rates to achieve smooth playback. If the client's decoding frame rate is too low, it may cause video freezes or the video playback speed is too fast, affecting the viewing experience.

[0054] In some embodiments, when the audio and video synchronization detection result is that the timestamp of the currently played audio frame is less than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than a preset threshold, the audio and video synchronization strategy is to reduce the decoding frame rate of the client so that the decoding frame rate of the client is less than the encoding frame rate of the server, and cache the video frames.

[0055] In some embodiments, the greater the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame, the greater the reduction in the decoding frame rate of the client.

[0056] In some embodiments, while reducing the decoding frame rate of the client, the difference between the timestamp of the currently playing video frame and the timestamp of the currently playing audio frame is obtained and calculated in real time; when the difference between the timestamp of the currently playing video frame and the timestamp of the currently playing audio frame is within a preset range, the reduction of the decoding frame rate of the client is stopped.

[0057] Exemplarily, assuming that the preset threshold is 1 second, after the audio and video playback time reaches 5 seconds, the timestamp of the currently played audio frame is 8 minutes and 5 seconds, and the timestamp of the currently played video frame is 8 minutes and 7 seconds. The timestamp of the currently played audio frame is less than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than 1 second. In this case, the audio and video synchronization strategy is to reduce the decoding frame rate of the client so that the decoding frame rate of the client is less than the encoding frame rate of the server.

[0058] Assuming that the encoding frame rate of the server in the previous example is 25 frames / second and the decoding frame rate of the client is 30 frames / second, the decoding frame rate of the client is reduced to 20 frames / second so that the decoding frame rate of the client is less than the encoding frame rate of the server.

[0059] In other embodiments, when the audio and video synchronization detection result is that the timestamp of the currently played audio frame is greater than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is greater than a preset threshold, the audio and video synchronization strategy is to send a first indication message to the server, and the first indication message is used to indicate to reduce the encoding frame rate of the server so that the client caches the video frame. In this case, it means that the decoding performance of the client is poor, part of the video data is discarded, resulting in a stuttering experience, so the client sends the first indication message to the server, indicating to reduce the encoding frame rate of the server, thereby ensuring the continuity and integrity of the encoded video data and achieving audio and video synchronization.

[0060] In some embodiments, the greater the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame, the greater the reduction in the encoding frame rate of the server.

[0061] In some embodiments, while reducing the encoding frame rate of the server, the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is obtained and calculated in real time; when the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is within a preset range, the reduction of the encoding frame rate of the server is stopped.

[0062] For example, assuming that the preset threshold is 1 second, after the audio and video playback time reaches 5 seconds, the timestamp of the currently played audio frame is 6 minutes and 5 seconds, and the timestamp of the currently played video frame is 6 minutes and 2 seconds. The timestamp of the currently played audio frame is greater than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than 1 second. Therefore, the server sends a first indication message to the client, indicating to reduce the encoding frame rate of the server, thereby ensuring the continuity and integrity of the encoded video data and achieving audio and video synchronization.

[0063] In some embodiments, the preset threshold can be customized or determined according to the decoding frame rate of the client and the encoding frame rate of the server. The higher the decoding frame rate, the smaller the preset threshold corresponding to ensure the effect of synchronous audio and video playback.

[0064] In other embodiments, when the audio and video synchronization detection result is that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range, the audio and video synchronization strategy is to adjust the decoding frame rate of the client and the encoding frame rate of the server to be equal.

[0065] Exemplarily, when the audio and video synchronization detection result is that the difference between the timestamp of the currently playing audio frame and the timestamp of the currently playing video frame is within a preset range, if the decoding frame rate of the client is greater than the encoding frame rate of the server, the decoding frame rate of the client is reduced so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0066] As another example, when the audio and video synchronization detection result is that the difference between the timestamp of the currently playing audio frame and the timestamp of the currently playing video frame is within a preset range, if the decoding frame rate of the client is lower than the encoding frame rate of the server, the decoding frame rate of the client is increased so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0067] As another example, when the audio and video synchronization detection result is that the difference between the timestamp of the currently playing audio frame and the timestamp of the currently playing video frame is within a preset range, if the decoding frame rate of the client is greater than the encoding frame rate of the server, the server is instructing to increase the encoding frame rate so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0068] As another example, when the audio and video synchronization detection result is that the difference between the timestamp of the currently playing audio frame and the timestamp of the currently playing video frame is within a preset range, if the decoding frame rate of the client is lower than the encoding frame rate of the server, the server is instructed to reduce the encoding frame rate so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0069] For example, after adjusting the decoding frame rate of the client or the encoding frame rate of the server, if the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is less than 1 second, and the decoding frame rate of the client is 20 frames / second and the encoding frame rate of the server is 25 frames / second, the decoding frame rate of the client is adjusted to 25 frames / second so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0070] Based on this, during the audio and video playback process, the client uses the audio data as a benchmark, and determines the client's decoding frame rate and the server's encoding frame rate based on the advance or lag of the audio data to ensure sufficient caching of video frames. At the start of video playback, the audio frames are cached first to ensure that the audio frames are not too far ahead of the video frames at the start. After the video playback is stable, if the sound lags behind the video, the client's decoding frame rate is controlled to achieve cache accumulation of video frames; if the sound is ahead of the video, the server's encoding frame rate is controlled to achieve cache accumulation of video frames, which improves the situation of audio and video playback asynchrony caused by unstable network transmission or different decoding performance of different terminals, and improves the user's viewing experience and satisfaction.

[0071] As an example, in a cloud computer scenario, the audio and video synchronization method specifically includes the following steps:

[0072] Step 1: The server identifies the playback scene of the audio and video and obtains the scene recognition result.

[0073] Step 2: The server sends the scene recognition result to the client.

[0074] Step 3: The server recognizes that the audio and video starts playing, notifies the client that the audio and video starts playing, and sends the encoded sound data and video data to it.

[0075] Step 4: Before playing the video, the client buffers 3 seconds of audio data corresponding to the video.

[0076] Step 5: During video playback, the server records the timestamps corresponding to the audio data and video data when capturing sound and images, and sends them to the client together with the audio and video streams after encoding.

[0077] Step 6: Before the audio and video playback duration reaches 5 seconds, the client periodically receives the second indication information sent by the server, where the second indication information carries the encoding frame rate of the server; the client adjusts the decoding frame rate of the client according to the second indication information so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

[0078] Step 7: After the audio and video playback duration reaches 5 seconds, periodically obtain the audio and video synchronization detection result; wherein the audio and video synchronization detection result is determined based on the timestamp of the currently played audio frame and the timestamp of the currently played video frame.

[0079] Step 8: Based on the audio and video synchronization detection result, the client executes the audio and video synchronization strategy corresponding to the audio and video synchronization detection result.

[0080] (1) When the audio and video synchronization detection result is that the timestamp of the currently played audio frame is less than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than a preset threshold, the audio and video synchronization strategy is to reduce the decoding frame rate of the client so that the decoding frame rate of the client is less than the encoding frame rate of the server.

[0081] (2) When the audio and video synchronization detection result is that the timestamp of the currently played audio frame is greater than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is greater than a preset threshold, the audio and video synchronization strategy is to send a first indication message to the server, and the first indication message is used to instruct the server to reduce the encoding frame rate so that the client caches the video frames.

[0082] (3) When the audio and video synchronization detection result is that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range, the audio and video synchronization strategy is to adjust the decoding frame rate of the client and the encoding frame rate of the server to be equal.

[0083] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the method. The following also shows an audio and video synchronization device for executing the audio and video synchronization method in any of the above embodiments and possible implementations thereof. It can be understood that in order to implement the audio and video synchronization method, the audio and video synchronization device includes a hardware structure and / or software module corresponding to each function; those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiment of the present application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0084] The embodiment of the present application can divide the audio and video synchronization device into functional modules according to the above method embodiment. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one functional module. The above integrated module can be implemented in the form of hardware or software. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0085] Figure 3The embodiment of the present application provides an audio and video synchronization device. The audio and video synchronization device 20 includes: a communication module 21 and a processing module 22.

[0086] The communication module 21 is used to obtain the audio and video synchronization detection result during the audio and video playback process;

[0087] The processing module 22 is used to execute an audio and video synchronization strategy corresponding to the audio and video synchronization detection result based on the audio and video synchronization detection result. The audio and video synchronization strategy is used to adjust the decoding frame rate of the client or the encoding frame rate of the server so that the client caches the video frame.

[0088] In some embodiments, the communication module 21 is used to obtain the timestamp of the currently played audio frame and the timestamp of the currently played video frame during the audio and video playback process; based on the timestamp of the currently played audio frame and the timestamp of the currently played video frame, determine the audio and video synchronization detection result.

[0089] In some embodiments, the processing module 22 is used to implement an audio and video synchronization strategy of reducing the decoding frame rate of the client so that the decoding frame rate of the client is less than the encoding frame rate of the server, and caching the video frames when the audio and video synchronization detection result is that the timestamp of the currently played audio frame is less than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than a preset threshold.

[0090] In some embodiments, the processing module 22 is used to, when the audio and video synchronization detection result is that the timestamp of the currently played audio frame is greater than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is greater than a preset threshold, the audio and video synchronization strategy is to send a first indication message to the server, and the first indication message is used to instruct the server to reduce the encoding frame rate so that the client caches the video frames.

[0091] In some embodiments, the processing module 22 is used to adjust the audio and video synchronization strategy to make the decoding frame rate of the client and the encoding frame rate of the server equal when the audio and video synchronization detection result is that the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is within a preset range.

[0092] In some embodiments, the communication module 21 is further used to periodically obtain the audio and video synchronization detection result after the audio and video playback duration reaches a first preset duration.

[0093] In some embodiments, the communication module 21 is used to periodically receive a second indication message sent by the server before the audio and video playback duration reaches a first preset duration, the second indication message carrying the encoding frame rate of the server; according to the second indication message, the decoding frame rate of the client is adjusted to make the decoding frame rate of the client equal to the encoding frame rate of the server.

[0094] In some embodiments, the processing module 22 is used to buffer the audio data corresponding to the video of a second preset duration before the video is played.

[0095] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, the embodiment of the present application also provides a possible structure of an electronic device, which is used to execute the audio and video synchronization method provided by the embodiment of the present application. Figure 4 As shown, the electronic device 300 includes: a communication interface 303, a processor 302 and a bus 304. Optionally, the electronic device may further include a memory 301.

[0096] The processor 302 may be a processor that implements or executes various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present application. The processor 302 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present application. The processor 302 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0097] The communication interface 303 is used to connect with other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0098] The memory 301 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0099] As a possible implementation, the memory 301 can exist independently of the processor 302, and the memory 301 can be connected to the processor 302 via a bus 304 to store instructions or program codes. When the processor 302 calls and executes the instructions or program codes stored in the memory 301, the audio and video synchronization method provided in the embodiment of the present application can be implemented.

[0100] In another possible implementation, the memory 301 may also be integrated with the processor 302. The bus 304 may be an extended industry standard architecture (EISA) bus, etc. The bus 304 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0101] Some embodiments of the present application provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), which stores computer program instructions. When the computer program instructions are executed on a computer, the computer executes the audio and video synchronization method as described in any of the above embodiments.

[0102] In an exemplary embodiment, the computer may be the above-mentioned audio and video synchronization device, and the present application does not limit the specific form of the computer.

[0103] In some examples, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes, etc.), optical disks (e.g., compact disks (CD), digital versatile disks (DVD), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this application may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0104] An embodiment of the present application provides a computer program product including instructions. When the computer program product is run on a computer, the computer is enabled to execute the audio and video synchronization method described in any of the above embodiments.

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An audio and video synchronization method, characterized in that: The method comprises: During audio and video playback, obtain audio and video synchronization detection results; Based on the audio and video synchronization detection result, an audio and video synchronization strategy corresponding to the audio and video synchronization detection result is executed, and the audio and video synchronization strategy is used to adjust the client video decoding frame rate or the server video encoding frame rate so that the client caches video frames.

2. The method according to claim 1, characterized in that The step of obtaining the audio and video synchronization detection result during the audio and video playback process includes: During audio and video playback, obtain the timestamp of the currently playing audio frame and the timestamp of the currently playing video frame; The audio and video synchronization detection result is determined based on the timestamp of the currently played audio frame and the timestamp of the currently played video frame.

3. The method according to claim 2, characterized in that When the audio and video synchronization detection result is that the timestamp of the currently played audio frame is less than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played video frame and the timestamp of the currently played audio frame is greater than a preset threshold, the audio and video synchronization strategy is to reduce the decoding frame rate of the client so that the decoding frame rate of the client is less than the encoding frame rate of the server, and cache the video frames.

4. The method according to claim 2, characterized in that: When the audio and video synchronization detection result is that the timestamp of the currently played audio frame is greater than the timestamp of the currently played video frame, and the difference between the timestamp of the currently played audio frame and the timestamp of the currently played video frame is greater than a preset threshold, the audio and video synchronization strategy is to send a first indication message to the server, and the first indication message is used to instruct to reduce the encoding frame rate of the server so that the client caches video frames.

5. The method according to claim 2, characterized in that: When the audio and video synchronization detection result is that the difference between the timestamp of the currently playing audio frame and the timestamp of the currently playing video frame is within a preset range, the audio and video synchronization strategy is to adjust the decoding frame rate of the client and the encoding frame rate of the server to be equal.

6. The method according to claim 1, characterized in that The step of obtaining the audio and video synchronization detection result during the audio and video playback process includes: After the audio and video playback duration reaches a first preset duration, the audio and video synchronization detection result is periodically obtained.

7. The method according to claim 6, characterized in that The method further comprises: Before the audio and video playback duration reaches the first preset duration, periodically receiving second indication information sent by the server, where the second indication information carries the encoding frame rate of the server; According to the second indication information, the decoding frame rate of the client is adjusted so that the decoding frame rate of the client is equal to the encoding frame rate of the server.

8. The method according to claim 1, characterized in that The method further comprises: Before the video is played, the audio data corresponding to the video is buffered for a second preset time length.

9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory for storing instructions executable by the processor; The processor is configured to execute the instruction so that the electronic device executes the audio and video synchronization method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the audio and video synchronization method according to any one of claims 1 to 8.

Images