Audio and video synchronous playing method and system

Abstract

The embodiment of the invention discloses an audio and video synchronous playing method and system, and the method comprises the steps: collecting video data and audio data through a video collection thread and an audio collection thread respectively; transmitting the video data and the audio data to a video coding thread and an audio coding thread respectively for coding, and storing the coded data in a video buffer area and an audio buffer area; wherein the size of the data buffer area is adjusted in real time according to performance indexes and the actual water level of the buffer area; the data buffer area comprises a video buffer area and an audio buffer area for synchronizing the video data and the audio data, and respectively inputting the synchronized video data and audio data into a video decoding thread and an audio decoding thread for decoding; and outputting and combining the decoded video data and audio data to obtain an audio and video stream, and packaging or transmitting the audio and video stream. And the data processing efficiency and the user experience are improved.

Description

Technical Field

[0001] The present invention relates to the field of multimedia, and in particular to a method and system for synchronously playing audio and video. Background Art

[0002] Traditional audio and video recording and playback systems usually adopt a single-threaded architecture, which executes operations such as acquisition, encoding, decoding, and rendering serially. This is inefficient and prone to problems such as freezes and delays.

[0003] In recent years, multithreading technology has been widely used in the field of audio and video processing, which can effectively improve system efficiency and real-time performance. However, existing multithreaded audio and video processing solutions usually implement multithreaded processing based on established rules, which makes it difficult to give full play to the advantages of multithreading, especially in scenarios with high requirements for real-time performance and stability, such as live webcasts and online conferences. Problems such as freezes, screen distortion, and delays are prone to occur.

[0004] There is currently no effective solution to the above problems in the prior art. Summary of the invention

[0005] To solve the above problems, the present invention provides a method and system for synchronous audio and video playback, which dynamically adjusts the buffer size through the collaboration of a strategy switching algorithm and a buffer size adjustment algorithm to ensure data processing efficiency and reduce latency, thereby solving the problem of performance degradation caused by unreasonable resource allocation in the multi-threaded audio and video processing process in the prior art.

[0006] To achieve the above-mentioned purpose, the present invention provides a method for synchronous audio and video playback, comprising: collecting video data from a camera through a video acquisition thread, and collecting audio data from a microphone through an audio acquisition thread; transmitting the video data and audio data to a video encoding thread and an audio encoding thread for encoding processing, respectively, and storing the encoded data in corresponding video buffers and audio buffers, respectively; wherein the size of the data buffer is adjusted in real time according to performance indicators and actual buffer water levels; the data buffer includes a video buffer and an audio buffer; synchronously processing the video data in the video buffer and the audio data in the audio buffer, and inputting the synchronized video data and audio data into a video decoding thread and an audio decoding thread for decoding processing, respectively; transmitting the decoded video data to a video rendering thread for processing, and transmitting the decoded audio data to an audio output thread for processing; merging the decoded audio data and video data to obtain an audio and video stream, and encapsulating or transmitting the audio and video stream.

[0007] Further optionally, the size of the data buffer is adjusted in real time according to the performance indicator and the actual buffer water level, including: obtaining the performance indicator and the actual buffer water level of the data buffer in real time; selecting the current buffer strategy of the data buffer from all buffer strategies according to the performance indicator; and calculating the current buffer size of the data buffer according to the actual buffer water level.

[0008] Further optionally, calculating the current buffer size of the data buffer based on the actual buffer water level includes: calculating a buffer water level deviation value based on the actual buffer water level and a target buffer water level; wherein the target buffer water level is determined based on a current buffer strategy; calculating an adjustment amount of the buffer size based on the buffer water level deviation value; and calculating the current buffer size based on the actual buffer water level and the adjustment amount.

[0009] Further optionally, the adjustment amount of the buffer size is calculated according to the buffer water level deviation value, and is calculated by the following formula:

[0010]

[0011] Where u(t) is the adjustment amount of the buffer size, K p , K i , K d is the coefficient, and e(t) is the buffer water level deviation value.

[0012] Further optionally, the synchronous processing of the video data in the video buffer and the audio data in the audio buffer includes: taking out the video data and audio data with timestamps from the video buffer and the audio buffer respectively through a message queue mechanism; and comparing and adjusting the timestamp of the video data with the timestamp of the audio data so that the video data and the audio data are played synchronously.

[0013] On the other hand, the present invention also provides an audio and video synchronous playback system, including: a data acquisition module, used to acquire video data from a camera through a video acquisition thread, and to acquire audio data from a microphone through an audio acquisition thread; an encoding module, used to transmit the video data and audio data to a video encoding thread and an audio encoding thread for encoding processing, respectively, and store the encoded data in corresponding video buffers and audio buffers, respectively; wherein the size of the data buffer is adjusted in real time according to performance indicators and actual buffer water levels; the data buffer includes a video buffer and an audio buffer; a decoding module, used to synchronize the video data in the video buffer and the audio data in the audio buffer, and input the synchronized video data and audio data to a video decoding thread and an audio decoding thread for decoding processing, respectively; a rendering output module, used to transmit the decoded video data to a video rendering thread for processing, and to transmit the decoded audio data to an audio output thread for processing; a multiplexing module, used to merge the decoded audio data and video data for audio and video data to obtain an audio and video stream, and encapsulate or transmit the audio and video stream.

[0014] Further optionally, the encoding module includes: a performance indicator acquisition submodule, used to acquire the performance indicator and actual buffer water level of the data buffer in real time; a buffer strategy switching submodule, used to select the current buffer strategy of the data buffer from all buffer strategies according to the performance indicator; and a buffer size calculation submodule, used to calculate the current buffer size of the data buffer according to the actual buffer water level.

[0015] Further optionally, the buffer size calculation submodule includes: a deviation value calculation unit, used to calculate the buffer water level deviation value based on the actual buffer water level and the target buffer water level; wherein the target buffer water level is determined according to the current buffer strategy; an adjustment amount calculation unit, used to calculate the adjustment amount of the buffer size based on the buffer water level deviation value; and a buffer size adjustment unit, used to calculate the current buffer size based on the actual buffer water level and the adjustment amount.

[0016] Further optionally, the adjustment amount of the buffer size is calculated according to the buffer water level deviation value, and is calculated by the following formula:

[0017]

[0018] Where u(t) is the adjustment amount of the buffer size, K p , K i , K d is the coefficient, and e(t) is the buffer water level deviation value.

[0019] Further optionally, the synchronous processing of the video data in the video buffer and the audio data in the audio buffer includes: a first synchronization submodule, used to take out the video data and audio data with timestamps from the video buffer and the audio buffer respectively through a message queue mechanism; a second synchronization submodule, used to compare and adjust the timestamp of the video data with the timestamp of the audio data so that the video data and the audio data are played synchronously.

[0020] The above technical solution has the following beneficial effects: by processing data through multi-threading, the data processing time is shortened and the system response speed is improved; by adaptively adjusting the data buffer, it is ensured that the data buffer size and data processing strategy can be dynamically adjusted according to the real-time status of the audio and video data stream, avoiding data buffer overflow or shortage and improving system stability; through more sophisticated data buffer management and priority control, it reduces freezes and delays and improves user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 is a flow chart of a method for synchronously playing audio and video provided by an embodiment of the present invention;

[0023] Figure 2 is a flow chart of a buffer size adjustment method provided by an embodiment of the present invention;

[0024] Figure 3 is a flow chart of a method for calculating buffer size provided by an embodiment of the present invention;

[0025] Figure 4 is a flow chart of an audio and video synchronization method provided by an embodiment of the present invention;

[0026] Figure 5 It is a structural diagram of an audio and video synchronous playback system provided by an embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of the structure of the encoding module provided by an embodiment of the present invention;

[0028] Figure 7 is a schematic diagram of the structure of a buffer size calculation submodule provided in an embodiment of the present invention;

[0029] Figure 8 It is a schematic diagram of the structure of a decoding module provided in an embodiment of the present invention.

[0030] Figure markings: 100-data acquisition module; 200-encoding module; 2001-performance indicator acquisition submodule; 2002-buffer strategy selection submodule; 2003-buffer size calculation submodule; 20031-deviation value calculation unit; 20032-adjustment amount calculation unit; 20033-buffer size adjustment unit; 300-decoding module; 3001-first synchronization submodule; 3002-second synchronization submodule; 400-rendering output module; 500-multiplexing module. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In order to solve the problem of poor playback effect in the prior art of audio and video synchronous playback, an embodiment of the present invention provides an audio and video synchronous playback method. Figure 1 is a flowchart of the audio and video synchronous playback method provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0033] S1. Collect video data from the camera through the video acquisition thread, and collect audio data from the microphone through the audio acquisition thread.

[0034] Create multiple threads that can be assigned to multiple processes for data processing to increase data processing speed.

[0035] The video acquisition thread acquires video data from a camera or other video source, and the audio acquisition thread acquires audio data from a microphone or other audio source. The video data and audio data are attached with acquisition timestamps for subsequent synchronous processing.

[0036] A timestamp is a numerical value used to identify the time when an event occurs, usually expressed in a precise time format.

[0037] S2. The video data and the audio data are transmitted to the video encoding thread and the audio encoding thread for encoding processing respectively, and the encoded data are stored in the corresponding video buffer and audio buffer respectively; wherein the size of the data buffer is adjusted in real time according to the performance index and the actual buffer water level; the data buffer includes a video buffer and an audio buffer.

[0038] During the encoding process, the video encoding thread encodes the collected video data into a video-specified format (such as H.264), while the audio encoding thread encodes the collected audio data into an audio-specified format (such as AAC). The encoded data is temporarily stored in a buffer, specifically storing the encoded video data in the video buffer and the encoded audio data in the audio buffer, to decouple the acquisition, encoding, and subsequent processing operations, and ensure the stability and continuity of data stream transmission.

[0039] The data buffer monitors the performance indicators of the system or buffer in real time, including: CPU load (percentage of CPU usage), memory usage (percentage of memory usage), network latency (the time difference from sending to receiving data, in milliseconds), buffer occupancy (the ratio of the number of frames of data stored in the current buffer to the total capacity of the buffer) and frame loss rate (the ratio of frames lost during video playback), etc.

[0040] The actual buffer level, that is, the amount of data currently stored in the data buffer, can be expressed in units of frames or bytes.

[0041] Performance indicators and actual buffer water levels are obtained through periodic sampling, combined with hardware sensors, network protocols (such as RTP / RTCP) or system call interfaces (such as operating system resource monitoring API) for real-time acquisition.

[0042] The comprehensive performance indicators and actual buffer water level can adjust the size of the data buffer in real time.

[0043] S3. Synchronize the video data in the video buffer and the audio data in the audio buffer, and input the synchronized video data and audio data into the video decoding thread and the audio decoding thread for decoding processing respectively.

[0044] The synchronization process is based on the timestamps of the video and audio data. The time difference of the data is calculated by comparing the timestamps, and the playback rate of the video data or audio data is dynamically adjusted to ensure synchronization of audio and video playback. The synchronized data is transmitted to the decoding thread, where the video decoding thread decodes the video data and the audio decoding thread decodes the audio data to provide decoded frame data for subsequent playback.

[0045] S4, transmitting the decoded video data to the video rendering thread for processing, and transmitting the decoded audio data to the audio output thread for processing.

[0046] The video rendering thread renders the decoded video frames to the display device, while the audio output thread outputs the decoded audio data to the speaker.

[0047] S5. Merge the decoded audio data and video data to obtain an audio and video stream, and encapsulate or transmit the audio and video stream.

[0048] Combine multiple audio signals and one video signal into one audio and video stream, which is packaged into a file in a specific format or transmitted in real time over the network.

[0049] As an optional implementation, when multiple threads are processed simultaneously, the priority of each thread can be dynamically adjusted according to the system resource usage and task importance.

[0050] For example, the system includes the following threads: video acquisition thread, audio acquisition thread, video encoding thread, audio encoding thread, video decoding thread, audio decoding thread, video rendering thread, and audio output thread. At this time, the priority adjustment strategy can be customized according to different application scenarios. The following are some example strategies:

[0051] When the CPU load is too high, lower the priority of the encoding thread and increase the priority of the decoding, rendering, and output threads to ensure smooth playback.

[0052] When the memory usage is too high, lower the priority of the encoding and decoding threads and release some cached data as much as possible.

[0053] When network latency is high, increase the priority of the acquisition thread and adjust the data buffer management strategy to prioritize smoothness.

[0054] When the data buffer occupancy is too low, the priority of the decoding thread is appropriately lowered to give the encoding thread a chance to fill the data buffer.

[0055] When the frame loss rate is too high, lower the priority of the encoding thread, increase the priority of the decoding thread, and consider reducing the video resolution or frame rate.

[0056] Priority adjustment methods include: directly setting priority: using the API provided by the operating system to directly set the priority of the thread; adjusting thread scheduling policy: adjusting the thread scheduling policy, such as setting threads with high real-time requirements to real-time scheduling policy; using priority queues: using priority queues to manage threads, and threads with high priorities are executed first.

[0057] As an optional implementation, Figure 2 is a flow chart of a buffer size adjustment method provided by an embodiment of the present invention. Figure 2 As shown, the size of the data buffer is adjusted in real time according to the performance index and the actual buffer water level, including:

[0058] S201, obtaining the performance index of the data buffer and the actual buffer water level in real time;

[0059] Get the system's CPU load and memory usage in real time.

[0060] The Kalman filter algorithm is used to predict network delays at the next moment based on previous network delay data, respond to changes in network conditions in advance, and prevent playback from being stuck or losing frames.

[0061] The exponentially weighted moving average algorithm is used to smooth the buffer occupancy and frame loss rate.

[0062] Exponentially Weighted Moving Average (EWMA): used to smooth performance indicators such as buffer occupancy and frame loss rate to avoid the impact of instantaneous fluctuations on strategy selection. The specific formula is as follows:

[0063] EWMA(t)=α*Indicator(t)+(1-α)*EWMA(t-1)

[0064] Among them, α is the smoothing coefficient, Indicator(t) is the value of the collected performance indicator, and EWMA(t) is the value of the smoothed performance indicator.

[0065] S202, selecting a current buffer strategy of the data buffer zone from among all buffer strategies according to the performance indicator;

[0066] The buffering strategy can be pre-set, for example, it can be divided into three modes, namely fast mode (small buffer, low delay, suitable for delay-sensitive scenarios, such as video conferencing, real-time interactive live broadcast), smooth mode (large buffer, can tolerate a certain delay, prioritize playback smoothness, suitable for scenarios with poor network conditions, such as online video playback), balanced mode (moderate buffer size, taking into account both delay and smoothness, suitable for scenarios with certain requirements for both delay and smoothness). Different buffering strategies correspond to different trigger conditions. The buffering strategy is adjusted in time according to the performance indicators at different times, and the buffer is adjusted accordingly according to the buffering strategy, such as dropping frames, reducing resolution, etc. Each buffering strategy corresponds to a switching condition. When the performance indicators meet the switching conditions, you can switch to the corresponding buffering strategy. Table 1 is an example of a specific switching condition.

[0067]

[0068]

[0069] Table 1

[0070] S203: Calculate the current buffer size of the data buffer according to the actual buffer water level.

[0071] After the strategy is switched, the buffer size needs to be adjusted according to the actual buffer level to meet the buffer size requirements of the buffer strategy. Of course, during the strategy application process, the actual buffer level needs to be monitored in real time and the buffer size needs to be adjusted according to the actual buffer level to ensure that the buffer size is within the target range of the buffer strategy.

[0072] As an optional implementation, the adjustment of the buffer size is achieved through a PID controller.

[0073] As an optional implementation, Figure 3 is a flow chart of a method for calculating the buffer size provided by an embodiment of the present invention. Figure 3 As shown, the current buffer size of the data buffer is calculated based on the actual buffer water level, including:

[0074] S2031. Calculate a buffer zone water level deviation value according to an actual buffer zone water level and a target buffer zone water level; wherein the target buffer zone water level is determined according to a current buffer strategy.

[0075] Get the actual buffer level of the current buffer in real time (for example, the amount of data stored in the current buffer, expressed in the number of frames or data bytes).

[0076] Get the target buffer level corresponding to the current buffer strategy. The target buffer level is the ideal range preset according to the selected buffer strategy (fast mode, smooth mode or balanced mode). For example: Fast mode: target level = 20 frames; Smooth mode: target level = 100 frames; Balanced mode: target level = 50 frames.

[0077] Water level deviation value = actual buffer zone water level - target buffer zone water level.

[0078] S2032. Calculate the adjustment amount of the buffer size according to the buffer water level deviation value.

[0079] Use a PID controller or other dynamic adjustment algorithm to calculate the adjustment amount of the buffer size based on the buffer water level deviation value.

[0080] S2033. Calculate the current buffer size according to the actual buffer water level and the adjustment amount.

[0081] The current buffer size is updated according to the actual buffer level and the calculated adjustment amount, and the formula is: current buffer size = actual buffer level + adjustment amount.

[0082] The adjusted buffer size is within a preset range (such as a minimum value of not less than 10 frames and a maximum value of not more than 200 frames). If the buffer size is lower than the minimum value, it is adjusted to the minimum value. If the buffer size is higher than the maximum value, it is adjusted to the maximum value. This restriction method prevents excessive changes in the buffer size from affecting the playback effect.

[0083] Assume that the current strategy is "balanced mode", the target buffer water level is 50 frames, and the actual buffer water level is 30 frames. Then the buffer water level deviation value e(t) = 50-30 = 20 frames, and the PID controller calculates the adjustment value u(t) according to the buffer water level deviation value e(t). If the calculated adjustment value u(t) = +5 frames, then the current buffer size is adjusted to 30+5 = 35 frames.

[0084] As an optional implementation, the adjustment amount of the buffer size is calculated according to the buffer water level deviation value, and is calculated by the following formula:

[0085]

[0086] Where u(t) is the adjustment amount of the buffer size, K p , K i , K d is the coefficient, and e(t) is the buffer water level deviation value.

[0087] K p is the proportional coefficient, which is used to adjust the response speed to the current deviation value; K i is the integral coefficient, which is used to eliminate the steady-state error and make the buffer water level closer to the target value; K d is the differential coefficient, which is used to suppress the violent fluctuation of the buffer water level. The integral form of e(t), τ is the integral variable, which is the conventional way of writing when integrating.

[0088] As an optional implementation, Figure 4 is a flow chart of an audio and video synchronization method provided by an embodiment of the present invention, such as Figure 4 As shown, the video data in the video buffer and the audio data in the audio buffer are synchronously processed, including:

[0089] S301 . Retrieve video data and audio data with time stamps from the video buffer and audio buffer respectively through a message queue mechanism.

[0090] Independent message queues are set for the video buffer and the audio buffer, namely, the video message queue and the audio message queue. The video message queue contains video frames, and the audio message queue contains audio segments. The message queue adopts a first-in-first-out (FIFO) mechanism to ensure that the video frames and audio segments are taken out in the order of acquisition.

[0091] S302: Compare and adjust the time stamp of the video data and the time stamp of the audio data to play the video data and the audio data synchronously.

[0092] For each video data and audio data taken out, read its timestamp T video and T audio .

[0093] Calculate the timestamp difference ΔT: ΔT = T video -T audio .

[0094] Then the adjustment logic is: according to the timestamp difference ΔT, adjust the playback rate or time offset of the video data or audio data to synchronize the two:

[0095] (1) If ΔT>0 (video timestamp is ahead): delay the video data to allow the audio data to catch up with the video data. Or insert a silent audio segment to compensate for the time difference.

[0096] (2) If ΔT < 0 (audio timestamp is ahead): delay the playback of audio data to allow the video data to catch up with the audio data. Or skip some video frames to quickly align the time difference.

[0097] (3) If ΔT=0: the video data and audio data have the same timestamps and are played synchronously.

[0098] The embodiment of the present invention also provides an audio and video synchronous playback system. Figure 5 is a structural diagram of an audio and video synchronous playback system provided by an embodiment of the present invention, such as Figure 5 As shown, the system includes:

[0099] The data acquisition module 100 is used to acquire video data from the camera through a video acquisition thread and to acquire audio data from the microphone through an audio acquisition thread;

[0100] Create multiple threads that can be assigned to multiple processes for data processing to increase data processing speed.

[0101] The video acquisition thread acquires video data from a camera or other video source, and the audio acquisition thread acquires audio data from a microphone or other audio source. The video data and audio data are attached with acquisition timestamps for subsequent synchronous processing.

[0102] A timestamp is a numerical value used to identify the time when an event occurs, usually expressed in a precise time format.

[0103] The encoding module 200 is used to transmit the video data and the audio data to the video encoding thread and the audio encoding thread for encoding processing respectively, and store the encoded data in the corresponding video buffer and the audio buffer respectively; wherein the size of the data buffer is adjusted in real time according to the performance index and the actual buffer water level; the data buffer includes a video buffer and an audio buffer.

[0104] During the encoding process, the video encoding thread encodes the collected video data into a video-specified format (such as H.264), while the audio encoding thread encodes the collected audio data into an audio-specified format (such as AAC). The encoded data is temporarily stored in a buffer, specifically storing the encoded video data in the video buffer and the encoded audio data in the audio buffer, to decouple the acquisition, encoding, and subsequent processing operations, and ensure the stability and continuity of data stream transmission.

[0105] The data buffer monitors the performance indicators of the system or buffer in real time, including: CPU load (percentage of CPU usage), memory usage (percentage of memory usage), network latency (the time difference from sending to receiving data, in milliseconds), buffer occupancy (the ratio of the number of frames of data stored in the current buffer to the total capacity of the buffer) and frame loss rate (the ratio of frames lost during video playback), etc.

[0106] The actual buffer level, that is, the amount of data currently stored in the data buffer, can be expressed in units of frames or bytes.

[0107] Performance indicators and actual buffer water levels are obtained through periodic sampling, combined with hardware sensors, network protocols (such as RTP / RTCP) or system call interfaces (such as operating system resource monitoring API) for real-time acquisition.

[0108] The comprehensive performance indicators and actual buffer water level can adjust the size of the data buffer in real time.

[0109] The decoding module 300 is used to synchronously process the video data in the video buffer and the audio data in the audio buffer, and input the synchronized video data and audio data to the video decoding thread and the audio decoding thread for decoding processing respectively;

[0110] The synchronization process is based on the timestamps of the video and audio data. The time difference of the data is calculated by comparing the timestamps, and the playback rate of the video data or audio data is dynamically adjusted to ensure synchronization of audio and video playback. The synchronized data is transmitted to the decoding thread, where the video decoding thread decodes the video data and the audio decoding thread decodes the audio data to provide decoded frame data for subsequent playback.

[0111] A rendering output module 400 is used to transmit the decoded video data to the video rendering thread for processing, and transmit the decoded audio data to the audio output thread for processing;

[0112] The video rendering thread renders the decoded video frames to the display device, while the audio output thread outputs the decoded audio data to the speaker.

[0113] The multiplexing module 500 is used to merge the decoded audio data and video data to obtain an audio and video stream, and encapsulate or transmit the audio and video stream.

[0114] Combine multiple audio signals and one video signal into one audio and video stream, which is packaged into a file in a specific format or transmitted in real time over the network.

[0115] As an optional implementation, when multiple threads are processed simultaneously, the priority of each thread can be dynamically adjusted according to the system resource usage and task importance.

[0116] For example, the system includes the following threads: video acquisition thread, audio acquisition thread, video encoding thread, audio encoding thread, video decoding thread, audio decoding thread, video rendering thread, and audio output thread. At this time, the priority adjustment strategy can be customized according to different application scenarios. The following are some example strategies:

[0117] When the CPU load is too high, lower the priority of the encoding thread and increase the priority of the decoding, rendering, and output threads to ensure smooth playback.

[0118] When the memory usage is too high, lower the priority of the encoding and decoding threads and release some cached data as much as possible.

[0119] When network latency is high, increase the priority of the acquisition thread and adjust the data buffer management strategy to prioritize smoothness.

[0120] When the data buffer occupancy is too low, the priority of the decoding thread is appropriately lowered to give the encoding thread a chance to fill the data buffer.

[0121] When the frame loss rate is too high, lower the priority of the encoding thread, increase the priority of the decoding thread, and consider reducing the video resolution or frame rate.

[0122] Priority adjustment methods include: directly setting priority: using the API provided by the operating system to directly set the priority of the thread; adjusting thread scheduling policy: adjusting the thread scheduling policy, such as setting threads with high real-time requirements to real-time scheduling policy; using priority queues: using priority queues to manage threads, and threads with high priorities are executed first.

[0123] As an optional implementation, Figure 6 is a schematic diagram of the structure of a buffer size adjustment module for buffer strategy selection provided by an embodiment of the present invention, such as Figure 6 As shown, the encoding module 200 includes:

[0124] The performance indicator acquisition submodule 2001 is used to acquire the performance indicator of the data buffer and the actual buffer water level in real time, and to acquire the CPU load and memory occupancy rate of the system in real time.

[0125] The Kalman filter algorithm is used to predict network delays at the next moment based on previous network delay data, respond to changes in network conditions in advance, and prevent playback from being stuck or losing frames.

[0126] The exponentially weighted moving average algorithm is used to smooth the buffer occupancy and frame loss rate.

[0127] Exponentially Weighted Moving Average (EWMA): used to smooth performance indicators such as buffer occupancy and frame loss rate to avoid the impact of instantaneous fluctuations on strategy selection. The specific formula is as follows:

[0128] EWMA(t)=α*Indicator(t)+(1-α)*EWMA(t-1)

[0129] Among them, α is the smoothing coefficient, Indicator(t) is the value of the collected performance indicator, and EWMA(t) is the value of the smoothed performance indicator.

[0130] The buffer strategy selection submodule 2002 is used to select a current buffer strategy for the data buffer from among all buffer strategies according to the performance indicator.

[0131] The buffering strategy can be pre-set, for example, it can be divided into three modes, namely fast mode (small buffer, low latency, suitable for scenarios sensitive to latency, such as video conferencing, real-time interactive live broadcast), smooth mode (large buffer, can tolerate a certain delay, give priority to ensuring playback smoothness, suitable for scenarios with poor network conditions, such as online video playback), balanced mode (moderate buffer size, taking into account both latency and smoothness, suitable for scenarios with certain requirements for both latency and smoothness). Different buffering strategies correspond to different trigger conditions. The buffering strategy is adjusted in time according to the performance indicators at different times, and the buffer is adjusted accordingly according to the buffering strategy, such as dropping frames, reducing resolution, etc.

[0132] Each buffer strategy corresponds to a switching condition. When the performance indicator meets the switching condition, the corresponding buffer strategy can be switched. Table 1 is an example of a specific switching condition.

[0133] The current buffer size calculation submodule 2003 is used to calculate the current buffer size of the data buffer according to the actual buffer water level.

[0134] After the strategy is switched, the buffer size needs to be adjusted according to the actual buffer level to meet the buffer size requirements of the buffer strategy. Of course, during the strategy application process, the actual buffer level needs to be monitored in real time and the buffer size needs to be adjusted according to the actual buffer level to ensure that the buffer size is within the target range of the buffer strategy.

[0135] As an optional implementation, the adjustment of the buffer size is achieved through a PID controller.

[0136] As an optional implementation, Figure 7 is a schematic diagram of the structure of a buffer size adjustment module for adjusting the buffer size provided by an embodiment of the present invention, such as Figure 7 As shown, the buffer size calculation submodule 2003 also includes:

[0137] The deviation value calculation unit 20031 is used to calculate the buffer water level deviation value according to the actual buffer water level and the target buffer water level; wherein the target buffer water level is determined according to the current buffer strategy;

[0138] Get the actual buffer level of the current buffer in real time (for example, the amount of data stored in the current buffer, expressed in the number of frames or data bytes).

[0139] Get the target buffer level corresponding to the current buffer strategy. The target buffer level is the ideal range preset according to the selected buffer strategy (fast mode, smooth mode or balanced mode). For example: Fast mode: target level = 20 frames; Smooth mode: target level = 100 frames; Balanced mode: target level = 50 frames.

[0140] Water level deviation value = actual buffer zone water level - target buffer zone water level.

[0141] An adjustment amount calculation unit 20032, used to calculate the adjustment amount of the buffer size according to the buffer water level deviation value;

[0142] Use a PID controller or other dynamic adjustment algorithm to calculate the adjustment amount of the buffer size based on the buffer water level deviation value.

[0143] The buffer size adjustment unit 20033 is used to calculate the current buffer size according to the actual buffer water level and the adjustment amount.

[0144] The current buffer size is updated according to the actual buffer level and the calculated adjustment amount, and the formula is: current buffer size = actual buffer level + adjustment amount.

[0145] The adjusted buffer size is within a preset range (such as a minimum value of not less than 10 frames and a maximum value of not more than 200 frames). If the buffer size is lower than the minimum value, it is adjusted to the minimum value. If the buffer size is higher than the maximum value, it is adjusted to the maximum value. This restriction method prevents excessive changes in the buffer size from affecting the playback effect.

[0146] Assume that the current strategy is "balanced mode", the target buffer water level is 50 frames, and the actual buffer water level is 30 frames. Then the buffer water level deviation value e(t) = 50-30 = 20 frames, and the PID controller calculates the adjustment value u(t) according to the buffer water level deviation value e(t). If the calculated adjustment value u(t) = +5 frames, then the current buffer size is adjusted to 30+5 = 35 frames.

[0147] As an optional implementation, the adjustment amount of the buffer size is calculated according to the buffer water level deviation value, and is calculated by the following formula:

[0148]

[0149] Where u(t) is the adjustment amount of the buffer size, K p , K i , K d is the coefficient, and e(t) is the buffer water level deviation value.

[0150] K p is the proportional coefficient, which is used to adjust the response speed to the current deviation value; K i is the integral coefficient, which is used to eliminate the steady-state error and make the buffer water level closer to the target value; K d is the differential coefficient, which is used to suppress the violent fluctuation of the buffer water level. The integral form of e(t), τ is the integral variable, which is the conventional way of writing when integrating.

[0151] As an optional implementation, Figure 8 is a schematic diagram of the structure of a decoding module provided by an embodiment of the present invention, such as Figure 8 As shown, the decoding module 300 includes:

[0152] The first synchronization submodule 3001 is used to fetch the video data and audio data with timestamps from the video buffer and the audio buffer respectively through a message queue mechanism;

[0153] Independent message queues are set for the video buffer and the audio buffer, namely, the video message queue and the audio message queue. The video message queue contains video frames, and the audio message queue contains audio segments. The message queue adopts a first-in-first-out (FIFO) mechanism to ensure that the video frames and audio segments are taken out in the order of acquisition.

[0154] The second synchronization submodule 3002 is used to compare and adjust the timestamp of the video data and the timestamp of the audio data so as to play the video data and the audio data synchronously.

[0155] For each video data and audio data taken out, read its timestamp T video and T audio .

[0156] Calculate the timestamp difference ΔT: ΔT = T video -T audio .

[0157] Then the adjustment logic is: according to the timestamp difference ΔT, adjust the playback rate or time offset of the video data or audio data to synchronize the two:

[0158] (1) If ΔT>0 (video timestamp is ahead): delay the video data to allow the audio data to catch up with the video data. Or insert a silent audio segment to compensate for the time difference.

[0159] (2) If ΔT < 0 (audio timestamp is ahead): delay the playback of audio data to allow the video data to catch up with the audio data. Or skip some video frames to quickly align the time difference.

[0160] (3) If ΔT=0: the video data and audio data have the same timestamps and are played synchronously.

[0161] The above technical solution has the following beneficial effects: by processing data through multi-threading, the data processing time is shortened and the system response speed is improved; by adaptively adjusting the buffer, the buffer size and data processing strategy can be dynamically adjusted according to the real-time status of the audio and video data stream, avoiding buffer overflow or shortage and improving system stability; through more sophisticated buffer management and priority control, the jamming and delay are reduced and the user experience is improved.

Claims

1. A method for synchronously playing audio and video, characterized in that: include: The video acquisition thread collects video data from the camera, and the audio acquisition thread collects audio data from the microphone; The video data and the audio data are transmitted to the video encoding thread and the audio encoding thread for encoding processing respectively, and the encoded data are stored in the corresponding video buffer and audio buffer respectively; wherein the size of the data buffer is adjusted in real time according to the performance index and the actual buffer water level; the data buffer includes the video buffer and the audio buffer; Synchronize the video data in the video buffer and the audio data in the audio buffer, and input the synchronized video data and audio data into the video decoding thread and the audio decoding thread for decoding processing respectively; The decoded video data is transferred to the video rendering thread for processing, and the decoded audio data is transferred to the audio output thread for processing; The decoded audio data and video data are combined to obtain an audio and video stream, and the audio and video stream is packaged or transmitted.

2. The method for synchronously playing audio and video according to claim 1, characterized in that: The size of the data buffer is adjusted in real time according to the performance index and the actual buffer water level, including: Obtain the performance indicators of the data buffer and the actual buffer water level in real time; Selecting a current buffering strategy for the data buffer from among all buffering strategies according to the performance indicator; The current buffer size of the data buffer is calculated according to the actual buffer water level.

3. The method for synchronously playing audio and video according to claim 2, characterized in that: The calculating the current buffer size of the data buffer according to the actual buffer water level includes: Calculating a buffer zone water level deviation value according to an actual buffer zone water level and a target buffer zone water level; wherein the target buffer zone water level is determined according to a current buffer strategy; Calculating the adjustment amount of the buffer size according to the buffer water level deviation value; The current buffer size is calculated according to the actual buffer water level and the adjustment amount.

4. The method for synchronously playing audio and video according to claim 3, characterized in that: The adjustment amount of the buffer size is calculated according to the buffer water level deviation value, and is calculated by the following formula: Where u(t) is the adjustment amount of the buffer size, K p , K i , K d is the coefficient, and e(t) is the buffer water level deviation value.

5. The method for synchronously playing audio and video according to claim 4, characterized in that: The synchronous processing of the video data in the video buffer and the audio data in the audio buffer includes: Retrieving the video data and the audio data with the timestamp from the video buffer and the audio buffer respectively through the message queue mechanism; Comparison and adjustment are performed based on the timestamp of the video data and the timestamp of the audio data, so that the video data and the audio data are played synchronously.

6. An audio and video synchronous playback system, characterized in that: include: A data acquisition module is used to acquire video data from a camera through a video acquisition thread and to acquire audio data from a microphone through an audio acquisition thread; The encoding module is used to transmit the video data and the audio data to the video encoding thread and the audio encoding thread for encoding processing, and store the encoded data in the corresponding video buffer and the audio buffer respectively; wherein the size of the data buffer is adjusted in real time according to the performance index and the actual buffer water level; the data buffer includes a video buffer and an audio buffer; A decoding module is used to synchronously process the video data in the video buffer and the audio data in the audio buffer, and input the synchronized video data and audio data into the video decoding thread and the audio decoding thread respectively for decoding processing; The rendering output module is used to transmit the decoded video data to the video rendering thread for processing, and transmit the decoded audio data to the audio output thread for processing; The multiplexing module is used to merge the decoded audio data and video data to obtain an audio and video stream, and encapsulate or transmit the audio and video stream.

7. The audio and video synchronous playback system according to claim 6, characterized in that: The encoding module comprises: The performance indicator acquisition submodule is used to obtain the performance indicators of the data buffer and the actual buffer water level in real time; A buffer strategy switching submodule, used for selecting a current buffer strategy of a data buffer zone from among all buffer strategies according to the performance indicator; The buffer size calculation submodule is used to calculate the current buffer size of the data buffer according to the actual buffer water level.

8. The audio and video synchronous playback system according to claim 7, characterized in that: The buffer size calculation submodule also includes: A deviation value calculation unit, used to calculate a buffer zone water level deviation value according to an actual buffer zone water level and a target buffer zone water level; wherein the target buffer zone water level is determined according to a current buffer strategy; An adjustment amount calculation unit, used for calculating the adjustment amount of the buffer size according to the buffer water level deviation value; The buffer size adjustment unit is used to calculate the current buffer size according to the actual buffer water level and the adjustment amount.

9. The audio and video synchronous playback system according to claim 8, characterized in that: The adjustment amount of the buffer size is calculated according to the buffer water level deviation value, and is calculated by the following formula: Where u(t) is the adjustment amount of the buffer size, K p , K i , K d is the coefficient, and e(t) is the buffer water level deviation value.

10. The audio and video synchronous playback system according to claim 9, characterized in that: The decoding module comprises: A first synchronization submodule, used for taking out the video data and the audio data with timestamps from the video buffer and the audio buffer respectively through a message queue mechanism; The second synchronization submodule is used to compare and adjust the timestamp of the video data and the timestamp of the audio data so as to play the video data and the audio data synchronously.

Images