Audio and video stream real-time processing method fusing key frame control, timestamp synchronization and cache optimization

Abstract

The application discloses a real-time processing method for audio and video stream fusing key frame control, timestamp synchronization and cache optimization. The method receives multiple types of audio and video data streams, extracts frame types, key frame identifiers, timestamps and media attribute parameters, and ensures synchronous processing of the audio and video stream. Through a key frame guiding mechanism, the first frame is ensured to be a key frame that can be independently decoded, avoiding decoding failure. A double-threshold timestamp jump detection and compensation technology is adopted to ensure accurate synchronization of the audio and video stream and avoid the problem of asynchronization caused by network fluctuations. Audio processing is performed through intelligent splicing and standardized packaging to solve the problem of audio frame breakage and improve the continuity of the audio stream. In combination with a memory management strategy of a single-stream dynamic buffer and a global memory pool, memory utilization is optimized, and fragmentation is reduced. Through priority scheduling and intelligent congestion control, low-delay transmission of real-time audio and video stream is ensured, and the stability of the system in a high-concurrency and network fluctuation environment is improved.

Description

Technical Field

[0001] This invention belongs to the field of multimedia data processing technology, specifically relating to a real-time audio and video stream processing method that integrates keyframe control, timestamp synchronization, and cache optimization. Background Technology

[0002] With the rapid development of internet communication technology, streaming media technology is increasingly widely used in real-time audio and video communication, live streaming, and remote monitoring. Users have an increasingly strong demand for the synchronous processing and stable playback of different mixed data streams (such as video and audio streams). In actual business operations, raw bitstreams from different manufacturers and types of equipment often exhibit significant differences in encoding formats, encapsulation methods, and even the specific implementation of transmission protocols. This diversity poses a severe challenge to achieving smooth synchronous processing and stable playback. The processing of multiple mixed data streams must balance real-time performance, integrity, and compatibility. Ensuring efficient transmission of different types of data while achieving accurate audio and video synchronization and multi-scenario adaptation has become one of the core research directions in the field of multimedia data processing.

[0003] Currently, existing streaming media systems primarily focus on processing media data of a single type and specific protocol (such as pure video or pure audio), lacking a universal processing design for different types of raw mixed data streams. Their core implementation mainly relies on mature open-source frameworks and standard protocols, specifically including the following two parts: In the stream reception stage, the system receives audio and video data streams through network protocols such as RTP (Real-Time Transport Protocol), RTSP (Real-Time Streaming Protocol), RTMP (Real-Time Messaging Protocol), or WebRTC, and uses mainstream multimedia frameworks such as FFmpeg and GStreamer to complete data capture and preliminary parsing; In the decoding stage, the received compressed encoded data in formats such as H.264 and AAC is decoded using open-source decoders such as libavcodec and OpenH264 to obtain directly processable raw frame data, ultimately achieving playback functionality.

[0004] However, the aforementioned existing technical solutions reveal several unavoidable shortcomings when facing scenarios involving the processing of multiple types of mixed data streams: First, they lack an effective keyframe control mechanism, failing to force the first frame to be an independently decodeable keyframe (such as a video I-frame) and to cache subsequent received keyframes. If the first frame is not a keyframe, it is necessary to obtain the keyframe through retransmission or long waiting times, which not only increases playback latency but may also cause decoder initialization failure. If subsequent keyframes are not cached, each user needs to re-parse the original bitstream to obtain keyframes when streaming video. This is especially problematic when many users are streaming video simultaneously, as re-parsering keyframes for each stream significantly impacts system resources and efficiency. Second, the timestamp synchronization strategy is rigid, relying on the device's local clock or a fixed clock. The timestamp accumulation method cannot cope with network jitter, device clock differences, and timestamp field corruption, which can easily cause audio and video playback desynchronization, screen jumps, or stuttering. Third, the audio frame fragmentation handling capability is weak. When network fluctuations cause audio fragments to be lost, the complete frame needs to be retransmitted, which reduces bandwidth utilization and causes audio playback interruptions. Fourth, the memory management method is unreasonable. Frequent dynamic memory allocation and release can easily lead to memory fragmentation, resulting in high system call overhead and difficulty in ensuring real-time performance and resource utilization in high-concurrency scenarios. Fifth, the multi-stream scheduling lacks flexibility. When multiple streams are running concurrently, an equal resource allocation strategy is adopted without considering the differences in real-time requirements of different business scenarios (such as real-time intercom streams and on-demand streams), resulting in increased latency and decreased throughput of high-priority real-time streams.

[0005] Existing streaming media processing technologies have significant shortcomings in keyframe control, timestamp synchronization, frame breakage handling, memory management, and stream scheduling for multi-type mixed data streams. They struggle to meet the high reliability, low latency, and high stability requirements of real-time audio and video communication and remote monitoring scenarios. This is especially true in applications such as real-time audio and video communication, remote interactive monitoring, and IoT live streaming with multi-sensor fusion, which demand stringent requirements for multi-type data stream synchronization, first-frame loading speed, and high concurrency stability. The existing technologies' "passive adaptation" rather than "active management" processing logic has become a core bottleneck restricting service experience and system performance. Therefore, a highly efficient real-time processing solution for multi-type audio and video streams is urgently needed to address these issues. Summary of the Invention

[0006] In view of the shortcomings of existing technologies in real-time processing of various types of audio and video streams, this invention provides a real-time audio and video stream processing method that integrates keyframe control, timestamp synchronization, and cache optimization. It aims to overcome the deficiencies of existing technologies, such as decoding failures caused by non-key frames in the first frame, audio and video desynchronization due to timestamp anomalies, low efficiency in audio frame breakage processing, fragmented memory allocation, and insufficient flexibility in multi-stream scheduling. This invention integrates intelligent keyframe guidance, dual-threshold timestamp jump detection and hierarchical compensation, fragmented audio cache splicing and standardized encapsulation, a two-layer memory management system coordinating a "single-stream precise buffer and global memory pool," and priority scheduling and intelligent congestion control based on business scenarios. This achieves rapid and reliable decoder initialization, millisecond-level precise audio and video synchronization, zero-loss transmission of audio segments, efficient reuse of memory resources, and low-latency assurance for high-real-time services. It comprehensively improves the reliability, real-time performance, compatibility, and resource utilization efficiency of processing various types of audio and video streams, meeting the high-specification application requirements of real-time audio and video communication, live streaming, remote monitoring, and other scenarios, and providing users with a stable and smooth media service experience.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: a real-time audio and video stream processing method that integrates keyframe control, timestamp synchronization, and buffer optimization, the method comprising the following steps:

[0008] a) Receive multiple types of audio and video data streams, receive data packets through network protocols, and extract frame type, keyframe identifier, timestamp, and media attribute parameters from the data packets;

[0009] b) Frame parsing and keyframe guidance: During session initialization, the system maintains a dual-state flag of "no keyframe received" and "keyframe received". If the current frame is not a keyframe, it is discarded and the system returns to the "blocked waiting" state until the first keyframe is received. The keyframe is then cached internally, and the state is updated to "keyframe received". The system then allows the keyframe and all subsequent frames to proceed. All subsequent keyframes are also cached. In a multi-user system, for example, when user A is playing a video, the system sends the first keyframe from the camera to the decoding library for decoding and playback. Simultaneously, user B is also playing a video. The decoding library can directly use all the keyframes cached during user A's playback, without needing to re-parse the complete stream data sent by the device. This greatly improves video playback speed and smoothness.

[0010] c) Timestamp synchronization and compensation: An independent timeline is established based on the timestamp of the first keyframe. Dual-threshold timestamp jump detection is performed. If a timestamp anomaly is detected, the global timestamp is updated with a fixed step size compensation.

[0011] d) For normal timestamps, the global smooth timestamp is updated by accurately accumulating the actual difference between adjacent frames to ensure precise synchronization of audio and video streams;

[0012] e) Audio splicing and encapsulation: When the length of audio data is less than that of a complete frame, it is stored in a dedicated audio buffer and the number of bytes is recorded. When new audio data arrives, intelligent splicing is performed. After splicing to the size of a complete frame, an adapted protocol header is dynamically added, a unified format audio frame is generated, and it is moved into the audio ready queue.

[0013] f) Intelligent cache management adopts a collaborative management strategy of single-stream dynamic buffer and global memory pool to dynamically adapt the buffer size and ensure that each data stream can obtain appropriate buffer resources;

[0014] g) Prioritize data streams according to business scenarios. High-priority streams have the right to preempt resources. Priority preemption scheduling strategy is adopted to ensure low-latency transmission of real-time audio and video streams. Stream scheduling and intelligent congestion control mechanisms are used to reduce the loss of non-critical frames.

[0015] Furthermore, the specific implementation of frame parsing and keyframe guidance includes:

[0016] a) Frame type parsing distinguishes the current frame as video, audio, serial data, static image, or general data by using the identifier field in the metadata;

[0017] b) The criterion for determining a keyframe is whether the frame is a key data unit that can be independently decoded;

[0018] c) In the "No key frame received" state, the system only performs non-key frame discard and blocking wait state return operations, and does not start any decoding or subsequent data processing process;

[0019] d) After the state switch, the system enters the normal receiving mode. Subsequent non-critical frames and critical frames pass through in the order of reception to build a complete and continuous data stream, ensuring the integrity and reliability of the initial stage of playback.

[0020] Furthermore, the specific implementation of timestamp synchronization and compensation includes: a) When initializing an independent timeline, the original timestamp of the first keyframe is used as the reference point to form a linear mapping relationship, without the need for additional calibration procedures;

[0021] b) Timestamp anomalies include rollback anomalies and positive mutation anomalies. When an anomaly occurs, the timestamp is accepted as the new baseline.

[0022] c) The fixed step size matches the regular playback rhythm of the media stream, ensuring that the global timestamp is monotonous and smooth. The normal timestamp is accurately accumulated according to the actual time difference between adjacent frames, faithfully reflecting the original time information and realizing timestamp reconstruction.

[0023] d) Audio and video streams maintain independent time bases, which is achieved by allocating independent time base storage units and global timestamp counters, thus preventing a single stream malfunction from affecting the synchronization of other streams;

[0024] e) The computational complexity of the entire processing is constant, supporting low-latency processing in high-concurrency scenarios and improving the synchronization stability of multi-source data streams.

[0025] Furthermore, the specific implementation of the dual-threshold timestamp transition detection algorithm includes:

[0026] a) The first threshold criterion is the detection of time-lapse anomalies: when the timestamp of the current frame is less than the timestamp of the previous valid frame, it is determined to be a timestamp time-lapse anomaly, corresponding to data out-of-order, network retransmission or timestamp generation error scenarios.

[0027] b) The second threshold criterion is positive mutation detection: when the difference between the timestamps of two adjacent frames exceeds the preset dynamic threshold, it is determined to be a positive mutation anomaly, corresponding to scenarios such as clock jump at the sending end, severe network jitter, or damage to the timestamp field.

[0028] c) The algorithm execution only includes difference calculation and integer comparison operations, without relying on historical data statistical modeling, to achieve fast and accurate anomaly detection;

[0029] d) The dynamic threshold can be dynamically adjusted according to the media stream type and frame rate to adapt to the time continuity constraints of different scenarios.

[0030] Furthermore, the specific implementation of audio splicing and encapsulation includes:

[0031] a) The dedicated audio buffer is separate from other types of data buffer space and only stores fragmented audio data caused by network packet splitting or jitter;

[0032] b) After new audio data arrives, the system first checks whether there are incomplete audio frames in the buffer, then performs the splicing operation. After accumulating to the size of a complete frame, the data is removed from the buffer, thus realizing intelligent splicing of audio data.

[0033] c) The dynamically added protocol header is in a standardized encapsulation format, containing an algorithm identifier field, a corrected global timestamp, and media attribute parameters;

[0034] d) Media attribute parameters are dynamically adjusted according to frame type.

[0035] Furthermore, the specific implementation of intelligent cache management includes:

[0036] a) The core of the dynamic adaptation strategy of single-stream dynamic buffer is the precise matching of buffer capacity and frame length. When a new frame arrives, it is determined whether the current buffer capacity is sufficient. If it is insufficient, the old buffer is released and a new buffer matching the frame length is allocated.

[0037] b) The global memory pool maintains a linked list of free memory blocks, recording the actual capacity of each free block. When requesting memory, free blocks are reused first, and new memory is requested from the system if no matching block is found.

[0038] c) After the data frame is processed, the memory block is returned to the memory pool. When the number of free blocks exceeds the preset threshold, redundant memory blocks are automatically released.

[0039] Furthermore, the specific implementation of flow scheduling and intelligent congestion control includes: a) priority marking based on the latency sensitivity of the business scenario;

[0040] b) A priority scheduling strategy is adopted, in which high-priority streams can obtain processing resources first to ensure low-latency transmission, while low-priority streams wait for system resources to become available before they can be processed.

[0041] c) The trigger condition for intelligent congestion control is that the length of each stream's independent buffer queue exceeds the dynamic threshold. The frame dropping strategy only applies to non-critical video frames, minimizing the impact on the playback experience while alleviating congestion.

[0042] The present invention adopts the above technical solution, and the specific beneficial effects are as follows:

[0043] The keyframe guidance mechanism ensures that the first keyframe can be successfully received and processed during session initialization, avoiding decoding failure due to the first frame being a non-keyframe, reducing playback latency, and ensuring the normal startup of audio and video streams.

[0044] By using timestamp synchronization and compensation technology and dual-threshold timestamp jump detection, timestamp anomalies (such as rollback or sudden changes) can be accurately handled, and baseline reset and compensation can be performed to ensure accurate synchronization of audio and video streams and avoid problems such as screen jumps or audio and video desynchronization caused by clock inconsistency.

[0045] The intelligent splicing mechanism ensures that when an incomplete audio frame is received, it can be spliced ​​and encapsulated into a complete audio frame in real time, improving the efficiency of audio processing, avoiding retransmission or delay problems caused by audio fragment loss, and improving bandwidth utilization and playback continuity.

[0046] By adopting a dynamic buffer management and global memory pool collaborative management strategy, the system avoids memory fragmentation and resource waste by accurately matching buffer capacity with frame size, thereby improving the system's processing capacity and memory reuse efficiency in high-concurrency scenarios and ensuring real-time processing performance and system stability.

[0047] By employing a priority scheduling strategy based on business scenarios, high-priority real-time audio and video streams can preempt processing resources, reducing the risk of low-latency transmission. Meanwhile, the intelligent congestion control strategy prioritizes discarding non-critical frames when the buffer queue overflows, ensuring the real-time performance and stability of core business data streams.

[0048] The technical solution of this invention can effectively process various types of data streams, including video, audio, GPS data, still images, and serial port data. Through a unified processing framework and multi-level optimization schemes, it ensures the synchronous, stable, and efficient transmission of different types of data streams in multiple scenarios, meeting the needs of high-reliability and high-real-time application scenarios such as real-time audio and video communication, remote monitoring, and live streaming. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the system architecture and audio / video data stream interaction transmission of the present invention.

[0051] Figure 2 This is a flowchart of the real-time audio and video stream processing method of the present invention.

[0052] Figure 3 This is a schematic diagram of frame parsing and keyframe guidance in this invention.

[0053] Figure 4 This is a schematic diagram of the timestamp jump detection and compensation method of the present invention. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0055] like Figure 1 As shown, the real-time audio and video stream processing method provided in this embodiment runs in a multi-layer service system, which includes a client, a client agent, a streaming media scheduling module, a streaming media service module, a device access service module, and audio and video source devices.

[0056] In a complete audio / video streaming session, data interaction follows these steps:

[0057] (1) The client initiates a stream request to the client proxy, and a data transmission channel (DTC) is established between the client proxy and the client.

[0058] (2) The client proxy forwards the request to the streaming media scheduling module, which then allocates it to a suitable streaming media service instance and creates a corresponding streaming channel on that instance;

[0059] (3) The streaming media service module initiates a streaming request to the target audio and video source device through the device access service module;

[0060] (4) The audio and video source device sends the original audio and video stream to the device access service module; the device access service module, as the core execution node of this method, performs a series of processes on the original stream in real time, such as key frame guidance, timestamp jump detection and compensation, audio splicing and encapsulation, and intelligent cache management.

[0061] (5) The processed audio and video streams with stable time base and complete frame structure are transmitted back to the client for playback via the streaming media service module and client proxy through the established DTC.

[0062] like Figures 2 to 4 As shown, this embodiment provides a real-time audio and video stream processing method that integrates keyframe control, timestamp synchronization, and cache optimization. The method includes the following steps:

[0063] a) Receive multiple types of audio and video data streams, receive data packets through network protocols, and extract frame type, keyframe identifier, timestamp, and media attribute parameters from the data packets;

[0064] b) Frame parsing and keyframe guidance: During session initialization, the system maintains a dual-state flag of "keyframe not received" and "keyframe received". If the current frame is not a key frame, the frame is discarded and the system returns to the "blocked waiting" state until the first key frame is received. The system then updates the state to "keyframe received" and allows the key frame and all subsequent frames to pass.

[0065] c) Timestamp synchronization and compensation: An independent timeline is established based on the timestamp of the first keyframe. Dual-threshold timestamp jump detection is performed. If a timestamp anomaly is detected, the global timestamp is updated with a fixed step size compensation.

[0066] d) For normal timestamps, the global smooth timestamp is updated by accurately accumulating the actual difference between adjacent frames to ensure precise synchronization of audio and video streams;

[0067] e) Audio splicing and encapsulation: When the length of audio data is less than that of a complete frame, it is stored in a dedicated audio buffer and the number of bytes is recorded. When new audio data arrives, intelligent splicing is performed. After splicing to the size of a complete frame, an adapted protocol header is dynamically added, a unified format audio frame is generated, and it is moved into the audio ready queue.

[0068] f) Intelligent cache management adopts a collaborative management strategy of single-stream dynamic buffer and global memory pool to dynamically adapt the buffer size and ensure that each data stream can obtain appropriate buffer resources;

[0069] g) Prioritize data streams according to business scenarios. High-priority streams have the right to preempt resources. Priority preemption scheduling strategy is adopted to ensure low-latency transmission of real-time audio and video streams. Stream scheduling and intelligent congestion control mechanisms are used to reduce the loss of non-critical frames.

[0070] like Figure 2 As shown, in the frame parsing and keyframe guidance stage, at the initial stage of data stream processing, the original network data packets first enter the "Frame Parsing and Keyframe Guidance" module. The system processes the data packets according to the dual-state flags of "No Keyframe Received" and "Keyframe Received" maintained during session initialization. The specific steps are as follows: The system extracts metadata such as frame type and keyframe identifier from the received network data packets; if the current frame is a non-keyframe, it is directly discarded and enters the "Blocked Waiting" state; if the received frame is the first keyframe, the state is updated to "Keyframe Received," and the system allows the frame and subsequent frames to proceed to the subsequent processing flow. This design ensures that when the first frame is a non-keyframe, it will not cause decoder initialization failure or long waiting time at the playback end, thereby improving the reliability of playback startup.

[0071] The data stream that has been allowed through the timestamp jump detection and compensation stage enters the "timestamp jump detection and compensation" stage. The specific process is as follows: First, the "receive timestamp" operation is performed to obtain the original timestamp in the data stream; then, the "dual threshold anomaly detection" module is entered to detect timestamp rollback anomalies (current frame timestamp < previous valid timestamp) and positive abrupt change anomalies (timestamp difference between adjacent frames exceeds a preset dynamic threshold); if the timestamp is abnormal, the "fixed step size compensation" operation is performed; if the timestamp is normal, the "precise value accumulation" operation is performed to update the global timestamp according to the actual time difference; finally, a smooth global timestamp is generated, and it is ensured that the audio and video streams each maintain an independent time base.

[0072] In the unified data encapsulation and audio splicing stage, the data stream that has completed timestamp processing enters the data encapsulation and audio splicing stage: For fragmented audio data (single reception length less than a complete audio frame), the system stores it in a dedicated buffer and records the number of buffered bytes; when new audio data arrives, the system checks whether there are incomplete audio frames in the buffer. If so, it performs intelligent splicing until a complete audio frame is spliced; the spliced ​​audio frame will dynamically add metadata such as algorithm identifier, corrected global timestamp, and media attribute parameters to form a unified format audio and video frame and move it into the ready queue.

[0073] In the intelligent caching and memory pool management phase, the encapsulated data stream enters the cache management phase, specifically including:

[0074] Single-stream dynamic buffer management: The buffer size is dynamically adjusted according to the frame length. If the current buffer capacity is insufficient, the old buffer is released and a new buffer matching the current frame data is allocated.

[0075] Global memory pool management: The system reuses memory blocks through a global memory pool. When requesting memory, it prioritizes obtaining free blocks from the pool. If a mismatch is found, it requests new memory from the system. After data processing is completed, memory blocks are returned to the memory pool. When the number of free blocks exceeds the threshold, redundant memory blocks are automatically released to reduce memory fragmentation.

[0076] In the stream scheduling and quality of service assurance phase, after the data stream completes cache management, it enters the stream scheduling phase. Specifically, this includes: marking each stream with a priority based on the business scenario of the data stream; streams with higher priority (such as real-time audio and video intercom streams) are given priority to obtain resources, while streams with lower priority (such as on-demand playback streams) wait for system resources to become available before processing; and monitoring the cache queue of each data stream in real time. If the queue length exceeds a dynamic threshold, intelligent congestion control is initiated, and non-critical video frames are discarded, thereby ensuring low-latency transmission of core services and system stability.

[0077] As one implementation method, the specific implementation of frame parsing and keyframe guidance in this embodiment includes:

[0078] a) Frame type parsing distinguishes the current frame as video, audio, serial data, static image, or general data by using the identifier field in the metadata;

[0079] b) The criterion for determining a keyframe is whether the frame is a key data unit that can be independently decoded;

[0080] c) In the "No key frame received" state, the system only performs non-key frame discard and blocking wait state return operations, and does not start any decoding or subsequent data processing process;

[0081] d) After the state switch, the system enters the normal receiving mode. Subsequent non-critical frames and critical frames pass through in the order of reception to build a complete and continuous data stream, ensuring the integrity and reliability of the initial stage of playback.

[0082] like Figure 3As shown, after receiving the raw stream data, the system first enters the "receive data frame" stage. During this stage, the system receives various types of audio and video data streams and parses them using network protocols. Subsequently, the system enters the "parse keyframe identifier" stage, parsing key information in the data packets, including frame type and keyframe identifier. If the current frame is a keyframe, the system checks its current status through the "system status detection" module. If the status is "no keyframe received," it updates to "keyframe received" and allows the keyframe to pass; if the current frame is not a keyframe, the system returns to the "blocking and waiting" state until the next keyframe is received. Upon receiving the first keyframe, the system enters the "allow keyframe" stage and then the time synchronization processing stage, continuing to execute subsequent frame processing and synchronization operations. If the system status is still "no keyframe received," it returns to the "blocking and waiting" state, waiting for the keyframe to arrive. Once the status changes to "keyframe received," subsequent non-keyframes (such as video P-frames and B-frames) and keyframes can pass through in an orderly manner, constructing a complete data stream. The system remains synchronized during this process and prepares to proceed to the next step of data synchronization and timestamp compensation.

[0083] An I-frame, also known as a keyframe, is a complete image frame in video encoding that does not depend on other frames. I-frames are self-contained and can be decoded independently; therefore, they are typically used as the basis for displaying the image during video playback.

[0084] A P-frame is a frame encoded based on the preceding frame (usually an I-frame or the previous P-frame). It records the differences between the previous and current frames, rather than the complete image. Therefore, P-frames are smaller in size and have higher compression efficiency.

[0085] B-frames are the most complex frames, relying on both the preceding and following frames for encoding, and are typically smaller than I-frames and P-frames. B-frames reduce redundant data through bidirectional prediction, further improving compression efficiency, but require information from adjacent frames for decoding.

[0086] After receiving raw streaming data, the system incorporates an intelligent frame parsing and keyframe guidance mechanism in the initial stage of data stream processing. Specifically, upon receiving raw network data packets, the system accurately extracts metadata information across multiple dimensions, including but not limited to frame type (precisely distinguishing video, audio, serial data, still images, or general data types through fields), keyframe identifiers (used to determine whether the frame is an independently decodeable video I-frame or other similar key data unit), and the original timestamp. Internally, the system maintains a status flag. In the "no keyframe received" state during session initialization, the system rigorously checks each received frame. If the current frame is not a keyframe, the system immediately discards it and returns a "blocking and waiting" status code, thus avoiding decoder initialization failures or prolonged black screen waiting at the playback end due to the first frame being a non-keyframe. Only when the system receives the first keyframe will it update its internal status to "keyframe received" and allow the frame to proceed, entering the subsequent timestamp synchronization and caching processing flow. At this point, the system switches to "normal reception" mode, allowing subsequent non-critical frames (such as video P-frames and B-frames) to be transmitted in an orderly manner along with the critical frames, constructing a complete and continuous data stream. This two-stage state machine design ensures reliable playback initiation while minimizing initial playback latency, thus enhancing the immediacy of the user experience.

[0087] As one implementation method, the specific implementation of timestamp synchronization and compensation in this embodiment includes: a) When the independent timeline is initialized, the original timestamp of the first keyframe is used as the reference point to form a linear mapping relationship without the need for additional calibration process;

[0088] b) Timestamp anomalies include rollback anomalies and positive mutation anomalies. When an anomaly occurs, the timestamp is accepted as the new baseline.

[0089] c) The fixed step size matches the regular playback rhythm of the media stream, ensuring that the global timestamp is monotonous and smooth. The normal timestamp is accurately accumulated according to the actual time difference between adjacent frames, faithfully reflecting the original time information and realizing timestamp reconstruction.

[0090] d) Audio and video streams maintain independent time bases, which is achieved by allocating independent time base storage units and global timestamp counters, thus preventing a single stream malfunction from affecting the synchronization of other streams;

[0091] e) The computational complexity of the entire processing is constant, supporting low-latency processing in high-concurrency scenarios and improving the synchronization stability of multi-source data streams.

[0092] like Figure 4 As shown, the specific implementation of timestamp jump detection and compensation includes the following steps:

[0093] When the system receives each new data frame, it first obtains the timestamp (Tsn) of the current frame. The system determines whether there is an anomaly in the timestamp by detecting the difference between the current timestamp and the previous timestamp (Tsn-1).

[0094] If the timestamp (Tsn) of the current frame is less than the previous timestamp (Tsn-1), a rollback exception is considered to have occurred. This usually indicates out-of-order data, network retransmission, or timestamp generation error. The system will mark this situation and enter the rollback exception handling process.

[0095] If the difference between the timestamp of the current frame and the timestamp of the previous frame (Tsn-1) is greater than 100 milliseconds (i.e., Tsn - Tsn-1 > 100ms), a positive abrupt change is considered to exist. This usually means that the sending device clock has jumped, there is network jitter, or the timestamp field is corrupted. The system will treat this change as a timestamp anomaly.

[0096] In case of rollback anomalies, the system sets the current frame's timestamp (Tsn) as the new baseline timestamp and performs "fixed-step compensation" to ensure the stability and smoothness of the global timestamp. The compensation step size is usually matched with the normal playback rhythm of the media stream, such as 40ms (adjusted according to actual conditions).

[0097] For positive mutation anomalies, the system also adopts a fixed step size compensation mechanism to adjust the timestamp to align it with the normal playback rhythm and avoid obvious jumps during playback.

[0098] Once the timestamps have been processed, the system updates the global timeline based on both the normal and compensated timestamps. Each audio and video stream maintains its own independent time base to avoid timestamp interference between different data streams.

[0099] As one implementation method, the specific implementation of the dual-threshold timestamp jump detection algorithm in this embodiment includes:

[0100] a) The first threshold criterion is the detection of time-lapse anomalies: when the timestamp of the current frame is less than the timestamp of the previous valid frame, it is determined to be a timestamp time-lapse anomaly, corresponding to data out-of-order, network retransmission or timestamp generation error scenarios.

[0101] b) The second threshold criterion is positive mutation detection: when the difference between the timestamps of two adjacent frames exceeds the preset dynamic threshold, it is determined to be a positive mutation anomaly, corresponding to scenarios such as clock jump at the sending end, severe network jitter, or damage to the timestamp field.

[0102] c) The algorithm execution only includes difference calculation and integer comparison operations, without relying on historical data statistical modeling, to achieve fast and accurate anomaly detection;

[0103] d) The dynamic threshold can be dynamically adjusted according to the media stream type and frame rate to adapt to the time continuity constraints of different scenarios.

[0104] Next, timestamp jump detection and compensation are performed, which includes the following four core technical steps:

[0105] The first step is rapid initialization. An independent timeline is established based on the first received timestamp. This design mathematically implements a linear mapping, avoiding complex initial calibration.

[0106] Secondly, to address the unavoidable timestamp anomalies in network transmission, this invention designs an efficient dual-threshold jump detection algorithm. The first criterion is timestamp backtracking detection, which checks whether the timestamp of a newly arrived frame is less than the timestamp of the previous valid frame recorded by the system. According to the physical law of "time irreversibility," any timestamp backtracking clearly indicates abnormalities such as data out of order, network retransmission, or timestamp generation errors. The second criterion is timestamp forward abrupt change detection, which checks whether the difference between the timestamps of two adjacent frames exceeds a preset dynamic threshold (e.g., 100 milliseconds). This threshold is set based on the basic principles of video encoding: at a normal frame rate (e.g., 25fps or 30fps), the display time interval between adjacent video frames is usually between 33 and 40 milliseconds. Therefore, if the difference exceeds this range, it usually indicates a clock jump at the sending device, severe network jitter, or a corrupted timestamp field. This detection process only requires one subtraction operation and two integer comparisons, with a computational complexity of O(1). The unique feature of this algorithm is that it does not rely on complex statistical modeling of historical data, but instead utilizes the inherent continuity constraints of media streams to quickly and accurately detect anomalies, providing a reliable basis for subsequent intelligent compensation.

[0107] Next is the intelligent compensation mechanism. For abnormal timestamps, the system uses a fixed step size (e.g., 40ms) to compensate and update the global timestamp; for normal timestamps, it precisely accumulates the actual difference. For abnormal timestamps, this invention employs a lightweight intelligent compensation mechanism, rather than costly global timeline reconstruction or data rearrangement strategies. The core idea of ​​this mechanism is "hierarchical processing" and "error localization." When a timestamp is determined to be abnormal, the system first uses it as a new reference timestamp to handle possible clock reference switching. Subsequently, a fixed step size (e.g., 40ms) is added to the globally smoothed timestamp maintained internally by the system to compensate, ensuring that the timestamp matches the playback rhythm of the media stream. This compensation strategy is mathematically equivalent to using an ideal timestamp sequence generated at a standard playback rate to replace the potentially erroneous original timestamp sequence. Its core advantage is that the computational cost of the compensation operation remains constant regardless of the magnitude of the abnormal timestamp deviation, thus avoiding complex recalculations triggered by a single outlier. For normal timestamps, the system employs a precise accumulation strategy to faithfully reflect their time information, thereby ensuring the long-term stability and monotonic smoothness of the output timeline.

[0108] Finally, video and audio streams maintain independent time bases and global timestamps respectively to prevent abnormal mutual interference between different types of media streams. The computational complexity of timestamp processing in this method is O(1), which can maintain nanosecond-level single-channel processing latency even in scenarios with thousands of concurrent streams, and ensure audio-visual synchronization accuracy of less than 10 milliseconds, achieving the best balance between processing efficiency and synchronization accuracy.

[0109] The working principle of timestamp detection here is as follows: This timestamp detection algorithm follows the design principles of lightweight and low latency, with the core being to ensure the monotonic smoothness of the global timestamp. It only performs local compensation for abnormal timestamps in a single frame, without performing high-complexity calculations such as global timeline reconstruction. This selective frame dropping strategy effectively alleviates the risk of congestion without excessively sacrificing the playback experience.

[0110] As one implementation method, the specific implementation of audio splicing and encapsulation in this embodiment includes:

[0111] a) The dedicated audio buffer is separate from other types of data buffer space and only stores fragmented audio data caused by network packet splitting or jitter;

[0112] b) After new audio data arrives, the system first checks whether there are incomplete audio frames in the buffer, then performs the splicing operation. After accumulating to the size of a complete frame, the data is removed from the buffer, thus realizing intelligent splicing of audio data.

[0113] c) The dynamically added protocol header is in a standardized encapsulation format, containing an algorithm identifier field, a corrected global timestamp, and media attribute parameters;

[0114] d) Media attribute parameters are dynamically adjusted according to frame type.

[0115] In the data encapsulation stage, the system adopts an adaptive multi-protocol unified encapsulation format. For a complete audio frame (as well as video frames and data frames that have completed the aforementioned processing) retrieved from the ready queue, the system dynamically adds the corresponding protocol header based on the frame type identifier and encoding format information obtained in the first stage. Taking audio as an example, the encapsulation header includes an algorithm identifier field (explicitly indicating that the encoding format is PCM, AAC, or G.711, etc.), a timestamp (from the globally smoothed timestamp corrected in the second stage), the number of channels, the sampling rate, bit depth, and other metadata, forming a standardized audio data packet.

[0116] During data transmission, when the length of audio data received in a single instance is less than a complete audio frame due to network packet fragmentation or jitter, the system will not discard it or trigger a retransmission request. Instead, it will store it in a temporary buffer specifically designed for audio. This buffer can store audio data segments of any size and precisely records the number of bytes currently cached in the data structure. When the next network data packet arrives, the system first checks whether there is an incomplete audio frame in the buffer. If so, it executes an intelligent splicing algorithm: calculates the remaining capacity of the buffer, copies the newly arrived data to the end of the cached data, and updates the cached length. This process continues until the cumulative data length reaches the preset size of a complete audio frame. At this point, the system determines that a logically complete audio frame has been generated, removes the frame from the temporary buffer, and places it in a first-in-first-out (FIFO) audio ready queue, awaiting subsequent transmission.

[0117] As one implementation method, the specific implementation of intelligent cache management in this embodiment includes:

[0118] a) The core of the dynamic adaptation strategy of single-stream dynamic buffer is the precise matching of buffer capacity and frame length. When a new frame arrives, it is determined whether the current buffer capacity is sufficient. If it is insufficient, the old buffer is released and a new buffer matching the frame length is allocated.

[0119] b) The global memory pool maintains a linked list of free memory blocks, recording the actual capacity of each free block. When requesting memory, free blocks are reused first, and new memory is requested from the system if no matching block is found.

[0120] c) After the data frame is processed, the memory block is returned to the memory pool. When the number of free blocks exceeds the preset threshold, redundant memory blocks are automatically released.

[0121] After data encapsulation, the data enters the cache management stage. At the data reception level of a single stream, the system adopts a buffer strategy that dynamically adjusts on demand. Each stream instance is allocated a basic buffer upon initialization. When a new data frame is received, the system checks in real time whether the current buffer capacity is sufficient to accommodate the frame data. If the capacity is insufficient, the system immediately releases the existing buffer and reallocates a new buffer whose size exactly matches the length of the frame data. This "precise matching" strategy effectively avoids long-term memory idle waste caused by an excessively large preset fixed buffer, and also eliminates the overhead of frequent memory reallocation and data copying caused by an excessively small buffer, achieving the best balance between memory utilization and performance. At the system-wide level, a unified memory pool is designed for efficient reuse. When all stream instances request memory blocks, they first attempt to obtain them from the global free memory pool. The memory pool records the actual capacity of each free memory block. When the requested memory size does not exceed the capacity of the existing free block, it is directly reused, avoiding duplicate allocation; only when there is no suitable free block in the pool or the requested size exceeds the capacity of all free blocks will new memory be temporarily requested from the system. Meanwhile, the system monitors the number of free blocks in the memory pool. When the number exceeds a preset threshold, it automatically releases some redundant memory blocks to prevent the memory pool from expanding indefinitely and consuming too many system resources. When a data frame is processed and released, the memory blocks it occupies are reclaimed into this memory pool for subsequent allocation, thus forming an efficient closed loop of memory usage.

[0122] As one implementation method, the specific implementation of flow scheduling and intelligent congestion control in this embodiment includes:

[0123] a) Priority marking is based on latency sensitivity classification according to business scenarios;

[0124] b) A priority scheduling strategy is adopted, in which high-priority streams can obtain processing resources first to ensure low-latency transmission, while low-priority streams wait for system resources to become available before they can be processed.

[0125] c) The trigger condition for intelligent congestion control is that the length of each stream's independent buffer queue exceeds the dynamic threshold. The frame dropping strategy only applies to non-critical video frames, minimizing the impact on the playback experience while alleviating congestion.

[0126] After completing the encapsulation and caching of streaming data, the system enters the stream scheduling and quality of service (QoS) assurance phase. First, the system performs fine-grained classification and priority marking of the input data streams. This classification is not only based on basic media types (such as video and audio) but also delves into specific business scenarios. For example, real-time audio and video intercom streams are marked as the highest priority because they are extremely sensitive to latency; real-time monitoring video streams are the next highest priority; while services with relatively relaxed real-time requirements, such as on-demand playback streams and file download streams, are marked as lower priority. This classification mechanism provides a basis for subsequent resource scheduling decisions. At the data transmission and processing scheduling level, the system implements a strict priority strategy. High-priority streams (such as real-time intercom streams) have resource preemption rights, and their data frames can be processed and transmitted immediately, ensuring minimal end-to-end latency. Low-priority streams (such as on-demand streams) must wait for system resources to become available, and their transmission requests may be temporarily blocked to make way for high-priority services. Simultaneously, to cope with sudden traffic surges or cache backlogs caused by insufficient downstream processing capacity, the system introduces proactive and intelligent congestion control and frame dropping strategies. The system maintains a frame buffer queue for each stream and monitors its length in real time. When the buffer queue length for a stream exceeds a dynamically calculated threshold, the system determines that the stream may face congestion risk. In this case, the system does not simply discard the newest or oldest data, but instead initiates intelligent selective frame dropping. The system prioritizes checking and discarding non-critical video frames in the queue. This "sacrifice pawns to save the king" strategy efficiently resolves congestion crises without excessively sacrificing user experience, ensuring the overall stability of the system and the service quality of high-priority streams, and greatly improving the performance of streaming media services.

[0127] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A real-time audio and video stream processing method integrating keyframe control, timestamp synchronization, and buffer optimization. Its features are, The method includes the following steps: a) Receive multiple types of audio and video data streams, receive data packets through network protocols, and extract frame type, keyframe identifier, timestamp, and media attribute parameters from the data packets; b) Frame parsing and keyframe guidance: When the session is initialized, the system maintains a dual-state flag of "keyframe not received" and "keyframe received". If the current frame is not a key frame, the frame is discarded and the system returns to the "blocked waiting" state until the first key frame is received. The system then updates the key frame to the "keyframe received" state and allows the key frame and all subsequent frames to pass. At the same time, the service will cache the received key frames internally. c) Timestamp synchronization and compensation: An independent timeline is established based on the timestamp of the first keyframe. Dual-threshold timestamp jump detection is performed. If a timestamp anomaly is detected, the global timestamp is updated with a fixed step size compensation. d) For normal timestamps, the global smooth timestamp is updated by accurately accumulating the actual difference between adjacent frames to ensure precise synchronization of audio and video streams; e) Audio splicing and encapsulation: When the length of audio data is less than that of a complete frame, it is stored in a dedicated audio buffer and the number of bytes is recorded. When new audio data arrives, intelligent splicing is performed. After splicing to the size of a complete frame, an adapted protocol header is dynamically added, a unified format audio frame is generated, and it is moved into the audio ready queue. f) Intelligent cache management adopts a collaborative management strategy of single-stream dynamic buffer and global memory pool to dynamically adapt the buffer size and ensure that each data stream can obtain appropriate buffer resources; g) Prioritize data streams according to business scenarios. High-priority streams have the right to preempt resources. Priority preemption scheduling strategy is adopted to ensure low-latency transmission of real-time audio and video streams. Stream scheduling and intelligent congestion control mechanisms are used to reduce the loss of non-critical frames.

2. The method according to claim 1, characterized in that, The specific implementation of frame parsing and keyframe guidance includes: a) Frame type parsing distinguishes whether the current frame is a video frame or an audio frame by using the identifier field in the metadata; b) The criterion for determining a keyframe is whether the frame is a key data unit that can be independently decoded; c) In the "No keyframe received" state, the system only performs non-keyframe discard and blocking wait state return operations, and does not start any decoding or subsequent data processing process; d) After the state switch, the system enters the normal receiving mode. Subsequent non-critical frames and critical frames pass through in the order of reception to build a complete and continuous data stream, ensuring the integrity and reliability of the initial stage of playback.

3. The method according to claim 1, characterized in that, The specific implementation of timestamp synchronization and compensation includes: a) When initializing an independent timeline, the original timestamp of the first keyframe is used as the reference point to form a linear mapping relationship without the need for additional calibration procedures; b) Timestamp anomalies include rollback anomalies and positive mutation anomalies. When an anomaly occurs, the timestamp is accepted as the new baseline. c) The fixed step size matches the regular playback rhythm of the media stream, ensuring that the global timestamp is monotonous and smooth. The normal timestamp is accurately accumulated according to the actual time difference between adjacent frames, faithfully reflecting the original time information and realizing timestamp reconstruction. d) Audio and video streams maintain independent time bases, which is achieved by allocating independent time base storage units and global timestamp counters, thus preventing a single stream malfunction from affecting the synchronization of other streams.

4. The method according to claim 3, characterized in that, The specific implementation of the dual-threshold timestamp jump detection algorithm includes: a) The first threshold criterion is the detection of time-lapse anomalies: when the timestamp of the current frame is less than the timestamp of the previous valid frame, it is determined to be a timestamp time-lapse anomaly, corresponding to data out-of-order, network retransmission or timestamp generation error scenarios. b) The second threshold criterion is positive mutation detection: when the difference between the timestamps of two adjacent frames exceeds the preset dynamic threshold, it is determined to be a positive mutation anomaly, corresponding to scenarios such as clock jump at the sending end, severe network jitter, or damage to the timestamp field. c) The algorithm execution only includes difference calculation and integer comparison operations, without relying on historical data statistical modeling, to achieve fast and accurate anomaly detection; d) The dynamic threshold can be dynamically adjusted according to the media stream type and frame rate to adapt to the time continuity constraints of different scenarios.

5. The method according to claim 1, characterized in that, The specific implementation of audio splicing and encapsulation includes: a) The dedicated audio buffer is separate from other types of data buffer space and only stores fragmented audio data caused by network packet splitting or jitter; b) After new audio data arrives, the system first checks whether there are incomplete audio frames in the buffer, then performs the splicing operation. After accumulating to the size of a complete frame, the data is removed from the buffer, thus realizing intelligent splicing of audio data. c) The dynamically added protocol header is in a standardized encapsulation format, containing an algorithm identifier field, a corrected global timestamp, and media attribute parameters; d) Media attribute parameters are dynamically adjusted according to frame type.

6. The method according to claim 1, characterized in that, The specific implementation of intelligent cache management includes: a) The core of the dynamic adaptation strategy of single-stream dynamic buffer is the precise matching of buffer capacity and frame length. When a new frame arrives, it is determined whether the current buffer capacity is sufficient. If it is insufficient, the old buffer is released and a new buffer matching the frame length is allocated. b) The global memory pool maintains a linked list of free memory blocks, recording the actual capacity of each free block. When requesting memory, free blocks are reused first, and new memory is requested from the system if no matching block is found. c) After the data frame is processed, the memory block is returned to the memory pool. When the number of free blocks exceeds the preset threshold, redundant memory blocks are automatically released.

7. The method according to claim 1, characterized in that, The specific implementation of flow scheduling and intelligent congestion control includes: a) priority marking based on the latency sensitivity of the business scenario; b) A priority scheduling strategy is adopted, in which high-priority streams can obtain processing resources first to ensure low-latency transmission, while low-priority streams wait for system resources to become available before they can be processed. c) The trigger condition for intelligent congestion control is that the length of each stream's independent buffer queue exceeds the dynamic threshold. The frame dropping strategy only applies to non-critical video frames, minimizing the impact on the playback experience while alleviating congestion.

Images