Method, system and terminal for multiplexing and synthesizing multiple heterogeneous audio-video streams, and storage medium
By maintaining the latest frame slots and managing reference counts through a hardware graphics engine, the problems of format unification and clock alignment in the processing of multi-channel heterogeneous audio and video streams are solved, achieving zero-copy multiplexing and synthesis, simplifying the system structure and reducing latency and resource consumption.
Patent Information
- Application Number
- CN202611496002.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-24
- Publication Date
- 2026-11-06
AI Technical Summary
Existing technologies for processing multi-channel heterogeneous audio and video streams suffer from problems such as redundant processing introduced by format unification, high latency introduced by clock alignment, wasted resources due to decoding and re-encoding, and copying by multiple consumers, resulting in excessive system complexity and resource consumption.
A hardware graphics engine is used to maintain the latest frame slot of each audio and video source stream. Zero-copy multiplexing and synthesis are achieved through cross-format conversion and reference counting management, avoiding clock alignment and format unification preprocessing. Heterogeneous source streams are directly accessed and format conversion is completed during the synthesis process.
It simplifies the system structure, reduces latency and complexity, improves screen availability and continuity, reduces memory copying overhead and resource consumption, and achieves zero-copy multi-channel audio and video stream processing.
Smart Images
Figure CN123055931A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio and video stream processing technology, and more specifically, to a method, system, terminal, and storage medium for multiplexing and synthesizing multiple heterogeneous audio and video streams. Background Technology
[0002] In a multi-channel audio and video monitoring recording and display system, multiple audio and video streams are typically accessed simultaneously: one type is the local source transmitted via a local high-speed bus (such as PCIe) from a programmable logic device (FPGA), and the other type is the remote source pulled via a network (such as RTMP / RTSP). The system needs to multiplex these heterogeneous source streams into the same screen (canvas) for splicing display and record them separately.
[0003] Existing technologies generally employ the following process when handling such heterogeneous multi-streams: 1. Format unification: First, decode each stream, and then use software or a separate image processing unit to unify the different pixel formats into the same format.
[0004] 2. Clock alignment: The clocks of each channel are aligned / resampled to the same clock domain through timestamp alignment and then synthesized.
[0005] 3. Decode and re-encode recording: When recording a network stream, first decode and then re-encode (or at least decode and then re-encapsulate) to obtain a recording file in a uniform format.
[0006] The existing technology has the following inherent defects: 1. Format unification introduces redundant processing: Local sources (e.g., UYVY format) and network sources (e.g., NV12 format) have different pixel formats. Existing technologies require explicit software format conversion or the addition of a separate image processing unit (e.g., VPSS) as an intermediate level, resulting in additional CPU usage, memory bandwidth and processing latency.
[0007] 2. Clock alignment introduces high latency: Although multiple sources are nominally 60fps, they are driven by independent clock sources (e.g., FPGA outputs 1080p60 local source with a 148.5MHz pixel clock, network source is constrained by remote encoder clock, and display is constrained by vsync). Each clock has a slight frequency drift. Existing technology believes that clock-level alignment must be performed for synthesis, which leads to a significant increase in buffer latency and implementation complexity. However, in the case of surveillance recording, the slight clock drift of multiple screens is acceptable.
[0008] 3. Decoding and re-encoding wastes resources when recording network streams: Decoding and re-encoding (or decoding and re-encapsulation) will consume the decoder (VDEC), encoder (VENC) and a large amount of memory bandwidth, while the recording requirement is only to "preserve the source stream itself", without changing its bitrate, resolution and frame rate.
[0009] 4. Multi-consumer copying: When the same frame of data needs to be used by multiple consumers for encoding, display, and recording at the same time, existing technologies often copy it for each consumer, which brings memory copying overhead and cache consistency issues.
[0010] Therefore, a method for multiplexing and synthesizing multiple heterogeneous audio and video streams that can solve the above-mentioned defects is needed. Summary of the Invention
[0011] The technical problem to be solved by the present invention is to provide a method for multiplexing and synthesizing multiple heterogeneous audio and video streams, a system for multiplexing and synthesizing multiple heterogeneous audio and video streams, a terminal for multiplexing and synthesizing multiple heterogeneous audio and video streams, and a computer-readable storage medium, in order to address the above-mentioned deficiencies of the prior art.
[0012] The technical solution adopted by this invention to solve its technical problem is: A method for multiplexing and synthesizing multiple heterogeneous audio and video streams is constructed, comprising the following steps: Receive multiple audio and video source streams, which include at least a local source stream transmitted locally and a network source stream transmitted over the network; Maintain a latest frame slot for each audio / video source stream and replace the previous frame in the latest frame slot with the latest arriving frame. When an audio / video source stream is interrupted, keep the previous frame in the latest frame slot of that stream unchanged. There is no clock alignment or resampling between the audio / video source streams, and the latest frame slots of each stream are updated independently. The hardware graphics engine scales and positions the latest frame slots of each audio and video source stream to a unified canvas, and simultaneously performs cross-format conversion between different pixel formats during the scaling and positioning process, so as to combine multiple audio and video source streams into a unified format canvas. Reference counting enables multiple consumers to share the corresponding frame data in the synthesized uniform format canvas and the video frame data in each latest frame slot: the encoding consumer, display consumer, and recording consumer each increment the reference count of the same frame data before using it, and decrement it after use, thus achieving zero-copy multiplexing.
[0013] The multi-channel heterogeneous audio and video stream multiplexing and synthesis method of the present invention, wherein replacing the previous frame in the latest frame slot with the latest arriving frame includes: Each audio / video source stream corresponds to a latest frame slot, which stores a reference to the latest frame of that audio / video source stream; When the latest frame arrives, the reference count of the latest frame is incremented by one, the latest frame is written to the latest frame slot, and the reference count of the previous frame in the latest frame slot is decremented by one, thus realizing the atomic replacement of the latest frame slot; The timestamps of the source frames of each audio and video source stream are uniformly converted to the same time unit when entering the latest frame slot for display and encoding.
[0014] The method for multiplexing and synthesizing multiple heterogeneous audio and video streams according to the present invention includes, in which, when a certain audio or video source stream is interrupted, keeping the previous frame in the latest frame slot of that stream unchanged includes: When no new frames arrive from a certain audio or video source stream for a period of time, the latest frame slot for that stream is not updated. The hardware graphics engine continues to use the previous frame in the latest frame slot of that stream for compositing, so that the image of that stream remains the last valid image. The update is resumed when the audio or video source stream is restored. The hardware graphics engine uses the display refresh signal as the synthesis master clock, and in each synthesis cycle, it successively takes the latest frame from each latest frame slot to participate in the synthesis; the determination duration of the time period is determined according to the nominal frame interval of the audio and video source stream.
[0015] The multi-channel heterogeneous audio and video stream multiplexing and synthesis method of the present invention includes, in which a hardware graphics engine scales and positions the frames in the latest frame slots of each audio and video source stream to a unified canvas, and simultaneously completes cross-format conversion between different pixel formats during the scaling and positioning process, so as to synthesize multiple audio and video source streams into a unified format canvas, including: The hardware graphics engine supports batch task mode: start a compositing task group, attach multiple scaling and positioning tasks within the compositing task group, each scaling and positioning task independently declares the pixel format of its source frame, source rectangle and target rectangle, and the target canvas is fixed to a certain format; when executing each scaling and positioning task, the hardware graphics engine automatically completes the conversion from the source format to the target canvas format while scaling and positioning. This also includes: pre-verifying the pixel format conversion combinations supported by each of the multiple processing cores of the hardware graphics engine; when executing the scaling and positioning task, selecting a processing core from the multiple processing cores that supports the source format to target format conversion required for the task to execute the task; the target rectangle of each scaling and positioning task is determined by a layout table, which records the position and size of each window under different source stream paths; when the source stream path changes, switching to the corresponding layout table while keeping the execution logic of the compositing task group unchanged.
[0016] The multi-channel heterogeneous audio and video stream multiplexing and synthesis method of the present invention, wherein the pixel format of the local source stream is a first format and the pixel format of the network source stream is a second format, the first format and the second format are respectively used as sources for different tasks, and the target is the same canvas; the hardware graphics engine enables the two different types of source streams to be synthesized into a unified canvas without software format conversion or intermediate image processing unit.
[0017] The method for multiplexing and synthesizing multiple heterogeneous audio and video streams described in this invention further includes zero-decoding pass-through recording of the network source stream: A branch point is set at the demultiplexing output, so that the video elementary stream and audio elementary stream obtained by demultiplexing the network source stream are sent to both the decoding / display path and the recording path, and the two paths do not block each other.
[0018] The multi-channel heterogeneous audio and video stream multiplexing and synthesis method of the present invention, wherein, If the video basic stream is already in Annex-B format with a start code, it is written to disk directly in the order of arrival. If the video basic stream is in length prefix format, the length prefix before each network abstraction layer unit is replaced with the start code before writing to disk, and the sequence parameter set and image parameter set obtained by demultiplexing are inserted before each keyframe. After the recording path is started, it waits for the first keyframe of the video basic stream to arrive before starting to write to disk. A video file that can be played directly is generated without decoding or re-encoding. If the audio primary stream is raw AAC, a 7-byte ADTS header is added before each audio frame to generate an independently playable audio file; the 7-byte ADTS header contains the sync word, audio object type, sample rate index, channel configuration, and frame length.
[0019] A multi-channel heterogeneous audio and video stream multiplexing and synthesis system is provided to implement the multi-channel heterogeneous audio and video stream multiplexing and synthesis method described above, comprising: The receiving module is used to receive multiple audio and video source streams, which include at least a local source stream transmitted locally and a network source stream transmitted over the network. The latest frame slot maintenance module is used to maintain a latest frame slot for each audio and video source stream and replace the previous frame in the latest frame slot with the latest arriving frame. When an audio or video source stream is interrupted, the previous frame in the latest frame slot of that stream remains unchanged. The hardware graphics engine is used to scale and position the latest frame slots of various audio and video source streams to a unified canvas, and simultaneously complete cross-format conversion between different pixel formats during the scaling and positioning process, and combine multiple audio and video source streams into a unified format canvas. The reference counting management module is used to enable multiple consumers to share the corresponding frame data in the synthesized unified format canvas and the video frame data in the latest frame slot of each channel through reference counting: the encoding consumer, display consumer, and recording consumer each increment the reference count of the same frame data before using it, and decrement it after use, thus achieving zero-copy multiplexing. The pass-through recording module is used to set a branch point at the demultiplexing output, so that the video primary stream and audio primary stream obtained by demultiplexing the network source stream are sent to both the decoding / display path and the recording path, and the two paths do not block each other.
[0020] A multi-channel heterogeneous audio and video stream multiplexing and synthesis terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.
[0021] A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described above.
[0022] The beneficial effects of this invention are as follows: 1. Heterogeneous source streams can be directly accessed: It is not required that each stream has the same pixel format or be in the same clock domain when it is accessed. The access side does not need to configure pre-processing such as format conversion or time base alignment for each stream, which simplifies the system structure.
[0023] 2. Minor differences in clock speeds between different channels will not cause image distortion or blockage: Each source stream maintains its own latest frame slot and updates independently. The synthesizer takes the latest frame in each synthesis cycle to participate in the synthesis. Minor differences in the frame rate of a certain channel will not accumulate and be transmitted between channels, nor will waiting for a certain channel block other channels. Compared with the solution that requires unifying all channels to the same clock domain, this invention does not require the introduction of an alignment buffer, which reduces the latency and implementation complexity of the synthesis link.
[0024] 3. Preserve the last valid image instead of a black screen when a single stream is interrupted: When the source stream is interrupted, the last frame on that stream is preserved and the slot is not cleared. The compositing end continues to use this frame to participate in the compositing during the interruption, avoiding black blocks on the entire screen and improving the availability and continuity of the screen in multi-channel monitoring display scenarios.
[0025] 4. Cross-format conversion is completed incidentally during scaling and positioning: The hardware graphics engine completes format conversion while performing scaling and positioning, and pixel format unification is no longer a processing step independent of compositing. Compared with the solution of decoding first and then converting by software or unifying the format by independent intermediate processing units such as VPSS, this independent step is eliminated, which correspondingly saves its CPU usage and the memory bandwidth occupied by at least one complete frame read and write, and shortens the processing chain.
[0026] 5. Canvas frame data is shared by multiple consumers without memory copying: Encoding, display, and recording consumers only increment the reference count of the same frame data before using it and decrement it after use, without copying the frame data itself, thus eliminating the memory copying overhead and cache consistency overhead in multi-consumer scenarios; at the same time, the frame data is not released or overwritten before all consumers have finished using it, and it is recycled and reused after use, achieving zero copying while ensuring the correct lifecycle. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a flowchart of a preferred embodiment of the multi-channel heterogeneous audio and video stream multiplexing and synthesis method of the present invention; Figure 2 This is a schematic diagram of the latest frame slot multiplexing and frame preservation method of the multi-channel heterogeneous audio and video stream multiplexing and synthesis method of the preferred embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware cross-format conversion and synthesis method for multiplexing and synthesizing multi-channel heterogeneous audio and video streams according to a preferred embodiment of the present invention; Figure 4 This is a flowchart of a preferred embodiment of the multi-channel heterogeneous audio and video stream multiplexing and synthesis method with zero-decoding pass-through recording. Figure 5 This is a multi-consumer reference counting lifecycle diagram of a multi-heterogeneous audio and video stream multiplexing and synthesis method according to a preferred embodiment of the present invention; Figure 6 This is a block diagram illustrating the principle of a multi-channel heterogeneous audio and video stream multiplexing and synthesis system according to a preferred embodiment of the present invention. Figure 7 This is a schematic diagram of the overall architecture of a multi-channel heterogeneous audio and video stream multiplexing and synthesis system according to a preferred embodiment of the present invention. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.
[0029] The preferred embodiment of the multi-channel heterogeneous audio and video stream multiplexing and synthesis method of the present invention, such as... Figure 1 As shown, see also Figures 2-5 This includes the following steps: S01: Receive multiple audio and video source streams, which include at least a local source stream transmitted locally and a network source stream transmitted over the network. The local source stream is transmitted from the programmable logic device (FPGA) via a local high-speed bus (such as PCIe, using DMA). The video frames of the local source stream are in the first format (such as the UYVY format of YUV422, 1920×1080, frame rate 60fps, driven by a 148.5MHz pixel clock). The network source stream is pulled by a network protocol (such as RTMP / RTSP) and then hardware decoded by a hardware decoder (VDEC) to obtain video frames. The video frames obtained by hardware decoding are in the second format (such as NV12 format, 1920×1080, frame rate approximately 60fps). The first format and the second format are different, and the local source stream and the network source stream are in different clock domains.
[0030] The transmission paths and pixel formats of the local source stream and the network source stream are different, but S01 to S04 do not require each channel to have the same pixel format or be in the same clock domain when it is accessed. Therefore, the access side does not need to configure pre-processing for format conversion or time base alignment for each channel, which simplifies the system structure.
[0031] S02: Maintain a latest frame slot for each audio / video source stream, and replace the previous frame in the latest frame slot with the latest arriving frame. When an audio / video source stream is interrupted, keep the previous frame in the latest frame slot unchanged. For example: Local source channels 0 to 3 correspond to the latest frame slots 0 to 3 respectively, and network source channels 0 to 3 correspond to the latest frame slots 4 to 7 respectively. Each latest frame slot is updated independently, and no clock alignment or resampling is performed between the audio and video source streams.
[0032] Each source stream maintains its own latest frame slot and updates independently. The synthesizer takes the latest frame in each synthesis cycle to participate in the synthesis. Small differences in the frame rate of a certain stream (such as small frequency drifts caused by independent clock sources but nominally 60fps) will not accumulate and be transmitted between streams, nor will other streams be blocked while waiting for one stream. Compared with the solution that requires unifying all streams to the same clock domain, this embodiment does not require the introduction of alignment buffers, which reduces the latency and implementation complexity of the synthesis link.
[0033] like Figure 2 As shown, replacing the previous frame in the latest frame slot with the most recently arrived frame includes: Each audio / video source stream corresponds to a latest frame slot, which stores a reference to the latest frame of that audio / video source stream; When the latest frame arrives, the reference count of the latest frame is incremented by one, the latest frame is written to the latest frame slot, and the reference count of the previous frame in the latest frame slot is decremented by one, thus realizing the atomic replacement of the latest frame slot; The timestamps of the source frames of each audio and video source stream are uniformly converted to the same time unit when entering the latest frame slot for display and encoding.
[0034] The latest frame slot stores a reference to the latest frame (such as a memory block handle). The increment and decrement operations of the reference count are completed under the protection of the slot lock. The timestamp units of the source frames may be different (such as nanoseconds for local sources and microseconds for network sources). When a frame enters the latest frame slot, it is uniformly converted to the same time unit (such as microseconds).
[0035] The atomic replacement order of incrementing by one and decrementing by one is adopted, so that there is no intermediate state of empty slots between the writing of new frames and the release of old frames. This avoids the replacement occurring while the hardware graphics engine is reading a slot, which would cause reading errors. The timestamp is unified in unit when entering the latest frame slot, so that subsequent display and encoding no longer need to distinguish between different sources, simplifying the time base processing.
[0036] like Figure 2 As shown, maintaining the previous frame in the latest frame slot of a certain audio or video source stream unchanged when the source stream is interrupted includes: When no new frames arrive from a certain audio or video source stream for a period of time, the latest frame slot for that stream is not updated. The hardware graphics engine continues to use the previous frame in the latest frame slot of that stream for compositing, so that the image of that stream remains the last valid image. The update is resumed when the audio or video source stream is restored. The hardware graphics engine uses the display refresh signal as the master clock for compositing, and in each compositing cycle, it successively retrieves the latest frame from each latest frame slot to participate in the compositing.
[0037] The time interval for determining when no new frames arrive within a certain period can be determined based on the nominal frame interval of the road, for example, by taking a multiple of the nominal frame interval; the hardware graphics engine uses the display refresh signal (such as vsync) as the synthesized master clock.
[0038] When a source stream is interrupted, the previous frame is retained and the slots are not cleared. During the interruption, the compositing end continues to use the last frame before the interruption to participate in the compositing, so that the picture retains the last valid picture instead of a black screen. This avoids black blocks appearing in the entire spliced picture due to a single interruption, and improves the availability and visual continuity of the picture in multi-channel monitoring display scenarios. The display refresh signal is used as the compositing master clock, so that the compositing rhythm is consistent with the display output, avoiding the accumulation or starvation caused by the mismatch between the canvas frame output and the display consumption rate.
[0039] S03: The hardware graphics engine scales and positions the latest frame slots of each audio and video source stream to a unified canvas, and simultaneously completes cross-format conversion between different pixel formats during the scaling and positioning process, so as to combine multiple audio and video source streams into a unified format canvas. Hardware graphics engines (such as RGA) scale and position the frames in each latest frame slot to a unified canvas, and simultaneously complete cross-format conversion between the first format and the second format to the unified canvas format during the scaling and positioning process. Pixel format unification is no longer a processing step independent of compositing. Compared with the scheme of decoding each source stream first, and then unifying it into the same format through software conversion or independent intermediate image processing units such as VPSS before compositing, this embodiment eliminates the independent format conversion step, thereby eliminating the CPU usage of this step and the memory bandwidth occupied by at least one complete frame read and write, and shortening the processing link from the source frame to the canvas.
[0040] Specifically, such as Figure 3 As shown: The hardware graphics engine supports batch task mode: start a compositing task group, attach multiple scaling and positioning tasks within the compositing task group, each scaling and positioning task independently declares the pixel format of its source frame, source rectangle and target rectangle, and the target canvas is fixed to a certain format; when executing each scaling and positioning task, the hardware graphics engine automatically completes the conversion from the source format to the target canvas format while scaling and positioning. This also includes: pre-verifying the pixel format conversion combinations supported by each of the multiple processing cores of the hardware graphics engine, and selecting the processing core that supports the source-to-target format conversion required for the task from among the multiple processing cores when performing scaling and positioning tasks.
[0041] The target rectangle for each scaling and positioning task is determined by the layout table. The layout table records the position and size of each window under different source streams (e.g., 4×2 for 8 streams, 480×540 per window; 2×2 for 4 streams, 960×540 per window). When the source streams change, the layout table is switched to the corresponding layout table while keeping the execution logic of the composition task group unchanged. When a certain processing core has limitations on specific format conversion, a core that supports the conversion (e.g., RGA3 core) is selected for execution.
[0042] Multiple scaling and positioning tasks declaring different source formats can be attached to a single compositing task group, enabling heterogeneous format source streams to be composited in the same hardware commit without the need to initiate separate hardware calls for each path; different paths can be adapted by switching according to the layout table, and the core logic does not need to be modified; the support for cross-format conversion among the multiple processing cores of the hardware graphics engine is not asymmetrical, and some cores have conversion restrictions for specific format combinations (such as between UYVY and NV12). Directly submitting the task will result in conversion failure. Therefore, it is necessary to verify the format conversion combinations supported by each core in advance and select the core that supports the combination during execution to ensure that the conversion failure count is zero (rgaFail=0 in actual testing).
[0043] like Figure 3As shown, the pixel format of the local source stream is the first format, and the pixel format of the network source stream is the second format. The first format and the second format serve as sources for different tasks, but the target is the same canvas. The hardware graphics engine enables the two different source streams to be combined into a unified canvas without software format conversion or intermediate image processing units.
[0044] The local source video frames in the latest frame slots 0-3 are in the first format (UYVY), and the network source video frames in the latest frame slots 4-7 are in the second format (NV12). Both serve as sources for different scaling and positioning tasks, with a unified canvas (e.g., a 1920×1080 UYVY canvas) as the target. The first format can be UYVY or NV16, the second format can be NV12 or NV21, and the unified canvas format can be UYVY or NV12.
[0045] Two different pixel format source streams are mixed into the same canvas as sources for different scaling and positioning tasks. The hardware graphics engine performs the conversion from each source format to the canvas format when performing scaling and positioning, thereby achieving heterogeneous format unification without CPU software conversion or intermediate image processing units such as VPSS. The system does not need to add a separate format conversion stage, saving device cost and board area.
[0046] S04: Reference counting enables multiple consumers to share the corresponding frame data in the synthesized uniform format canvas: the encoding consumer, display consumer, and recording consumer each increment the reference count of the same frame data before using it, and decrement it after use, thus achieving zero-copy multiplexing.
[0047] Frame data is not released or overwritten until all consumers have finished using it. When the reference count reaches zero, the memory occupied is reclaimed for reuse by subsequent canvas frames. Canvas frames can be managed using a buffer pool rotation method, and canvas frames with a reference count of zero are returned to the pool for reuse.
[0048] like Figure 5As shown, the lifecycle of the reference count is as follows: when a canvas frame is generated, the reference count is set to an initial value (e.g., 1); the encoding consumer and the display consumer, as direct consumers of the canvas frame, increment the reference count of the canvas frame when using it and decrement it when finished using it; the recording consumer works on the output bitstream of the encoding consumer, and by incrementing the reference count of the canvas frame, it ensures that the canvas frame it depends on is not released or overwritten before the encoding is completed; each consumer decrements the reference count after use, and after all consumers have finished consuming the canvas frame, the reference count is returned to zero and the memory is reclaimed for reuse by subsequent canvas frames. Among them, the recording consumer refers to the party that encodes and records a uniform format canvas (i.e., splicing recording), which is a different recording method from the zero-decoding pass-through recording for network source streams described later; the direct consumers of the canvas frame are the encoding consumer and the display consumer. The recording consumer works on the output bitstream of the encoding consumer. It ensures that the canvas frame it depends on is not released or overwritten before the encoding is completed by incrementing the reference count of the canvas frame. Thus, the three form a "encoding-display-recording" reference chain for the same canvas frame. The canvas frame remains valid until any consumer has finished using it.
[0049] Furthermore, the video frame data in the latest frame slot is also shared through reference counting: during the interruption of a source stream, the last frame in the latest frame slot of that stream continues to survive because it is still referenced by the compositor and the reference count does not return to zero, thus continuing to be used for compositing during the interruption, and seamlessly connecting after the source stream is restored; that is, the preservation of frames during interruption is achieved by keeping the frames alive through reference counting, and the latest frame slot, hardware cross-format conversion, and reference counting form a synergistic effect that no single feature can provide.
[0050] The encoding consumer, display consumer, and recording consumer only increment or decrement the reference count of the same frame data, without copying the frame data itself, thus eliminating the memory copying overhead and cache consistency overhead in multi-consumer scenarios. At the same time, the frame data is not released or overwritten before all consumers have finished using it, achieving zero-copy sharing while ensuring the correct lifecycle of the frame data.
[0051] like Figure 4 As shown, it also includes zero-decoding pass-through recording of the network source stream: A branch point is set at the demultiplexing output, and the video and audio elementary streams obtained by demultiplexing the network source stream are sent to both the decoding / display path and the recording path, with the two paths not blocking each other; the network source stream enters the latest frame slot to participate in synthesis and also branches and passes through the recording at the demultiplexing point. The two processes the same network source stream together, which constitutes the common inventive concept of this invention.
[0052] After the network source stream is demultiplexed by a demultiplexer (such as FFmpeg), a video elementary stream (such as H.264) and an audio elementary stream (such as AAC) are obtained. A branch point is set at the demultiplexing output. The same elementary stream data is sent to both the decoding / display path (after hardware decoding, it enters the latest frame slot to participate in synthesis) and the recording path to write to disk. When the recording queue is full, the current data is discarded without blocking the demultiplexing thread and the decoding / display path.
[0053] Network source stream recording does not occupy the decoder and encoder, does not change the bitrate, resolution and frame rate of the source stream, and the recorded content is completely consistent with the source stream; the recording path and the decoding / display path do not block each other, data is discarded when the recording side is congested without affecting the display, and the integrity of the recorded file is not affected when the display side is abnormal.
[0054] Preferred, If the video primary stream is already in Annex-B format with a start code, it is written directly to the disk in the order of arrival, generating a video file that can be played directly without decoding or re-encoding. If the audio primary stream is raw AAC, a 7-byte ADTS header is added before each audio frame to generate an independently playable audio file; the 7-byte ADTS header contains the sync word, audio object type, sample rate index, channel configuration, and frame length.
[0055] When the video primary stream is already in Annex-B format with a start code, its sequence parameter set (SPS) and image parameter set (PPS) are already located before the keyframes. It can be directly written to the disk in the order of arrival to generate a playable video file. If the video primary stream obtained by demultiplexing is in length prefix format, it can be processed by the bitstream layer (the length prefix is replaced with the start code, and the sequence parameter set and image parameter set are inserted before the keyframes) to be converted to a format with a start code before being written to the disk. This processing does not involve decoding and re-encoding. After the recording path is started, it waits for the first keyframe of the video primary stream to arrive before starting to write to the disk. When the audio primary stream is raw AAC, its audio-specific configuration (AudioSpecificConfig) is stored in the decoder parameter area. The configuration, sample rate index, and channel configuration are obtained by parsing the audio-specific configuration. The frame length is determined by the sum of the audio frame length and the 7-byte ADTS header length.
[0056] The video and audio elementary streams are processed by the bitstream layer and then written directly to the disk, generating a video file that can be played directly and an audio file that can be played independently. The recording process does not introduce decoding and re-encoding, which preserves the original quality of the source stream and avoids additional occupation of hardware decoders and encoders. After recording starts, it waits for the first keyframe before recording begins to avoid the beginning of the recorded file being undecoded due to the lack of reference frames.
[0057] In other preferred embodiments of the present invention, the parameters may take the following non-limiting values: The RK3588 data processing chip integrates 4 local FPGA sources (1080p UYVY@60fps, 148.5MHz pixel clock, via PCIe) and 4 RTMP network sources (1080p@60fps, via VDEC) into a unified 4×2 canvas HDMI display, and records 9 channels (4 local + 4 network + 1 splicing).
[0058] Actual test results (continuous recording for 23 hours, 6 minutes, and 35 seconds, or 83,195 seconds, with 9 video streams recorded):
[0059] Note: The fps field (e.g., 60.2) in the actual test log is the instantaneous value sampled by the code according to the statistical time window; the FPGA local source frame rate is nominally 60fps (driven by a 148.5MHz pixel clock), but there is a slight fluctuation during operation, and it is not a fixed 60 frames; 19,946,192 frames ÷ 4 channels ÷ 83195 seconds = 59.94fps, and the deviation from the nominal 60fps is within the sampling accuracy range. The average value of each channel is 59.94fps.
[0060] Data Meaning: The above tests show that in a complex scenario of 23 hours and 6 minutes with 9 concurrent video streams (including heterogeneous UYVY+NV12 format mixing, network pass-through recording, and splicing encoding), all indicators such as cross-format conversion failure (rgaFail), canvas exhaustion (canvasEmpty), splicing encoding failure (vencFail), frame drop (encDrop), frame drop in network pass-through recording, and disk writing failure (err) are all 0. That is, the "hardware cross-format conversion + latest frame slot + pass-through recording" mechanism can run stably for a long time without format conversion failure or recording / display failure.
[0061] Direct evidence of interruption recovery: During the test, the third network source (net3) experienced one interruption and reconnection (reconn=1). Within the interruption window, no new frames arrived on this channel, and the latest frame slot retained its last frame before the interruption for synthesis. This last frame continued to survive because it was still referenced by the synthesizer and the reference count did not return to zero. After the reconnection was restored, the channel continued to write to the disk with drop=0 and err=0 (only the cumulative frame count was slightly less than the other three channels due to the interruption window), directly confirming the technical features of "interruption frame preservation + decoupling of recording and display".
[0062] It should be noted that the specific device, protocol, and bus names such as RK3588, RGA, FFmpeg, RTMP, and PCIe are merely examples listed to illustrate the feasibility of this invention and do not constitute a limitation on the scope of protection of this invention; the core concept of this invention can also be applied to other data processing chips with hardware graphics engines and audio / video processing capabilities.
[0063] A multi-channel heterogeneous audio and video stream multiplexing and synthesis system is provided to implement the multi-channel heterogeneous audio and video stream multiplexing and synthesis method described above. Figure 6 As shown, see also Figure 7 ,include: The receiving module 100 is used to receive multiple audio and video source streams, which include at least a local source stream transmitted locally and a network source stream transmitted over the network. The latest frame slot maintenance module 101 is used to maintain a latest frame slot for each audio and video source stream, and replace the previous frame in the latest frame slot with the latest arriving frame. When an audio or video source stream is interrupted, the previous frame in the latest frame slot of that stream remains unchanged. The hardware graphics engine 102 is used to scale and position the frames in the latest frame slots of each audio and video source stream to a unified canvas, and simultaneously complete cross-format conversion between different pixel formats during the scaling and positioning process, and combine multiple source streams into a unified format canvas. The reference counting management module 103 is used to enable multiple consumers to share the corresponding frame data in the synthesized uniform format canvas through reference counting: the encoding consumer, display consumer, and recording consumer each increment the reference count of the same frame data before using it, and decrement it after use, thus achieving zero-copy multiplexing. The pass-through recording module 104 is used to set a branch point at the demultiplexing output, so that the video primary stream and audio primary stream obtained by demultiplexing the network source stream are sent to both the decoding / display path and the recording path, and the two paths do not block each other.
[0064] The receiving module and the latest frame slot maintenance module enable heterogeneous and asynchronous multi-source streams to be stably used without pre-format conversion and time base alignment. The system does not need to configure a pre-processing unit for each stream, which simplifies the access side structure and reduces the synthesis delay.
[0065] The hardware graphics engine eliminates format conversion in the scaling and positioning process, so the system does not need to add software conversion steps or independent intermediate image processing units such as VPSS, saving CPU usage, memory bandwidth and device costs.
[0066] The reference counting management module enables the same canvas frame data to be shared by encoding consumers, display consumers, and recording consumers without memory copying, eliminating the copying overhead and cache consistency overhead in multi-consumer scenarios. Furthermore, the frame data is not released or overwritten before the reference count reaches zero, ensuring the correctness of sharing.
[0067] The pass-through recording module makes the recording and display paths independent of each other. Recording does not occupy the decoder and encoder and retains the original bitrate, resolution and frame rate of the source stream. Congestion or abnormality on one side will not affect the other side.
[0068] Each module is divided according to its function and has a single responsibility. They can be adjusted or replaced independently, which facilitates portability and expansion on different hardware platforms.
[0069] A multi-channel heterogeneous audio and video stream multiplexing and synthesis terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.
[0070] A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method described above.
[0071] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A method for multiplexing and synthesizing multiple heterogeneous audio and video streams, characterized in that, Includes the following steps: Receive multiple audio and video source streams, which include at least a local source stream transmitted locally and a network source stream transmitted over the network; Maintain a latest frame slot for each audio / video source stream and replace the previous frame in the latest frame slot with the latest arriving frame. When an audio / video source stream is interrupted, keep the previous frame in the latest frame slot of that stream unchanged. There is no clock alignment or resampling between the audio / video source streams, and the latest frame slots of each stream are updated independently. The hardware graphics engine scales and positions the latest frame slots of each audio and video source stream to a unified canvas, and simultaneously performs cross-format conversion between different pixel formats during the scaling and positioning process, so as to combine multiple audio and video source streams into a unified format canvas. Reference counting enables multiple consumers to share the corresponding frame data in the synthesized uniform format canvas and the video frame data in each latest frame slot: the encoding consumer, display consumer, and recording consumer each increment the reference count of the same frame data before using it, and decrement it after use, thus achieving zero-copy multiplexing.
2. The method for multiplexing and synthesizing multiple heterogeneous audio and video streams according to claim 1, characterized in that, Replacing the previous frame in the latest frame slot with the most recently arrived frame includes: Each audio / video source stream corresponds to a latest frame slot, which stores a reference to the latest frame of that audio / video source stream; When the latest frame arrives, the reference count of the latest frame is incremented by one, the latest frame is written to the latest frame slot, and the reference count of the previous frame in the latest frame slot is decremented by one, thus realizing the atomic replacement of the latest frame slot; The timestamps of the source frames of each audio and video source stream are uniformly converted to the same time unit when entering the latest frame slot for display and encoding.
3. The method for multiplexing and synthesizing multiple heterogeneous audio and video streams according to claim 1, characterized in that, When a certain audio or video source stream is interrupted, keeping the previous frame in the latest frame slot of that stream unchanged includes: When no new frames arrive from a certain audio or video source stream for a period of time, the latest frame slot for that stream is not updated. The hardware graphics engine continues to use the previous frame in the latest frame slot of that stream for compositing, so that the image of that stream remains the last valid image. The update is resumed when the audio or video source stream is restored. The hardware graphics engine uses the display refresh signal as the synthesis master clock, and in each synthesis cycle, it successively takes the latest frame from each latest frame slot to participate in the synthesis; the determination duration of the time period is determined according to the nominal frame interval of the audio and video source stream.
4. The method for multiplexing and synthesizing multiple heterogeneous audio and video streams according to claim 1, characterized in that, The hardware graphics engine scales and positions the latest frames from each audio and video source stream onto a unified canvas, and simultaneously performs cross-format conversion between different pixel formats during the scaling and positioning process, so as to combine multiple audio and video source streams into a unified format canvas, including: The hardware graphics engine supports batch task mode: start a compositing task group, attach multiple scaling and positioning tasks within the compositing task group, each scaling and positioning task independently declares the pixel format of its source frame, source rectangle and target rectangle, and the target canvas is fixed to a certain format; when executing each scaling and positioning task, the hardware graphics engine automatically completes the conversion from the source format to the target canvas format while scaling and positioning. This also includes: pre-verifying the pixel format conversion combinations supported by each of the multiple processing cores of the hardware graphics engine; when executing the scaling and positioning task, selecting a processing core from the multiple processing cores that supports the source format to target format conversion required for the task to execute the task; the target rectangle of each scaling and positioning task is determined by a layout table, which records the position and size of each window under different source stream paths; when the source stream path changes, switching to the corresponding layout table while keeping the execution logic of the compositing task group unchanged.
5. The method for multiplexing and synthesizing multiple heterogeneous audio and video streams according to claim 4, characterized in that, The local source stream has a pixel format of a first format, and the network source stream has a pixel format of a second format. The first format and the second format serve as sources for different tasks, but the target is the same canvas. The hardware graphics engine enables the two different source stream formats to be combined into a unified canvas without software format conversion or intermediate image processing units.
6. The method for multiplexing and synthesizing multiple heterogeneous audio and video streams according to claim 1, characterized in that, It also includes zero-decoding pass-through recording of network source streams: A branch point is set at the demultiplexing output, so that the video elementary stream and audio elementary stream obtained by demultiplexing the network source stream are sent to both the decoding / display path and the recording path, and the two paths do not block each other.
7. The method for multiplexing and synthesizing multiple heterogeneous audio and video streams according to claim 6, characterized in that, If the video basic stream is already in Annex-B format with a start code, it is written to disk directly in the order of arrival. If the video basic stream is in length prefix format, the length prefix before each network abstraction layer unit is replaced with the start code before writing to disk, and the sequence parameter set and image parameter set obtained by demultiplexing are inserted before each keyframe. After the recording path is started, it waits for the first keyframe of the video basic stream to arrive before starting to write to disk. A video file that can be played directly is generated without decoding or re-encoding. If the audio primary stream is raw AAC, a 7-byte ADTS header is added before each audio frame to generate an independently playable audio file; the 7-byte ADTS header contains the sync word, audio object type, sample rate index, channel configuration, and frame length.
8. A multi-channel heterogeneous audio and video stream multiplexing and synthesis system, used to implement the multi-channel heterogeneous audio and video stream multiplexing and synthesis method as described in any one of claims 1 to 7, characterized in that, include: The receiving module is used to receive multiple audio and video source streams, which include at least a local source stream transmitted locally and a network source stream transmitted over the network. The latest frame slot maintenance module is used to maintain a latest frame slot for each audio and video source stream and replace the previous frame in the latest frame slot with the latest arriving frame. When an audio or video source stream is interrupted, the previous frame in the latest frame slot of that stream remains unchanged. The hardware graphics engine is used to scale and position the latest frame slots of various audio and video source streams to a unified canvas, and simultaneously complete cross-format conversion between different pixel formats during the scaling and positioning process, and combine multiple audio and video source streams into a unified format canvas. The reference counting management module is used to enable multiple consumers to share the corresponding frame data in the synthesized unified format canvas and the video frame data in the latest frame slot of each channel through reference counting: the encoding consumer, display consumer, and recording consumer each increment the reference count of the same frame data before using it, and decrement it after use, thus achieving zero-copy multiplexing. The pass-through recording module is used to set a branch point at the demultiplexing output, so that the video primary stream and audio primary stream obtained by demultiplexing the network source stream are sent to both the decoding / display path and the recording path, and the two paths do not block each other.
9. A multi-channel heterogeneous audio and video stream multiplexing and synthesis terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.