A low-delay video encoding and transmission control method

Abstract

The application discloses a low-delay video encoding and sending control method, and belongs to the field of video encoding control and wireless data sending scheduling. The method receives line-by-line video data and writes the video data into a line buffer; the number of written lines is counted, and when reaching a preset threshold, video encoding is triggered, without waiting for the completion of the whole frame reception; in the encoding process, data is output according to the encoding segmentation granularity, and a sending trigger signal is generated when each segment is completed, and the corresponding encoding data is sent to a wireless sending queue. Through the double-layer trigger mechanism, parallel pipeline operation of video input, encoding processing and data sending is realized. The preset threshold can be dynamically adjusted according to factors such as video resolution, frame rate, network bandwidth and target delay. The method reduces the encoding start time from the frame level to the line level, eliminates the frame end waiting time, reduces the overall system delay, and can meet the ultra-low delay application requirement in the wireless video transmission scene.

Description

Technical Field

[0001] This invention relates to video encoding control and wireless data transmission scheduling, and more particularly to a low-latency video encoding and transmission scheduling method. Background Technology

[0002] In wireless video transmission systems, the latency of video encoding and transmission directly affects user experience and system performance. Existing video encoding and transmission systems typically employ the following control methods: Waiting for the complete reception of the video frame; Start video encoding; Generate a complete frame or a complete slice; Then, the data will be sent in a unified manner.

[0003] This traditional control method has the following problems: First, encoding starts by relying on the end-of-frame signal, and encoding can only begin after the entire frame of data has been received, resulting in a delay of at least one frame cycle. Second, the input and encoding are performed in a serial execution mode, meaning that video data reception and encoding processing cannot be carried out in parallel. Third, the overall latency is at least at the frame level. For high frame rate videos (such as 60fps and 120fps), the frame-level latency can reach 16.7ms-8.3ms, which cannot meet the requirements of ultra-low latency applications. Fourth, it cannot achieve fine-grained low-latency control; the granularity of latency control is limited to the frame level.

[0004] In the prior art, patent CN102625089B discloses an OFDM uncompressed transmission technology for UWB physical layer systems. Its core emphasis is on the physical layer architecture of transmission, primarily addressing physical layer transmission issues rather than coding and scheduling mechanisms. This technology does not involve improvements at the coding and scheduling level, such as line-level triggering or low-latency pipelined control.

[0005] CN103716680A patent discloses a video frame Ethernet WiFi forwarding technology, but this technology adopts a scheme without line-level encoding start-up, threshold triggering, segmented real-time transmission control and encoding scheduling mechanism, and only performs simple frame forwarding. In essence, it is still based on the processing mode of complete frames.

[0006] Patent CN109286818A discloses a technology for dynamically adjusting encoder parameters based on communication link status. Its core is to predict the buffer based on link status, calculate the target bit budget, and adjust the quantization parameter (QP), thus solving the problem of "bandwidth fluctuation → bit rate adaptation." The control logic of this technology is still based on frame structure for encoding initiation and does not involve: Line buffer threshold trigger; Encoding does not depend on a complete frame to begin; Input and encoding parallel startup mechanism; Row-level encoding pipeline.

[0007] Existing video encoding and transmission control schemes are primarily based on frame-level processing, where encoding initiation depends on the end-of-frame signal, and encoding and transmission are typically executed serially. These schemes lack a dynamic scheduling mechanism based on video input progress, making it impossible to initiate encoding processing in advance during video data reception or to coordinate the triggering of encoding and transmission processes. This results in overall system latency being limited by frame-level granularity, failing to meet the requirements for low-latency video transmission. Therefore, an encoding and transmission control mechanism independent of the end-of-frame signal is needed to achieve parallel execution of video input, encoding, and transmission, significantly reducing system latency. Summary of the Invention

[0008] Purpose of the invention: The purpose of this invention is to provide a low-latency video encoding and transmission control method based on input progress triggering. By constructing a two-layer scheduling mechanism that combines encoding start triggering and encoding data transmission triggering, video input, encoding processing and data transmission form an overlapping pipeline structure in time. This allows encoding to start in advance and encoding data to be transmitted in real time before the video frame has been received, significantly reducing the end-to-end latency of the system.

[0009] Technical solution: A low-latency video encoding and transmission control method, comprising the following steps: S1. Receive line-by-line video data and write the video line data into the line buffer; S2. Count the number of rows that have been written to the row buffer; S3. When the number of written rows reaches a preset threshold, video encoding is started, wherein the start of video encoding does not depend on the completion of receiving a complete video frame; S4. During the video encoding process, the encoded data is output according to the encoding segment granularity; S5. When each encoded segment is completed, a send trigger signal is generated; S6. Based on the transmission trigger signal, send the corresponding coded segment data to the wireless transmission queue; S7. If the video input is not completed, continue to execute steps S2 to S6.

[0010] Furthermore, the preset threshold is less than or equal to the total number of lines in the video frames, and is dynamically adjusted according to the video encoding format and system latency requirements. This design reduces the encoding start time from the frame level to the line level, allowing for flexible adjustment of the encoding triggering timing based on actual needs.

[0011] Furthermore, the granularity of the encoding segmentation can be adopted in different ways, such as segmentation based on the encoding macroblock line, segmentation based on the encoding slice, segmentation based on a fixed data size, or segmentation based on a fixed number of lines, in order to adapt to the requirements of different encoding standards and transmission protocols.

[0012] Furthermore, step S6 further includes: after generating the transmission trigger signal, writing the corresponding encoded segment data from the encoding output buffer into the wireless transmission queue; after the writing is completed, releasing the space of the transmitted encoded segment data in the encoding output buffer. This design achieves rapid flow of encoded data and efficient utilization of storage resources.

[0013] Furthermore, the method further includes: simultaneously executing steps S1 and S2, allocating a storage address in the line buffer for each video line data; establishing a line count counter to record the number of video line data already written to the line buffer in real time. This design ensures the accuracy and real-time performance of the line count.

[0014] Furthermore, the sliding window is a processing window that moves forward as new video line data is written, and the encoding module continuously encodes the continuous video line data based on this sliding window.

[0015] Furthermore, the method further includes: when a video frame end signal is detected, writing the remaining video line data that has not reached a preset threshold into a line buffer; and encoding the remaining video line data after the current encoding task is completed. This design ensures the integrity of the video frame and the continuity of the encoding.

[0016] Furthermore, the row buffer is a circular buffer or a chained buffer, which supports simultaneous data writing and encoding / reading operations, enabling efficient parallel data reading and writing.

[0017] Furthermore, the method further includes: in step S3, when a preset threshold is reached, an encoding start signal is generated; the encoding start signal triggers the encoder to start encoding the written video line data; during the encoding process, the line buffer continues to receive subsequent video line data, realizing parallel execution of video input and encoding. This design forms a continuous pipelined encoding structure, improving system processing efficiency.

[0018] Furthermore, the preset threshold is set based on a comprehensive consideration of one or more factors, including video frame resolution, video frame rate, network bandwidth conditions, encoder processing capability, and target latency requirements, to ensure optimal system performance.

[0019] This invention decouples encoding start control from sending trigger control and binds it to the input data progress to form a collaborative scheduling mechanism based on data flow progress. Beneficial effects

[0020] Compared with the prior art, the present invention has the following beneficial effects: (1) Encoding start forward: Since encoding start is triggered based on the threshold of the number of lines already written in the line buffer, rather than relying on the video frame end signal, the encoding start time is advanced from the frame level to the line level.

[0021] (2) Eliminate frame-level waiting: Encoding processing can be started before the video frame has been received, thereby eliminating the delay caused by waiting for the entire frame data to be received.

[0022] (3) Forming a pipeline for parallel processing: By parallel execution and time overlap of video input, encoding processing and data transmission, the data retention time in the system is reduced.

[0023] (4) Reduce end-to-end latency: Due to the early start and overlapping execution of the encoding and transmission processes, the overall system latency is reduced from the frame level to the millisecond level.

[0024] (5) Two-layer trigger control structure: The first layer is triggered by the row number threshold to start the encoding, and the second layer is triggered by the segment completion to send, forming a flexible and efficient encoding scheduling control mechanism, which changes the traditional serial scheduling method at the control logic level.

[0025] (6) Improve scheduling flexibility: Through joint scheduling of preset thresholds and coding segmentation granularity, the balance between latency and throughput can be dynamically optimized according to system performance and network conditions. Attached Figure Description

[0026] Figure 1 This is a diagram of the encoding and transmission architecture of the present invention. Figure 2 This is a schematic diagram of the parallel pipelined encoding and transmission process of the present invention. Figure 3 This is a flowchart of the encoding and transmission control of the present invention.

[0027] Figure 4 This is a diagram of the dual-layer trigger control mechanism of the present invention.

[0028] Figure 3 In this process, video encoding is initiated when a preset threshold is reached by counting the number of lines written to the line buffer, without relying on the reception of complete video frames. During encoding, a transmission trigger signal is generated after each encoded segment is completed, and the encoded segment data is written to the wireless transmission queue, enabling parallel execution of input and encoding.

[0029] Figure 4 In this system, the first trigger layer initiates encoding based on a cache line count threshold; the second trigger layer triggers transmission based on the completion of encoded segments. This dual-layer triggering mechanism enables parallel execution of video input, encoding, and transmission, thereby reducing system latency. Detailed Implementation

[0030] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Example

[0031] This embodiment uses H.264 video encoding at 1920×1080 resolution and 60fps as an example to illustrate the specific implementation of the present invention.

[0032] System parameter settings: Video resolution: 1920×1080, 1080 lines per frame Video frame rate: 60fps Preset threshold: 60 lines (i.e., encoding will start when 60 lines are reached). Encoding segmentation granularity: Based on encoding slices, each slice consists of 16 lines.

[0033] The specific implementation steps are as follows: Step 1: After the system starts up, it begins receiving video data line by line. The video data is input line by line, with each line containing 1920 pixels of luminance and chrominance data.

[0034] Step 2: Write the received video line data into the line buffer. This embodiment uses a circular buffer, which can store 200 lines of video data. A line count counter is also maintained to record the number of video lines written to the line buffer in real time.

[0035] Step 3: When the number of lines written recorded by the line count counter reaches the preset threshold of 60, an encoding start signal is immediately generated to start the video encoder for encoding. At this time, only about 5.56% of the video frame data (60 / 1080) has been received, far from a complete frame, but the encoder has already started working.

[0036] Step 4: The encoder encodes the 60 lines of video data that have been written. The encoding process uses the H.264 standard and segments the data into slices, with each slice encoding unit consisting of 16 lines of data.

[0037] Step 5: Once the first slice (16 lines) is encoded, a transmit trigger signal is immediately generated. This signal triggers the transmit module to read the encoded slice data from the encoding output buffer and write it into the wireless transmit queue.

[0038] Step 6: While the encoder continues to process subsequent slices, the wireless transmission module reads the already transmitted slice data from the transmission queue and transmits it through the wireless channel.

[0039] Step 7: While the encoder continues encoding, the line buffer continues to receive subsequent video line data. When the number of lines written reaches the preset threshold again, a new encoding start is triggered.

[0040] The above process results in the following parallel structure: The video input continuously receives data from lines 61-120. The encoder is encoding lines 1-60 of data; The sending module is sending the completed slice data.

[0041] This parallel structure significantly reduces the overall processing latency of video frames. Traditional methods require waiting for all 1080 lines to be received before encoding begins, resulting in a latency of at least 16.7ms; this invention begins encoding after the 60th line is received, reducing the latency to approximately 1.67ms. This latency reduction is mainly due to the earlier encoding start time and the overlapping execution of encoding and transmission processes. Example

[0042] This embodiment uses H.265 video encoding at 3840×2160 resolution (4K) and 30fps as an example to illustrate the adaptive threshold adjustment function of the present invention.

[0043] System parameter settings: Video resolution: 3840×2160, 2160 lines per frame Video frame rate: 30fps Encoding standard: H.265 / HEVC Network bandwidth: 10Mbps Target latency: no higher than 10ms.

[0044] Preset threshold dynamic adjustment logic: Calculate the preset threshold based on network bandwidth and target latency.

[0045] Assume the average encoding time per line is 0.005ms and the encoder's processing capacity is 200 lines / ms.

[0046] Considering the total number of lines of 2160, a preset threshold of 500 lines (approximately 23% of the frame lines) is set to ensure that the encoder can work continuously without interruption.

[0047] Specific implementation steps: Step 1-Step 2: Similar to Example 1, receive line-by-line video data and write it to the line buffer, while counting the number of lines written.

[0048] Step 3: When the line count counter reaches 500, an encoding start signal is generated to start the H.265 encoder.

[0049] Step 4: The encoder encodes the 500 lines of data that have been written. H.265 encoding uses a CTU (Coding Tree Unit) structure, where each CTU contains a 64×64 pixel block. In this embodiment, every 64 lines of data are treated as one encoding segment.

[0050] Step 5: Once the first encoded segment (64 lines) is completed, a send trigger signal is immediately generated. Due to the network bandwidth limitation of 10Mbps, the sending module adopts a variable-speed sending strategy, dynamically adjusting the sending rate according to the current network congestion situation.

[0051] Step 6: The sending module reads the encoded segmented data from the encoding output buffer and determines the transmission timing based on the current network conditions. It sends immediately when channel conditions are good; and buffers appropriately when the channel is congested.

[0052] Step 7: Continue executing steps 2 through 6 until the entire frame of data has been processed.

[0053] By dynamically adjusting preset thresholds, this embodiment can maintain low end-to-end latency even under network bandwidth constraints. Test data shows that the average latency is reduced from 33.3ms (one frame period) in the traditional method to 8-12ms, meeting the target latency requirements. Example

[0054] This embodiment uses WiFi wireless video transmission as an application scenario to illustrate the performance of the present invention in a real wireless environment.

[0055] System parameter settings: Video resolution: 1280×720, 720 lines per frame Video frame rate: 30fps Wireless channel: WiFi 5GHz, bandwidth 20MHz Preset threshold: 36 lines (5% of frame lines) Encoding segmentation granularity: Based on a fixed data volume, each segment is 10KB. Specific implementation steps: Step 1: The WiFi receiver module begins receiving progressive video data. Due to potential packet loss and jitter in the wireless channel, data arrival times are uneven.

[0056] Step 2: Write the received video line data to the line buffer. A linked buffer structure is used, with each buffer node storing one line of data. A line count counter is maintained to record the number of validly written lines in real time.

[0057] Step 3: When the number of valid write lines reaches 36, an encoding start signal is immediately generated to start the video encoder. At this time, only about 5% of the video frame data has been received, but the encoder has already started working, greatly improving processing efficiency.

[0058] Step 4: The encoder encodes the 36 lines of data that have been written. The H.264 encoding standard is used, with each 10KB of data forming an encoded segment.

[0059] Step 5: Once the first encoded segment is completed, a send trigger signal is immediately generated. The send module reads the segment data from the encoded output buffer.

[0060] Step 6: The transmitting module sends the encoded segmented data to the receiving end via a WiFi wireless channel. Because segmented real-time transmission is used, even if the transmission of one segment fails, it will not affect the transmission of other segments.

[0061] Step 7: Continue executing steps 2 to 6 to maintain stable low-latency transmission even when there is packet loss and jitter in the WiFi channel.

[0062] Performance test results: In a WiFi environment, the average latency of traditional methods is 33.3ms (one frame period), and the maximum latency can reach more than 50ms (affected by channel jitter); Using the method of this invention, the average latency is reduced to 3-5ms, the maximum latency does not exceed 8ms, and the latency fluctuation is significantly reduced.

[0063] Meanwhile, because segmented real-time transmission is adopted, even if packet loss occurs, only the corresponding segment is affected, and the entire frame of data will not be retransmitted, thus improving the robustness of the system. Example

[0064] This embodiment illustrates the performance comparison of the present invention under different preset thresholds, so as to demonstrate the optimization space for threshold selection.

[0065] Test conditions: Video resolution: 1920×1080, 1080 lines per frame Video frame rate: 60fps Encoding standard: H.264 Test variables: Preset thresholds (set to 30 rows, 60 rows, 90 rows, 120 rows, and 150 rows respectively). Test metrics: average latency, encoder clock speed / utilization, system throughput The test results are shown in the table below: 30 lines 0.83 650MHz / 55% 15.2 60 lines 1.67 600MHz / 68% 18.5 90 lines 2.50 500MHz / 82% 20.1 120 lines 3.33 400MHz / 91% 21.3 150 lines 4.17 400MHz / 96% 21.8 The test results show that: The smaller the preset threshold, the lower the latency, but the higher the encoder frequency required, the lower the utilization rate, which may lead to frequent encoder start-stop and increase system power consumption. The higher the preset threshold, the higher the encoder utilization and the higher the system throughput, but the latency will also increase accordingly. Within the range of 60-120 lines, the system performance is relatively balanced, ensuring both low latency and high encoder utilization. Taking all factors into consideration, this application scenario selects 90 lines as the preset threshold, achieving an average latency of 2.5ms and an encoder utilization rate of 82%, thus achieving optimal performance. This example demonstrates the importance of preset thresholds. In practical applications, optimization should be performed based on specific needs (latency priority or throughput priority).

[0066] It should be noted that the preset threshold, encoding segmentation granularity and related parameters in the above embodiments are only examples. Those skilled in the art can adjust them according to specific system performance, latency requirements and network conditions, without affecting the encoding and transmission collaborative scheduling mechanism based on input progress triggering of the present invention.

[0067] Furthermore, the above embodiments are primarily used to illustrate the technical solutions of the present invention, and not to limit the scope of protection of the present invention. Those skilled in the art can make various modifications or equivalent substitutions without departing from the concept of the present invention, and all such modifications or substitutions should fall within the scope of protection of the present invention. The scope of protection of the present invention is defined by the appended claims.

Claims

1. A low-latency video encoding and transmission control method, characterized in that, include: Receive progressive video data and write it to the line buffer; Real-time statistics are performed based on the number of video lines already written in the line buffer; When the number of written lines reaches a preset threshold, an encoding start signal is generated before the video frame has been fully received, so as to trigger the video encoder to encode the written video line data, and the encoding start is not triggered by the video frame end signal. During video encoding, the line buffer continues to receive subsequent video line data, enabling video input and video encoding to be performed in parallel. The video encoder segments the video data according to a preset encoding segment granularity, and generates a transmission trigger signal corresponding to each encoding segment when the encoding segment is completed. According to the transmission trigger signal, the corresponding coded segment data is written from the coded output buffer to the wireless transmission queue, and the corresponding buffer space is released after transmission; Even when video frame data is not fully received, the line counting, encoding, and sending steps are continuously executed, thereby forming a pipelined parallel processing structure for video input, video encoding, and data transmission.

2. The method according to claim 1, characterized in that, The method includes: The first triggering mechanism is used to trigger video encoding startup based on the number of lines written in the line buffer reaching a preset threshold. The second triggering mechanism is used to trigger the transmission of encoded data when the encoding segmentation is completed; The first triggering mechanism and the second triggering mechanism are independent of each other but work together.

3. The method according to claim 1, characterized in that, The video encoding process does not depend on video frame boundary information and does not wait for the complete reception of full video frame data.

4. The method according to claim 1, characterized in that, The video input module, video encoding module, and data transmission module operate in parallel in a pipeline manner, and their execution intervals overlap in time.

5. The method according to claim 1, characterized in that, Video encoding is performed continuously based on a sliding window in the line buffer, which is dynamically updated as video line data is written.

6. The method according to claim 1, characterized in that, The preset threshold is dynamically determined based on system latency constraints and encoding processing capabilities.

7. The method according to claim 1, characterized in that, The encoding segmentation granularity is any one or a combination of the following: Segmentation based on encoded macroblock lines; Segmentation based on encoded slices; Segmentation based on a fixed data size; Segmentation based on a fixed number of rows.

8. The method according to claim 1, characterized in that, The sending process includes: After generating the transmit trigger signal, the corresponding coded segment data is written from the coded output buffer to the wireless transmit queue; Release the corresponding encoded output buffer space after writing is complete.

9. The method according to claim 1, characterized in that, At the end of a video frame, for the remaining video lines that have not reached the preset threshold, encoding processing is performed after the current encoding task is completed.

10. The method according to claim 1, characterized in that, The row buffer is a buffer structure that supports parallel read and write operations, including a circular buffer or a chained buffer.

Images