A video encoding apparatus, method, system, program product, and storage medium

Abstract

The application discloses a video coding device, method, system, program product and storage medium, and relates to the technical field of video processing.In the application, the video coding device can be connected with a host through a bus interface, and at least two video channels are arranged between the video coding device and the host.A processor, a programmable logic device, a memory, a direct memory access controller and a channel management register can be arranged in the video coding device.The channel management register stores channel configurations of the video channels, and the programmable logic device is provided with processing units corresponding to the video channels.In actual application, the processor can set an encoder for the processing units of the video channels according to the channel configurations, and the direct memory access controller can allocate transmission bandwidth for the video channels according to the channel configurations.In this way, the original video data of the video channels can be subjected to different coding and transmission processing, so that the flexibility and scalability of multi-video stream processing can be improved.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to a video encoding apparatus, method, system, program product, and storage medium. Background Technology

[0002] With the continuous updates and development of internet content, video streaming has largely replaced text, images, and other formats, becoming the primary content format on the internet. Due to the large size of raw video data, efficient encoding and compression of raw video data is of great significance for the storage and distribution of video content.

[0003] In related technologies, hardware acceleration modules have been developed to accelerate video encoding and processing to improve efficiency. However, these technologies still suffer from poor flexibility and scalability when processing multiple video streams, making it difficult to perform differentiated encoding and transmission processing on multiple video streams. Summary of the Invention

[0004] This invention provides a video encoding device, method, system, program product, and storage medium, which can ensure that the video encoding device performs different encoding and transmission processing on various raw video data sent by the host according to the requirements, thereby improving the flexibility and scalability of multi-video stream processing.

[0005] To solve the above-mentioned technical problems, the present invention provides a video encoding device, which is connected to a host via a bus interface and has at least two video channels between it and the host.

[0006] The video encoding device includes:

[0007] The system includes a processor, a programmable logic device, memory, a direct memory access controller, and a channel management register. The programmable logic device contains the processing unit for each video channel, and the channel management register records the configuration parameters of each video channel.

[0008] The processor is used to configure the encoder of the processing unit of the video channel according to the configuration parameters; to receive the raw video data corresponding to the video channel sent by the host and transmit it to the corresponding processing unit;

[0009] The processing unit is used to encode the raw video data and write the encoded video data into memory;

[0010] The direct memory access controller is used to allocate the transmission bandwidth corresponding to each video channel according to the configuration parameters, and to transfer the encoded video data of each video channel from memory to the host according to the transmission bandwidth.

[0011] The present invention also provides a video encoding method applied to the above-mentioned video encoding apparatus, the method comprising:

[0012] The processor configures the encoder for each video channel processing unit based on the configuration parameters of each video channel recorded in the channel management register.

[0013] The processor receives the raw video data corresponding to the video channel from the host and transmits it to the corresponding processing unit;

[0014] The processing unit encodes the raw video data and writes the encoded video data into memory.

[0015] The Direct Memory Access Controller allocates the transmission bandwidth corresponding to each video channel according to the configuration parameters, and transmits the encoded video data of each video channel from memory to the host according to the transmission bandwidth.

[0016] The present invention also provides a video encoding system, including a host and the aforementioned video encoding device;

[0017] The host is used to send raw video data to the video encoding device and receive encoded video data sent by the video encoding device.

[0018] A video encoding device is used to encode raw video data and send the encoded video data to the host.

[0019] The present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the above-described video encoding method.

[0020] The present invention also provides a non-volatile computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the above-described video encoding method.

[0021] This invention provides a video encoding device. The video encoding device is connected to a host computer via a bus interface and has at least two video channels with the host computer. The video encoding device includes: a processor, a programmable logic device, memory, a direct memory access controller, and a channel management register. The programmable logic device includes processing units for each video channel, and the channel management register records configuration parameters for each video channel. The processor is used to configure the encoders of the processing units of the video channels according to the configuration parameters; receive raw video data corresponding to the video channels from the host computer and transmit it to the corresponding processing units; the processing units are used to encode the raw video data and write the encoded video data into memory; the direct memory access controller is used to allocate the transmission bandwidth corresponding to each video channel according to the configuration parameters and transmit the encoded video data of each video channel from memory to the host computer according to the transmission bandwidth.

[0022] The beneficial effects of this invention are as follows: The video encoding device provided by this invention can be connected to a host via a bus interface and has at least two video channels with the host. This video encoding device includes a processor, a programmable logic device (PLD), memory, a direct memory access controller (DMI), and a channel management register. The PLD includes processing units for each video channel, and the channel management register records the configuration parameters of each video channel. In practical use, the processor can be used to configure the encoders of the processing units of the video channels according to the configuration parameters, and to receive the raw video data corresponding to each video channel from the host and transmit it to the corresponding processing unit; the processing unit can be used to encode the raw video data and write the encoded video data into memory; the DMI can be used to allocate the transmission bandwidth corresponding to each video channel according to the configuration parameters, and to transmit the encoded video data of each video channel from memory to the host according to the transmission bandwidth. Therefore, by setting corresponding processing units for each video channel and configuring the encoder configuration and transmission bandwidth of each video channel individually, this invention can ensure that each video data stream receives different encoding and transmission processing, thereby improving the flexibility and scalability of multi-video stream processing.

[0023] The present invention also provides a video encoding method, system, program product, and storage medium, which have the above-mentioned beneficial effects. Attached Figure Description

[0024] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a structural block diagram of a video encoding device provided in an embodiment of the present invention;

[0026] Figure 2 A flowchart of a video encoding method provided in an embodiment of the present invention;

[0027] Figure 3 This is a structural block diagram of a video encoding system provided in an embodiment of the present invention;

[0028] Figure 4 This is a structural block diagram of another video encoding device provided in an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram illustrating the connection between a host computer and a video encoding device according to an embodiment of the present invention;

[0030] Figure 6A flowchart of another video encoding method provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0032] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0033] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] With the continuous updates and development of internet content, video streaming has largely replaced text and images, becoming the primary content format on the internet. Due to the large size of raw video data, efficient encoding and compression of raw video data is crucial for the storage and distribution of video content. In related technologies, hardware acceleration modules have emerged to accelerate video encoding and processing to improve efficiency. However, these technologies still suffer from poor flexibility and scalability in processing multiple video streams, making it difficult to perform differentiated encoding and transmission processing for multiple video streams.

[0035] In view of this, the present invention provides a video encoding device that can perform different encoding and transmission processing on various original video data sent by the host according to the requirements, thereby improving the flexibility and scalability of multi-video stream processing.

[0036] For easier understanding, please refer to Figure 1 , Figure 1This is a structural block diagram of a video encoding device provided in an embodiment of the present invention. This video encoding device can be connected to a host computer via a bus interface (e.g., PCIe interface, Peripheral Component Interconnect express, high-speed serial computer expansion bus standard), and at least two video channels are provided between it and the host computer. This video encoding device may include:

[0037] The system comprises a processor 1, a programmable logic device 2, memory 3, a direct memory access controller 4, a channel management register 5, and a bus interface 6. The programmable logic device 2 may contain processing units 21 for each video channel. The bus interface 6 can be connected to the channel management register 5 and the direct memory access controller 4. The channel management register 5 can also be connected to the processor 1 and the direct memory access controller 4. The processor 1 can also be connected to the programmable logic device 2 and the direct memory access controller 4. The programmable logic device 2 can also be connected to memory 3, and the direct memory access controller 4 can also be connected to memory 3.

[0038] Understandably, the processor 1, programmable logic device 2, memory 3, direct memory access controller 4, channel management register 5, and bus interface 6 can be connected via a bus structure. For example, to improve the transmission rate within a video encoding device, the above connections can be implemented based on the AXI interconnect bus (Advanced eXtensible Interface). To reduce complexity, the channel management register 5 and bus interface 6 can be specifically connected based on the lightweight AXI bus (AXIlite).

[0039] It should be noted that, Figure 1 The video encoding device shown is only its basic form. In other possible cases, the processor 1, programmable logic device 2, memory 3, direct memory access controller 4, channel management register 5, or bus interface 6 may have different internal structures. For example, the bus interface 6 may include a bus controller, and the memory 3 may include a DDR controller (Double Data Rate, DDR memory). Furthermore, the video encoding device may also include other structures, such as network ports and external memory, which can be configured according to actual application requirements.

[0040] Additionally, channel management register 5 can record configuration parameters for each video channel. These configuration parameters are adjustable parameters that indicate the encoding and transmission requirements of each video channel. For example, configuration parameters may include encoder configuration (such as encoding algorithm) and transmission configuration information (such as transmission priority). The configuration parameters in the channel management register can be configured by the host, allowing for convenient and flexible adjustment of the encoding and transmission requirements of each video channel.

[0041] The following are the specific uses of processor 1, processing unit 21, and direct memory access controller 4:

[0042] Processor 1 is used to configure the encoder of the processing unit 21 of the video channel according to the configuration parameters; receive the raw video data corresponding to the video channel sent by the host, and transmit it to the corresponding processing unit 21.

[0043] The processing unit 21 is used to encode the raw video data and write the encoded video data into memory 3.

[0044] The direct memory access controller 4 is used to allocate the transmission bandwidth corresponding to each video channel according to the configuration parameters, and to transfer the encoded video data of each video channel from memory 3 to the host according to the transmission bandwidth.

[0045] It is worth noting that the programmable logic device 2 can implement hardware processing logic for various video processing methods using a hardware description language (HDL). Furthermore, when the processor 1 configures the encoder, it can switch the video processing method corresponding to each processing unit 21 through RTL (Register Transfer Level) reconstruction to provide a specific video encoding method for the video channel. For example, the hardware processing logic for video encoding standards such as H.264 / H.265 / AV1 can be implemented using a hardware description language, and flexible switching between H.264 / H.265 / AV1 video encoding standards can be achieved through RTL reconstruction.

[0046] As can be seen in this embodiment, the channel management register 5 can store the configuration parameters corresponding to each video channel. The processor 1 can configure the encoder of the processing unit 21 of each video channel according to the configuration parameters, such as setting the video preprocessing method and encoding algorithm to be executed by each processing unit 21. The direct memory access controller 4 can allocate the transmission bandwidth corresponding to each video channel according to the configuration parameters. Then, when the host sends the raw video data (such as YUV image data, where Y represents luminance and U and V represent chrominance) corresponding to each video channel, the processor 1 can schedule the raw video data to the processing unit 21 corresponding to each video channel. The processing unit 21 can encode the raw video data of the corresponding video channel according to the encoder configuration provided by the processor 1, and write the obtained encoded video data (such as H.264 encoded data, H.265 encoded data, AV1 encoded data, etc.) into memory 3. The direct memory access controller can transfer the encoded video data of each video channel from memory 3 to the host according to the allocated transmission bandwidth. As can be seen, this embodiment can flexibly adjust the encoding method and data transmission rate of each video channel, enabling the video encoding device to perform targeted encoding and transmission processing on the raw video data of each video channel, thereby improving the flexibility and scalability of multi-video stream processing.

[0047] In another embodiment, there are at least two processing units 21 for each video channel. The processor 1 can also be used for:

[0048] The hardware resource utilization rate of each processing unit 21 in the video channel is detected;

[0049] Based on the hardware resource utilization rate, the raw video data is transmitted to the processing unit 21 with the lowest load.

[0050] In this embodiment, each video channel has at least two processing units 21, meaning that each video channel has at least two schedulable and parallel processing units 21. To improve the video processing efficiency of each video channel, the processor 1 can detect the hardware resource utilization rate of each processing unit 21 in the video channel. Subsequently, the load pressure of each processing unit 21 can be determined based on the hardware resource utilization rate, and the original video data can be transmitted to the processing unit 21 with the lowest load. This effectively balances the load pressure of each processing unit 21 in the video channel, thereby effectively improving video encoding efficiency.

[0051] It should be noted that this embodiment does not limit how the processor 1 detects the hardware resource utilization of the processing unit 21. For example, the processor 1 can detect the utilization of various hardware resources (processing unit utilization, cache utilization, DMA queue depth, DMA, Direct Memory Access) of the processing unit 21 by reading the status register of each processing unit 21.

[0052] Furthermore, processor 1 can also be used for:

[0053] The total load rate of all processing units 21 in the video channel is determined based on the hardware resource utilization rate.

[0054] Determine if the total load rate is greater than a preset threshold;

[0055] If the total load rate is determined to be greater than the preset threshold, the direct memory access controller 4 is controlled to write the raw video data into memory.

[0056] The direct memory access controller 4 controls the transfer of raw video data from memory 3 to the corresponding processing unit 21.

[0057] In this embodiment, considering that all processing units 21 of the video channel may be under high overall load, the raw video data sent by the host may be difficult to process in a timely manner. Therefore, when the processor 1 obtains the hardware resource utilization rate corresponding to each processing unit 21 of the video channel, it can further determine the total load rate corresponding to all processing units of the video channel. Subsequently, the processor 1 can determine whether the total load rate is greater than a preset threshold. If it is greater, it can control the direct memory access controller to write the raw video data into memory for temporary storage. Subsequently, the processor 1 can control the direct memory access controller to transfer the raw video data from memory to the corresponding processing unit 21. In this way, when all processing units 21 of the video channel are under high load, the processor 1 can first schedule the raw video data to memory 3, thereby relieving the pressure on the processing units 21.

[0058] It should be noted that this embodiment does not limit how the processor 1 determines the above total load rate. It can be set according to the actual application requirements. For example, the hardware resource utilization of each processing unit 21 can be averaged to obtain the above total load rate.

[0059] In another embodiment, the programmable logic device 2 may further include a backup processing unit. The processor 1 may also be used for:

[0060] Read the status register corresponding to the processing unit 21 of the video channel, and determine whether the processing unit 21 of the video channel is faulty based on the status register value;

[0061] If the video channel processing unit 21 fails, the original video data of the video channel will be transmitted to the backup processing unit.

[0062] In this embodiment, considering that the processing unit 21 corresponding to the video channel may be damaged, thus affecting the processing of the original video data, a backup processing unit can be set in the programmable logic device 2. In this case, the processor 1 can read the status register corresponding to the processing unit 21 of the video channel and determine whether the processing unit 21 is faulty based on the status register value. If the processing unit 21 is determined to be faulty, the processor 1 can transmit the original video data of the video channel to the backup processing unit, so that the backup processing unit can continue the video encoding processing, thereby improving the reliability of the video encoding processing.

[0063] In another embodiment, the processing unit 21 can also be used for:

[0064] Scene recognition is performed on the raw video data to determine the image scene corresponding to the raw video data;

[0065] The encoding parameters are adjusted according to the image scene, and the original video data is encoded according to the adjusted encoding parameters.

[0066] In this embodiment, to effectively balance image quality and image compression rate, the processing unit 21 can also perform scene recognition when encoding the original video data to obtain the image scene corresponding to the original video data, such as a moving scene or a static scene. Furthermore, the processing unit 21 can dynamically adjust the encoding parameters (such as quantization parameters, frame type, reference frame, bitrate, etc.) according to the image scene, thereby achieving a balance between image quality and compression rate.

[0067] In another embodiment, to better perform video encoding, the processing unit 21 can also be functionally divided into two types: a preprocessing unit and an encoding unit. The preprocessing unit is used to preprocess the raw video data, such as denoising, scaling, and color adjustment. The encoding unit is used to encode the preprocessed raw video data to obtain encoded video data. Furthermore, each video channel has at least two encoding units to facilitate scheduling and parallel processing.

[0068] Furthermore, to improve coding efficiency, the preprocessing unit, processor, and coding unit may also have the following uses:

[0069] The preprocessing unit is also used to divide the video image frames in the raw video data into at least two sub-blocks;

[0070] Processor 1 is used to schedule sub-blocks to the various coding units of the video channel;

[0071] The encoding unit is also used for parallel encoding of sub-blocks.

[0072] In this embodiment, to fully utilize the parallel processing capabilities of the programmable logic device 2, the preprocessing unit can pre-divide the video image frame into multiple sub-blocks according to a preset macroblock size. Subsequently, the processor 1 can schedule these sub-blocks to various encoding units, which then perform parallel encoding processing on these sub-blocks. Thus, when processing each video image frame, this embodiment can fully utilize various encoding units for parallel encoding processing, thereby effectively improving processing efficiency.

[0073] In another embodiment, the processing unit 21 can also be used for:

[0074] Perform resolution adjustment and / or frame rate conversion on the raw video data to convert the raw video data into at least two output video data with different resolutions and / or different frame rates;

[0075] Encode each output video data and write the encoded video data corresponding to each output video data into memory 3;

[0076] Direct memory access controller 4 can also be used for:

[0077] The encoded video data corresponding to each output video data is transferred from memory 3 to the host.

[0078] In this embodiment, to facilitate the conversion of a single input video stream into multiple output video streams with different resolutions and frame rates to flexibly meet the user's viewing needs, the processing unit 21 can also perform resolution adjustment processing (such as downsampling or interpolation) and frame rate conversion processing (such as frame duplication or frame dropping) on ​​the original video data to convert the original video data into multiple output video data with different resolutions and / or frame rates. For example, the resolution of the input video can be dynamically adjusted from 4K (3840×2160) to 1080p (1920×1080) or 720p (1280×720). Through frame rate conversion algorithms (such as frame duplication or frame dropping), the frame rate of the input video can be adjusted from 60fps to 30fps or 15fps to adapt to the needs of different bandwidths and display devices. Furthermore, after the processing unit 21 completes the encoding processing of each output video data, the direct memory access controller 4 can transfer the encoded video data corresponding to each output video data from the memory 3 to the host.

[0079] In another embodiment, the configuration parameters include the priority of the video channel, which is either high or low priority; the direct memory access controller 4 can also be used for:

[0080] Set the theoretical upper limit of transmission bandwidth for each priority level;

[0081] Based on the theoretical upper limit of transmission bandwidth, the encoded video data of each priority video channel is transferred from memory to the host.

[0082] Detect the actual usage value of transmission bandwidth corresponding to each priority level;

[0083] When the actual usage of high-priority transmission bandwidth is less than or equal to the theoretical upper limit of high-priority transmission bandwidth, the unused high-priority transmission bandwidth will be allocated to low-priority transmission bandwidth.

[0084] When the actual usage of high-priority transmission bandwidth exceeds the theoretical upper limit of high-priority transmission bandwidth, the transmission of encoded video data of low-priority video channels is stopped, and the low-priority transmission bandwidth is allocated to high-priority channels.

[0085] In this embodiment, each video channel can be assigned a priority, and the Direct Memory Access Controller 4 can allocate different transmission bandwidths to video channels with different priorities. For example, high-priority video streams (such as live streams) can be allocated 80% of the transmission bandwidth, while low-priority video streams (such as recording streams) can be allocated 20% of the transmission bandwidth. When the bandwidth requirement of a high-priority stream decreases, the Direct Memory Access Controller 4 can automatically allocate the remaining bandwidth to low-priority streams to ensure efficient utilization of system resources; when a high-priority stream needs to occupy more transmission bandwidth (such as occupying more than 80% of the transmission bandwidth), the transmission of low-priority streams can be interrupted to ensure that video streams with high real-time requirements are processed first.

[0086] Based on the above embodiments, the video encoding method based on the above video encoding apparatus will be described below. For ease of understanding, please refer to... Figure 2 , Figure 2 A flowchart illustrating a video encoding method provided in an embodiment of the present invention. This method, applied to the aforementioned video encoding apparatus, may include:

[0087] S201. The processor configures the encoder for each video channel processing unit according to the configuration parameters of each video channel recorded in the channel management register.

[0088] S202, The processor receives the raw video data corresponding to the video channel sent by the host and transmits it to the corresponding processing unit;

[0089] S203. The processing unit encodes the raw video data and writes the encoded video data into memory.

[0090] S204, the direct memory access controller allocates the transmission bandwidth corresponding to each video channel according to the configuration parameters, and transmits the encoded video data of each video channel from memory to the host according to the transmission bandwidth.

[0091] As can be seen, the video encoding device provided by this invention can be connected to a host via a bus interface and has at least two video channels with the host. This video encoding device includes a processor, a programmable logic device (PLD), memory, a direct memory access controller (DMI), and a channel management register. The PLD contains processing units for each video channel, and the channel management register records the configuration parameters of each video channel. In practical use, the processor can be used to configure the encoders of the processing units of the video channels according to the configuration parameters, and to receive the raw video data corresponding to each video channel from the host and transmit it to the corresponding processing unit. The processing unit can be used to encode the raw video data and write the encoded video data into memory. The DMI can be used to allocate the transmission bandwidth corresponding to each video channel according to the configuration parameters, and to transmit the encoded video data of each video channel from memory to the host according to the transmission bandwidth. Therefore, by setting corresponding processing units for each video channel and configuring the encoder configuration and transmission bandwidth of each video channel individually, this invention can ensure that each video data stream receives different encoding and transmission processing, thereby improving the flexibility and scalability of multi-video stream processing.

[0092] Optionally, the configuration parameters in the channel management register can be configured by the host, allowing the host to flexibly adjust the encoding and transmission requirements of each video channel.

[0093] Optionally, the video channel processing unit comprises at least two; the method may also include:

[0094] The processor detects the hardware resource utilization of each processing unit in the video channel;

[0095] Based on hardware resource utilization, the raw video data is transmitted to the processing unit with the lowest load.

[0096] In this embodiment, the processor can detect the hardware resource utilization rate of each processing unit in the video channel, and then determine the load pressure of each processing unit 21 based on the hardware resource utilization rate, and transmit the original video data to the processing unit 21 with the lowest load. This can effectively balance the load pressure of each processing unit 21 in the video channel, thereby effectively improving the video encoding efficiency.

[0097] Optionally, this method may also include:

[0098] The processor determines the total load rate of all processing units in the video channel based on hardware resource utilization.

[0099] Determine if the total load rate is greater than a preset threshold;

[0100] If the total load rate is determined to be greater than the preset threshold, the direct memory access controller is controlled to write the raw video data into memory.

[0101] The direct memory access controller controls the transfer of raw video data from memory to the corresponding processing unit.

[0102] In this embodiment, processor 1 can determine whether the total load rate of all processing units in the video channel is greater than a preset threshold. If it is, processor 1 can control the direct memory access controller to write the raw video data into memory for temporary storage. Subsequently, processor 1 can control the direct memory access controller to transfer the raw video data from memory to the corresponding processing unit. In this way, when all processing units 21 of the video channel are under high load, processor 1 can first schedule the raw video data to memory 3, thereby relieving the pressure on the processing units.

[0103] Optionally, the programmable logic device further includes a backup processing unit; the method may also include:

[0104] The processor reads the status register corresponding to the processing unit of the video channel and determines whether the processing unit of the video channel is faulty based on the status register value.

[0105] If the processing unit of the video channel fails, the original video data of the video channel will be transmitted to the backup processing unit.

[0106] In this embodiment, the processor can read the status register corresponding to the processing unit of the video channel and determine whether the processing unit is faulty based on the status register value. If the processing unit is determined to be faulty, the processor can transmit the raw video data of the video channel to a backup processing unit so that the backup processing unit can continue video encoding processing, thereby improving the reliability of video encoding processing.

[0107] Optionally, the processing unit includes a preprocessing unit and an encoding unit, and the encoding unit for the video channel includes at least two units; the method may also include:

[0108] The preprocessing unit divides the video image frames in the raw video data into at least two sub-blocks;

[0109] The processor schedules sub-blocks to the various coding units of the video channel;

[0110] The encoding unit performs parallel encoding processing on the sub-blocks.

[0111] In this embodiment, to better perform video encoding, the processing unit can be functionally divided into two types: a preprocessing unit and an encoding unit. To fully utilize the parallel processing capabilities of the programmable logic device, the preprocessing unit can pre-divide the video image frame into multiple sub-blocks according to a preset macroblock size. Subsequently, the processor can schedule these multiple sub-blocks to various encoding units, which then perform parallel encoding processing on these sub-blocks. Thus, when processing each video image frame, this embodiment can fully utilize various encoding units for parallel encoding processing, thereby effectively improving processing efficiency.

[0112] Optionally, this method may also include:

[0113] The processing unit performs resolution adjustment and / or frame rate conversion on the raw video data to convert the raw video data into at least two output video data with different resolutions and / or different frame rates;

[0114] The processing unit encodes each output video data and writes the encoded video data corresponding to each output video data into memory.

[0115] The direct memory access controller transfers the encoded video data corresponding to each output video data from memory to the host.

[0116] In this embodiment, the processing unit can perform resolution adjustment and / or frame rate conversion on the original video data to convert the original video data into multiple output video data with different resolutions and / or different frame rates, so as to flexibly meet user needs.

[0117] Optionally, this method may also include:

[0118] The processing unit performs scene recognition on the raw video data to determine the image scene corresponding to the raw video data;

[0119] The encoding parameters are adjusted according to the image scene, and the original video data is encoded according to the adjusted encoding parameters.

[0120] In this embodiment, the processing unit can automatically identify image scenes in the original video data and dynamically adjust the encoding parameters (such as quantization parameters, frame type, reference frame, bit rate, etc.) according to the image scene, thereby achieving a balance between image quality and compression rate.

[0121] Optionally, the configuration parameters include the priority of the video channel, which can be either high or low priority; this method may also include:

[0122] The direct memory access controller sets the theoretical upper limit of the transmission bandwidth corresponding to each priority level.

[0123] Based on the theoretical upper limit of transmission bandwidth, the encoded video data of each priority video channel is transferred from memory to the host.

[0124] Detect the actual usage value of transmission bandwidth corresponding to each priority level;

[0125] When the actual usage of high-priority transmission bandwidth is less than or equal to the theoretical upper limit of high-priority transmission bandwidth, the unused high-priority transmission bandwidth will be allocated to low-priority transmission bandwidth.

[0126] When the actual usage of high-priority transmission bandwidth exceeds the theoretical upper limit of high-priority transmission bandwidth, the transmission of encoded video data of low-priority video channels is stopped, and the low-priority transmission bandwidth is allocated to high-priority channels.

[0127] In this embodiment, the direct memory access controller can set corresponding theoretical upper limits for transmission bandwidth for different priorities of video channels, and can provide data transmission services for video channels of each priority based on these theoretical upper limits. When the actual usage of transmission bandwidth for a high-priority channel is less than or equal to its theoretical upper limit, it means that the transmission bandwidth allocated to the high-priority video channel has not been exhausted. In this case, the unused transmission bandwidth of the high-priority channel can be allocated to the low-priority channel to ensure transmission efficiency. When the actual usage of transmission bandwidth for a high-priority channel is greater than its theoretical upper limit, it means that the high-priority video channel will occupy more bandwidth. In this case, the transmission of encoded video data from the low-priority video channel can be stopped, and the low-priority transmission bandwidth can be allocated to the high-priority channel to ensure priority transmission of the high-priority video channel.

[0128] Based on the above embodiments, please refer to Figure 3 , Figure 3 This is a structural block diagram of a video encoding system provided in an embodiment of the present invention. The video encoding system may include a host 20 and a video encoding device 10;

[0129] The host 20 is used to send raw video data to the video encoding device and receive encoded video data sent by the video encoding device.

[0130] The video encoding device 10 is used to encode the raw video data and send the encoded video data to the host.

[0131] Alternatively, host 20 can also be used for:

[0132] The configuration parameters of each video channel are written to the channel management register in the video encoding device through a preset interface.

[0133] Specifically, the preset interface can be an API interface.

[0134] Based on the above embodiments, the video encoding device, video encoding method and video encoding system will be fully described below with reference to specific schematic diagrams.

[0135] This invention proposes a method and system for improving video processing performance. The system enables high-speed data interaction with an x86 server via a high-bandwidth, low-latency PCIe interface. The video acceleration system employs a modular design, integrating a PCIe controller, DMA controller, embedded CPU, FPGA hardware encoder, DDR controller, and on-chip interconnect module. Through video channel management registers, the system can dynamically allocate and manage multiple video streams, ensuring that each video data stream is transmitted along a predetermined path. The on-chip interconnect module utilizes AXI bus technology to achieve efficient data transmission between modules, significantly reducing system latency and improving overall performance.

[0136] Please refer to Figure 4 , Figure 4 A structural block diagram of another video encoding device provided in an embodiment of the present invention. This device may include the following key components and their connection methods:

[0137] (1) PCIe controller: Located at the top layer of the system, it is responsible for high-speed data exchange with the external x86 host CPU.

[0138] (2) Video Channel Management Register: It is connected to the PCIe controller through the AXI Lite interface. By configuring the register, the video channel can be dynamically managed to achieve dynamic allocation and resource scheduling of multiple video streams.

[0139] (3) CPU and Firmware: Controls the operation of the entire system, including the startup and configuration of the video preprocessing module and the encoding module. It is responsible for encoding channel management and encoder configuration for different channels. The embedded CPU implements a load balancing algorithm, monitors FPGA resource utilization, dynamically allocates encoding tasks, and prioritizes allocating high-complexity frames to idle modules during processing. Through an exception recovery mechanism, when an FPGA module fails, the CPU automatically switches to a backup encoding channel to ensure system reliability. Deep integration with the host side allows the FFmpeg plugin to provide customized APIs, enabling host software to directly call hardware acceleration modules without modifying existing code.

[0140] (4) On-chip interconnect module (AXI interconnect bus): connects the embedded CPU, video preprocessing module, encoding module and DMA controller to ensure high-speed data flow within the system.

[0141] (5) FPGA-implemented video preprocessing and encoding module: Responsible for preprocessing and compressing the input video data to reduce data volume and improve transmission efficiency. FPGA can implement parallel processing and optimization of encoding algorithms through hardware description language (HDL). Block parallel processing can divide video frames into multiple regions, and distributed processing is implemented inside the FPGA. Adaptive quantization parameters dynamically adjust encoding parameters according to scene complexity to achieve a balance between image quality and compression rate. Supports multiple encoding standards, and flexible switching between video encoding standards such as H.264 / H.265 / AV1 can be achieved through RTL reconstruction.

[0142] (6) Multi-channel DDR controller and external DDR memory: Provides sufficient storage space and high-speed data read and write capabilities for caching and storing video data.

[0143] (7) DMA Controller: The DMA controller employs a parallel transmission mechanism, efficiently transferring data between the PCIe controller, on-chip interconnect module, and DDR controller without consuming CPU resources. It can process multiple video streams simultaneously or execute multiple data transfer tasks concurrently, significantly improving the speed and efficiency of video processing. This enhances the efficiency of video data transfer. Through scheduling algorithms, the DMA controller can dynamically adjust transmission priorities according to the needs of the video streams, ensuring real-time transmission of high-priority video streams. Furthermore, the DMA controller supports flexible configuration of transmission parameters, such as transmission direction, transmission size, and transmission speed, to adapt to different video processing needs and application scenarios. The DMA module allows for flexible configuration of transmission parameters, such as transmission direction, transmission size, and transmission speed, to adapt to different video processing needs and application scenarios.

[0144] (8) At the software level, video processing software is used on the x86 server for encoding scheduling, command issuance and other tasks. The original video stream is decoded by the video processing software, and the decoded YUV image is transmitted to the video acceleration system through the PCIe bus. After receiving the YUV image, the embedded CPU and firmware system perform encoding task scheduling and control the FPGA encoder to encode the stream in formats such as H.264, H.265 and AV1. Finally, the encoded stream is cached in DDR or directly transmitted back to the host server via PCIe and encapsulated into various video formats such as MP4 for video streaming.

[0145] Please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the connection between a host and a video encoding device according to an embodiment of the present invention. In this diagram, Host refers to the host machine, FFmpeg is the software on the host machine, API is the interface called by the software, Drivers is the driver program for the SoC (Sort of Entities), and SoC is the video encoding device.

[0146] For further details, please refer to... Figure 6 , Figure 6 A flowchart of another video encoding method provided in an embodiment of the present invention is shown below. The complete steps of the method are as follows:

[0147] The implementation and workflow of a video acceleration system involve the collaborative work of multiple components to ensure efficient processing and transmission of video data.

[0148] 1. Video Data Reception and Transmission. The host (x86 CPU) first sends video data to the video acceleration system via the PCIe interface. This data may come from various input sources, such as cameras, video files, or other video input devices. The CPU and firmware implementation is as follows:

[0149] (1) Video data is transmitted to the video acceleration system through the PCIe interface. It can be directly entered into the FPGA preprocessing module and encoder module through the scheduling of the CPU and firmware system. Alternatively, the data can be efficiently moved from the host to the external DDR memory of the system through the DMA controller as a cache or preprocessing.

[0150] (2) The embedded CPU runs a load balancing algorithm. By periodically reading the status registers of each module of the FPGA (such as processing unit utilization, cache utilization, DMA queue depth), it monitors the FPGA resource utilization in real time and dynamically allocates coding tasks to maximize throughput. The task allocation strategy is to allocate new tasks to the processing unit with the lowest current utilization. When the utilization of a processing unit exceeds a threshold (such as 80%), the allocation of new tasks to that unit is suspended until its load decreases.

[0151] 2. Video Data Storage. Video data is transmitted to the video acceleration system. First, the video channel management register allocates and manages each video channel, ensuring that each video data stream is transmitted along the predetermined path. Through flexible channel allocation strategies, the system can dynamically adjust resource allocation based on the number of video streams and processing requirements.

[0152] The DDR controller module manages the storage of this data in external DDR memory. Video data is temporarily stored in DDR memory to ensure data integrity and accessibility, awaiting further processing.

[0153] 3. Video preprocessing. The embedded CPU and firmware system manage the encoding channels, configure encoders for different channels, perform pre-analysis calculations (scene detection, frame type decision, etc.), manage reference frames, control frame rate and bit rate, and configure the FPGA preprocessing module to perform operations such as noise reduction, scaling, and color adjustment according to preprocessing requirements.

[0154] The preprocessing module performs the above processing on the video data cached in the DDR memory. That is, the input frame is transmitted to the preprocessing module of the FPGA via the AXI bus and divided according to the preset macroblock size. The segmentation strategy supports dynamic adjustment (for example, 8×8 macroblocks are used to improve accuracy in high motion scenes, while 16×16 or 32×32 can be used in other scenes). The preprocessed data is then stored in the DDR memory again.

[0155] 4. Video data encoding. Based on instructions from the host or scheduling within the video acceleration system, the DMA controller transfers video data from the DDR memory to the encoding module implemented on the FPGA.

[0156] (1) The encoding module executes video compression algorithms, such as H.264, H.265, and AV1. The FPGA encoding module compresses and encodes video data according to predetermined encoding parameters (such as resolution, bit rate, compression standard, etc.). During the encoding process, the FPGA adopts a block-parallel processing algorithm to divide the input video frame into multiple sub-blocks (such as 16×16 or 32×32 macroblocks). Each sub-block is assigned to an independent processing unit for parallel encoding. By utilizing the parallel processing capability of the FPGA, complex video encoding algorithms can be executed quickly, thereby achieving efficient video encoding.

[0157] (2) The FPGA-based video acceleration system not only supports traditional video encoding operations but also innovatively introduces the function of generating multiple video streams from a single video stream. Through a configurable scaling engine and frame rate controller, the FPGA performs real-time downsampling or interpolation processing on the input video to generate multiple output streams with different resolutions. This invention uses a bilinear interpolation algorithm, for example, dynamically adjusting the resolution of the input video from 4K (3840×2160) to 1080p (1920×1080) or 720p (1280×720). Through frame rate conversion algorithms (such as frame duplication or frame dropping), the frame rate of the input video is adjusted from 60fps to 30fps or 15fps to adapt to the needs of different bandwidths and display devices. Based on the network bandwidth and the performance of the client device, the FPGA automatically selects the optimal combination of resolution and frame rate for output, ensuring the smoothness and image quality of the video stream. Through the collaborative work of hardware and software, the system can dynamically generate multiple output videos from a single input video stream. This feature is suitable for scenarios with high real-time requirements such as live streaming and video conferencing, and can significantly reduce bandwidth requirements and improve video transmission efficiency.

[0158] 5. Encoded Video Data. After encoding, the video data can be stored again in DDR memory or directly transferred back to the host via the DMA controller. If long-term storage or subsequent processing is required, the encoded video data can also be written to external storage devices. When the encoded video data needs to be sent back to the host or other devices, the DMA controller transfers the data from the DDR memory to the PCIe controller. The PCIe controller then sends the data back to the host CPU via the PCIe interface for further processing by the FFMPEG software on the host or direct output to the display device. The system contains multiple DDR controllers, each managing the storage of one or more video data streams. The system can simultaneously store and process multiple video data streams, and each DDR controller can independently read and write different video data streams.

[0159] 6. DMA Scheduling Algorithm Implementation. The DMA controller employs a weighted round-robin algorithm to allocate bandwidth based on the priority of video streams. High-priority video streams (such as live streams) are allocated 80% of the transmission resources, while low-priority video streams (such as recorded streams) are allocated 20%. When the bandwidth demand of a high-priority stream decreases, the DMA controller automatically allocates the remaining bandwidth to low-priority streams, ensuring efficient utilization of system resources. High-priority streams can interrupt the transmission of low-priority streams, ensuring that video streams with high real-time requirements are processed first. The DMA transmission optimization mechanism achieves zero-copy transmission. Through an address mapping mechanism, DMA directly accesses memory, avoiding data copying. Simultaneously, the DMA controller integrates arbitration logic and supports priority queue management, with each queue independently configured with transmission parameters (such as burst length and transmission direction).

[0160] 7. Video Data Return. The host receives encoded video data via the PCIe interface. This data can be used for further processing, such as video editing, transcoding, or direct output to a display device. Video data can also be sent to other devices or streaming media servers via the network interface for user viewing or further distribution.

[0161] Throughout the process, the on-chip interconnect module and DMA controller within the video acceleration system ensure high-speed data transmission between modules, guaranteeing the smoothness and real-time performance of video processing. The embedded CPU monitors and manages the entire video processing flow, ensuring data correctness and timing accuracy. Meanwhile, the FPGA-implemented video preprocessing and encoding modules leverage their parallel processing capabilities to improve video processing efficiency and performance. This architecture offloads host-side video processing tasks to the video acceleration system, effectively handling large amounts of video data, meeting the high-performance requirements of modern video applications, and ensuring users receive high-quality video content promptly.

[0162] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program configured to execute the steps in any of the above-described video encoding method embodiments at runtime.

[0163] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0164] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described video encoding method embodiments.

[0165] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described video encoding method embodiments.

[0166] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0167] The foregoing has provided a detailed description of the video encoding apparatus, method, system, program product, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A video encoding device, characterized in that, The video encoding device is connected to the host via a bus interface, and at least two video channels are provided between the device and the host. The video encoding device includes: The system includes a processor, a programmable logic device, memory, a direct memory access controller, and a channel management register. The programmable logic device includes processing units for each of the video channels. The channel management register records configuration parameters for each of the video channels. The configuration parameters in the channel management register are configured by the host and are adjustable parameters used to indicate the encoding and transmission requirements of the video channels. The processor is configured to configure the encoder of the processing unit of the video channel according to the configuration parameters, and switch the hardware processing logic corresponding to each processing unit through register transfer level reconfiguration; wherein, the hardware processing logic corresponds to the video encoding standard, and the programmable logic device implements the hardware processing logic of multiple video encoding standards through a hardware description language, and switches the video encoding standard corresponding to each processing unit through register transfer level reconfiguration when the processor configures the encoder; and receives the raw video data corresponding to the video channel sent by the host, and transmits it to the processing unit corresponding to the video channel. The processing unit is used to encode the original video data and write the encoded video data into the memory; The direct memory access controller is used to allocate transmission bandwidth corresponding to each of the video channels according to the configuration parameters, and to transmit the encoded video data of each of the video channels from the memory to the host according to the transmission bandwidth; The configuration parameters include the priority of the video channel, which can be either high or low priority. The direct memory access controller is further configured to: Set the theoretical upper limit value of the transmission bandwidth corresponding to each priority level; Based on the theoretical upper limit of the transmission bandwidth, the encoded video data of each video channel of the specified priority is transmitted from the memory to the host. Detect the actual usage value of the transmission bandwidth corresponding to each of the priorities; When the actual usage value of the high-priority transmission bandwidth is less than or equal to the theoretical upper limit value of the high-priority transmission bandwidth, the unused high-priority transmission bandwidth is allocated to the low-priority transmission bandwidth. When the actual usage of the high-priority transmission bandwidth exceeds the theoretical upper limit of the high-priority transmission bandwidth, the transmission of encoded video data of the low-priority video channel is stopped, and the low-priority transmission bandwidth is allocated to the high-priority channel.

2. The video encoding apparatus according to claim 1, characterized in that, The video channel has at least two processing units; The processor is also used for: Detect the hardware resource utilization rate of each processing unit in the video channel; Based on the hardware resource utilization rate, the raw video data is transmitted to the processing unit with the lowest load.

3. The video encoding apparatus according to claim 2, characterized in that, The processor is also used for: The total load rate of all processing units in the video channel is determined based on the hardware resource utilization rate. Determine whether the total load rate is greater than a preset threshold; If it is determined that the total load rate is greater than the preset threshold, then the direct memory access controller is controlled to write the original video data into the memory; The direct memory access controller is controlled to transfer the raw video data from the memory to the corresponding processing unit.

4. The video encoding apparatus according to claim 1, characterized in that, The programmable logic device further includes a backup processing unit; The processor is also used for: Read the status register corresponding to the processing unit of the video channel, and determine whether the processing unit of the video channel is faulty based on the status register value; If the processing unit of the video channel fails, the original video data of the video channel will be transmitted to the backup processing unit.

5. The video encoding apparatus according to claim 1, characterized in that, The processing unit includes a preprocessing unit and an encoding unit, and the video channel has at least two encoding units; The preprocessing unit is used to divide the video image frames in the original video data into at least two sub-blocks; The processor is used to schedule the sub-blocks to the respective encoding units of the video channel; The encoding unit is used to perform parallel encoding processing on the sub-block.

6. The video encoding apparatus according to claim 1, characterized in that, The processing unit is further configured to: The original video data is subjected to resolution adjustment processing and / or frame rate conversion processing to convert the original video data into at least two output video data with different resolutions and / or different frame rates; The output video data is encoded, and the encoded video data corresponding to each output video data is written into the memory. The direct memory access controller is further configured to: The encoded video data corresponding to each of the output video data is transferred from the memory to the host.

7. A video encoding method, characterized in that, Applied to the video encoding apparatus as described in any one of claims 1 to 6, the method comprises: The processor configures the encoders of the processing units for each video channel according to the configuration parameters of each video channel recorded in the channel management register, and switches the hardware processing logic corresponding to each processing unit through register transfer level reconfiguration. The configuration parameters in the channel management register are configured by the host and are adjustable parameters indicating the encoding and transmission requirements of the video channel. These parameters include the priority of the video channel, which can be either high or low priority. The hardware processing logic corresponds to a video encoding standard. The programmable logic device implements the hardware processing logic for multiple video encoding standards using a hardware description language, and switches the video encoding standard corresponding to each processing unit through register transfer level reconfiguration when the processor configures the encoder. The processor receives the raw video data corresponding to the video channel from the host and transmits it to the processing unit corresponding to the video channel. The processing unit encodes the original video data and writes the encoded video data into memory. The direct memory access controller allocates the transmission bandwidth corresponding to each video channel according to the configuration parameters, and transmits the encoded video data of each video channel from the memory to the host according to the transmission bandwidth; The direct memory access controller sets the theoretical upper limit of the transmission bandwidth corresponding to each priority level; The direct memory access controller transmits the encoded video data of each priority video channel from the memory to the host according to the theoretical upper limit of the transmission bandwidth; The direct memory access controller detects the actual usage value of the transmission bandwidth corresponding to each priority. When the actual usage value of the high-priority transmission bandwidth is less than or equal to the theoretical upper limit of the high-priority transmission bandwidth, the direct memory access controller allocates the unused high-priority transmission bandwidth to the low-priority transmission bandwidth. When the actual usage of the high-priority transmission bandwidth exceeds the theoretical upper limit of the high-priority transmission bandwidth, the direct memory access controller stops transmitting the encoded video data of the low-priority video channel and allocates the low-priority transmission bandwidth to the high-priority channel.

8. A video encoding system, characterized in that, Includes a host computer and a video encoding device as described in any one of claims 1 to 6; The host is used to write configuration parameters of each video channel into the channel management register in the video encoding device; send raw video data to the video encoding device; and receive encoded video data sent by the video encoding device. The video encoding device is used to encode the original video data and send the encoded video data to the host.

9. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the video encoding method as described in claim 7.

Images