Video coding device, method and system, program product and storage medium

Abstract

The invention discloses a video coding device, method and system, a program product and a storage medium, and relates to the technical field of video processing. In the application, the video coding device can be connected with a host through a bus interface, and at least two video channels are arranged between the video coding device and the host. A processor, a programmable logic device, a memory, a direct memory access controller and a channel management register can be arranged in the video coding device. Wherein channel configuration of each video channel is stored in the channel management register, and a processing unit corresponding to each video channel is arranged in the programmable logic device. In practical application, the processor can perform encoder setting on the processing unit of each video channel according to channel configuration, and the direct memory access controller can distribute transmission bandwidth for each video channel according to the channel configuration. In this way, it can be ensured that the original video data of each video channel can be subjected to different encoding processing and transmission processing, and therefore the flexibility and expandability of multi-video-stream processing can be improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video encoding device, method, system, program product and storage medium. Background Art

[0002] With the continuous development of internet content, video streaming has essentially replaced text, images, and other forms of content, becoming the primary form of internet content. Due to the large size of raw video data, efficient encoding and compression of raw video data is crucial for the storage and distribution of video content.

[0003] To improve video encoding processing efficiency, related technologies have introduced methods to accelerate video encoding using hardware acceleration modules. However, these technologies still suffer from poor flexibility and scalability when processing multiple video streams, making it difficult to differentially encode and transmit multiple video streams. Summary of the Invention

[0004] The present invention provides a video encoding device, method, system, program product and storage medium, which can ensure that the video encoding device performs different encoding and transmission processing on each channel of original video data sent by the host according to needs, thereby improving the flexibility and scalability of multi-video stream processing.

[0005] To solve the above technical problems, the present invention provides a video encoding device, which is connected to a host via a bus interface and has at least two video channels set between the device and the host;

[0006] The video encoding device comprises:

[0007] A processor, a programmable logic device, a memory, a direct memory access controller, and a channel management register, wherein the programmable logic device includes a processing unit for each video channel, and the channel management register records configuration parameters of each video channel;

[0008] The processor is used to configure the encoder of the processing unit of the video channel according to the configuration parameters; receive the original video data corresponding to the video channel sent by the host and transmit it to the corresponding processing unit;

[0009] A processing unit, configured to encode the original video data and write the encoded video data into a memory;

[0010] The direct memory access controller is used to allocate the transmission bandwidth corresponding to each video channel according to the configuration parameters, and transmit the encoded video data of each video channel from the memory to the host according to the transmission bandwidth.

[0011] The present invention further provides a video encoding method, which is applied to the above-mentioned video encoding device, and the method includes:

[0012] The processor performs encoder configuration on the processing unit of each video channel according to the configuration parameters of each video channel recorded in the channel management register;

[0013] The processor receives the original video data corresponding to the video channel sent by the host and transmits it to the corresponding processing unit;

[0014] The processing unit encodes the original video data and writes the encoded video data into the memory;

[0015] The direct memory access controller allocates a transmission bandwidth corresponding to each video channel according to configuration parameters, and transmits the encoded video data of each video channel from the memory to the host according to the transmission bandwidth.

[0016] The present invention also provides a video encoding system, comprising a host and the above-mentioned video encoding device;

[0017] A host, configured to send original video data to the video encoding device and receive encoded video data sent by the video encoding device;

[0018] The video encoding device is used to encode the original video data and send the encoded video data to the host.

[0019] The present invention also provides a computer program product, comprising a computer program or instructions, which implements the above-mentioned video encoding method when executed by a processor.

[0020] The present invention also provides a non-volatile computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the above-mentioned video encoding method is implemented.

[0021] The present invention provides a video encoding device, which is connected to a host via a bus interface and has at least two video channels set between the device and the host. The video encoding device comprises: a processor, a programmable logic device, a memory, a direct memory access controller, and a channel management register. The programmable logic device includes a processing unit for each video channel, and the channel management register records configuration parameters of each video channel. The processor is used to perform encoder configuration on the processing unit of the video channel according to the configuration parameters; receive original video data corresponding to the video channel issued by the host and transmit it to the corresponding processing unit; the processing unit is used to encode the original video data and write the encoded video data into the memory; the direct memory access controller is used to allocate a transmission bandwidth corresponding to each video channel according to the configuration parameters, and transmit the encoded video data of each video channel from the memory to the host according to the transmission bandwidth.

[0022] The beneficial effects of the present invention are as follows: the video encoding device provided by the present invention can be connected to a host via a bus interface and set up at least two video channels with the host. The video encoding device includes a processor, a programmable logic device, a memory, a direct memory access controller, and a channel management register. The programmable logic device includes a processing unit for each video channel, and the channel management register records the configuration parameters of each video channel. In actual use, the processor can be used to configure the encoder of the processing unit of the video channel according to the configuration parameters, and receive raw video data corresponding to the video channel from the host and transmit it to the corresponding processing unit; the processing unit can be used to encode the raw video data and write the encoded video data to the memory; the direct memory access controller can be used to allocate the transmission bandwidth corresponding to each video channel according to the configuration parameters, and transmit the encoded video data of each video channel from the memory to the host according to the transmission bandwidth. As can be seen, by setting up a corresponding processing unit for each video channel and separately configuring the encoder configuration and transmission bandwidth of each video channel, the present invention can ensure that each channel of video data can receive different encoding and transmission processing, thereby improving the flexibility and scalability of multi-video stream processing.

[0023] The present invention also provides a video encoding method, system, program product and storage medium, which have the above beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 A structural block diagram of a video encoding device provided by an embodiment of the present invention;

[0026] Figure 2 A flowchart of a video encoding method provided by an embodiment of the present invention;

[0027] Figure 3 A structural block diagram of a video encoding system provided by an embodiment of the present invention;

[0028] Figure 4 A structural block diagram of another video encoding device provided by an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a connection between a host terminal and a video encoding device provided by an embodiment of the present invention;

[0030] Figure 6This is a flowchart of another video encoding method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0032] It should be noted that, in the description of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence.

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0034] With the continuous development of internet content, video streaming has essentially replaced text, images, and other forms of content, becoming the primary form of internet content. Due to the large size of raw video data, efficient encoding and compression of raw video data is crucial for the storage and distribution of video content. In related technologies, methods have emerged to improve video encoding processing efficiency by using hardware acceleration modules. However, these technologies still suffer from poor flexibility and scalability in processing multiple video streams, making it difficult to differentially encode and transmit multiple video streams.

[0035] In view of this, in order to address the technical problem of how to differentially encode and transmit multiple video streams, the present invention can provide a video encoding device, which can ensure that the video encoding device performs different encoding and transmission processing on each channel of original video data sent by the host according to needs, thereby improving the flexibility and scalability of multi-video stream processing.

[0036] For easier understanding, please refer to Figure 1 , Figure 1This is a block diagram of the structure of a video encoding device provided by an embodiment of the present invention. This video encoding device can be connected to a host computer via a bus interface (e.g., a PCIe interface, Peripheral Component Interconnect Express, a high-speed serial computer expansion bus standard) and has at least two video channels set up between the host computer and the device. This video encoding device may include:

[0037] Processor 1, programmable logic device 2, memory 3, direct memory access controller 4, channel management register 5, and bus interface 6. Programmable logic device 2 may include processing units 21 for each video channel. Bus interface 6 may be connected to channel management register 5 and direct memory access controller 4. Channel management register 5 may also be connected to processor 1 and direct memory access controller 4. Processor 1 may also be connected to programmable logic device 2 and direct memory access controller 4. Programmable logic device 2 may also be connected to memory 3. Direct memory access controller 4 may also be connected to memory 3.

[0038] It is understood that the processor 1, programmable logic device 2, memory 3, direct memory access controller 4, channel management register 5, and bus interface 6 can be connected via a bus structure. For example, to improve the transmission rate within the video encoding device, the above connection relationship can be implemented based on an AXI interconnect bus (Advanced eXtensible Interface). To reduce complexity, the channel management register 5 and bus interface 6 can be connected based on a lightweight AXI bus (AXIlite).

[0039] It should be noted that Figure 1 The video encoding device shown is only a basic form. In other possible scenarios, the processor 1, programmable logic device 2, memory 3, direct memory access controller 4, channel management register 5, or bus interface 6 may have different internal structures. For example, the bus interface 6 may include a bus controller, and the memory 3 may include a DDR controller (Double Data Rate Synchronous Dynamic Random Access Memory) and DDR memory. Furthermore, the video encoding device may also include other structures, such as a network port and external memory, which can be configured according to actual application requirements.

[0040] Additionally, channel management register 5 can record configuration parameters for each video channel. Configuration parameters are adjustable parameters that indicate the encoding and transmission requirements of a video channel. For example, these parameters may include encoder configuration (such as the encoding algorithm) and transmission configuration information (such as transmission priority). The configuration parameters in the channel management register can be configured by the host, allowing the host to flexibly adjust the encoding and transmission requirements of each video channel.

[0041] The specific uses of the processor 1, the processing unit 21, and the direct memory access controller 4 are as follows:

[0042] The processor 1 is configured to perform encoder configuration on the processing unit 21 of the video channel according to the configuration parameters; receive the original video data corresponding to the video channel sent by the host, and transmit it to the corresponding processing unit 21.

[0043] The processing unit 21 is configured to encode the original video data and write the encoded video data into the memory 3 .

[0044] The direct memory access controller 4 is used to allocate a transmission bandwidth corresponding to each video channel according to configuration parameters, and transmit the encoded video data of each video channel from the memory 3 to the host according to the transmission bandwidth.

[0045] It is worth noting that the programmable logic device 2 can implement hardware processing logic for multiple video processing methods through a hardware description language (HDL). When the processor 1 is configured for the encoder, the programmable logic device 2 can switch the video processing method corresponding to each processing unit 21 through RTL reconstruction to provide a specific video encoding method for the video channel. For example, the hardware processing logic for video encoding standards such as H.264 / H.265 / AV1 can be implemented through a hardware description language, and flexible switching of video encoding standards such as H.264 / H.265 / AV1 can be achieved through RTL reconstruction (Register Transfer Level).

[0046] As can be seen in this embodiment, the channel management controller 5 can store configuration parameters corresponding to each video channel. The processor 1 can configure the encoder for the processing unit 21 of each video channel based on these configuration parameters, for example, by setting the video preprocessing method and encoding algorithm required for each processing unit 21. The direct memory access controller 4 can allocate transmission bandwidth corresponding to each video channel based on these configuration parameters. Furthermore, when the host sends raw video data corresponding to each video channel (e.g., YUV image data, where Y represents luminance and U and V represent chrominance), the processor 1 can dispatch this raw video data to the processing unit 21 corresponding to each video channel. The processing unit 21 can encode the raw video data of the corresponding video channel based on the encoder configuration provided by the processor 1 and write the resulting encoded video data (e.g., H.264 encoded data, H.265 encoded data, AV1 encoded data, etc.) into the memory 3. The direct memory access controller can transfer the encoded video data of each video channel from the memory 3 to the host based on the allocated transmission bandwidth. It can be seen that this embodiment can flexibly adjust the encoding method and data transmission rate of each video channel, so that the video encoding device can perform targeted encoding and transmission processing on the original video data of each video channel, thereby improving the flexibility and scalability of multi-video stream processing.

[0047] In another embodiment, there are at least two processing units 21 for each video channel. The processor 1 can also be used to:

[0048] Detecting the hardware resource usage of each processing unit 21 of the video channel;

[0049] According to the hardware resource usage, the original video data is transmitted to the processing unit 21 with the lowest load.

[0050] In this embodiment, each video channel corresponds to at least two processing units 21, meaning that each video channel has at least two processing units 21 that can be scheduled and operated in parallel. To improve the video processing efficiency of each video channel, the processor 1 can detect the hardware resource utilization of each processing unit 21 in the video channel. Subsequently, the load pressure of each processing unit 21 can be determined based on the hardware resource utilization, and the original video data can be transmitted to the processing unit 21 with the lowest load. This effectively balances the load pressure of each processing unit 21 in the video channel, thereby effectively improving video encoding efficiency.

[0051] It should be noted that this embodiment does not limit how the processor 1 detects the hardware resource utilization of the processing unit 21. For example, the processor 1 can detect the utilization of various hardware resources (processing unit utilization, cache occupancy, DMA queue depth, DMA, Direct Memory Access) by the processing unit 21 by reading the status register of each processing unit 21.

[0052] Furthermore, the processor 1 may also be configured to:

[0053] Determine the total load rate corresponding to all processing units 21 of the video channel according to the hardware resource utilization rate;

[0054] Determine whether the total load rate is greater than a preset threshold;

[0055] If it is determined that the total load rate is greater than the preset threshold, the direct memory access controller 4 is controlled to write the original video data into the memory;

[0056] The direct memory access controller 4 is controlled to transfer the original video data from the memory 3 to the corresponding processing unit 21 .

[0057] In this embodiment, considering that all processing units 21 of the video channel may be in a situation of overall high load, the original video data sent by the host may be difficult to be processed in a timely manner. Therefore, when the processor 1 obtains the hardware resource utilization rate corresponding to each processing unit 21 of the video channel, it can further determine the total load rate corresponding to all processing units of the video channel. Subsequently, the processor 1 can determine whether the total load rate is greater than a preset threshold. If it is, the direct memory access controller can be controlled to write the original video data to the memory to temporarily store the original video data. Subsequently, the processor 1 can control the direct memory access controller to transfer the original video data from the memory to the corresponding processing unit 21. In this way, when all processing units 21 of the video channel are in a high load situation, the processor 1 can first schedule the original video data to the memory 3, thereby alleviating the pressure on the processing unit 21.

[0058] It should be noted that this embodiment does not limit how the processor 1 determines the above-mentioned total load rate, which can be set according to actual application requirements. For example, the hardware resource utilization rate of each processing unit 21 can be averaged to obtain the above-mentioned total load rate.

[0059] In another embodiment, the programmable logic device 2 may further include a backup processing unit. The processor 1 may also be used to:

[0060] Read the status register corresponding to the processing unit 21 of the video channel, and determine whether the processing unit 21 of the video channel is faulty according to the status register value;

[0061] If the processing unit 21 of the video channel fails, the original video data of the video channel is transmitted to the backup processing unit.

[0062] In this embodiment, considering that the processing unit 21 corresponding to the video channel may be damaged, thereby affecting the processing of the raw video data, this embodiment can also provide a backup processing unit in the programmable logic device 2. In this case, the processor 1 can read the status register corresponding to the processing unit 21 of the video channel and determine whether the processing unit 21 is faulty based on the status register value. If the processing unit 21 is determined to be faulty, the processor 1 can transfer the raw video data of the video channel to the backup processing unit, so that the backup processing unit can continue the video encoding processing, thereby improving the reliability of the video encoding processing.

[0063] In another embodiment, the processing unit 21 may also be configured to:

[0064] Perform scene recognition on the original video data to determine the image scene corresponding to the original video data;

[0065] The encoding parameters are adjusted according to the image scene, and the original video data is encoded according to the adjusted encoding parameters.

[0066] In this embodiment, to effectively balance image quality and image compression rate, the processing unit 21 can also perform scene recognition on the raw video data when encoding it, and obtain the image scene corresponding to the raw video data, such as a motion scene, a static scene, etc. Furthermore, the processing unit 21 can dynamically adjust encoding parameters (such as quantization parameter, frame type, reference frame, bit rate, etc.) based on the image scene, thereby achieving a balance between image quality and compression rate.

[0067] In another embodiment, to better perform video encoding, the processing unit 21 can be functionally divided into two types: a preprocessing unit and an encoding unit. The preprocessing unit is used to preprocess the raw video data, such as performing noise reduction, scaling, and color adjustment. The encoding unit is used to encode the preprocessed raw video data to generate encoded video data. Furthermore, each video channel has at least two encoding units to facilitate scheduling and parallel processing.

[0068] Furthermore, to improve coding efficiency, the pre-processing unit, processor, and coding unit may also have the following uses:

[0069] The pre-processing unit is further configured to divide the video image frame in the original video data into at least two sub-blocks;

[0070] Processor 1, configured to dispatch sub-blocks to respective coding units of a video channel;

[0071] The coding unit is also used to perform parallel coding processing on the sub-blocks.

[0072] In this embodiment, to fully utilize the parallel processing capabilities of programmable logic device 2, the pre-processing unit can pre-divide the video image frame into multiple sub-blocks according to a preset macroblock size. The processor 1 can then dispatch these multiple sub-blocks to the various encoding units, which then perform parallel encoding processing on these multiple sub-blocks. This allows each encoding unit to fully utilize the parallel encoding capabilities of each video frame, effectively improving processing efficiency.

[0073] In another embodiment, the processing unit 21 may also be configured to:

[0074] Performing resolution adjustment processing and / or frame rate conversion processing on the original video data to convert the original video data into at least two channels of output video data with different resolutions and / or different frame rates;

[0075] Encode each output video data, and write the encoded video data corresponding to each output video data into memory 3;

[0076] Direct memory access controller 4 can also be used for:

[0077] The encoded video data corresponding to each output video data is transmitted from the memory 3 to the host.

[0078] In this embodiment, to facilitate the conversion of a single input video stream into multiple output video streams with different resolutions and frame rates to flexibly meet user viewing needs, the processing unit 21 can also perform resolution adjustment processing (such as downsampling or interpolation) and frame rate conversion processing (such as frame duplication or frame dropping) on the original video data to convert the original video data into multiple output video streams with different resolutions and / or frame rates. For example, the input video resolution can be dynamically adjusted from 4K (3840×2160) to 1080p (1920×1080) or 720p (1280×720). Using a frame rate conversion algorithm (such as frame duplication or frame dropping), the input video frame rate can be adjusted from 60fps to 30fps or 15fps to accommodate different bandwidth and display device requirements. Furthermore, after the processing unit 21 completes encoding processing for each output video stream, the direct memory access controller 4 can transfer the encoded video data corresponding to each output video stream from the memory 3 to the host.

[0079] In another embodiment, the configuration parameters include the priority of the video channel, which is high priority or low priority; the direct memory access controller 4 can also be used to:

[0080] Set the theoretical upper limit of the transmission bandwidth corresponding to each priority;

[0081] According to the theoretical upper limit of the transmission bandwidth, the encoded video data of the video channel of each priority level is transmitted from the memory to the host;

[0082] Detect the actual usage value of the transmission bandwidth corresponding to each priority;

[0083] When the actual usage of the high-priority transmission bandwidth is less than or equal to the theoretical upper limit of the high-priority transmission bandwidth, the unused transmission bandwidth of the high-priority is allocated to the low-priority group.

[0084] When the actual usage value of the high-priority transmission bandwidth is greater than the theoretical upper limit of the high-priority transmission bandwidth, the transmission of the encoded video data of the low-priority video channel is stopped, and the low-priority transmission bandwidth is allocated to the high-priority channel.

[0085] In this embodiment, each video channel can be prioritized, and the DMA controller 4 can set different transmission bandwidths for video channels of different priorities. For example, a high-priority video stream (such as a live stream) can be allocated 80% of the transmission bandwidth, while a low-priority video stream (such as a recorded stream) can be allocated 20%. When the bandwidth demand of the high-priority stream decreases, the DMA controller 4 automatically allocates the remaining bandwidth to the low-priority stream, ensuring efficient utilization of system resources. When the high-priority stream requires more transmission bandwidth (for example, when it occupies more than 80% of the transmission bandwidth), the transmission of the low-priority stream can be interrupted, ensuring that video streams with high real-time requirements are processed first.

[0086] Based on the above embodiment, the video encoding method based on the above video encoding device is introduced below. For easy understanding, please refer to Figure 2 , Figure 2 This is a flow chart of a video encoding method provided by an embodiment of the present invention. This method is applied to the above-mentioned video encoding device and may include:

[0087] S201. The processor performs encoder configuration on the processing unit of each video channel according to the configuration parameters of each video channel recorded in the channel management register;

[0088] S202: The processor receives the original video data corresponding to the video channel sent by the host and transmits it to the corresponding processing unit;

[0089] S203, the processing unit encodes the original video data and writes the encoded video data into the memory;

[0090] S204 , the direct memory access controller allocates a transmission bandwidth corresponding to each video channel according to the configuration parameters, and transmits the encoded video data of each video channel from the memory to the host according to the transmission bandwidth.

[0091] As can be seen, the video encoding device provided by the present invention can be connected to a host via a bus interface and at least two video channels are set up between the host and the device. The video encoding device includes a processor, a programmable logic device, a memory, a direct memory access controller, and a channel management register. The programmable logic device includes a processing unit for each video channel, and the channel management register records the configuration parameters of each video channel. In actual use, the processor can be used to configure the encoder of the processing unit of the video channel according to the configuration parameters, and receive raw video data corresponding to the video channel from the host and transmit it to the corresponding processing unit; the processing unit can be used to encode the raw video data and write the encoded video data to the memory; the direct memory access controller can be used to allocate the transmission bandwidth corresponding to each video channel according to the configuration parameters, and transmit the encoded video data of each video channel from the memory to the host according to the transmission bandwidth. As can be seen, by setting a corresponding processing unit for each video channel and separately configuring the encoder configuration and transmission bandwidth of each video channel, the present invention can ensure that each channel of video data can receive different encoding and transmission processing, thereby improving the flexibility and scalability of multi-video stream processing.

[0092] Optionally, the configuration parameters in the channel management register can be configured by the host, which allows the host to flexibly adjust the encoding and transmission requirements of each video channel.

[0093] Optionally, the video channel includes at least two processing units; the method may further include:

[0094] The processor detects the hardware resource usage of each processing unit of the video channel;

[0095] Based on hardware resource usage, raw video data is transferred to the processing unit with the least load.

[0096] In this embodiment, the processor can detect the hardware resource utilization rate of each processing unit of the video channel, and then determine the load pressure of each processing unit 21 based on the hardware resource utilization rate, and transmit the original video data to the processing unit 21 with the lowest load. This can effectively balance the load pressure of each processing unit 21 of the video channel, thereby effectively improving the video encoding efficiency.

[0097] Optionally, the method may further include:

[0098] The processor determines the total load rate corresponding to all processing units of the video channel according to the hardware resource utilization rate;

[0099] Determine whether the total load rate is greater than a preset threshold;

[0100] If it is determined that the total load rate is greater than a preset threshold, controlling the direct memory access controller to write the original video data into the memory;

[0101] Controls the direct memory access controller to transfer raw video data from the memory to the corresponding processing unit.

[0102] In this embodiment, processor 1 can determine whether the total load rate corresponding to all processing units of the video channel is greater than a preset threshold. If so, it can control the direct memory access controller to write the raw video data to the memory for temporary storage. The processor can then control the direct memory access controller to transfer the raw video data from the memory to the corresponding processing unit. In this way, when all processing units 21 of the video channel are under high load, the processor can first dispatch the raw video data to memory 3, thereby alleviating the pressure on the processing units.

[0103] Optionally, the programmable logic device further includes a spare processing unit; and the method may further include:

[0104] The processor reads the status register corresponding to the processing unit of the video channel and determines whether the processing unit of the video channel is faulty according to the value of the status register;

[0105] If the processing unit of the video channel fails, the original video data of the video channel is transmitted to the backup processing unit.

[0106] In this embodiment, the processor can read the status register corresponding to the processing unit of the video channel and determine whether the processing unit has failed based on the status register value. If the processing unit is determined to be faulty, the processor can transfer the original video data of the video channel to the backup processing unit, so that the backup processing unit can continue the video encoding processing, thereby improving the reliability of the video encoding processing.

[0107] Optionally, the processing unit includes a preprocessing unit and an encoding unit, and the encoding unit of the video channel includes at least two. The method may further include:

[0108] The pre-processing unit divides the video image frame in the original video data into at least two sub-blocks;

[0109] The processor dispatches the sub-blocks to the various coding units of the video channel;

[0110] The coding unit performs parallel coding processing on the sub-blocks.

[0111] In this embodiment, to better perform video encoding, the processing units can be functionally divided into two types: a pre-processing unit and an encoding unit. To fully utilize the parallel processing capabilities of programmable logic devices, the pre-processing unit can pre-divide the video image frame into multiple sub-blocks according to a preset macroblock size. The processor can then dispatch these multiple sub-blocks to the various encoding units, which then perform parallel encoding processing on these multiple sub-blocks. In this way, when processing each video image frame, this embodiment can fully utilize the various encoding units for parallel encoding processing, thereby effectively improving processing efficiency.

[0112] Optionally, the method may further include:

[0113] The processing unit performs resolution adjustment processing and / or frame rate conversion processing on the original video data to convert the original video data into at least two channels of output video data with different resolutions and / or different frame rates;

[0114] The processing unit encodes each output video data and writes the encoded video data corresponding to each output video data into the memory;

[0115] The direct memory access controller transfers the encoded video data corresponding to each output video data from the memory to the host.

[0116] In this embodiment, the processing unit may perform resolution adjustment processing and / or frame rate conversion processing on the original video data to convert the original video data into multiple output video data with different resolutions and / or different frame rates to flexibly meet user needs.

[0117] Optionally, the method may further include:

[0118] The processing unit performs scene recognition on the original video data to determine the image scene corresponding to the original video data;

[0119] The encoding parameters are adjusted according to the image scene, and the original video data is encoded according to the adjusted encoding parameters.

[0120] In this embodiment, the processing unit can automatically identify the image scene in the original video data, and dynamically adjust the encoding parameters (such as quantization parameters, frame type, reference frame, bit rate, etc.) according to the image scene, so as to achieve the effect of balancing image quality and compression rate.

[0121] Optionally, the configuration parameter includes a priority of the video channel, which is a high priority or a low priority; the method may further include:

[0122] The direct memory access controller sets the theoretical upper limit of the transmission bandwidth corresponding to each priority level;

[0123] According to the theoretical upper limit of the transmission bandwidth, the encoded video data of the video channel of each priority level is transmitted from the memory to the host;

[0124] Detect the actual usage value of the transmission bandwidth corresponding to each priority;

[0125] When the actual usage of the high-priority transmission bandwidth is less than or equal to the theoretical upper limit of the high-priority transmission bandwidth, the unused transmission bandwidth of the high-priority is allocated to the low-priority group.

[0126] When the actual usage value of the high-priority transmission bandwidth is greater than the theoretical upper limit of the high-priority transmission bandwidth, the transmission of the encoded video data of the low-priority video channel is stopped, and the low-priority transmission bandwidth is allocated to the high-priority channel.

[0127] In this embodiment, the direct memory access controller can set corresponding theoretical upper limits of transmission bandwidth for different priorities of video channels, and can provide data transmission services for video channels of each priority level based on the theoretical upper limit of transmission bandwidth. When the actual usage value of the high-priority transmission bandwidth is less than or equal to the theoretical upper limit of the high-priority transmission bandwidth, it means that the transmission bandwidth set for the high-priority video channel has not been exhausted. At this time, the unused transmission bandwidth of the high-priority can be allocated to the low-priority to ensure transmission efficiency. When the actual usage value of the high-priority transmission bandwidth is greater than the theoretical upper limit of the high-priority transmission bandwidth, it means that the high-priority video channel will occupy more bandwidth. At this time, the transmission of the encoded video data of the low-priority video channel can be stopped, and the low-priority transmission bandwidth can be allocated to the high-priority to ensure priority transmission of the high-priority video channel.

[0128] Based on the above examples, please refer to Figure 3 , Figure 3 This is a structural block diagram of a video encoding system provided by an embodiment of the present invention. The video encoding system may include a host 20, a video encoding device 10;

[0129] The host 20 is used to send original video data to the video encoding device and receive encoded video data sent by the video encoding device;

[0130] The video encoding device 10 is used to encode original video data and send the encoded video data to a host.

[0131] Optionally, the host 20 may also be used to:

[0132] The configuration parameters of each video channel are written into the channel management register in the video encoding device through a preset interface.

[0133] Specifically, the preset interface may be an API interface.

[0134] Based on the above embodiments, the above video encoding device, video encoding method and video encoding system are fully introduced below based on specific schematic diagrams.

[0135] This paper proposes a method and system for improving video processing performance. This system uses a high-bandwidth, low-latency PCIe interface for high-speed data exchange with an x86 server. The video acceleration system utilizes a modular design, integrating a PCIe controller, a DMA controller, an embedded CPU, an FPGA hardware encoder, a DDR controller, and an on-chip interconnect module. Using video channel management registers, the system dynamically allocates and manages multiple video streams, ensuring that each video stream is transmitted along a predetermined path. The on-chip interconnect module utilizes AXI bus technology to achieve efficient data transmission between modules, significantly reducing system latency and improving overall performance.

[0136] Please refer to Figure 4 , Figure 4 This is a block diagram of another video encoding device provided by an embodiment of the present invention. This device may include the following key components and their connection methods:

[0137] (1) PCIe controller: Located at the top layer of the system, responsible for high-speed data exchange with the external x86 host CPU.

[0138] (2) Video channel management register: connected to the PCIe controller through the AXI Lite interface, the dynamic management of the video channel can be realized by configuring the register to achieve dynamic allocation and resource scheduling of multiple video streams.

[0139] (3) CPU and firmware: Controls the operation of the entire system, including the startup and configuration of the video preprocessing module and the encoding module. Responsible for encoding channel management, encoder configuration for different channels, etc. The embedded CPU implements a load balancing algorithm, monitors FPGA resource utilization, dynamically allocates encoding tasks, and prioritizes high-complexity frames to idle modules during processing. Through the exception recovery mechanism, when the FPGA module fails, the CPU automatically switches to the backup encoding channel to ensure system reliability. Deeply integrated with the host side, the FFmpeg plug-in can provide a customized API, allowing the host side software to directly call the hardware acceleration module without modifying the original code.

[0140] (4) On-chip interconnect module (AXI interconnect bus): connects the embedded CPU, video pre-processing module, encoding module and DMA controller to ensure high-speed data flow within the system.

[0141] (5) FPGA-implemented video preprocessing and encoding module: Responsible for preprocessing and compression encoding of input video data to reduce data volume and improve transmission efficiency. FPGA can implement parallel processing and optimization of encoding algorithms through hardware description language (HDL). Block parallel processing can divide video frames into multiple regions, and distributed processing is implemented within the FPGA. Adaptive quantization parameters can dynamically adjust encoding parameters according to scene complexity to achieve a balance between image quality and compression rate. Supports multiple encoding standards, and flexible switching of video encoding standards such as H.264 / H.265 / AV1 can be achieved through RTL reconstruction.

[0142] (6) Multi-channel DDR controller and external DDR memory: Provides sufficient storage space and high-speed data read and write capabilities for caching and storing video data.

[0143] (7) DMA controller: The DMA controller uses a parallel transmission mechanism to efficiently transfer data between the PCIe controller, on-chip interconnect module, and DDR controller without occupying CPU resources. It can process multiple video streams or perform multiple data transmission tasks simultaneously, significantly improving the speed and efficiency of video processing. It improves the efficiency of moving video data. Through the scheduling algorithm, the DMA controller can dynamically adjust the transmission priority according to the needs of the video stream to ensure the real-time transmission of high-priority video streams. In addition, the DMA controller supports flexible transmission parameter configuration, such as transmission direction, transmission size, transmission speed, etc., to adapt to different video processing requirements and application scenarios. The DMA module allows flexible configuration of transmission parameters, such as transmission direction, transmission size, transmission speed, etc., to adapt to different video processing requirements and application scenarios.

[0144] (8) At the software level, video processing software is used on the x86 server to perform encoding scheduling, command issuance, and other tasks. The original video code stream will be decoded by the video processing software, and the decoded YUV image will be transmitted to the video acceleration system through the PCIe bus. After receiving the YUV image, the embedded CPU and firmware system will schedule the encoding task and control the FPGA encoder to encode code streams in formats such as H.264, H.265, and AV1. Finally, the encoded code stream will be cached in DDR, or directly transmitted back to the host server by PCIe, and encapsulated into various video formats such as MP4 through the protocol for video streaming.

[0145] Please refer to Figure 5 , Figure 5 This is a schematic diagram of a connection between a host and a video encoding device provided by an embodiment of the present invention. Here, Host is the host, FFmpeg is the software in the host, API is the interface called by the software, Drivers is the driver of SoC, and SoC is the video encoding device.

[0146] For further information, please refer to Figure 6 , Figure 6 This is a flowchart of another video encoding method provided by an embodiment of the present invention. The complete steps of the method are as follows:

[0147] The implementation and workflow of a video acceleration system involves the collaborative work of multiple components to ensure efficient processing and transmission of video data.

[0148] 1. Video data reception and transmission. The host (X86 CPU) first sends video data to the video acceleration system via the PCIe interface. This data can come from a variety of input sources, such as cameras, video files, or other video input devices. The CPU and firmware system implementation is as follows:

[0149] (1) Video data is transmitted to the video acceleration system through the PCIe interface. It can directly enter the FPGA pre-processing module and encoder module through the scheduling of the CPU and firmware system. It can also efficiently move the data from the host side to the system's external DDR memory through the DMA controller for cache or pre-processing.

[0150] (2) The embedded CPU runs a load balancing algorithm, periodically reading the status registers of each FPGA module (such as processing unit utilization, cache occupancy, and DMA queue depth) to monitor FPGA resource utilization in real time and dynamically allocate encoding tasks to maximize throughput. The task allocation strategy is to assign new tasks to the processing unit with the lowest utilization. When the utilization of a processing unit exceeds a threshold (such as 80%), the allocation of new tasks to that unit is suspended until its load decreases.

[0151] 2. Video Data Storage. Video data is transmitted to the video acceleration system. First, the video channel management registers allocate and manage individual video channels, ensuring that each channel of video data is transmitted along the designated path. Flexible channel allocation strategies allow the system to dynamically adjust resource allocation based on the number of video streams and processing requirements.

[0152] The DDR controller module manages the storage of this data in the external DDR memory. Video data is temporarily stored in the DDR memory to ensure data integrity and accessibility, awaiting further processing.

[0153] 3. Video preprocessing. The embedded CPU and firmware system manages encoding channels, configures encoders for different channels, performs pre-analysis calculations (scene detection, frame type determination, etc.), manages reference frames, and controls frame rate and bit rate. The FPGA preprocessing module is configured based on preprocessing requirements to perform operations such as denoising, scaling, and color adjustment.

[0154] The preprocessing module performs the aforementioned processing on the video data cached in the DDR memory. The input frame is transmitted to the FPGA preprocessing module via the AXI bus and segmented according to the preset macroblock size. The segmentation strategy supports dynamic adjustment (for example, high-motion scenes use 8×8 macroblocks to improve accuracy, while other scenes can use 16×16 or 32×32). After preprocessing, the data is stored again in the DDR memory.

[0155] 4. Video data encoding: Based on host instructions or scheduling within the video acceleration system, the DMA controller transfers video data from the DDR memory to the encoding module implemented in the FPGA.

[0156] (1) The encoding module executes video compression algorithms such as H.264, H.265, and AV1. The FPGA encoding module compresses and encodes video data according to predetermined encoding parameters (such as resolution, bit rate, compression standard, etc.). During the encoding process, the FPGA uses a block-based parallel processing algorithm to divide the input video frame into multiple sub-blocks (such as 16×16 or 32×32 macroblocks). Each sub-block is assigned to an independent processing unit for parallel encoding. By utilizing the parallel processing capabilities of the FPGA, complex video encoding algorithms can be quickly executed, thereby achieving efficient video encoding.

[0157] (2) The video acceleration system implemented based on FPGA not only supports traditional video encoding operations, but also innovatively introduces the function of deriving multiple videos from one video. Through the configurable scaling engine and frame rate controller, FPGA performs real-time downsampling or interpolation processing on the input video to generate multiple output streams with different resolutions. The present invention adopts a bilinear interpolation algorithm, for example, dynamically adjusts the resolution of the input video from 4K (3840×2160) to 1080p (1920×1080) or 720p (1280×720). Through the frame rate conversion algorithm (such as frame repetition or frame drop), the frame rate of the input video is adjusted from 60fps to 30fps or 15fps to adapt to the requirements of different bandwidths and display devices. According to the network bandwidth and the performance of the client device, FPGA automatically selects the optimal resolution and frame rate combination output to ensure the smoothness and image quality of the video stream. Through the collaborative work of software and hardware, the system can dynamically derive one input video into multiple output videos. This feature is suitable for scenarios with high real-time requirements such as live broadcasts and video conferencing, which can significantly reduce bandwidth requirements and improve video transmission efficiency.

[0158] 5. Encoded video data. After encoding is completed, the video data can be stored in the DDR memory again, or directly transferred back to the host through the DMA controller. If long-term storage or subsequent processing is required, the encoded video data can also be written to an external storage device. When the encoded video data needs to be sent back to the host or other device, the DMA controller will transfer the data from the DDR memory to the PCIe controller. The PCIe controller then sends the data back to the host CPU through the PCIe interface, and the FFMPEG software on the host side further processes it or directly outputs it to the display device. The system contains multiple DDR controllers, each of which manages the storage of one or more channels of video data. The system can store and process multiple channels of video data at the same time, and each DDR controller can independently read and write different video data.

[0159] 6. Implementation of DMA scheduling algorithm. The DMA controller adopts a weighted polling algorithm to allocate bandwidth according to the priority weight of the video stream. High-priority video streams (such as live streams) are allocated 80% of the transmission resources, and low-priority video streams (such as recording streams) are allocated 20% of the transmission resources. When the bandwidth demand of high-priority streams decreases, the DMA controller automatically allocates the remaining bandwidth to low-priority streams to ensure efficient utilization of system resources. High-priority streams can interrupt the transmission of low-priority streams to ensure that video streams with high real-time requirements are given priority. The DMA transmission optimization mechanism realizes zero-copy transmission. Through the address mapping mechanism, DMA directly accesses the memory to avoid data copying. At the same time, the DMA controller integrates arbitration logic and supports priority queue management. Each queue independently configures transmission parameters (such as burst length and transmission direction).

[0160] 7. Video data transmission. The host receives encoded video data via the PCIe interface. This data can be used for further processing, such as video editing, transcoding, or direct output to a display device. Video data can also be sent to other devices or streaming servers via a network interface for user viewing or further distribution.

[0161] Throughout the entire process, the on-chip interconnect modules and DMA controllers within the video acceleration system ensure high-speed data transmission between modules, guaranteeing smooth and real-time video processing. The embedded CPU is responsible for monitoring and managing the entire video processing process, ensuring data correctness and timing accuracy. The video pre-processing and encoding modules implemented by the FPGA utilize its parallel processing capabilities to improve the efficiency and performance of video processing. Through this architectural design, host-side video processing tasks can be offloaded to the video acceleration system, which can effectively process large amounts of video data, meet the high-performance requirements of modern video applications, and ensure that users can receive high-quality video content in a timely manner.

[0162] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any one of the above-mentioned video encoding method embodiments when running.

[0163] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0164] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any one of the above-mentioned video encoding method embodiments are implemented.

[0165] An embodiment of the present invention further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned video encoding method embodiments are implemented.

[0166] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0167] The above describes in detail the video encoding device, method, system, program product, and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The above examples are intended only to facilitate understanding of the method and core concepts of the present invention. It should be noted that those skilled in the art may make various improvements and modifications to the present invention without departing from the principles of the present invention, and such improvements and modifications fall within the scope of protection of the present invention.

Claims

1. A video encoding device, characterized in that: The video encoding device is connected to the host through a bus interface, and at least two video channels are set between the device and the host; The video encoding device comprises: A processor, a programmable logic device, a memory, a direct memory access controller, and a channel management register, wherein the programmable logic device includes a processing unit for each of the video channels, and the channel management register records configuration parameters of each of the video channels; The processor is configured to perform encoder configuration on the processing unit of the video channel according to the configuration parameters; receive the original video data corresponding to the video channel sent by the host, and transmit the data to the corresponding processing unit; The processing unit is configured to encode the original video data and write the encoded video data into the memory; The direct memory access controller is configured to allocate a transmission bandwidth corresponding to each of the video channels according to the configuration parameters, and transmit the encoded video data of each of the video channels from the memory to the host according to the transmission bandwidth.

2. The video encoding device according to claim 1, wherein The configuration parameters in the channel management register are configured by the host.

3. The video encoding device according to claim 1, wherein The video channel has at least two processing units; The processor is further configured to: Detecting the hardware resource usage of each processing unit of the video channel; The original video data is transmitted to the processing unit with the lowest load according to the hardware resource usage rate.

4. The video encoding device according to claim 3, wherein The processor is further configured to: Determine the total load rate corresponding to all processing units of the video channel according to the hardware resource utilization rate; Determining whether the total load rate is greater than a preset threshold; If it is determined that the total load rate is greater than the preset threshold, controlling the direct memory access controller to write the original video data into the memory; The direct memory access controller is controlled to transfer the raw video data from the memory to a corresponding processing unit.

5. The video encoding device according to claim 1, wherein The programmable logic device further comprises a spare processing unit; The processor is further configured to: Reading a status register corresponding to the processing unit of the video channel, and determining whether the processing unit of the video channel is faulty according to the status register value; If the processing unit of the video channel fails, the original video data of the video channel is transmitted to the backup processing unit. The video encoding device according to claim 1 , wherein: The processing unit includes a pre-processing unit and an encoding unit, and the number of encoding units for the video channel is at least two; The pre-processing unit is configured to divide the video image frame in the original video data into at least two sub-blocks; The processor is configured to dispatch the sub-blocks to the respective encoding units of the video channel; The encoding unit is configured to perform parallel encoding processing on the sub-blocks.

7. The video encoding device according to claim 1, wherein The processing unit is further configured to: Performing resolution adjustment processing and / or frame rate conversion processing on the original video data to convert the original video data into at least two channels of output video data with different resolutions and / or different frame rates; performing encoding processing on each of the output video data, and writing the encoded video data corresponding to each of the output video data into the memory; The direct memory access controller is further configured to: The encoded video data corresponding to each of the output video data is transmitted from the memory to the host.

8. The video encoding device according to claim 1, wherein The configuration parameters include the priority of the video channel, where the priority is high priority or low priority; The direct memory access controller is further configured to: Setting the theoretical upper limit of the transmission bandwidth corresponding to each priority; transmitting the encoded video data of the video channels of each priority from the memory to the host according to the theoretical upper limit of the transmission bandwidth; Detecting an actual usage value of the transmission bandwidth corresponding to each priority; When the actual usage value of the high-priority transmission bandwidth is less than or equal to the theoretical upper limit of the high-priority transmission bandwidth, allocating the unused transmission bandwidth of the high-priority to the low-priority; When the actual usage value of the high-priority transmission bandwidth is greater than the theoretical upper limit of the high-priority transmission bandwidth, transmission of the encoded video data of the low-priority video channel is stopped, and the low-priority transmission bandwidth is allocated to the high-priority channel.

9. A video encoding method, characterized in that: Applied to the video encoding device according to any one of claims 1 to 8, the method comprises: The processor performs encoder configuration on the processing unit of each video channel according to the configuration parameters of each video channel recorded in the channel management register; The processor receives the original video data corresponding to the video channel sent by the host and transmits it to the corresponding processing unit; The processing unit encodes the original video data and writes the encoded video data into a memory; The direct memory access controller allocates a transmission bandwidth corresponding to each of the video channels according to the configuration parameters, and transmits the encoded video data of each of the video channels from the memory to the host according to the transmission bandwidth.

10. A video coding system, characterized in that: comprising a host and a video encoding device according to any one of claims 1 to 8; The host is configured to send original video data to the video encoding device and receive encoded video data sent by the video encoding device; The video encoding device is used to encode the original video data and send the encoded video data to the host.

11. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the video encoding method according to claim 9 is implemented.

Images