An AI pre-processing method and system based on video hard decoding

Abstract

This invention provides an AI preprocessing method and system based on video hardware decoding, comprising: a VPU video decoding unit built into an NPU chip decoding and scaling the input compressed video stream, outputting scaled RGB planar format frame data; the VPU video decoding unit transmitting the scaled RGB planar format frame data to a DMA controller via a hardware handshake mechanism; and the DMA controller performing vertical padding and normalization on the scaled RGB planar format frame data, storing the processed data in a tensor storage area for subsequent AI inference model inference. This invention integrates vertical padding and normalization into the DMA transfer process, which is completed in real-time by hardware during data transfer, without any intermediate buffers or CPU intervention. This directly eliminates multi-level copy overhead and CPU computational burden, significantly reducing end-to-end latency and overall power consumption.

Description

Technical Field

[0001] This invention relates to the field of AI data processing, and more specifically, to an AI preprocessing method and system based on video hardware decoding. Background Technology

[0002] With the rapid development of edge AI and real-time video intelligent analysis, NPU (Neural Processing Unit) chips need to efficiently process large amounts of video streams, quickly preparing the decoded video data as AI tensor input. Traditional solutions typically employ a multi-step software processing flow: after the VPU (Video Processing Unit) or video decoder outputs YUV / NV12 or other formats, the CPU or a general-purpose VPP (Video Post-Processing) module performs format conversion (to RGBplanar), calculates bounding padding (especially vertical padding to adapt to models with fixed input resolution), and performs normalization (e.g., dividing by 255.0 or normalization by mean and standard deviation), before writing to the tensor buffer through multiple memory copies. These steps lead to wasted memory bandwidth, increased end-to-end latency, and high CPU resource consumption, ultimately preventing the hardware performance from being fully utilized.

[0003] The existing technical solutions most similar to this patent include:

[0004] Some NPU chips (such as the Sophgo BM168x series) have VPP post-processing modules that support clipping, scaling, and padding before converting to tensors. However, these modules typically rely on software-driven calls. The VPU decoding output to the tensor still involves an intermediate buffer and CPU intervention, making it impossible to achieve a closed loop where DMA directly completes vertical padding, hardware real-time normalization, and writes the output to the tensor storage area after RGB planar decoding.

[0005] NVIDIA Jetson or Intel-related VPU / NVDEC solutions can perform hardware decoding and support some image processing, but padding, normalization, and transfer to tensors mostly rely on the CUDA kernel or additional copies, and have not formed a dedicated NPU-DMA-tensor hardware pipeline.

[0006] While other VPU products support decoding and 2D processing (including fill filters), fill and normalization are mostly general software configuration operations. They have not been hardware-integrated and optimized for NPU-specific scenarios such as "unchanged width, only vertical fill + RGB planar", and the normalization operation has not been integrated into the DMA transfer process. Summary of the Invention

[0007] This invention addresses the technical problems existing in the prior art by providing an AI preprocessing method and system based on video hardware decoding, which can solve the problem of low efficiency in the existing process from video decoding to AI tensor processing.

[0008] According to a first aspect of the present invention, an AI preprocessing method based on video hardware decoding is provided, comprising:

[0009] The VPU video decoding unit built into the NPU chip decodes and scales the input compressed video stream, and outputs scaled RGB planar format frame data.

[0010] The VPU video decoding unit transmits the scaled RGB planar format frame data to the DMA controller through a hardware handshake mechanism. The VPU output buffer and the DMA input buffer are designed to share the same storage area.

[0011] The DMA controller performs vertical padding and normalization on the scaled RGB planar format frame data, and stores the processed data in the tensor storage area.

[0012] Based on the above technical solution, the present invention can also be improved as follows.

[0013] Optionally, the VPU video decoding unit built into the NPU chip decodes and scales the input compressed video stream, outputting scaled RGB planar format frame data, including:

[0014] The NPU chip receives compressed video streams from external input and sends the compressed video streams to the VPU video decoding unit built into the NPU chip.

[0015] The VPU video decoding unit decodes the compressed video stream based on the configured decoding parameters and outputs decoded RGB planar format frame data, which includes three completely independent plane data: R, G, and B.

[0016] The hardware scaling unit integrated in the VPU video decoding unit scales the RGB planar format frame data according to the principle of "fixed width and maintaining the original aspect ratio" to obtain scaled RGB planar format frame data.

[0017] Optionally, the hardware scaling unit scales the RGB planar format frame data according to the principle of "fixed width, maintaining the original aspect ratio" to obtain scaled RGB planar format frame data, including:

[0018] Determine the target width, and calculate the scaling factor based on the target width and the original video width;

[0019] Calculate the scaled target height based on the original video height and the scaling factor;

[0020] The RGB planar format frame data is scaled to the target width and target height, and the scaled RGB planar format frame data is output.

[0021] Optionally, the DMA controller performs vertical padding and normalization on the scaled RGB planar format frame data, and stores the processed data in a tensor storage area, including:

[0022] After receiving the scaled RGB planar format frame data, the DMA controller immediately loads the pre-configured or dynamically generated DMA descriptor;

[0023] Configure the DMA registers according to the DMA descriptor;

[0024] The scaled RGB planar format frame data is vertically padded and normalized based on the configured DMA controller, and the processed data is stored in the corresponding tensor storage area.

[0025] Optionally, the DMA descriptor includes a source address, a destination address, size information, vertical padding parameters, a transfer mode, and normalization parameters;

[0026] The source address includes the base address of the scaled RGB planar format frame data, including the base addresses of the three independent planes: R, G, and B.

[0027] The target address is a pre-allocated AI tensor base address;

[0028] The size information includes the scaled target width and target height;

[0029] The vertical padding parameters include the number of top padding rows (top_padding), the number of bottom padding rows (bottom_padding), and the padding pixel value (padding_value).

[0030] The transmission mode is set to a vertical scanning mode, with the transmission width of each line fixed at the target width, and filling is only performed in the vertical direction.

[0031] The normalization parameters include a 1-bit hardware normalization enable bit and a 2-bit normalization mode selection bit.

[0032] Optionally, the normalization mode selection bit specifically includes:

[0033] When the normalization mode selection bit is 00, the normalization calculation formula is: output pixel value = original pixel value divided by 255.0, which maps the original pixel value from 0 to 255 to the range of 0 to 1.

[0034] When the normalization mode selection bit is 01, the normalization calculation formula is: Output pixel value = (Original pixel value - mean[channel]) / std[channel], where channel represents the R, G, and B channels, mean[channel] represents the mean pixel value of the corresponding channel, and std[channel] represents the standard deviation of the pixel value of the corresponding channel.

[0035] When the normalization mode selection bit is 10, the normalization calculation formula is: Output pixel value = (Original pixel value × scale[channel]) + bias[channel], where scale[channel] is the scaling factor of the corresponding channel and bias[channel] is the bias coefficient of the corresponding channel.

[0036] When the normalization mode selection bit is 11: no normalization processing is performed, the original data is retained, and the retained original data includes mean[channel]), std[channel], scale[channel] and bias[channel], as well as the data type output to the tensor storage area.

[0037] Optionally, the process of vertically padding and normalizing the scaled RGB planar format frame data based on the configured DMA controller, and storing the processed data in the corresponding tensor storage area, includes:

[0038] Generate and transmit top_padding row padding data, with each row having a fixed width equal to the target width and pixel values ​​uniformly set to the padding pixel values;

[0039] Transmit the target height of the line raw scaling data output by the VPU video decoding unit, wherein the corresponding line data of the R, G, and B plane pixel data are synchronously transported;

[0040] Generate and transmit bottom_padding row padding data, with each row having the same width as the target width;

[0041] The processed data is stored in the tensor storage area pointed to by the target address.

[0042] Optionally, the DMA controller performs vertical padding and normalization on the scaled RGB planar format frame data, stores the processed data in a tensor storage area, and then further includes:

[0043] Once the tensor acceleration unit of the NPU chip detects that the data in the tensor storage area is ready through hardware signals or status registers, it immediately starts the pre-loaded AI model inference task to infer the data. The AI ​​model inference task includes object detection task, image classification task, and semantic segmentation task.

[0044] Optionally, for multiple video streams,

[0045] The VPU video decoding unit built into the NPU chip decodes and scales the input compressed video stream, and outputs scaled RGB planar format frame data.

[0046] The VPU video decoding unit transmits the scaled RGB planar format frame data to the DMA controller through a hardware handshake mechanism. The VPU output buffer and the DMA input buffer are designed to share the same storage area.

[0047] The DMA controller performs vertical padding and normalization on the scaled RGB planar format frame data, and stores the processed data in the tensor storage area.

[0048] According to a second aspect of the present invention, an AI preprocessing system for video based on video hardware decoding is provided, including an NPU chip, wherein the NPU chip integrates a VPU video decoding unit and a DMA controller;

[0049] The VPU video decoding unit is used to decode and scale the input compressed video stream, and output scaled RGB planar format frame data; and to pass the scaled RGB planar format frame data to the DMA controller through a hardware handshake mechanism, wherein the VPU output buffer and the DMA input buffer are designed to share the same storage area.

[0050] The DMA controller is used to perform vertical padding and normalization on the scaled RGB planar format frame data, and store the processed data in the tensor storage area.

[0051] Optionally, the NPU chip integrates a VPU video decoding unit and a multi-channel DMA controller;

[0052] The VPU video decoding unit is used to decode and scale each input compressed video stream, output scaled RGB planar format frame data; and to pass the scaled RGB planar format frame data to the corresponding DMA controller through a hardware handshake mechanism. The VPU output buffer and the DMA input buffer are designed to share the same storage area.

[0053] Each of the DMA controllers is used to perform vertical padding and normalization on the scaled RGB planar format frame data, and store the processed data in the tensor storage area.

[0054] This invention provides an AI preprocessing method and system based on video hardware decoding. It directly outputs RGB planar format data through VPU hardware decoding within the NPU chip. Combined with a dedicated DMA controller, it simultaneously performs vertical (top / bottom) padding (maintaining the original width) and real-time hardware normalization during transmission. The processed data is then directly transferred to a pre-allocated AI tensor storage area via DMA, achieving an end-to-end preprocessing method from video decoding to normalized tensor input. This method eliminates the need for frequent CPU intervention and offers advantages such as high efficiency, low latency, high bandwidth utilization, and ease of multi-channel concurrency, providing strong support for the real-time performance of video AI inference. Attached Figure Description

[0055] Figure 1 A flowchart of an AI preprocessing method based on video hardware decoding is provided as an embodiment of the present invention;

[0056] Figure 2 The overall flowchart of an AI preprocessing method based on video hardware decoding is provided in one embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of an AI preprocessing system based on video hardware decoding, provided as an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0059] In existing video AI preprocessing, the input of RGB or YUV format data decoded by the VPU to the AI ​​tensor involves multiple levels of memory copying, format conversion, vertical padding calculations, and normalization (e.g., dividing by 255.0 or normalizing by mean and standard deviation). These operations are typically performed by the CPU or a general post-processing module, resulting in high end-to-end latency, significant memory bandwidth waste, and heavy CPU load. These problems are particularly pronounced when variable-resolution video streams require letterbox-style vertical padding (with a constant width, mainly filling the top and bottom portions) to adapt to the fixed input size of the AI ​​model.

[0060] Based on this, the present invention provides an AI preprocessing method based on video hardware decoding. After the VPU video decoder directly outputs RGB planar format, the DMA controller simultaneously performs vertical padding (width unchanged, only top / bottom padding) and real-time hardware normalization during data transmission. The processed data is then transferred to the AI ​​tensor storage area using DMA. This hardware pipeline approach significantly reduces end-to-end latency, saves memory bandwidth, and reduces CPU intervention, providing more efficient, lower-power, and higher real-time preprocessing capabilities for edge video AI applications.

[0061] Figure 1 The following is a flowchart illustrating an AI preprocessing method based on video hardware decoding according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, the method includes the following steps:

[0062] Step 1: The VPU video decoding unit built into the NPU chip decodes and scales the input compressed video stream, and outputs scaled RGB planar format frame data.

[0063] Understandably, the NPU chip receives compressed video streams from external inputs. These video streams can be any mainstream encoding format such as H.264, HEVC / H.265, or AV1, and support single or multiple concurrent inputs.

[0064] The video stream is sent to the VPU video decoding unit built into the NPU chip. The VPU video decoding unit is configured with decoding parameters, including the decoding output format being RGB planar (R, G, and B three completely independent plane data), rather than the traditional YUV packed format.

[0065] The VPU video decoding unit decodes the compressed video stream according to the configured decoding parameters and outputs RGB planar format frame data, including three completely independent plane data: R, G, and B.

[0066] The VPU video decoding unit integrates a hardware scaling unit during or after decoding to scale the decoded RGB planar format frame data according to the principle of "fixed width and maintaining the original aspect ratio".

[0067] During the scaling process, the target width is fixed to the target width (target_width) required by the AI ​​inference model (typically 640 pixels, but can also be configured to 384, 512, etc.). The scaling factor is calculated as: scaling factor = target width / original video width (scale_factor = target_width / decoded_width). The scaled target height is calculated as: target height = original video height * scaling factor (new_height = decoded_height × scale_factor), ensuring that the image is not stretched and the original aspect ratio is preserved (e.g., a 1080P video of 1920×1080 scaled to a width of 640 and a height of approximately 360).

[0068] Based on the configured target width and target height, the RGB planar format frame data output by the VPU video decoding unit is scaled, and the scaled RGB planar format frame data is output. Its width is the target width and its height is the new height calculated after scaling. Each pixel plane has an independent base address and row stride (the stride must meet the alignment requirements of the NPU memory controller, such as 64-byte or 128-byte alignment).

[0069] Step 2: The VPU video decoding unit transmits the scaled RGB planar format frame data to the DMA controller through a hardware handshake mechanism.

[0070] Understandably, the VPU decoding and scaling completion signals are directly transmitted to the DMA controller via a hardware handshake mechanism, avoiding CPU involvement in intermediate state queries or parameter calculations. The VPU output buffer and the DMA input buffer can be designed to share the same storage area, providing a foundation for subsequent DMA transfers.

[0071] In this embodiment of the invention, the VPU video decoding unit in the NPU chip directly outputs RGB planar format and is tightly coupled to the DMA controller in hardware. This allows the decoder to directly transmit the completion signal to the DMA controller through a dedicated hardware handshake mechanism, avoiding CPU intervention in intermediate state queries or parameter calculations.

[0072] Step 3: The DMA controller performs vertical padding and normalization on the scaled RGB planar format frame data, and stores the processed data in the tensor storage area.

[0073] Understandably, upon receiving scaled RGB planar frame data from the VPU video decoding unit, the DMA controller immediately loads a pre-configured or dynamically generated DMA descriptor. The DMA controller integrates a dedicated vertical padding engine, supporting a hardware processing mode of "unchanged width, only top / bottom padding," maintaining the original width and padding only the top and bottom portions to adapt to the fixed input resolution of the AI ​​model. Furthermore, the DMA controller integrates a hardware normalization engine, performing real-time normalization on each channel of the RGB planar during data transmission (supporting multiple modes such as division by 255.0, mean-std normalization, scale+bias, etc.), and then writing the normalized data after padding to a pre-allocated AI tensor storage area via DMA.

[0074] The DMA descriptor defines vertical fill parameters and normalization parameters. The DMA descriptor is shown in Table 1.

[0075] Table 1 DMA Descriptors

[0076]

[0077] The DMA descriptor must contain the following detailed fields:

[0078] (1) Source address: The base address of the RGB planar format data output by the VPU, including three independent planar base addresses: src_base_addr_R, src_base_addr_G, and src_base_addr_B;

[0079] (2) Target address: The pre-allocated base address of the AI ​​tensor (dst_tensor_base_addr), which corresponds to the fixed size of the AI ​​model input tensor (such as [1, 3, target_height, target_width] in NCHW format, where target_height is usually equal to target_width, for example, 640).

[0080] (3) Size information: target_width (target width, fixed), new_height (effective image height after scaling, calculated based on the original video aspect ratio);

[0081] (4) Vertical padding parameters: top_padding (number of top padding rows), bottom_padding (number of bottom padding rows), padding_value (padding pixel value, default 0 indicates black padding, supports zero padding, edge pixel copying or constant value padding and other modes); The vertical padding parameters are automatically calculated and determined by the DMA hardware based on new_height and the target height of the model;

[0082] (5) Transmission mode: vertical scan mode (line-by-line vertical scan), the transmission width of each line is fixed at target_width (keeping the width unchanged, only filling in the vertical direction);

[0083] (6) Normalization parameters:

[0084] norm_enable (1 bit): Hardware normalization enable bit (1 indicates enabled);

[0085] norm_mode (2 bits): Normalization mode selection, as follows:

[0086] 00: Divide by 255.0 (maps pixel values ​​from 0 to 255 to the range of 0 to 1);

[0087] 01: mean-std normalization, the calculation formula is: output pixel value = (original pixel value - mean[channel]) / std[channel], where channel represents the R, G, B channels, mean[channel] represents the mean pixel value of the corresponding channel, and std[channel] represents the standard deviation of the pixel value of the corresponding channel (FP16 or fixed-point format);

[0088] 10: Affine transformation (scale + bias processing), the calculation formula is: output pixel value = (original pixel value × scale[channel]) + bias[channel], where, scale[channel] is the scaling factor of the corresponding channel, and bias[channel] is the bias coefficient of the corresponding channel (FP16 or fixed point format).

[0089] 11: Retain (without normalization):

[0090] mean[channel]), std[channel], scale[channel], and bias[channel]: Normalization parameters for the R, G, and B channels respectively (stored in FP16 or fixed-point format);

[0091] output_data_type (2 bits): The data type of the output tensor (00: FP16; 01: FP32; 10: INT8 quantization; 11: Reserved).

[0092] Other control fields: stride (row step size, supports 64-byte or 128-byte alignment), next_desc_addr (next descriptor address, supports descriptor chaining).

[0093] After generating the DMA descriptor, the DMA controller is configured accordingly. Based on the configured DMA controller, the scaled RGB planar format frame data is vertically padded and normalized. Specifically, top_padding row padding data is generated and transmitted, with each row width fixed to the target width and pixel values ​​uniformly set to the padding pixel values. The original scaled row data of the target height output by the VPU video decoding unit is transmitted, and the corresponding row data of the R, G, and B plane pixel data are synchronously transported to ensure channel alignment. Bottom_padding row padding data is also generated and transmitted, with each row width also being the target width.

[0094] Throughout the process, the DMA controller integrates a dedicated vertical fill engine and a normalization engine (composed of a small number of vector multiply-accumulate units) to perform "fill + normalization" operations on each transmitted pixel in real time. The normalized data is then directly written to the tensor storage area in the format required by the model (FP16 / INT8, etc.). All operations are completed by the hardware engine during data transfer, without requiring the CPU to perform any intermediate calculations, format conversions, normalizations, or additional data transfers, effectively avoiding multiple memory copies, cache pollution, and bandwidth waste.

[0095] After the DMA controller completes the transfer, the scaled, vertically filled, and normalized data is directly located in the storage space corresponding to the AI ​​tensor, without any intermediate buffer, virtual address mapping, or additional copy operation.

[0096] Once the Tensor Core (or AI Engine) of the NPU chip detects that the data is ready through hardware signals or status registers, it immediately starts the pre-loaded AI model inference task (such as object detection, image classification, semantic segmentation, etc.).

[0097] The end-to-end latency from video input to tensor availability is significantly reduced, supporting automatic adaptation of variable resolution video streams (such as 1080P, 720P, etc.) to a fixed input model. Processing completion information can be fed back to the upper-layer software via a status register or completion interrupt, but the core data pipeline operates in a closed hardware loop throughout, with the CPU only needing to perform initial configuration and final result reading, greatly reducing CPU load.

[0098] See Figure 3 The diagram shows a schematic of an AI preprocessing system based on video hardware decoding according to an embodiment of the present invention. The system includes an NPU chip 31, which integrates a VPU video decoding unit 311 and a DMA controller 312.

[0099] The VPU video decoding unit 311 is used to decode and scale the input compressed video stream, and output scaled RGB planar format frame data; and to pass the scaled RGB planar format frame data to the DMA controller 312 through a hardware handshake mechanism, wherein the VPU output buffer and the DMA input buffer are designed to share the same storage area.

[0100] The DMA controller 312 is used to perform vertical padding and normalization on the scaled RGB planar format frame data, and store the processed data in the tensor storage area.

[0101] Among them, the NPU chip 31 integrates a multi-channel VPU video decoding unit 311 and a multi-channel DMA controller 312;

[0102] Each VPU video decoding unit 311 is used to decode and scale each input compressed video stream, output scaled RGB planar format frame data; and to pass the scaled RGB planar format frame data to the corresponding DMA controller 312 through a hardware handshake mechanism. The VPU output buffer and the DMA input buffer are designed to share the same storage area.

[0103] Each of the DMA controllers 312 is used to perform vertical padding and normalization on the scaled RGB planar format frame data, and store the processed data in the tensor storage area.

[0104] It is understood that the AI ​​preprocessing system based on video hardware decoding provided by the present invention corresponds to the AI ​​preprocessing method based on video hardware decoding provided in the foregoing embodiments. The relevant technical features of the AI ​​preprocessing system based on video hardware decoding can be referred to the relevant technical features of the AI ​​preprocessing method based on video hardware decoding, and will not be repeated here.

[0105] This invention provides an AI preprocessing method and system based on video hardware decoding. It directly outputs RGB planar format via VPU hardware decoding, integrates a vertical fill engine and a normalization engine with a DMA controller, and utilizes a closed-loop hardware pipeline of VPU-DMA-tensor storage. Compared to existing video AI preprocessing solutions, this method offers the following significant advantages:

[0106] First, existing technologies require the CPU or a general-purpose VPP module to sequentially complete format conversion, padding parameter calculation, normalization processing, and multiple memory copies. These software-driven multi-step operations inevitably lead to repeated memory bandwidth occupation, end-to-end latency accumulation, and increased CPU load. In contrast, this patent integrates vertical padding and normalization processing into the DMA transfer process, which is completed in real time by the hardware while data is being transferred, without any intermediate buffer or CPU intervention. This directly eliminates the overhead of multi-level copying and the CPU computational burden, significantly reducing end-to-end latency and overall power consumption.

[0107] Secondly, existing general-purpose filling and normalization engines are mostly general-purpose 2D processing or software-driven, and are not optimized for the special scenario of "unchanged width, only vertical filling + RGB planar + channel-based normalization" in NPU chips, resulting in low hardware utilization and poor real-time performance. This patent, however, uses a dedicated DMA descriptor field and hardware engine to accurately support letterbox-style vertical filling and real-time hardware normalization of variable resolution video streams (supporting multiple normalization modes), enabling different resolution inputs to be efficiently adapted to the fixed input size of AI models, greatly improving the targeting and execution efficiency of the hardware pipeline.

[0108] Furthermore, since the core processing flow is entirely completed by DMA triggered by the VPU hardware handshake, the CPU only needs to perform initial configuration and final result reading. Therefore, this patent performs excellently in multi-channel video concurrency scenarios, significantly reducing single-channel CPU usage and supporting high-density real-time video stream processing.

[0109] In summary, this patent has significant advantages in reducing latency, saving bandwidth, reducing power consumption, and improving real-time performance. It is particularly suitable for edge video AI application scenarios such as security monitoring, vehicle vision, and industrial inspection, providing efficient and reliable hardware-level support for improving the real-time performance of NPU chip video AI inference.

[0110] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0111] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0116] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An AI preprocessing method based on video hardware decoding, characterized in that, include: The VPU video decoding unit built into the NPU chip decodes and scales the input compressed video stream, and outputs scaled RGB planar format frame data. The VPU video decoding unit transmits the scaled RGB planar format frame data to the DMA controller through a hardware handshake mechanism. The VPU output buffer and the DMA input buffer are designed to share the same storage area. The DMA controller performs vertical padding and normalization on the scaled RGB planar format frame data, and stores the processed data in the tensor storage area for AI inference model to perform inference.

2. The AI ​​preprocessing method based on video hardware decoding according to claim 1, characterized in that, The VPU video decoding unit built into the NPU chip decodes and scales the input compressed video stream, outputting scaled RGB planar format frame data, including: The NPU chip receives compressed video streams from external input and sends the compressed video streams to the VPU video decoding unit built into the NPU chip. The VPU video decoding unit decodes the compressed video stream based on the configured decoding parameters and outputs decoded RGB planar format frame data, which includes three completely independent plane data: R, G, and B. The hardware scaling unit integrated in the VPU video decoding unit scales the RGB planar format frame data according to the principle of "fixed width and maintaining the original aspect ratio" to obtain scaled RGB planar format frame data.

3. The AI ​​preprocessing method based on video hardware decoding according to claim 2, characterized in that, The hardware scaling unit scales the RGB planar format frame data according to the principle of "fixed width, maintaining the original aspect ratio," resulting in scaled RGB planar format frame data, including: Determine the target width, and calculate the scaling factor based on the target width and the original video width; Calculate the scaled target height based on the original video height and the scaling factor; The RGB planar format frame data is scaled to the target width and target height, and the scaled RGB planar format frame data is output.

4. The AI ​​preprocessing method based on video hardware decoding according to claim 1, characterized in that, The DMA controller performs vertical padding and normalization on the scaled RGB planar format frame data, and stores the processed data in the tensor storage area, including: After receiving the scaled RGB planar format frame data, the DMA controller immediately loads the pre-configured or dynamically generated DMA descriptor; Configure the DMA registers according to the DMA descriptor; The scaled RGB planar format frame data is vertically padded and normalized based on the configured DMA controller, and the processed data is stored in the corresponding tensor storage area.

5. The AI ​​preprocessing method based on video hardware decoding according to claim 4, characterized in that, The DMA descriptor includes source address, destination address, size information, vertical padding parameters, transfer mode, and normalization parameters; The source address includes the base address of the scaled RGB planar format frame data, including the base addresses of the three independent planes: R, G, and B. The target address is a pre-allocated AI tensor base address; The size information includes the scaled target width and target height; The vertical padding parameters include the number of top padding rows (top_padding), the number of bottom padding rows (bottom_padding), and the padding pixel value (padding_value). The transmission mode is set to a vertical scanning mode, with the transmission width of each line fixed at the target width, and filling is only performed in the vertical direction. The normalization parameters include a 1-bit hardware normalization enable bit and a 2-bit normalization mode selection bit.

6. The AI ​​preprocessing method based on video hardware decoding according to claim 5, characterized in that, The normalization mode selection bit specifically includes: When the normalization mode selection bit is 00, the normalization calculation formula is: output pixel value = original pixel value divided by 255.0, which maps the original pixel value from 0 to 255 to the range of 0 to 1. When the normalization mode selection bit is 01, the normalization calculation formula is: Output pixel value = (Original pixel value - mean[channel]) / std[channel], where channel represents the R, G, and B channels, mean[channel] represents the mean pixel value of the corresponding channel, and std[channel] represents the standard deviation of the pixel value of the corresponding channel. When the normalization mode selection bit is 10, the normalization calculation formula is: Output pixel value = (Original pixel value × scale[channel]) + bias[channel], where scale[channel] is the scaling factor of the corresponding channel and bias[channel] is the bias coefficient of the corresponding channel. When the normalization mode selection bit is 11: no normalization is performed, the original data is retained, and the retained original data includes mean[channel], std[channel], scale[channel] and bias[channel], as well as the data type output to the tensor storage area.

7. The AI ​​preprocessing method based on video hardware decoding according to claim 5, characterized in that, The process of vertically padding and normalizing the scaled RGB planar format frame data based on the configured DMA controller, and storing the processed data into the corresponding tensor storage area, includes: Generate and transmit top_padding row padding data, with each row having a fixed width equal to the target width and pixel values ​​uniformly set to the padding pixel values; Transmit the target height of the line raw scaling data output by the VPU video decoding unit, wherein the corresponding line data of the R, G, and B plane pixel data are synchronously transported; Generate and transmit bottom_padding row padding data, with each row having the same width as the target width; The processed data is stored in the tensor storage area pointed to by the target address.

8. The AI ​​preprocessing method based on video hardware decoding according to claim 1, characterized in that, The DMA controller performs vertical padding and normalization on the scaled RGB planar format frame data, stores the processed data in the tensor storage area, and then further includes: Once the tensor acceleration unit of the NPU chip detects that the data in the tensor storage area is ready through hardware signals or status registers, it immediately starts the pre-loaded AI model inference task to infer the data. The AI ​​model inference task includes object detection task, image classification task, and semantic segmentation task.

9. An AI preprocessing system based on video hardware decoding, characterized in that, Includes an NPU chip, which integrates a VPU video decoding unit and a DMA controller; The VPU video decoding unit is used to decode and scale the input compressed video stream, and output scaled RGB planar format frame data; and to pass the scaled RGB planar format frame data to the DMA controller through a hardware handshake mechanism, wherein the VPU output buffer and the DMA input buffer are designed to share the same storage area. The DMA controller is used to perform vertical padding and normalization on the scaled RGB planar format frame data, and store the processed data in the tensor storage area.

10. The AI ​​preprocessing system based on video hardware decoding according to claim 9, characterized in that, The NPU chip integrates a VPU video decoding unit and a multi-channel DMA controller. The VPU video decoding unit is used to decode and scale each input compressed video stream and output scaled RGB planar format frame data. The scaled RGB planar format frame data is then passed to the corresponding DMA controller via a hardware handshake mechanism. The VPU output buffer and the DMA input buffer are designed to share the same storage area. Each of the DMA controllers is used to perform vertical padding and normalization on the scaled RGB planar format frame data, and store the processed data in the tensor storage area.

Images