GPU-based full-link zero-copy video decoding and unreal engine rendering method

Abstract

The present application relates to the technical field of image processing, and more particularly to a video decoding and Unreal Engine rendering method based on GPU full-link zero-copy, which comprises the following steps: step one, hardware decoding of an input video code stream on a GPU, and construction of a motion vector description buffer for storing compressed domain motion vector information in macroblock order; step two, reading of the motion vector description buffer by a calculation shader on the GPU to generate a dynamic special effect control buffer corresponding to each macroblock; and step three, in the post-processing material of the Unreal Engine rendering module, taking a shared video texture resource as an input texture and outputting a video picture. The present application realizes unified management of pixel data and motion data without relying on a host processor and without excessively occupying video memory bandwidth, significantly reduces delay, and improves special effect stability and direction consistency.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a video decoding and Unreal Engine rendering method based on GPU end-to-end zero-copy. Background Technology

[0002] With the rapid development of technologies such as video processing, real-time rendering, and virtual reality, the demand for directly integrating high-quality video content into 3D rendering scenes is becoming increasingly common. Especially with the widespread application of graphics engines like Unreal Engine in virtual shooting, real-time compositing, live streaming, and immersive interactive content production, achieving a seamless connection between video decoding and rendering with low latency and an efficient processing chain has become a key industry concern. Existing video decoding solutions typically use the host processor to perform decoding, copying the decoded pixel data to host memory, and then transmitting it to the graphics processing unit's video memory via the graphics interface for rendering. While this method offers high compatibility, in high-resolution video and high-frame-rate scenarios, frequent memory copying not only consumes significant bandwidth but also introduces additional latency, making synchronization between decoding and rendering more difficult. Furthermore, with the increasing number of dynamic effects in real-time rendering, such as motion blur, ghosting, velocity lines, or frame interpolation, the rendering stage often relies on motion information. However, traditional video decoding only uses motion vectors for inter-frame prediction, failing to provide motion vectors to the rendering end in a structured manner. This forces the engine to rely on additional calculations, such as optical flow estimation, when motion information is needed. These pixel-level optical flow reconstruction methods are typically expensive and have no advantage for latency-sensitive real-time systems.

[0003] Several technologies have attempted to reduce data copying overhead. For example, they utilize the graphics processing unit's hardware video decoding unit to directly write the decoding results to video memory, reducing the transfer between the host and video memory; or they employ a shared buffer mechanism, allowing the video decoding results to be directly read by the rendering end in some form. While these technologies can alleviate bandwidth pressure to some extent, several problems remain. First, most solutions still primarily focus on zero-copy transmission of pixel data, failing to extract motion information directly usable by the rendering end from the compressed domain. This results in the rendering end being unable to fully utilize the motion vectors already calculated during the hardware decoding stage. Second, these solutions lack a structured mapping method from compressed domain motion information to rendered effects. Even if the rendering end obtains motion vectors, it is difficult to directly use them to drive effects, requiring the engine to perform additional inferences for each frame or even each pixel, increasing the rendering burden. Third, existing solutions typically do not construct a unified data structure for organizing motion vectors, leading to issues such as coordinate mismatches, inconsistent frame references, and insufficient data precision when using motion information on the engine side, resulting in unstable effects.

[0004] On the other hand, although some technologies have proposed using optical flow inference or image differencing methods to generate dynamic effects, these methods themselves require a large amount of computing power. Furthermore, most optical flow inference operations performed on graphics processing units (GPUs) require extensive texture sampling across multiple channels, which can easily create performance bottlenecks in complex scenes. Simultaneously, optical flow methods often fail to work stably in high-noise encoded videos, resulting in jerky or broken effects that negatively impact the real-time rendering experience. Moreover, existing rendering solutions often rely on multiple rendering passes or post-processing steps when handling dynamic effects, such as applying motion blur, then overlaying velocity lines, and finally performing frame reprojection. This not only increases the depth of the rendering pipeline but also complicates resource management. Summary of the Invention

[0005] The main objective of this invention is to provide a GPU-based end-to-end zero-copy video decoding and Unreal Engine rendering method. By constructing a structured motion vector description buffer within the graphics processing unit and performing region-level aggregation of macroblock-level motion information, a dynamic effects control buffer is generated that can directly drive effects such as adaptive motion blur, object trailing, velocity lines, and inter-frame interpolation reprojection. This allows the rendering end to efficiently obtain the required motion features without additional inference. This method achieves unified management of pixel and motion data without relying on the host processor or excessively consuming video memory bandwidth, significantly reducing latency, improving the stability and directional consistency of effects, and making the visuals of fast-moving scenes more natural and coherent. It also possesses high scalability and real-time performance, meeting the stringent requirements of high-resolution and high-frame-rate video in real-time rendering.

[0006] To solve the above problems, the technical solution of the present invention is implemented as follows:

[0007] A GPU-based end-to-end zero-copy video decoding and Unreal Engine rendering method includes the following steps:

[0008] Step 1: Hardware decoding of the input video stream is performed on the GPU, shared video texture resources are generated in the GPU memory, and a motion vector description buffer is constructed to store compressed domain motion vector information in macroblock order.

[0009] Step 2: On the GPU, the motion vector description buffer is read through the computation shader, and the compressed domain motion vector region hierarchical aggregation and dynamic effect mapping algorithm is executed to generate dynamic effect control buffers that correspond one-to-one with each macroblock.

[0010] Step 3: In the post-processing material of the Unreal Engine rendering module, the shared video texture resource is used as the input texture, the dynamic effect control buffer is used as the structured buffer input, and adaptive motion blur, object trailing, speed line effect and inter-frame interpolation reprojection effect are applied to the screen pixels, and the video image is output.

[0011] Furthermore, in step one, a linearly contiguous motion vector description buffer is created in the GPU memory. Macroblocks in the current video frame are numbered from left to right and from top to bottom, with the numbers increasing sequentially from the beginning. The number of each macroblock is decremented by one to serve as the element index of the motion vector description buffer. The motion vector description buffer is divided into several record units. Each record unit includes eight fields stored in the form of four-byte integers. These fields sequentially store the signed integer representation of the horizontal motion vector component of the corresponding macroblock, the signed integer representation of the vertical motion vector component, the combined value of the width and height pixels, the reference frame index, the horizontal pixel coordinates of the top-left pixel of the macroblock in the shared video texture resource, the vertical pixel coordinates of the top-left pixel of the macroblock in the shared video texture resource, the video channel identifier, and a set of status flags for dynamic effects control.

[0012] Furthermore, in step two, the computation shader performs a single-block motion feature calculation process for each recording unit in the motion vector description buffer, including: reading the signed integer representations of the horizontal motion vector component and the vertical motion vector component from the current recording unit, obtaining the absolute values ​​of the two components and adding them to obtain the basic motion intensity value, dividing the basic motion intensity value into a first motion intensity level, a second motion intensity level, and a third motion intensity level, and dividing the motion direction of the macroblock into multiple discrete direction indices, including the horizontal direction, the vertical direction, and the diagonal direction, according to the sign and absolute value relationship of the two components.

[0013] Furthermore, the multiple discrete direction indices include a first direction index, a second direction index, a third direction index, a fourth direction index, a fifth direction index, a sixth direction index, a seventh direction index, and an eighth direction index, which are used to represent the movement directions toward the right, the left, the bottom, the top, and the four diagonal directions, respectively.

[0014] Furthermore, in step two, the computation shader divides the current video frame into several rectangular local regions based on the horizontal and vertical pixel coordinates of the top-left corner pixel of each recording unit in the motion vector description buffer within the shared video texture resource. Each rectangular local region covers a preset number of consecutive macroblocks. The shader then sequentially traverses the macroblock recording units covered by each rectangular local region, counts the number of macroblocks corresponding to the first motion intensity level, the second motion intensity level, and the third motion intensity level, as well as the number of macroblocks corresponding to each discrete direction index. The motion intensity level with the most occurrences is selected as the regional motion intensity level of the rectangular local region, and the discrete direction index with the most occurrences is selected as the regional dominant direction index of the rectangular local region. The regional motion intensity level and the regional dominant direction index are combined into a regional motion description value and written into the regional intermediate buffer.

[0015] Furthermore, in step two, a dynamic effects control buffer corresponding one-to-one with the motion vector description buffer is created in the GPU memory. Each record unit in the dynamic effects control buffer includes eight fields stored in the form of four-byte integers, which are used to store the dynamic effects type mask value, the signed integer value of the horizontal component of the motion blur sampling step size, the signed integer value of the vertical component of the motion blur sampling step size, the unsigned integer value of the number of motion blur sampling points, the combination value of the number of object shadow superpositions and the number of pixels of shadow interval, the combination value of the number of velocity lines and the number of pixels of velocity line length, the combination value of the horizontal component of the inter-frame interpolation reprojection offset and the vertical component of the inter-frame interpolation reprojection offset, and the reference sampling mode identifier. Different bits of the dynamic effects type mask value are used to indicate whether the adaptive motion blur effect, object shadow effect, velocity line effect, and inter-frame interpolation reprojection effect are enabled.

[0016] Furthermore, when the computational shader writes fields to the dynamic effects control buffer, it sets the parameters based on the regional motion intensity level and the dominant direction index of the rectangular local region to which each macroblock belongs. Specifically, when the regional motion intensity level is the first motion intensity level, the pixel length of both the horizontal and vertical components of the motion blur sampling step is set to one pixel, and the number of motion blur sampling points is set to three. When the regional motion intensity level is the second motion intensity level, the pixel length of both the horizontal and vertical components of the motion blur sampling step is set to two pixels, and the number of motion blur sampling points is set to five. The parameters for the third motion intensity level and so on are further specified. At level 1, the pixel length of both the horizontal and vertical components of the motion blur sampling step size is set to three pixels, and the number of motion blur sampling points is set to seven. When the dynamic effect type mask value indicates that the object trailing effect is enabled, the number of object trailing superpositions is set to three, five, and seven times respectively according to the region motion intensity level, and the number of trailing interval pixels is set to be the same as the corresponding motion blur sampling step size pixel length. When the dynamic effect type mask value indicates that the velocity line effect is enabled and the region dominant direction index is a diagonal direction index, the number of velocity lines is set to four or six, and the number of velocity line length pixels is set to be the same as the number of macroblock width pixels or twice the number of macroblock width pixels.

[0017] Furthermore, in steps two and three, the shared video texture resources, motion vector description buffers, and dynamic effects control buffers are all located in the GPU memory and are directly shared between the hardware video decoding unit and the Unreal Engine rendering module through GPU-side resource handles. During the entire method execution process, the shared video texture resources, motion vector description buffers, and dynamic effects control buffers are not copied from the GPU memory to the CPU memory, nor are they copied between different GPUs.

[0018] Furthermore, in step three, the shared video texture resource is bound as the input texture of the post-processing material shader, and the dynamic effects control buffer is bound as the structured buffer input of the post-processing material shader. For each pixel on the screen, the post-processing material shader determines the number of the macroblock in which the pixel is located based on the horizontal and vertical pixel coordinates of the pixel in the shared video texture resource and accesses the corresponding record unit in the dynamic effects control buffer. When the dynamic effects type mask value indicates that the adaptive motion blur effect is enabled, multiple motion blur sampling positions are calculated on the shared video texture resource starting from the current pixel position along the direction determined by the horizontal and vertical components of the motion blur sampling step size. The pixel color is read at each motion blur sampling position and accumulated to form the adaptive motion blur effect. When the dynamic effects type mask value indicates that the object trailing effect is enabled, multiple trailing sampling positions are generated at the current pixel position along the direction corresponding to the region's dominant direction index based on the number of object trailing superpositions and the number of trailing interval pixels. The pixel color is read at each trailing sampling position and superimposed with the pixel color of the adaptive motion blur effect to form the object trailing effect.

[0019] The video decoding and Unreal Engine rendering method based on GPU end-to-end zero-copy in this invention has the following advantages: This invention establishes a unified data link that resides within the graphics processing unit throughout the entire process between video decoding and real-time rendering. This allows pixel data and compressed domain motion vectors generated during the decoding stage to directly participate in the rendering process in a structured manner, thereby significantly reducing latency and redundant computation. This invention writes macroblock-level motion information into a fixed-format motion vector description buffer and then generates a dynamic effects control buffer through region-level aggregation. This enables the rendering end to obtain motion-related features in a consistent, clear, and low-cost manner, avoiding the blurring instability, direction drift, and performance overhead caused by relying on optical flow inference or historical frame inference in traditional rendering. Based on the integrated region-level motion information, the dynamic effects control buffer of this invention enables effects such as motion blur, object trailing, velocity lines, and inter-frame interpolation reprojection to present a hierarchical and continuous visual effect according to the motion direction and intensity, reducing blocky edges, breakage marks, and abrupt changes in effect intensity. By employing a unified data structure, this invention enables the rendering end to perform special effects calculations simply by reading fields, reducing the inference logic within the shader and improving the stability of the rendering stage. Furthermore, this invention completely avoids the back-and-forth transmission of motion information and pixel data between host memory and graphics processing unit memory, maintaining a zero-copy characteristic throughout the processing chain and meeting the stringent real-time requirements of high-resolution, high-frame-rate video scenes. By introducing compressed domain motion vectors into the rendering side, this invention allows existing dynamic effects to achieve higher expressiveness with lower resource costs, resulting in clearer object edges and more natural motion trajectories in fast-moving scenes, while significantly reducing processing latency and improving visual coherence. Overall, this invention not only solves the problem of data disconnect between decoding and rendering but also provides a scalable and configurable special effects mapping mechanism, making video content appear more realistic, detailed, and adaptable to a wider range of applications within the Unreal Engine. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the principle of discretized polar coordinates for motion vector intensity and direction, provided in an embodiment of the present invention.

[0021] Figure 2 A schematic diagram of the register-level structure of the motion vector description buffer (MVDB) provided in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram illustrating the structure and mask definition of the Dynamic Effects Control Buffer (DECB) provided in an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] A GPU-based end-to-end zero-copy video decoding and Unreal Engine rendering method includes the following steps:

[0025] Step 1: Hardware decoding of the input video stream is performed on the GPU, shared video texture resources are generated in the GPU memory, and a motion vector description buffer is constructed to store compressed domain motion vector information in macroblock order.

[0026] Step 2: On the GPU, the motion vector description buffer is read through the computation shader, and the compressed domain motion vector region hierarchical aggregation and dynamic effect mapping algorithm is executed to generate dynamic effect control buffers that correspond one-to-one with each macroblock.

[0027] Step 3: In the post-processing material of the Unreal Engine rendering module, the shared video texture resource is used as the input texture, the dynamic effect control buffer is used as the structured buffer input, and adaptive motion blur, object trailing, speed line effect and inter-frame interpolation reprojection effect are applied to the screen pixels, and the video image is output.

[0028] In a typical implementation, the input video stream is first received by the host processor and written to a contiguous memory region directly accessible by the GPU. This memory region can be mapped to the GPU's address space via page-locked memory or direct memory access. The purpose of this is to reduce the round-trip movement of the stream between host memory and GPU memory, allowing subsequent hardware video decoding units to directly read the stream data from addresses visible to the GPU, ensuring the overall data path is as short as possible and reducing latency.

[0029] After receiving the video stream, the hardware video decoding unit parses the header information segment by segment according to the video encoding format to obtain configuration data such as image width, image height, macroblock division method, number of reference frames, and motion vector precision. Taking an image width of 1920 pixels, an image height of 1080 pixels, and a macroblock size of 16 pixels by 16 pixels as an example, the number of macroblocks is 120 horizontally and 68 vertically, for a total of 8160 macroblocks. Based on this parsed data, the hardware video decoding unit allocates several frame buffers in the GPU memory. Each frame buffer stores one frame of decoded video pixel data, and a corresponding motion information data area is reserved for each frame for subsequent extraction of compressed domain motion vector information.

[0030] The generation of shared video texture resources is based on the aforementioned frame buffer. The hardware video decoding unit writes the decoded luminance and planar chrominance components into the frame buffer in video memory according to a predetermined pixel format. The pixel format can be either a 420 format (separate luminance and chrominance) or a 422 format (separate luminance and chrominance). The system creates a texture view for each frame buffer at the GPU's resource management layer, binding the frame buffer's video memory address, line-aligned byte count, pixel format, and other information to the texture view object. The texture view object, as a shared video texture resource, only exposes the texture handle and resolution information. In subsequent rendering stages, Unreal Engine can directly sample pixels from the decoded frame buffer using this texture handle, without needing to copy an additional texture internally within the GPU. This approach ensures that the pixel data generated during the decoding stage and the pixel data used during the rendering stage point to the same video memory region, avoiding unnecessary copying at the source.

[0031] The construction of the motion vector description buffer is synchronized with the shared video texture resources. Each macroblock in the image is uniquely determined by its row and column position in the macroblock grid. The system uses a left-to-right, top-to-bottom macroblock numbering method, assigning a number starting from 1 to each macroblock in each frame. Continuing with the resolution mentioned above, the leftmost macroblock in the first row is numbered 1, the second macroblock in the first row is numbered 2, the 120th macroblock in the first row is numbered 120, the leftmost macroblock in the second row is numbered 121, and so on until the last macroblock is numbered 8160. To facilitate quick location of the corresponding motion vector record within the GPU using the macroblock number, the system uses a linear storage method within the motion vector description buffer, subtracting 1 from the macroblock number as the array index. This creates a one-to-one correspondence between the macroblock number and the buffer record. Subsequent computation shaders only need to perform a single multiplication and addition calculation based on the macroblock number to obtain the record address. The access logic is simple and the access pattern is continuous, which is beneficial for GPU cache hits.

[0032] The motion vector description buffer itself is allocated as a linear, contiguous storage space in GPU memory, with a total length equal to the number of macroblocks multiplied by the length of a single record unit. Each record unit is a fixed 32 bytes long, containing eight fields stored as 4-byte integers. The first field stores the signed integer value of the horizontal motion vector component of the current macroblock, and the second field stores the signed integer value of the vertical motion vector component of the current macroblock. To directly reflect the size information of the macroblock in the image, the third field uses a 32-bit unsigned integer to represent the number of pixels for the width and height of the macroblock. The first 16 bits represent the number of pixels for the width, and the next 16 bits represent the number of pixels for the height. This combination reduces the number of fields while retaining two independent values, facilitating subsequent reading in the compute shader and separating the width and height through shift and bitwise AND operations, while ensuring a compact buffer structure and aligned access.

[0033] The fourth field stores the reference frame index value, using a signed integer. When parsing the bitstream, the hardware video decoding unit determines the reference frame source for each macroblock, such as referencing the previous frame, the two previous frames, or a specific keyframe. Writing the reference frame index directly into the motion vector description buffer facilitates subsequent inter-frame interpolation and reprojection processes in finding the sampling position in the reference frame texture resource based on the reference frame index. Furthermore, it allows inference of the motion source without accessing the bitstream, reducing coupling between different processing stages.

[0034] The fifth and sixth fields store the horizontal and vertical coordinates of the macroblock's top-left pixel in the shared video texture resource, respectively. The horizontal coordinate is counted starting from the leftmost pixel of the image, and the vertical coordinate is counted starting from the topmost pixel, both in pixels. For example, with a 16-pixel by 16-pixel macroblock, the first macroblock in the first row has a horizontal coordinate of 0 and a vertical coordinate of 0, while the second macroblock in the first row has a horizontal coordinate of 16 and a vertical coordinate of 0. In this way, during subsequent shading, as long as the macroblock number is known, the position of the top-left pixel of the macroblock can be directly obtained by looking up the table. Combined with the pixel offset within the macroblock, the screen coordinates can be quickly converted to texture sampling coordinates. This pre-writing of coordinates reduces the number of repeated integer division and multiplication operations in the shader, improving the performance of inter-frame processing.

[0035] The seventh field stores the video channel identifier, using integers to represent different video channels. For example, it is uniformly set to 0 for a single video stream, while different channels use different identifier values ​​when processing multiple video streams in parallel. The significance of introducing video channel identifiers is that macroblock motion information from multiple video streams can be recorded simultaneously in the same motion vector description buffer. In subsequent calculations, the set of macroblocks to be processed is selected by the channel identifier, thus supporting multiple video streams sharing a single set of computation shader logic.

[0036] The 8th field is a set of status flags, stored as a 32-bit unsigned integer, where different bits represent different status information. For example, the first bit can indicate whether the current macroblock is located in an image edge region, the second bit can indicate whether the current macroblock participates in dynamic effects calculations, the third bit can indicate whether the motion vector is reliable, and the fourth bit can indicate whether the current macroblock belongs to a fast-moving region. The remaining bits are reserved for expanding other states. By encoding these Boolean states into a single field, rich conditional judgment criteria can be provided for subsequent dynamic effects control without increasing the buffer length.

[0037] In one specific embodiment, after the graphics processing unit (GPU) completes hardware decoding of the input video stream, the decoded video frames are stored in the GPU's video memory in the form of a luminance / chrominance plane corresponding to the encoding format. To reduce host-side involvement and avoid additional data transfer, this embodiment does not perform any format conversion in the host memory. Instead, the GPU's compute shader processes the decoded video frame data in the video memory for color space conversion. Specifically, a target texture buffer area is allocated in the video memory for each video frame. The compute shader traverses each pixel position of the current frame in a thread group manner, reads the luminance and chrominance components from the source luminance and chrominance planes according to a predetermined sampling rule, calculates the three color components and one transparency component in the target color space based on the read luminance and chrominance components according to a preset color transformation matrix and offset relationship, and writes them into the target texture buffer area in a unified pixel format. By completing the color space conversion internally within the GPU, it is no longer necessary to output multiple pixel formats at the decoding end, nor is it necessary to repeatedly perform color conversion calculations on the Unreal Engine side, thereby shortening the processing time of the entire chain. After the conversion is complete, the converted frame data is written to the texture cache. The texture cache can be a 2D texture object provided by the graphics interface or an external texture resource, storing frame data that is already in the color space required by the Unreal Engine rendering pipeline. In subsequent rendering stages, the Unreal Engine rendering pipeline uses this texture cache as a material input texture and binds it to the post-processing material or scene material. Pixel colors are directly read from the texture cache for rendering output using regular texture sampling instructions, without needing to access the original decoding buffer or perform additional color conversions. This embodiment can be used in conjunction with the aforementioned motion vector-based dynamic effects control process. That is, the shared video texture resource is obtained from the aforementioned compute shader color space conversion, and the dynamic effects control buffer is generated by region-level aggregation of compressed domain motion vectors. Unreal Engine uses both the texture cache and the dynamic effects control buffer in a single post-processing pass, achieving integrated processing of color space conversion, zero-copy texture output, and motion vector-based dynamic effects rendering.

[0038] When a hardware video decoding unit completes decoding of a frame, it generates macroblock-level motion vectors and reference frame information during the motion compensation phase of the internal decoding process. The system configures a callback logic or dedicated transmission path for the hardware video decoding unit to synchronously write the motion vectors and related macroblock information generated during decoding into the motion vector description buffer. For example, in some implementations, the hardware video decoding unit first writes the raw motion vector data into an internally formatted intermediate buffer, and then uses a compute shader on the GPU to traverse the intermediate buffer, calculating the motion vector, reference frame index, and macroblock position of each macroblock into the aforementioned eight fields, and writing them into the motion vector description buffer in macroblock number order. This unifies the proprietary motion vector formats provided by different manufacturers' hardware video decoding units into a fixed description method, facilitating consistent use in upper-layer applications.

[0039] In another optional implementation, the hardware video decoding unit itself supports directly outputting macroblock-level motion information in a preset format. When initializing the hardware video decoding unit, the system passes the memory address of the motion vector description buffer and the record unit length to the hardware video decoding unit, which then writes the data block by block according to a predefined field order. This avoids the overhead of intermediate buffers and additional compute shaders, further reducing latency and memory bandwidth consumption.

[0040] The lifecycles of shared video texture resources and motion vector description buffers are designed for synchronous management. In the actual implementation, the system allocates a set of resources for each decoded frame, including a shared video texture resource and a corresponding motion vector description buffer. After a new frame is decoded, the system submits the texture handle and motion vector description buffer handle of the current frame to the rendering side through a circular queue or frame indexing. The resources of the previous frame are remarked as reusable after rendering is completed. This paired management method ensures that the rendering logic using a particular frame can directly access the motion vector description buffer that corresponds exactly to that frame at any time, achieving a one-to-one correspondence between pixel data and motion information and avoiding mixing between different frames.

[0041] In some scenarios, the macroblock size is not fixed at 16 pixels by 16 pixels. For example, in video coding formats that support variable block partitioning, an image may contain blocks of 8 pixels by 8 pixels or 32 pixels by 32 pixels. To accommodate these situations, one approach is to still use the largest block as the macroblock benchmark in the motion vector description buffer, for example, uniformly using 16 pixels by 16 pixels as the logical macroblock, treating smaller blocks as part of the logical macroblock, and adding a bit in the status flag field to indicate the actual block size. Another approach is to directly write the actual block size into the combination of the width and height pixel counts in the third field, and the compute shader adjusts the texture sampling range based on this block size when using motion information. Both approaches maintain the overall layout of the motion vector description buffer, facilitating the reuse of the same set of subsequent processing logic across different coding formats.

[0042] Through the above process, the input video stream is hardware-decoded on the GPU. The resulting pixel data resides in the GPU memory as a shared video texture resource, while macroblock-level motion information resides in the same memory space as a fixed-structure, consistent-order motion vector description buffer. The decoding stage does not copy pixel data or motion vector data to host memory or between different GPUs. This ensures that the basic data required for subsequent processing is generated and organized entirely within the GPU, providing a stable, low-latency data foundation for subsequent dynamic effects calculations and Unreal Engine rendering.

[0043] In a typical implementation, step two is completed on the graphics processing unit (GPU) via a compute shader program. Once the video data of the current frame has been decoded and the motion vector description buffer and shared video texture resources are ready in the GPU's video memory, the host processor submits a compute shader scheduling command to the GPU. The compute shader uses the number of macroblocks as the total number of threads. Each thread is responsible for processing the motion information of one macroblock and participates in the aggregation calculation of the rectangular local region in subsequent stages, ultimately generating a dynamic effects control buffer recording unit corresponding to each macroblock.

[0044] The computational shader first reads the record unit of the current macroblock from the motion vector description buffer. The first field of the record unit is the signed integer value of the horizontal motion vector component, and the second field is the signed integer value of the vertical motion vector component. The computational shader takes the absolute value of each of these components and adds them together to obtain the base motion intensity value. In a specific implementation, the base motion intensity value can be divided into three levels: 0 to 15 for the first level, 16 to 63 for the second level, and a value of at least 64 for the third level. This grading method allows slow, medium, and high-speed motion to receive significantly different processing intensities in subsequent dynamic effects. Weak motion areas only require light blur and ghosting to avoid excessive lag, while strong motion areas, if processed with a light intensity, will appear to tear or stutter. Increasing the motion intensity level can significantly enhance the performance of dynamic effects.

[0045] refer to Figure 2 , Figure 2 This is a schematic diagram of the register-level structure of the motion vector description buffer (MVDB) provided in an embodiment of the present invention. Figure 2 As shown, the buffer is allocated as a contiguous linear memory space in the graphics processing unit's video memory to store the raw motion information extracted from the video stream. Figure 2 The internal bit-field layout of a single recording unit is described in detail. Each rectangular bar represents a field within that recording unit, and the entire recording unit strictly adheres to a 32-byte (256-bit) alignment standard. This fixed-length 32-byte design is based on deep optimization of GPU memory access characteristics, ensuring merged access during concurrent reads by the compute shader and maximizing memory bandwidth utilization. Specifically, byte offset 0x00 of this recording unit stores the horizontal motion vector component (MV_X), in 32-bit signed integer format, representing the number of pixels the macroblock has shifted horizontally; positive values ​​represent rightward movement, and negative values ​​represent leftward movement. Following this, offset 0x04 stores the vertical motion vector component (MV_Y), also a 32-bit signed integer; positive values ​​represent downward movement, and negative values ​​represent upward movement. These two fields constitute the most basic motion description, directly derived from the motion compensation unit of the hardware decoder. Offset 0x08 is a composite field storing the macroblock's size information. To save space, this field allocates the high 16 bits of the 32-bit integer to the macroblock width and the low 16 bits to the macroblock height. For example, for a standard 16x16 macroblock, both the high and low bits of this field are set to 16. This compact packing method allows shaders to obtain both the width and height simultaneously with a single read and simple bitwise operations, reducing the number of memory access instructions.

[0046] Offset 0x0C stores the reference frame index, indicating which frame the current macroblock's motion vector is calculated relative to. This is particularly important for processing multi-reference frame encoded videos. Offsets 0x10 and 0x14 store the absolute coordinates Pos_X and Pos_Y of the macroblock's top-left corner within the entire video frame, respectively. Although these coordinates can be calculated using the macroblock number, pre-calculating and storing them here saves the overhead of repeated calculations in each subsequent pixel coloring thread, reflecting a "space-for-time" optimization strategy. Offset 0x18 contains the channel identifier ID, used to distinguish which video stream the current macroblock belongs to in scenarios involving parallel decoding of multiple video streams. Finally, offset 0x1C contains a 32-bit set of status flags. Figure 2 The text provides a detailed explanation of this field, with each bit representing a specific Boolean state. For example, Bit 0 might indicate "whether it is an edge macroblock," Bit 1 "whether the motion vector is valid," and Bit 3 "whether it is a fast-moving region," etc. This bit-field design allows the system to convey rich contextual information without increasing memory overhead, providing fine-grained control for subsequent adaptive effects decisions. The entire MVDB structure is not only a data container for the decoder's output but also a core data bus connecting the underlying decoding hardware with the upper-layer rendering algorithm.

[0047] While obtaining the motion intensity level, the computation shader also maps the macroblock's motion direction to discrete direction indices based on the signs and absolute values ​​of the horizontal and vertical motion vector components. A feasible approach is to divide the direction indices into eight values, representing the horizontal, vertical, and four diagonal directions. For example, when the absolute value of the horizontal direction is greater than or equal to the absolute value of the vertical direction and the horizontal component is positive, the direction index is set to the first direction index, corresponding to the right direction; when the absolute value of the horizontal direction is greater than or equal to the absolute value of the vertical direction and the horizontal component is negative, the direction index is set to the second direction index, corresponding to the left direction; when the absolute value of the vertical direction is greater than the absolute value of the horizontal direction and the vertical component is positive, the direction index is set to the third direction index, corresponding to the bottom direction; and when the absolute value of the vertical direction is greater than the absolute value of the horizontal direction and the vertical component is negative, the direction index is set to the fourth direction index, corresponding to the top direction. When the absolute values ​​of the horizontal and vertical directions are close, the motion direction is assigned to the four diagonal directions (bottom right, top right, bottom left, and top left) based on their sign combination, corresponding to the fifth to eighth direction indices. This discretization process can map any direction to a finite number of direction indices while maintaining the approximate accuracy of the motion direction. This makes it easier to use integer values ​​to control the sampling path in the shader, avoids complex trigonometric function calculations for each pixel, reduces computational load, and improves data consistency.

[0048] refer toFigure 1 , Figure 1 This is a schematic diagram illustrating the principle of discretized polar coordinates for motion vector intensity and direction provided in an embodiment of the present invention. Figure 1 As shown in the diagram, this schematic illustrates how the system maps high-precision floating-point or fixed-point motion vectors acquired during the decoding stage into discretized state indices that facilitate subsequent rendering processing. Throughout the entire end-to-end zero-copy video decoding and rendering process, to reduce the computational load on the graphics processing unit during the pixel shading stage and ensure the visual consistency of dynamic effects, the system does not directly use the original motion vector values ​​for real-time calculation per pixel. Instead, it pre-quantizes and grades the motion vectors in polar coordinates during the computation shader stage. Figure 1 The left side displays a two-dimensional polar coordinate plane, with the origin (0,0) representing a macroblock in a completely stationary state. Radial outwards from the origin, the system defines several concentric rings as threshold boundaries for intensity grading. The innermost circular region represents a stationary or slightly moving region, where the motion vector magnitude is less than a first preset threshold (e.g., 1.5 pixels). Macroblocks falling into this region are marked as intensity level 0, and typically, motion blur or other effects are not applied to maintain image sharpness. The first outward-extending ring region represents a weak motion region, corresponding to intensity level I, with its motion vector magnitude between the first and second thresholds (e.g., 4.5 pixels); the second ring region represents a medium motion region, corresponding to intensity level II, with the magnitude between the second and third thresholds (e.g., 7.5 pixels); the outermost region represents a strong motion region, corresponding to intensity level III, indicating that the object is moving at high speed. Simultaneous with intensity grading is the discretization of direction. Figure 1 As shown, the polar coordinate plane is uniformly divided into eight sectors, each covering an angle range of 45 degrees, corresponding to eight discrete direction indices (Idx1 to Idx8). Specifically, the horizontal rightward direction corresponds to index 1, the vertical upward direction corresponds to index 4, the vertical downward direction corresponds to index 3, the horizontal leftward direction corresponds to index 2, and the four diagonal directions correspond to indices 5, 6, 7, and 8 respectively. This eight-directional division covers the main motion trends in screen space. Figure 1 The example demonstrates a motion vector (example MV) whose endpoint coordinates fall within the lower right sector and are located within the first annular region. According to the mapping logic of this invention, this vector is quantized as a combination of "intensity I" and "direction index Idx6 (lower right)". This means that in subsequent effects rendering, regardless of the actual minute fluctuations of the vector, the system will uniformly call the lower right direction, low-intensity blur sampling parameters, thereby avoiding effect flickering caused by vector noise.

[0049] Relying solely on the motion information of a single macroblock to generate dynamic effects control parameters can lead to significant differences in processing intensity and direction between adjacent macroblocks, easily resulting in blocky boundaries on the screen. To address this, the concept of rectangular local regions is introduced in the implementation, hierarchically aggregating the motion information of multiple adjacent macroblocks. The computation shader divides the current video frame into several rectangular local regions based on the horizontal and vertical coordinates of the macroblock's top-left corner pixel, provided by the 5th and 6th fields in the motion vector description buffer. Each rectangular local region covers a fixed number of consecutive macroblocks. For example, given an image width of 1920 pixels, a height of 1080 pixels, and a macroblock size of 16 pixels by 16 pixels, each rectangular local region can be defined to contain 8 macroblock widths and 8 macroblock heights, meaning one rectangular local region covers a 128-pixel by 128-pixel image area. This size ensures that the motion features within the region have statistical significance without being too large and neglecting local motion details.

[0050] To obtain the region motion intensity level and the region dominant direction index, the computation shader counts the macroblocks within each rectangular local region. For each region, the computation shader traverses all macroblock recording units covered within the region, counting the number of macroblocks at the first, second, and third motion intensity levels, and simultaneously counting the number of macroblocks corresponding to the eight direction indices. Then, the motion intensity level with the most occurrences is selected as the region motion intensity level, and the direction index with the most occurrences is selected as the region dominant direction index. If there are cases with the same number of macroblocks, they can be selected according to a preset priority order, for example, prioritizing the third motion intensity level before selecting the second motion intensity level; the direction indices are similarly set up with a priority table. Through this majority-principle approach, the overall motion state of each rectangular local region is summarized by a region motion intensity level and a region dominant direction index. The majority principle ensures that a small number of abnormal macroblocks will not have a decisive impact on the effect control of the entire region, thereby reducing flickering caused by noise, coding errors, or single-pixel anomalies and enhancing the stability of the visual effect.

[0051] Region-level motion description values ​​can be stored in a separate region-level intermediate buffer. The length of the region-level intermediate buffer is equal to the number of rectangular local regions, with each region corresponding to a record unit. Each record unit includes two fields stored as 4-byte integers: the first field stores the region's motion intensity level, and the second field stores the region's dominant direction index. In one implementation, the region-level intermediate buffer is generated by the first compute shader scheduler, and the second compute shader scheduler uses the region-level intermediate buffer and the motion vector description buffer together to generate the dynamic effects control buffer. In another implementation, the two-stage computation is completed through a single compute shader scheduler. While calculating the motion features of a single block, the thread participates in the statistics of its local region through shared memory or atomic operations, and writes back the region-level motion description values ​​after the statistics are completed.

[0052] After completing the region-level statistics, the system creates a dynamic effects control buffer in the graphics processing unit's video memory. The dynamic effects control buffer corresponds one-to-one with the motion vector description buffer, and the number of record units equals the number of macroblocks. Each record unit is fixed at 32 bytes in length and contains eight fields stored as 4-byte integers. The first field stores the dynamic effects type mask value, using different binary bits to indicate whether adaptive motion blur, object trailing, velocity line, and inter-frame interpolation reprojection effects are enabled. For example, the first bit indicates whether adaptive motion blur is enabled, the second bit indicates whether object trailing is enabled, the third bit indicates whether velocity line is enabled, and the fourth bit indicates whether inter-frame interpolation reprojection is enabled. The remaining bits are reserved for expanding other effects controls. When setting these bits, the compute shader can refer to the flag information in the status flag set field of the motion vector description buffer. For example, when a macroblock is marked as a fast-moving region in the status flag set, adaptive motion blur, object trailing, and velocity line effects can be enabled simultaneously; when a macroblock is marked as a static edge region, only the inter-frame interpolation reprojection effect can be enabled.

[0053] The second and third fields store the signed integer values ​​of the horizontal and vertical components of the motion blur sampling step size, respectively. These two components jointly determine the direction and spacing for multi-point sampling during subsequent upsampling of the shared video texture resource. In a specific mapping method, when the region motion intensity level is the first motion intensity level, the pixel length of both the horizontal and vertical components is set to 1 pixel; when the region motion intensity level is the second motion intensity level, the pixel length of both the horizontal and vertical components is set to 2 pixels; and when the region motion intensity level is the third motion intensity level, the pixel length of both the horizontal and vertical components is set to 3 pixels. The step size direction is determined based on the region's dominant direction index. For example, when the region's dominant direction index corresponds to the lower right direction, the horizontal component is set to a positive value, and the vertical component is also set to a positive value. The intuitive effect of this mapping method is that the more intense the motion, the larger the sampling step size, and the longer the blur stretch, which can create a noticeable drag effect in high-speed motion regions, while the blur range in slow motion regions is shorter, helping to preserve details.

[0054] The fourth field stores an unsigned integer value for the number of motion blur sampling points. The number of sampling points determines how many points are selected for color composition along the sampling path, with the step size as the interval. In one example, the first motion intensity level corresponds to 3 sampling points, the second motion intensity level corresponds to 5 sampling points, and the third motion intensity level corresponds to 7 sampling points. The number of sampling points is used in conjunction with the step size so that more sampling points are accumulated in space in high motion intensity areas, forming a wider blur band, while only a few samplings are performed in low motion intensity areas to avoid over-blurring. By discretizing the sampling step size and the number of sampling points to fixed integer values, the control logic in the shader can be significantly simplified, and the blur intensity of different macroblocks presents a stepped change, which is more visually acceptable to the audience.

[0055] The fifth field stores a combination of the number of times the object's motion blur is overlaid and the number of pixels at the motion blur interval. In one specific implementation, the lower 16 bits can represent the number of times the motion blur is overlaid, and the higher 16 bits can represent the number of pixels at the motion blur interval. When the region's motion intensity level is the first level, the number of times the motion blur is overlaid is set to 3, and the number of pixels at the motion blur interval is set to the same as the pixel length of the motion blur sampling step. When the region's motion intensity level is the second level, the number of times the motion blur is overlaid is set to 5; and when the region's motion intensity level is the third level, the number of times the motion blur is overlaid is set to 7. This design allows for more layers of motion blur as the motion is more intense, enabling viewers to more clearly perceive the object's trajectory, making it particularly suitable for action games or sports scenarios. Furthermore, keeping the number of pixels at the motion blur interval consistent with the motion blur sampling step ensures that the motion blur layers are aligned with the main blur direction, reducing ghosting and misalignment.

[0056] The sixth field stores a combined value of the number of velocity lines and the number of pixels for the velocity line length. Similarly, the number of velocity lines can be stored in the lower byte, and the number of pixels for the velocity line length can be stored in the higher byte. In one implementation, when the region's motion intensity level is the second or third motion intensity level, and the region's dominant direction index is a diagonal direction index, the number of velocity lines is set to 4 or 6, and the number of pixels for the velocity line length is set to be equal to the number of pixels for the macroblock width or twice the number of pixels for the macroblock width. The reason for favoring diagonal velocity lines is that the human eye is often less sensitive to diagonal motion than to horizontal and vertical motion. Strengthening the diagonal motion trajectory with velocity lines can compensate for this perceptual weakness. In the horizontal or vertical direction, the speed line effect can be enabled or disabled depending on the specific application requirements. For example, in scenes with a lot of interface element movement, the image can be kept clean by using only motion blur and trailing shadows.

[0057] The seventh field stores the combined value of the horizontal and vertical components of the inter-frame interpolation reprojection offset. The signs and directions of these two components are consistent with the motion vector components, and the pixel length can be set to the same as the motion blur sampling step size. This ensures that during inter-frame interpolation reprojection, the reprojection sampling position is offset by the same distance along the motion direction. This effectively extracts the content corresponding to the current macroblock position from the reference frame based on the macroblock's motion direction, thereby generating compensated images without using historical textures and complex interpolation models, mitigating detail loss caused by rapid motion.

[0058] The eighth field stores the reference sampling mode identifier, which indicates the sampling method on the reference frame texture resource during inter-frame interpolation reprojection effects. For example, different integer values ​​can be used to represent sampling using only the current reference frame, sampling using two different reference frames simultaneously, or using special filtering sampling for edge regions. Through the reference sampling mode identifier, subsequent shaders can select different sampling strategies based on the macroblock's motion characteristics and state flag set, improving reprojection quality while keeping computational costs under control.

[0059] In some optional implementations, parameters such as the threshold for the base motion intensity value, the size of the rectangular local region, the number of velocity lines, and their length can be adjusted according to the application scenario. For example, in high frame rate, low resolution video scenarios, the threshold for the base motion intensity value can be appropriately reduced, allowing more regions to be divided into second and third motion intensity levels to enhance dynamic performance. In high resolution, high detail video playback scenarios, the number of macroblocks covered by the rectangular local region can be reduced to improve the accuracy of region division and avoid using the same effect configuration for large areas, thus obscuring local details. Regardless of the specific values ​​used, the structure of the motion vector description buffer, the region-level intermediate buffer, and the dynamic effect control buffer remains consistent, and the upper-level rendering logic can be adapted to different application scenarios without modification.

[0060] Through the aforementioned hierarchical aggregation and dynamic effects mapping process of compressed domain motion vector regions, each macroblock in a video frame corresponds to a dynamic effects control buffer recording unit. The dynamic effects type information, adaptive motion blur effect control parameters, object trailing effect control parameters, velocity line effect control parameters, and inter-frame interpolation reprojection effect control parameters contained in the recording unit are all calculated directly by the computation shader within the graphics processing unit based on the motion vector description buffer and the region-level intermediate buffer; the data path does not pass through host memory. From a macroscopic perspective, this step converts the compressed domain motion information generated in the decoding stage into effect control data that can be directly used in the rendering stage, establishing a correlation between compressed domain motion vectors and dynamic effects performance. This allows subsequent Unreal Engine post-processing materials to apply visual effects matching the motion state to each macroblock and even each pixel without adding extra data copies.

[0061] In a typical implementation, after completing geometry rendering and basic color compositing, Unreal Engine's rendering pipeline triggers a full-screen post-processing pass. This post-processing pass binds a shared video texture resource as the input texture for the pixel shader, and simultaneously binds the dynamic effects control buffer to a structured buffer that the pixel shader can randomly access. In this way, each screen pixel, during post-processing shading, can synchronously access the video pixel color of the current frame and the dynamic effects control parameters corresponding to its macroblock, thus enabling the overlay of multiple dynamic effects on the same frame without increasing data copying.

[0062] refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the structure and mask definition of the Dynamic Effects Control Buffer (DECB) provided in an embodiment of the present invention. Figure 3As shown, DECB is the output result after computation shader processing, and it is directly used as input data bound to the post-processing material in Unreal Engine. Similar to MVDB, each recording unit of DECB also strictly occupies 32 bytes of video memory space and corresponds one-to-one with a macroblock in the video frame. Figure 3 This shows the eight core fields of the buffer and their layout in memory; these fields directly control the visual presentation of the final image. The first field at the starting offset is the effect type mask. This is a 32-bit bitmask field. Figure 3 The bottom left corner provides a detailed breakdown. Different bits act as "master switches" for different effects. For example, bit 0 controls the on / off state of adaptive motion blur, bit 1 controls the object trailing effect, bit 2 controls the velocity line effect, and bit 3 controls the inter-frame interpolation reprojection effect. The computation shader dynamically sets these flags based on the macroblock's motion intensity and region statistics. For instance, in static regions, all effect flags may be set to zero, allowing the pixel shader to skip cumbersome sampling calculations and significantly saving GPU computing power; while in high-speed motion regions, relevant flags are set to one, triggering the corresponding rendering logic. The next two fields are the horizontal blur step size and the vertical blur step size, respectively. These two fields are generated from the original motion vector after denoising and discretization mapping, defining the offset increment of texture coordinates during motion blur sampling. Following this is the number of sampling points field, which determines the smoothness of the blur. The following three fields employ a high-low bit compression storage strategy to accommodate more parameters: offset 0x10 stores the motion blur effect parameters, with the high 16 bits representing the number of motion blur layers and the low 16 bits representing the motion blur interval; offset 0x14 stores the velocity line parameters, including the number and length of lines; and offset 0x18 stores the reprojection offset vector, used for inter-frame detail restoration. The last field is the sampling mode enumeration value, indicating which texture filtering strategy the shader should use.

[0063] When rendering a frame, Unreal Engine generates a render target based on the target output resolution. For example, if the output resolution is 1920 pixels by 1080 pixels, the post-processing material shading process will execute on each of these 1920 by 1080 screen pixels one by one. For the current screen pixel, the shading process first calculates the horizontal and vertical pixel coordinates in the shared video texture resource based on screen space coordinates. If the video content completely fills the screen proportionally, the screen coordinates can be directly mapped to the video pixel coordinates linearly. If there are black areas at the top, bottom, left, or right, an offset is subtracted before mapping, and the image is scaled according to the width and height of the actual display area of ​​the video content. After mapping, the current screen pixel can accurately find the corresponding source video pixel position.

[0064] After obtaining the video pixel coordinates, the shader program calculates the macroblock number of the current pixel based on the macroblock size. For example, with a macroblock size of 16 pixels by 16 pixels and an image width of 1920 pixels, there are 120 macroblocks horizontally and 68 macroblocks vertically. The shader program divides the horizontal pixel coordinates by 16 and takes the integer part to obtain the horizontal index of the macroblock; it also divides the vertical pixel coordinates by 16 and takes the integer part to obtain the vertical index of the macroblock. Then, it multiplies the vertical index of the macroblock by the number of macroblocks per row (120), adds the horizontal index of the macroblock, and finally adds 1 to obtain the macroblock number. The macroblock number minus 1 is used as the index of the motion effects control buffer array. This index allows direct access to the record unit corresponding to the current pixel's macroblock. Because the macroblock partitioning uses the same numbering method as the motion vector description buffer and the motion effects control buffer, the entire lookup process involves only a small number of division and multiplication operations, resulting in high efficiency.

[0065] After reading the record unit of the motion effects control buffer, the shader first parses the motion effects type mask value of the first field. Different bits of the motion effects type mask value indicate whether adaptive motion blur, object trailing, velocity line effects, and inter-frame interpolation reprojection effects are enabled. The shader determines whether to perform the corresponding effect calculation for the current pixel by performing a bitwise AND operation to check if each bit is 1. The advantage of this approach is that for large static areas or areas where effects are unnecessary, related calculations can be completely skipped, reducing unnecessary workload, improving rendering performance, and preserving image detail.

[0066] The implementation of adaptive motion blur relies on the second, third, and fourth fields in the dynamic effect control buffer recording unit. The second field stores the signed integer value of the horizontal component of the motion blur sampling step size, the third field stores the signed integer value of the vertical component of the motion blur sampling step size, and the fourth field stores the number of motion blur sampling points. When the dynamic effect type mask value indicates that adaptive motion blur needs to be enabled for the current macroblock, the pixel coloring program determines the blur sampling direction based on the horizontal and vertical components of the motion blur sampling step size, starting from the current video pixel coordinates. For example, when the horizontal component of the motion blur sampling step size is 3 and the vertical component is 0, the blur direction is to the right horizontally, with a step size of 3 pixels; when the horizontal component is 2 and the vertical component is 2, the blur direction is to the lower right, with a step size of 2 pixels in both the horizontal and vertical directions.

[0067] The shader generates several sampling positions sequentially along this direction, with the number of samples matching the number of motion blur sampling points. For example, if the number of motion blur sampling points is 5, the shader will select 5 positions, including the current pixel, for sampling. Each sample reads the pixel color at the corresponding position from the shared video texture resource, accumulates these color values ​​one by one on the color channel, divides the accumulated result by the number of sampling points to obtain the average color, and then limits the average color to a valid color range. By extending the sampling positions along the motion direction and controlling the blur range according to the motion blur sampling step size and the number of sampling points, adaptive motion blur produces a longer blur trail in high-speed motion areas and only a shorter, softer blur in low-speed motion areas. The human eye sees a continuous stretching effect along the motion trajectory, which reduces high-frequency flicker and enhances the sense of speed.

[0068] The object motion blur effect is primarily based on the fifth field in the dynamic effect control buffer recording unit. This fifth field contains the number of object motion blur stackings and the number of pixels between the stackings. When the dynamic effect type mask value indicates that the object motion blur effect is enabled, the pixel shader, after completing adaptive motion blur, generates multiple motion blur sampling positions based on the number of pixels between the stackings and the dominant motion direction of the region (this direction is implicitly contained in the signs of the horizontal and vertical components of the motion blur sampling step). For example, if the number of stackings is 5 and the number of pixels between the stackings is the same as the motion blur step size in pixels, the pixel shader will shift the current pixel along the motion direction by 1, 2, 3, 4, and 5 times the number of pixels between the stackings, resulting in 5 motion blur positions. At each motion blur position, the pixel color is read from the shared video texture resource and superimposed on the pre-calculated motion blur result color. The superimposed color is then limited to the range allowed by the color channel. Because these motion blur samples are located behind the motion trajectory, they are superimposed to form a series of afterimages distributed along the direction of motion. Viewers will naturally perceive these afterimages as the trajectory left by a fast-moving object, thus enhancing the sense of motion, without the need to save historical frame data from multiple time points.

[0069] The velocity line effect utilizes the sixth field in the dynamic effect control buffer recording unit, namely the number of velocity lines and the number of pixels for the velocity line length. When the corresponding bit of the dynamic effect type mask value indicates that the velocity line effect needs to be enabled, the pixel shader first determines the number of velocity lines distributed along the motion direction near the current pixel based on the number of velocity lines. For example, when the number of velocity lines is 4, two velocity lines can be placed on each side of the center line in the motion direction. The number of pixels for the velocity line length determines the maximum length of each velocity line extending along the motion direction. For example, when the number of pixels for the velocity line length equals the number of pixels for the macroblock width, one velocity line covers the current macroblock width.

[0070] In the actual calculation, the pixel shader defines a sampling path for each velocity line. This path extends from the current pixel in the same direction as the motion blur, with several sampling points. The spacing between the sampling points can be the same as the motion blur sampling step size, or slightly smaller to achieve a more continuous line feel. At each sampling point, the shader reads the pixel color from the shared video texture resource and superimposes an enhanced luminance component onto the output color of the current pixel, for example, enhancing only the luminance component while keeping the color hue unchanged. Because the sampling path of the velocity lines is highly aligned with the direction of motion and its length is usually greater than the macroblock size, the human eye perceives these bright bands stretched along the direction of motion as airflow or light trails around a high-speed moving object, thus creating a strong sense of dynamism in the image. The velocity lines are arranged along the direction of motion rather than in random directions because the human eye is extremely sensitive to bright lines extending along the direction of motion; this visual cue can significantly enhance the sense of speed without introducing additional geometry.

[0071] The implementation of the inter-frame interpolation reprojection effect relies on fields 7 and 8 in the dynamic effect control buffer recording unit. Field 7 contains the horizontal and vertical components of the inter-frame interpolation reprojection offset, and field 8 is the reference sampling mode identifier. When the dynamic effect type mask value indicates that the inter-frame interpolation reprojection effect is enabled, the pixel shader calculates the reprojection sampling position based on the current video pixel coordinates, according to the direction and pixel length specified by the horizontal and vertical components of the inter-frame interpolation reprojection offset. For example, when the horizontal offset is 2 and the vertical offset is 1, the reprojection sampling position is offset 2 pixels to the right and 1 pixel down relative to the current pixel. Subsequently, the pixel shader calculates the corresponding reference sampling position from one or more reference frame texture resources based on the reference sampling mode identifier. The reference sampling mode identifier can specify using the most recent reference frame, or using two reference frames from different time points simultaneously, or employing a special interpolation strategy in edge regions.

[0072] After determining the reprojection sampling position and the reference sampling position, the pixel shading program reads the pixel color at the corresponding position from the shared video texture resource of the current frame and the texture resource of the reference frame, respectively. These colors are then superimposed or averaged to form the auxiliary pixel color used for inter-frame interpolation reprojection. This auxiliary pixel color is further synthesized with the pixel color that has already been superimposed with adaptive motion blur effect, object trailing effect, and velocity line effect to obtain the final output pixel color. By sampling from the reference frame at the position corresponding to the motion direction, the inter-frame interpolation reprojection effect can fill in the details lost due to insufficient sampling in fast-moving areas, such as tearing of object edges or local blurring. Since the offset direction is consistent with the motion vector direction, the texture content in the reference frame is closer to the real position of the object after motion in the current frame, and the synthesized image looks coherent and natural.

[0073] Throughout the post-processing material execution, shared video texture resources, dynamic effects control buffers, and potentially used reference frame texture resources all reside in the graphics processing unit's video memory. The pixel shader accesses this data using both texture sampling and buffer reading, eliminating the need to transfer any intermediate results back to host memory during the rendering phase or to transfer data between different graphics processing units. Once all screen pixels in a frame have completed the calculations for the aforementioned adaptive motion blur, object trailing, velocity line, and inter-frame interpolation reprojection effects, the rendering target stores the video image with superimposed dynamic effects, which can be directly output to the display device or used as input for subsequent rendering stages.

[0074] In some optional implementations, Unreal Engine's post-processing materials can selectively apply certain effects as needed. For example, for devices with lower performance, only adaptive motion blur and simplified object trailing effects can be enabled, while speed line effects and inter-frame interpolation reprojection effects can be turned off, thereby reducing the workload of the computation shader. Configuration can also be tailored to the scene type. For instance, in real-time shooting scenes primarily featuring human performances, inter-frame interpolation reprojection effects are emphasized to maintain the clarity of character outlines, while in racing or flight action scenes, speed line effects and high-intensity motion blur effects are emphasized to highlight the sense of speed. Regardless of the configuration, post-processing materials always obtain the effect control parameters for each macroblock through the dynamic effects control buffer, completing the effect overlay within a unified rendering flow, achieving end-to-end zero-copy processing from the decoded domain motion vector to the display's dynamic effects.

[0075] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video decoding and Unreal Engine rendering method based on GPU end-to-end zero-copy, characterized in that, The method includes the following steps: Step 1: Hardware decoding of the input video stream is performed on the GPU, shared video texture resources are generated in the GPU memory, and a motion vector description buffer is constructed to store compressed domain motion vector information in macroblock order. Step two: On the GPU, the motion vector description buffer is read through the computation shader, and the compressed domain motion vector region hierarchical aggregation and dynamic effect mapping algorithm is executed to generate dynamic effect control buffers corresponding one-to-one with each macroblock. The compressed domain motion vector region hierarchical aggregation and dynamic effect mapping algorithm includes: the computation shader performs a single-block motion feature calculation process for each recording unit in the motion vector description buffer, including: reading the signed integer representations of the horizontal motion vector component and the vertical motion vector component from the current recording unit, obtaining the absolute values ​​of the two components respectively, and adding them to obtain the basic motion intensity value, and then setting the motion intensity base value... The base values ​​are divided into three motion intensity levels: a first motion intensity level, a second motion intensity level, and a third motion intensity level. Based on the signs and absolute values ​​of the two components, the motion direction of the macroblock is divided into multiple discrete direction indices, including horizontal, vertical, and diagonal directions. The calculation shader, based on the horizontal and vertical pixel coordinates of the top-left pixel of each recording unit in the motion vector description buffer within the shared video texture resource, divides the current video frame into several rectangular local regions. Each rectangular local region covers a preset number of consecutive macroblocks. The shader then sequentially traverses the macroblock recording units covered by each rectangular local region, counting the first... The number of macroblocks corresponding to the motion intensity level, the second motion intensity level, and the third motion intensity level, as well as the number of macroblocks corresponding to each discrete direction index, are used to select the motion intensity level with the most occurrences as the regional motion intensity level of the rectangular local region. The discrete direction index with the most occurrences is selected as the regional dominant direction index of the rectangular local region. The regional motion intensity level and the regional dominant direction index are combined into a regional-level motion description value and written to the regional-level intermediate buffer. A dynamic effects control buffer, corresponding one-to-one with the motion vector description buffer, is created in the GPU memory. Each record unit in the dynamic effects control buffer includes eight words stored in four-byte integer form. The segment is used to store, in sequence, the dynamic effect type mask value, the signed integer value of the horizontal component of the motion blur sampling step size, the signed integer value of the vertical component of the motion blur sampling step size, the unsigned integer value of the number of motion blur sampling points, the combination value of the number of object shadow superpositions and the number of pixels of shadow interval, the combination value of the number of velocity lines and the number of pixels of velocity line length, the combination value of the horizontal component of the inter-frame interpolation reprojection offset and the vertical component of the inter-frame interpolation reprojection offset, and the reference sampling mode identifier. Different bits of the dynamic effect type mask value are used to indicate whether adaptive motion blur effect, object shadow effect, velocity line effect and inter-frame interpolation reprojection effect are enabled. Step 3: In the post-processing material of the Unreal Engine rendering module, the shared video texture resource is used as the input texture, the dynamic effect control buffer is used as the structured buffer input, and adaptive motion blur, object trailing, speed line effect and inter-frame interpolation reprojection effect are applied to the screen pixels, and the video image is output.

2. The method according to claim 1, characterized in that, In step one, a linearly contiguous motion vector description buffer is created in the GPU memory. Macroblocks in the current video frame are numbered from left to right and from top to bottom, with the numbers increasing sequentially from the beginning. The number of each macroblock is decremented by one to serve as the element index of the motion vector description buffer. The motion vector description buffer is divided into several record units. Each record unit includes eight fields stored in the form of four-byte integers. These fields sequentially store the signed integer representation of the horizontal motion vector component of the corresponding macroblock, the signed integer representation of the vertical motion vector component, the combined value of the width and height pixels, the reference frame index, the horizontal pixel coordinates of the top-left pixel of the macroblock in the shared video texture resource, the vertical pixel coordinates of the top-left pixel of the macroblock in the shared video texture resource, the video channel identifier, and a set of status flags for dynamic effects control.

3. The method according to claim 1, characterized in that, Multiple discrete direction indices include a first direction index, a second direction index, a third direction index, a fourth direction index, a fifth direction index, a sixth direction index, a seventh direction index, and an eighth direction index, which are used to represent the motion directions toward the right, the left, the bottom, the top, and the four diagonal directions, respectively.

4. The method according to claim 1, characterized in that, When the computational shader writes fields to the dynamic effects control buffer, it sets the parameters based on the region motion intensity level and the region dominant direction index of the rectangular local area to which each macroblock belongs. Specifically, when the region motion intensity level is the first motion intensity level, the pixel length of both the horizontal and vertical components of the motion blur sampling step is set to one pixel, and the number of motion blur sampling points is set to three. When the region motion intensity level is the second motion intensity level, the pixel length of both the horizontal and vertical components of the motion blur sampling step is set to two pixels, and the number of motion blur sampling points is set to five. When the region motion intensity level is the third motion intensity level... The pixel lengths of both the horizontal and vertical components of the motion blur sampling step are set to three pixels, and the number of motion blur sampling points is set to seven. When the dynamic effect type mask value indicates that the object trailing effect is enabled, the number of object trailing superpositions is set to three, five, and seven times respectively according to the region motion intensity level, and the number of trailing interval pixels is set to be the same as the corresponding motion blur sampling step pixel length. When the dynamic effect type mask value indicates that the velocity line effect is enabled and the region dominant direction index is a diagonal direction index, the number of velocity lines is set to four or six, and the number of velocity line length pixels is set to be the same as the number of macroblock width pixels or twice the number of macroblock width pixels.

5. The method according to claim 1, characterized in that, In steps two and three, the shared video texture resources, motion vector description buffer, and dynamic effects control buffer are all located in the GPU memory and are directly shared between the hardware video decoding unit and the Unreal Engine rendering module through GPU-side resource handles. During the entire method execution process, the shared video texture resources, motion vector description buffer, and dynamic effects control buffer are not copied from the GPU memory to the CPU memory, nor are they copied between different GPUs.

6. The method according to claim 5, characterized in that, In step three, the shared video texture resource is bound as the input texture of the post-processing material shader, and the dynamic effects control buffer is bound as the structured buffer input of the post-processing material shader. For each pixel on the screen, the post-processing material shader determines the number of the macroblock in which the pixel is located based on the horizontal and vertical pixel coordinates of the pixel in the shared video texture resource and accesses the corresponding record unit in the dynamic effects control buffer. When the dynamic effects type mask value indicates that the adaptive motion blur effect is enabled, multiple motion blur sampling positions are calculated on the shared video texture resource starting from the current pixel position along the direction determined by the horizontal and vertical components of the motion blur sampling step size. The pixel color is read at each motion blur sampling position and accumulated to form the adaptive motion blur effect. When the dynamic effects type mask value indicates that the object ghosting effect is enabled, multiple ghosting sampling positions are generated at the current pixel position along the direction corresponding to the region's dominant direction index based on the number of object ghosting superpositions and the number of ghosting interval pixels. The pixel color is read at each ghosting sampling position and superimposed with the pixel color of the adaptive motion blur effect to form the object ghosting effect.

Images