System and method for game-generated motion vectors

Abstract

Systems and methods for game generated motion vectors are disclosed. Systems and methods for integrated graphics rendering are also disclosed. In some embodiments, the systems and methods utilize a graphics engine, a video coding engine, and a remote client coding engine to render graphics over a network. The systems and methods involve generating a per-pixel motion vector that is converted to a per-block motion vector at a graphics engine. The graphics engine injects the block motion vectors into a video coding engine such that the video coding engine can convert the vectors into encoded video data for transmission to a remote client coding engine.

Description

[0001] This invention patent application is a divisional application of the invention patent application with international application number PCT / US2018 / 028544, international application date April 20, 2018, application number 201880040171.3 entering the Chinese national phase, and name “System and method for motion vector generation for games”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Application No. 62 / 488,526, filed on April 21, 2017, and U.S. Provisional Application No. 62 / 596,325, filed on December 8, 2017. Background Art

[0004] Remote gaming applications, where a server-side game is controlled by a client-side player, have attempted to encode the video output from a three-dimensional (3D) graphics engine in real time using existing or custom encoders. However, the interactive nature of video games, particularly the player feedback loop between video output and player input, makes game video streaming more sensitive to latency than traditional video streaming. Existing video encoding methods offer a reduction in encoding time at the expense of computational effort, while few alternatives exist. A new approach that integrates the encoding process into the video rendering process can significantly reduce encoding time while also reducing computational effort, improving the quality of the encoded video, and preserving the original bitstream data format to maintain interoperability with existing hardware devices.

[0005] Existing video coding standards only incorporate color and temporal information from image sequences to improve video encoding time, size, or quality. Some coding standards, such as those in the MPEG family of standards, use computationally intensive block-based motion estimation methods to approximate object motion based on the color data contained in the video. These block-based motion estimation methods have historically significantly reduced the size of the encoded video, but are a source of significant latency in real-time video streaming environments.

[0006] Integrating the encoding process into the video rendering process provides access to additional data sources that can be used to improve encoding. For example, some 3D graphics engines, such as those included in game engines, may already generate motion vectors that perfectly describe the motion of every pixel on each video frame. By providing the final rendered frame and injecting appropriately formatted motion vector data into the encoder, the most computationally complex and time-consuming step in the video encoder—motion estimation—can be skipped for each frame. Additionally, the motion vectors provided by the graphics engine are more accurate than those estimated by block-based motion estimation algorithms, which improves the quality of the encoded video.

[0007] Video encoding and real-time graphics rendering have traditionally operated separately and independently. By integrating the graphics engine and encoder to leverage their respective strengths, encoding time can be reduced enough to support highly latency-sensitive streaming applications.

[0008] These and other attendant advantages of the present invention will become apparent from the technical shortcomings described hereinafter.

[0009] For example, U.S. Patent Application Publication No. 2015 / 0228106A1 (“the '106 Publication”) discloses a technique for decoding video data to generate a sequence of decoded blocks of a video image. This technique allows each decoded block of the video image to be used as a separate texture for a corresponding polygon of a geometric surface as the decoded block is generated by a codec engine. The '106 Publication describes an integration between a codec engine and a 3D graphics engine, wherein the codec engine decodes the encoded video data to generate a video image to be mapped, and the 3D graphics engine partially renders a display image by performing texture mapping of the video image to the geometric surface. However, this technique is insufficient compared to the present invention, at least because it does not disclose or utilize a graphics engine that provides appropriately formatted motion vector data for injection into the video codec engine and the resulting rendered frame, so that the video codec engine does not need to perform any motion estimation before transmitting the encoded video data to a remote client encoding engine. In contrast, the present invention's improvements to computer technology reduce encoding time and computational complexity, improve the quality of the encoded video, and preserve the original bitstream data format to maintain interoperability.

[0010] U.S. Patent Application Publication No. 2011 / 0261885A1 (the "'885 Publication") discloses a system and method for reducing bandwidth by integrating motion estimation and macroblock encoding. In this system, motion estimation can be performed using acquired video data to generate motion estimation-related information, including motion vectors. These motion vectors can be mapped to the current macroblock using the corresponding video data cached in a buffer. As such, the '885 Publication technique is insufficient compared to the present invention, at least because: it does not disclose or utilize a graphics engine that provides appropriately formatted motion vector data for injection into a video codec engine and a final rendered frame, such that the video codec engine does not need to perform any motion estimation before transmitting the encoded video data to a remote client encoding engine. Therefore, the '885 Publication technique does not provide the reduction in encoding time and computational effort, as well as the improvement in encoded video quality, provided by the present invention.

[0011] From the above discussion of the prior art in the art, it is apparent that there is a need in the art to improve upon current computer technology related to video encoding in a gaming environment. Summary of the Invention

[0012] It is therefore an object of the exemplary embodiments disclosed herein to address the shortcomings of the art and to provide a system and method for graphics generation that transmits encoded video data using a networked server architecture running a graphics engine, a video codec engine, and a remote client encoding engine, wherein the graphics engine provides appropriately formatted motion vector data for injection into the video codec engine and the final rendered frame.

[0013] It is another object of the present invention to provide a system and method for graphics generation, wherein a video codec engine does not need to perform any motion estimation before transmitting the encoded video data to a remote client encoding engine.

[0014] Another object of the present invention is to provide a system and method for graphics generation, wherein a graphics engine converts per-pixel motion vectors into per-block motion vectors.

[0015] It is a further object of the present invention to provide a system and method for graphics generation, wherein per-pixel motion vectors are generated by adding the per-pixel motion vectors to the camera velocity using a compute shader to obtain a per-pixel result, and wherein the per-pixel result is stored in a motion vector buffer.

[0016] It is a further object of the present invention to provide a system and method for graphics generation, wherein each block of motion vector data is injected into a video encoding engine along with chroma subsampled video frames by a graphics engine in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be better understood, and a more complete appreciation of the invention and many of its attendant advantages will be readily obtained, by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0018] Figure 1 is a block diagram illustrating a 3D graphics engine rendering video for encoding and transmission to a client.

[0019] Figure 2 is shown by injecting the motion vectors generated by the 3D graphics engine into Figure 4 A flowchart of the steps required in the modified encoding process to reduce latency;

[0020] Figure 3 is a diagram illustrating the conversion of per-pixel motion vectors generated in a graphics engine to per-macroblock motion vectors for injection into an encoding engine;

[0021] Figure 4 It shows the Figure 1 Flowchart of the changes required for the video encoding process used in . DETAILED DESCRIPTION

[0022] In describing the preferred embodiments of the present invention illustrated in the accompanying drawings, specific terminology will be employed for the sake of clarity. However, the present invention is not intended to be limited to the specific terminology so selected, and it should be understood that each specific term includes all technical equivalents that operate in a similar manner to achieve a similar purpose. For illustrative purposes, several preferred embodiments of the present invention have been described, and it is understood that the present invention may be embodied in other forms not specifically shown in the accompanying drawings.

[0023] In applications where a 3D graphics engine is rendering video for real-time encoding and transmission, the graphics engine and encoder can be more tightly coupled to reduce overall computation time and overhead. For each video frame, the per-pixel motion vector data already generated by the graphics engine can be converted to per-block motion vector data and injected into the codec engine, bypassing the motion estimation step, which is the single most complex and computationally intensive step in the encoding process. In graphics engines using reconstruction filters in a realistic motion blur approach, per-pixel motion vectors can already be calculated for each video frame. The conversion from per-pixel motion vectors to per-block motion vectors can be performed by finding the average vector for each 16×16 pixel macroblock. This conversion is performed in the 3D graphics engine, so only a small portion of the original motion vector data needs to be transferred from the 3D graphics engine to the encoding engine. This also helps reduce memory bandwidth consumption in cases where the graphics and encoding engines do not share memory. Injecting per-block motion vectors into the codec engine completely skips the motion estimation step without significantly changing the rest of the encoding process.

[0024] Figures 1 to 4 Example techniques are shown for improving video encoding in video streaming applications where a 3D graphics engine generates accompanying motion vector data during the process of rendering video frames.

[0025] Figure 1An example system is shown in which video is rendered and encoded for transmission to a remote client 116. A 3D graphics engine 100, running in memory 106 on a server architecture 120, passes video and supplemental motion vector information about the rendered video frames to a codec engine (referred to herein as a codec or encoder) 102, which generates an encoded bitstream 108 for transmission to a client computer system 116. The server architecture 120 is any combination of hardware or software capable of supporting the functionality of a graphics engine and a codec engine. In the example given, the graphics engine 100 can be implemented as, for example, a GPU executing video game software 104 loaded into a computer-readable memory 106, while the codec engine 102 can be implemented as a CPU running video encoding software. The encoding engine 102 generates encoded video data 108 for transmission to a remote client computer system 116, which includes a remote encoding engine (codec) 110, which decodes the bitstream for playback on a display 114 driven by a display controller 112. The remote client computer system 116 is any combination of hardware, device, or software capable of decoding and displaying the encoded bitstream 108 .

[0026] Figure 2 The steps required to achieve faster encoding times by reusing existing supplemental data from the rendering process during video encoding are shown. In step 202, the supplemental data must first be generated as a normal operating function of the graphics engine 100 located at the server 120. As GPUs become more powerful and prevalent, real-time per-pixel motion vector generation has become a common feature of modern video game engines. During the rendering of 2D video frames from a 3D scene, the 3D graphics engine generates auxiliary outputs during the color generation process to be used as input to subsequent post-processing processes. The auxiliary outputs may include information written to three memory locations: accumulation, color, or velocity buffers, which are each allocated for temporarily storing information about pixel depth, pixel color, and pixel motion.

[0027] In a common implementation of motion blur, called a reconstruction filter for realistic motion blur, the per-pixel velocity from a velocity buffer is first downsampled into a smaller number of slices, where each slice is assumed to be the maximum velocity from the pixel group. The slices are then masked using the per-pixel depth in the accumulation buffer, and the result is applied to the per-pixel color in the color buffer to generate the motion blur. There are several variations of the reconstruction filter approach that improve fidelity, performance, or both, but the idea remains similar, and the velocity buffer contains the per-pixel motion between two adjacent frames. Although "velocity" is the term used in graphics engine terminology and "motion vector" is the term used in video encoding terminology, the terms are functionally equivalent, and per-pixel velocity and per-pixel motion vector are the same thing. The velocity buffer contains supplementary data in the form of pixel motion vectors that will be reused during the video encoding process.

[0028] In step 204, the graphics engine 100 at the server 120 converts the per-pixel motion vectors into per-block motion vectors based on the macroblock size to be used for encoding. The H.264 codec uses 16x16 pixel macroblocks by default and can optionally be further subdivided. The 256 per-pixel motion vectors can be averaged together to provide a single average vector as the per-block motion vector. Figure 3 This process is described in further detail.

[0029] In step 206, the per-macroblock motion vector information is injected into the encoding engine / encoder 102 located at the server 120, thereby circumventing the motion estimation step. In a software implementation of the encoder, the motion estimation step can be completely disabled, thereby significantly saving CPU computation time. The time saved in the CPU should be sufficient to offset the additional time required to calculate the mean vector in the GPU (in step 204) and transfer it to the CPU.

[0030] In step 208, encoding starts with a motion compensation step (step 208) since the per-block motion vectors provided by the graphics engine 100 are interchangeable with the motion vectors calculated in the typical motion estimation step. Figure 4 As described in further detail, the remainder of the video encoding process does not differ significantly from the typical motion compensation, residual calculation, and encoding steps performed by coding standards using motion estimation techniques.

[0031] Figure 3The conversion from per-pixel motion vectors to per-macroblock motion vectors that occurs in the graphics engine 100 is further detailed. During the color generation phase, the 3D graphics engine 100, located on the server 120, generates per-pixel motion vectors and stores this data in a velocity buffer 300, also located on the server 120. The velocity buffer 300 may only contain data for dynamic objects, excluding motion information imparted by the player's camera motion. To obtain motion vector information for each pixel in image space, the compute shader 302 combines the vectors in the velocity buffer 300 with the camera velocities of all static objects not already contained in the velocity buffer and stores the per-pixel result in a motion vector buffer 304. The camera velocity is a 2D projection of the rotational and translational camera motion during the frame. Specific graphics engines may use slightly different methods to calculate these per-pixel motion vectors for the entire screen space, but the concept remains the same.

[0032] The default macroblock size used by the H.264 encoder is 16×16, but it can be subdivided into smaller sizes, down to 4×4. Figure 3 In the example of , 4x4 macroblocks 306 are used as a simplifying case, but the method should be extrapolated to match the macroblock size used in the encoder. For a 4x4 macroblock 306, 16 per-pixel motion vectors 308 are stored in the motion vector buffer 304. These per-pixel motion vectors 308 need to be transformed 312 into a single per-macroblock motion vector 310 that can be injected into the encoder for use as shown in FIG. Figure 4 The motion compensation shown is shown. The arithmetic mean of a set of per-pixel vectors 308 is a transformation 312 method with low computational complexity and short computation time.

[0033] Optional modifications can be made to the arithmetic mean transform 312 to improve quality at the expense of additional computational complexity or computational effort. For example, a vector median filtering technique can be used to remove discontinuities in the macroblock vector domain prior to the arithmetic mean calculation to ensure that the per-macroblock motion vector 310 represents the majority of pixels in the macroblock 306. Because the final per-macroblock motion vector is derived from the pixel-perfect motion vector initially calculated based on known object motion data, the per-macroblock motion vector is always a more accurate representation than motion vectors calculated by existing block-based motion estimation algorithms, which can only derive motion based on pixel color data.

[0034] Figure 4 shows that by putting Figure 1 The motion vector generated by the graphics engine 100 of the server 120 is injected into Figure 1The resulting bitstream of the encoded video data 108 is transmitted to the remote client computer system 116 as described in detail below. Figure 4 The method shown illustrates the encoding process of a single inter frame, specifically a P frame as defined by the MPEG family of video codec standards. Intra (I frame) generation is not altered as motion compensation 406 is not performed in I frame generation. Once the chroma subsampled video frame 402 and per-block motion vector data 404 are available, they are transferred from the graphics engine 100. The game generated motion vectors 404 are used to circumvent the motion vector generation that would otherwise occur in a typical motion estimation 426 step, as described in the H.264 / MPEG-4 AVC standard. The motion estimation 426 step is skipped and can be disabled in a software implementation of the encoding engine. Skipping the block based motion estimation 426 step will significantly reduce encoding time, which will greatly offset the overhead of combining Figure 3 The time described for converting the velocity buffer data into the appropriate format.

[0035] The motion vectors 404, which have been converted to the appropriate macroblock size, can be used directly without any changes to the motion compensation 406. The results of the motion compensation 406 are combined with the input chroma subsampled video frame 402 to form a residual image 430, which is typically processed by residual transform & scaling 408, quantization 410, and scanning 412 steps, which typically occur within existing hardware or software video encoders.

[0036] If the decoding standard selected for the embodiment requires a deblocking step, then the deblocking step must be performed. Deblocking settings 420 and deblocked images 428 are calculated by applying the coding standard's algorithm to perform inverse quantization 414, inverse transform & scaling 416, and then deblocking 418. The scanned coefficients 412 are combined with the deblocking settings 420 and encoded in an entropy encoder 422, and then transmitted as a bitstream 108 to the remote client computer system 116 for decoding at the remote client computer system's codec 110. The deblocked images 428 become the input for motion compensation 406 for the next frame. The bitstream (including the encoded video data) 108 retains the same format as defined by the coding standard used in the embodiment, such as H.264 / MPEG-4 AVC. This example is specific to the H.264 / MPEG-4 AVC standard and can generally be used for similar coding standards that use motion estimation 426 and motion compensation 406 techniques.

[0037] Example 1: Benchmarking shows reduced encoding time

[0038] The motion estimation step in traditional H.264-compatible encoding is often the most computationally complex and time-consuming step. As discussed in this article, reusing motion vectors generated by the game can significantly reduce encoding time.

[0039] In the test environment, the graphics engine produces output at a resolution of 1280×720 at 60 frames per second. Encoding times are captured from the x264 encoder running a single thread. Running a single-threaded encoder will result in longer encoding times than actual usage, but the measurements are normalized to one core to make them directly comparable. Encoding times were first measured using unmodified motion estimation within the encoder, and then re-measured in the same environment with game-generated motion estimation enabled.

[0040] A low-motion area was selected, encompassing the first-person player view of the player's hands, weapon, and a fixed wall. The player's hands and weapon were cycled through a slight "wobble" animation, producing a small amount of pixel motion in a relatively small screen space. The results of this test are shown in Table 1 below, which shows latency results with and without the game-generated motion estimation technique described in this article. In the low-intensity scenario, where game-generated motion estimation was disabled, the unmodified encoding time was 12ms. When game-generated motion estimation was enabled, the encoding time was reduced by 3ms, to 9ms. Similar latency reductions were shown for the average and high-motion intensity scenarios, with a 17.6% latency reduction for the average scenario and a 15% to 30% latency reduction for the high-latency scenarios. These results demonstrate a significant latency reduction when game-generated motion estimation is enabled.

[0041]

[0042] Table 1: Latency results at different exercise intensities

[0043] The test environment also shows that there is an additional overhead when converting the per-pixel motion vectors generated by the game into per-macroblock motion vectors for the encoder. However, this overhead is significantly less than the encoding time reduction described in the previous section. Using the graphics engine to generate video at a resolution of 1280×720, the conversion from per-pixel to per-macroblock motion vectors takes 0.02ms. The measured encoder time savings are three orders of magnitude greater than the time cost of encoding using the motion vectors generated by the game.

[0044] The foregoing description and accompanying drawings should be understood as merely illustrative of the principles of the present invention. The present invention is not limited to the preferred embodiment and may be embodied in a variety of ways as will be apparent to those skilled in the art. Numerous applications of the present invention will readily occur to those skilled in the art. Therefore, it is not intended to limit the present invention to the specific examples disclosed or to the exact architecture and operation shown and described. All suitable modifications and equivalents may be employed within the scope of the present invention.

Claims

1. A computer-implemented method for encoding video in a system comprising a graphics engine and a video encoder, the method comprising: Using a graphics engine, converting the one or more motion vectors into one or more per-block motion vectors; transmitting the one or more per-block motion vectors to the video encoder; as well as Using the video encoder, encoding a video frame to generate encoded video data, wherein encoding the video frame comprises: circumventing motion estimation for one or more blocks of the video frame based on the one or more per-block motion vectors; using the one or more per-block motion vectors in motion compensation for the one or more blocks of the video frame; and The encoded video data is output in a bitstream.

2. The computer-implemented method of claim 1 , wherein: The one or more motion vectors being converted are one or more per-pixel motion vectors.

3. The computer-implemented method of claim 1 , further comprising: At the graphics engine, one or more motion vectors of a dynamic object are stored in a velocity buffer prior to performing the conversion.

4. The computer-implemented method of claim 3 , further comprising: Using a compute shader of the graphics engine, one or more motion vectors of dynamic objects stored in the velocity buffer are combined with camera velocity information to generate the one or more per-pixel motion vectors.

5. The computer-implemented method of claim 2, wherein: The converting includes performing a vector median filtering technique on the one or more per-pixel motion vectors.

6. The computer-implemented method of claim 2, further comprising: At the graphics engine, the one or more per-pixel motion vectors are calculated based on known object motion data.

7. The computer-implemented method of claim 2, wherein: The converting includes performing an arithmetic mean transform on the one or more per-pixel motion vectors.

8. The computer-implemented method of claim 1 , wherein: The one or more per-block motion vectors are compatible with the H.264 / MPEG-4 AVC standard.

9. A graphics generation system comprising a graphics engine and a video encoder, wherein: The graphics engine is configured to: converting the one or more motion vectors into one or more per-block motion vectors; and injecting the one or more per-block motion vectors into the video encoder; and The video encoder is configured to encode a video frame, thereby generating encoded video data, the video encoder being configured, as part of encoding the video frame: circumventing motion estimation for one or more blocks of the video frame based on the one or more per-block motion vectors; using the one or more per-block motion vectors in motion compensation for the one or more blocks of the video frame; as well as The encoded video data is output in a bitstream.

10. The system according to claim 9, wherein: The one or more motion vectors being converted are one or more per-pixel motion vectors.

11. The system according to claim 10, wherein: The graphics engine is further configured to store one or more motion vectors of the dynamic object in a velocity buffer.

12. The system according to claim 11, wherein The graphics engine is further configured to combine, using a compute shader, one or more motion vectors of dynamic objects stored in the velocity buffer with camera velocity information to generate the one or more per-pixel motion vectors.

13. The system according to claim 10, wherein: The graphics engine is configured to perform a vector median filtering technique on the one or more per-pixel motion vectors.

14. The system according to claim 10, wherein: The graphics engine is further configured to calculate the one or more per-pixel motion vectors based on known object motion data.

15. The system according to claim 9, wherein: The one or more per-block motion vectors are compatible with the H.264 / MPEG-4 AVC standard.

16. The system according to claim 9, wherein: The graphics engine is configured to perform an arithmetic mean transform on the one or more per-pixel motion vectors.

17. A computer-implemented method for encoding video in a system comprising a graphics engine and a video encoder, the method comprising: receiving, at the video encoder, motion information from the graphics engine; At the video encoder, receiving a video frame to be encoded; as well as Using the video encoder, encoding the video frame to generate encoded video data, wherein encoding the video frame comprises: circumventing motion estimation for one or more blocks of the video frame in response to receiving the motion information from the graphics engine; using one or more per-block motion vectors in motion compensation for the one or more blocks of the video frame, the one or more per-block motion vectors being based on the motion information from the graphics engine; and The encoded video data is output in a bitstream.

18. The method according to claim 17, wherein Each of the one or more per-block motion vectors indicates motion of a macroblock of the video frame.

19. The method according to claim 17, wherein Each of the one or more per-block motion vectors indicates motion of a 4x4 block of the video frame.

20. The method according to claim 17, wherein The one or more per-block motion vectors are generated by the graphics engine based on information generated by the graphics engine for the game.

Images