Method and system for motion vectors in 3D graphics

Abstract

Various methods are provided for generating motion vectors in the context of 3D computer-generated images. An exemplary method includes generating, for each pixel of one or more objects to be rendered in a current frame, a 1-phase motion vector (MV1) and a 0-phase motion vector (MV0), each MV1 and MV0 having an associated depth value, thereby forming an MV1 texture and an MV0 texture; converting the MV1 texture into a set of MV1 blocks and converting the MV0 texture into a set of MV0 blocks; and outputting the set of MV1 blocks and the set of MV0 blocks for image processing.

Description

Technical Field

[0001] The implementation of the subject matter disclosed in this paper relates to the field of three-dimensional (3D) computer graphics, and more specifically to how to improve the rendering of interpolated frames in a 3D environment. Background Technology

[0002] Over the years, advancements in computer processing power have made real-time video rendering (such as in video games or certain animations) increasingly complex. For instance, early video games were characterized by pixelated sprites moving across a fixed background, while contemporary video games feature realistic 3D scenes filled with characters. Meanwhile, the miniaturization of processing components has enabled mobile devices (such as handheld video game consoles and smartphones) to effectively support real-time rendering of high frame rates and high resolution video.

[0003] 3D graphics video can be output at various frame rates and screen resolutions. It may be desirable to convert a video with 3D graphics from one frame rate (and / or resolution) to another. To save computational power while still increasing the frame rate, interpolated frames can be used instead of rendering all frames within the video. Interpolated frames can be efficiently generated using motion vectors (also referred to as MVs in this paper), which track the positional differences of objects between the current frame (CF) and the previous frame (PF). Summary of the Invention

[0004] This document discloses an implementation scheme for calculating and transferring motion vectors in the context of 3D computer graphics. In one example, a method includes: generating a 1-phase motion vector (MV1) and a 0-phase motion vector (MV0) for each pixel of one or more objects to be rendered in the current frame, each MV1 and MV0 having an associated depth value, thereby forming an MV1 texture and an MV0 texture; converting the MV1 texture into a set of MV1 blocks and the MV0 texture into a set of MV0 blocks; and outputting the set of MV1 blocks and the set of MV0 blocks for image processing. Attached Figure Description

[0005] Figure 1 An exemplary diagram of a computing system is shown.

[0006] Figure 2 This is an exemplary block diagram schematically illustrating a pipeline used for transmitting and using motion vector data.

[0007] Figure 3 An exemplary 3D coordinate system and two points in 3D space are shown.

[0008] Figure 4 An exemplary 2D coordinate system and two points in 2D space are shown.

[0009] Figure 5 A flowchart illustrating an example of a method for creating 2.5D blocks MV0 and 2.5D blocks MV1 and outputting them to an image processing module is shown.

[0010] Figure 6 A flowchart illustrating a method for calculating block motion vectors using pixel-by-pixel motion vectors as input, according to an example of this disclosure, is shown.

[0011] Figure 7 This schematically illustrates an example of converting a pixel-by-pixel MV into a block MV.

[0012] Figure 8 Three different methods for dividing data into two separate channels according to examples of this disclosure are illustrated schematically. Detailed Implementation

[0013] This document describes systems and methods for calculating motion vectors for use in frame interpolation, frame rate conversion, or other actions. As previously explained, motion vectors are generated to track the positional differences of an object between the current frame (CF) and the previous frame (PF). As explained herein, frame interpolation can be performed using two types of motion vectors: 1-phase MV (MV1) and 0-phase MV (MV0). MV0 represents motion from PF to CF, and MV1 represents motion from CF to PF. MVs are generated for each pixel (or group of pixels) on the screen, thus forming a texture or set of MVs for the pixels on the screen. As used herein, a texture is defined as a mapping from the set of pixels in a frame to a set of one or more numbers (e.g., components of a vector or a single number).

[0014] If motion vectors are used during frame rate conversion, a typical rendering engine outputs only a two-dimensional MV1 texture. Therefore, the texture contains no depth content and only includes information about changes in relative screen position, as seen in a virtual camera's frame of reference. The depth of the per-pixel motion vectors tells how to calculate the 2D components of the block motion vectors. Block motion vectors can represent the average of the motion vectors of a block of pixels (e.g., a five-by-five pixel block) and can be used for frame interpolation or other image processing tasks to, for example, reduce processing requirements. Scene regions within a specific depth range are called the foreground (closer to the camera), background (farther from the camera), or intermediate range (between the foreground and background). In image processing, it may be desirable to determine which of the following depth ranges dominates each pixel block: foreground, background, or intermediate range.

[0015] As an example, two objects can be positioned at different distances from the (virtual) camera or viewpoint. If the two objects move in the same direction at equal world-space distances, the farther object will appear to have moved less distance in eye space, creating a parallax effect where the object farther from the viewpoint appears to have moved less than the object closer to the viewpoint.

[0016] When most pixels in a block are in the background, most pixels will have small motion vectors (MV) because MV is evaluated from the camera / viewpoint perspective. If a small number of pixels in the block are, for example, in the foreground, the foreground pixels will have larger motion vector magnitudes. If all motion vectors within the block are averaged, the (relatively smaller) MV of the foreground will dominate the average MV. This can distort the relatively small, noticeable motion of background pixels, thus favoring the MV of foreground pixels. The magnitude of the background MV can also be strongly affected by camera movement. Pixels in the background can also have large MVs, potentially increasing the error of the average in undesirable ways.

[0017] By including depth information in the pixel MV, the dominant depth range of each block can be resolved: foreground, background, or intermediate range. Motion vector values ​​within the block that do not fall within the dominant range can then be ignored to facilitate evaluation of the average value of pixels only within the dominant range. When the block is dominated by background pixels, the resulting motion vectors more closely match intra-frame motion.

[0018] The added depth information also provides additional flexibility to the image processing module. The depth component of MV can, for example, be used as input to a weighting function, which can then be used to apply different levels of image correction to regions at different depths.

[0019] Therefore, according to the implementation described herein, a depth texture can be attached to MV1, which allows for more accurate frame interpolation by taking into account depth variations. Furthermore, an MV0 texture with depth can be similarly generated. Both MV0 and MV1 can be used as input to a frame rate conversion algorithm, which facilitates the interpolation step.

[0020] Furthermore, each frame can consist of two types of objects: objects with motion vectors and objects without motion vectors. Objects characterized by motion vectors can include moving characters or other objects, the user's view, and portions of the game map. Objects without motion vectors can include, for example, smoke effects, full-screen or partial screen scene transitions (e.g., fade-in / fade-out and wipe), overlays (such as graphical user interface elements and gun reticles), and / or particle effects. Improved image processing can be performed by separating objects with and without motion vectors. Traditionally, algorithms attempt to exclude screen areas characterized by objects without motion vectors. However, this approach is not perfect and can lead to the blending of nearby objects during frame rate conversion. Therefore, separating objects with and without motion vectors before transmission to the image processor reduces artifacts caused by conventionally known exclusion methods. Object separation produces two channels, which together constitute the frame data.

[0021] Traditionally, motion vectors are also sampled on a pixel-by-pixel basis, with each pixel on the screen having an associated MV0 and MV1. However, the sampling resolution of motion vectors can be dynamically reduced or increased. Reducing the resolution can also reduce the computing power required for MV computation. Since many devices (e.g., smartphones) have limited computing resources and battery life, reducing computational costs can save processing power and battery life. As described herein, "pixel-by-pixel" or "each pixel" may not refer to an individual pixel, but rather to a set of pixels in the context of evaluating the motion vector. In some implementations, low-resolution MV generation can be performed by reducing the sampling resolution when computing the MV. For example, the MV can be calculated only for every fourth pixel in the x-direction and every fourth pixel in the y-direction.

[0022] First, the motion vectors described herein are generated on a pixel-by-pixel basis, and then transformed into a block form with depth texture. Since objects with and without motion vectors can be separated within each frame, the block motion vectors can then be transmitted to image processing module 222 along with frame data containing either objects with or without motion vectors. One of the channels may include an alpha layer, which can inform the image processing module how to blend objects with and without motion vectors when rendering a frame. The image processing module can use the block motion vectors to perform visual enhancements, such as frame rate conversion.

[0023] By separating objects with and without MV, the exclusion algorithm may not be necessary. Therefore, even in regions blurred by particle effects, separation allows for the generation of interpolated frame data. Including depth information in the pixel MV allows for more accurate block MV calculations because blocks dominated by background pixels are better represented than by averaging.

[0024] This document describes aspects of the disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments disclosed herein. To the extent that such block diagrams, flowcharts, and / or examples contain one or more functions and / or operations, those skilled in the art will understand that each function and / or operation within such block diagrams, flowcharts, or examples can be implemented individually and / or collectively using a wide range of hardware, software, firmware, or virtually any combination thereof via computer-readable instructions. The described systems are exemplary in nature and may include additional elements and / or omitted elements. The subject matter of this disclosure includes all novel and non-obvious combinations and sub-combinations of the various systems and configurations disclosed, as well as other features, functions, and / or characteristics. Therefore, the methods can be performed by executing instructions stored on a machine-readable storage medium using one or more logical means (e.g., a processor) in conjunction with one or more additional hardware elements (such as storage devices, memories, hardware network interfaces / antennas, switches, actuators, clock circuits, etc.). The described methods and associated actions can also be performed in various sequences, in parallel, and / or simultaneously, other than the sequence described herein. The processor of the logical subsystem can be single-core or multi-core, and the program executing on it can be configured for parallel or distributed processing. The logical subsystem may optionally include individual components distributed across two or more devices, which can be remotely located and / or configured for coordinated processing. One or more aspects of the logical subsystem can be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

[0025] Figure 1 An example of a computer system 100 is schematically depicted, which may include one or more processors 110 (e.g., a central processing unit (CPU) and / or a graphics processing unit (GPU)), volatile and / or non-volatile memory 120 (e.g., random access memory (RAM) and / or one or more hard disk drives (HDDs)). The computer system may also include one or more displays 130, which may include any number of visual interface technologies. Furthermore, exemplary embodiments may include a user interface 140 (e.g., a keyboard, computer mouse, touchscreen, controller, etc.) to allow a user to provide input to the computer system. In some embodiments, the computer system may be a mobile phone or a tablet computer.

[0026] Figure 2 A block diagram 200 schematically illustrates a pipeline (including components capable of generating and processing MVs) for generating and using MVs. Block diagram 200 includes a game engine 202 that outputs one or more 3D models. Game engine 202 can be configured to generate and output 3D models 204 to be rendered, specifying the desired positions of objects and any possible particle effects. Game engine 202 generates multiple image frame data sequentially, which may include CF data 208 and PF data 206. Previous frame data may be derived from a cached version of the current frame (e.g., game engine 202 may only output the current frame data and may retrieve previous frame data from memory). Therefore, the current frame data becomes the previous frame data for the next frame to be rendered. Ultimately, one or more interpolated frames can be generated between CF and PF. 3D model 204 is output to a 3D motion vector module, which in this instance may be a software development kit and is therefore referred to as a game SDK 210. The 3D motion vector module uses an internal rendering matrix and information from the 3D model to generate 3D pixel MV0 textures 212 and 3D pixel MV1 textures 214. 3D pixel MV1 texture 214 may include multiple 3D MV1 textures, one for each pixel or group of pixels. (e.g., for a given pixel) Each 3D MV1 may include changes in the vertical, horizontal, and depth positions of an object at that pixel from a previous frame to the current frame. 3D pixel MV0 texture 212 may include multiple 3D MV0 textures, one for each pixel or group of pixels. (e.g., for a given pixel) Each MV0 may include changes in the vertical, horizontal, and depth positions of an object at that pixel from a previous frame to the current frame.

[0027] Because of the non-exclusive projection of objects from the PF to the CF, the process of generating 3D pixel MV0 texture 212 and 3D pixel MV1 texture 214 may include generating multiple possible MVs for one or more pixels. In the 2D domain, a pixel in the PF can be projected onto multiple pixels in the CF. The multiple possible MVs are then compared with depth textures and other objects in the 3D scene for double verification of the MVs. Double verification is the process of selecting the correct MV from the multiple MVs. One exemplary method is to compare the depth buffers of 3D pixel MV0 212 and 3D pixel MV1 214 with the depth textures of the PF and CF, respectively.

[0028] The (double-confirmed) 3D pixel MV0 texture 212 and (double-confirmed) 3D pixel MV1 texture 214 can then be output to the block motion vector module. In this example, the block motion vector module can be a software development kit and therefore may be referred to as the algorithm SDK 216. The block motion vector module can process the input to generate 2.5D block textures MV0 218 and 2.5D block MV1 texture 220, which are textures that can be used to generate interpolated frames. Block texturing is a process of averaging the motion vector values ​​within each of a plurality of blocks (as described below). Figure 5 (As explained). The 2.5D block MV0 texture 218 and 2.5D block MV1 texture 220 are output to an image processing module 222, which may or may not be a separate piece of hardware from one or more other processors 110. The image processing module 222 also receives CF data 208 and PF data 206. CF data can be transmitted over two physically or logically separate channels, one channel for objects with MV and the other channel for objects without MV. The 2.5D block MV0 texture 218 and 2.5D block MV1 texture 220 can be included within the channel containing objects with MV. One channel may include an alpha layer that provides data on how to mix objects with and without MV. The separated data can be transmitted to the image processing module 222 over two physically or logically separate channels.

[0029] The image processing module 222 can then use the 2.5D block MV0 texture 218 and the 2.5D block MV1 texture 220 as input and use PF data 206 and CF data 208 to perform an interpolation step or frame rate conversion. The image processing module 222 can output a previous frame (PF) 224, one or more interpolated frames 228, and a current frame 226, which can then be visually displayed on the display 130 (in the listed order). Therefore, generating interpolated frames 228 allows for an increase in frame rate.

[0030] The game engine 202, game SDK 210, and algorithm SDK 216 can each execute on the same or different processors 110 of the computer 100 according to instructions stored in memory 120. In at least some instances, the image processing module 222 may be a separate piece of hardware from the game engine 202, game SDK 210, and algorithm SDK 216.

[0031] As used herein, the terms "system" or "module" can include hardware and / or software systems that operate to perform one or more functions. For example, a module or system can include a computer processor, controller, or other logic-based device that performs operations based on instructions stored on a tangible and non-transitory computer-readable storage medium, such as computer memory. Alternatively, a module or system can include hardwired devices that perform operations based on the device's hardwired logic. The various modules or units shown in the accompanying drawings may represent hardware that operates based on software or hardwired instructions, software that directs the hardware to perform operations, or a combination thereof.

[0032] It should be noted that the techniques discussed in this article are applicable not only to games, but also to any animation rendering of 3D models, although the advantages offered by this method are most significant in real-time rendering scenarios.

[0033] Figure 3 A 3D coordinate system 300 is shown with two 3D points, Q_PF 302 and Q_CF 304. The two points have x, y, and z coordinates, where Q_PF 302 has coordinates (x0, y0, z0) and Q_CF has coordinates (x1, y1, z1). The z-value is also called depth, and in some instances, it is provided by a separate texture or may need to be calculated using methods involving, for example, rays from the camera position to the point's location. Based on these two points in space, two 3D motion vectors can be calculated using, for example, the following formula:

[0034] MV0=(x0-x1, y0-y1, z0-z1)=(Δx0, Δy0, Δz0),

[0035] MV1=(x1-x0, y1-y0, z1-z0)=(Δx1, Δy1, Δz1)=-MV0.

[0036] According to these definitions, MV0 represents the 3D transformation an object undergoes from Q_PF 302 to Q_CF 304. On the other hand, MV1 represents the transformation in the opposite direction from Q_CF 304 to Q_PF 302.

[0037] Although Figure 3The 3D motion vectors represent points in 3D space, but these points are projected onto a 2D display for viewing. In typical instances, the motion vectors are generated as 2D textures, which are then used to generate visual enhancements and create interpolated frames. In addition to regular motion image (MV) information, the implementation described herein also uses depth textures, but in a different way than in pure 3D MV. For example, this depth texture can be calculated by drawing a ray from the camera's position to the location of an object, such as as observed from a close clip. A close clip, as defined herein, is the nearest plane in 3D space where the object is still visible. Objects closer than the close clip are not visible in the scene. In some instances, the game engine provides the depth information used in the algorithm.

[0038] Therefore, the method for calculating 3D MV0 and 3D MV1 can be applied to generate the above information. Figure 2 The 3D pixel MV discussed is then double-verified according to a process described in more detail below. For example, the double-verified 3D pixel MV0 and double-verified 3D pixel MV1 can be converted into 2.5D block MV0 and 2.5D block MV1, and then the 2.5D block MV0 and the 2.5D block MV1 are output to the image processing module for frame rate conversion.

[0039] Figure 4 This demonstrates how 2.5D motion vectors can be calculated, and thus shows a 2D coordinate system 400 with projections of Q_PF 402 and Q_CF 404. These projected points only have x and y coordinates. Projection can be performed using a rendering matrix applied to points in 3D space (such as the one mentioned above). Figure 2 The rendering matrix discussed is used to perform this. A 2D coordinate system of 400 can represent, for example, the view position of a pixel on a 2D screen as viewed from the perspective of a virtual camera. Using the projected coordinates of two points and a depth value, a 2.5D motion vector can be calculated. It should be noted that these are different from 3D motion vectors. A 2.5D motion vector can be calculated, for example,

[0040] MV0 2.5D = (Δx0, Δy0, z0),

[0041] MV1 2.5D = (Δx1, Δy1, z1) ≠ -MV0 2.5D .

[0042] It should be noted that in both cases, the depth changes between frames are not recorded; instead, the original depth values ​​z1 and z0 are used.

[0043] Figure 5A flowchart illustrating a method 500 for generating 2.5D block motion vectors is shown. Method 500 can be executed according to instructions stored in computer memory (such as memory 120) and executed on one or more processors (such as processor 110). In some instances, method 500 can be... Figure 2 The game SDK 210 and / or algorithm SDK 216 are executed. At 502, the CF and PF frame pixels and the view projection matrix (e.g., a rendering matrix) are obtained. In some instances, the CF and PF frame pixels and the view projection matrix are received from the game SDK 210 or the game engine 202. At 504, the depth textures of the CF and PF are calculated. The depth textures may include the depth variation of the corresponding object from CF to PF for each pixel (for the MV1 texture) and the depth variation from the corresponding object PF to CF for each pixel (for the MV0 texture). At 506, a double-verification process is applied to double-verify the MV1 and MV0 of each 3D pixel of the texture. Double-verification selects the correct motion vector from one or more possible choices based on factors such as pixel occlusion and may contribute to rendering more accurate motion vectors. To double-verify the MV1 or MV0 of the selected pixel, the method may verify that the coordinates of the selected pixel in the current frame match the coordinates of the pixel mapped back to the previous frame using the MV1 or MV0 of the selected pixel, where the coordinates of the selected pixel include depth coordinates. MV0 can be verified by projecting the PF pixel onto the CF using MV0. If the projected pixel matches (or nearly matches, such as within a threshold range of the corresponding pixel in the CF) the corresponding pixel in the CF, then MV0 is verified. MV1 can be verified similarly by projecting a pixel from the CF onto the PF and comparing the projected pixel with the corresponding pixel in the PF. If both MV0 and MV1 are verified, then MV is considered correct.

[0044] For example, in a 2D domain, a pixel in the PF can be projected onto several pixels in the CF (or, if an object / pixel in the PF is covered by an object in the CF, then no pixel in the CF can be projected onto a pixel in the PF). In a 3D domain, there is a one-to-one correspondence between each pixel. The depth buffer can then be used to determine which MV is correct.

[0045] Depth_buffer[i] = cf_mv_z_i + depth_i; if Depth_PF = Depth_buffer[j], then the buffered MV_j is the correct MV0.

[0046] Alternatively, the MV (coordinates X0, Y0, depth0) of a pixel in the PF is (delta_x0, delta_y0, delta_z0). The corresponding pixel in the CF may have coordinates (X1, Y1, depth1). If X1 = X0 + delta_x; Y1 = Y0 + delta_y; and depth1 = depth0 + delta_z, then this pixel is said to be double-identified. Any pixel that is not double-identified is outside the double-identified region. In other words, if the MV (with coordinates X1, Y1, depth1) of a pixel in the CF is (delta_x1, delta_y1, delta_z1), then if delta_x0 = -delta_x1, delta_y0 = -delta_y1, and delta_z0 = -delta_z1, then pixels Q_CF and Q_PF are double-identified.

[0047] If a pixel / MV is determined to be in an unconfirmed region, any unconfirmed MV1 and / or MV0 can be adjusted based on the most recent double-confirmed MV1 and / or MV0 (or the most recent double-confirmed MV1 or MV0 of the same object) to transform one or more unconfirmed MV1 and / or MV0 into one or more double-confirmed MV1 and / or MV0. For example, for a given pixel P0, the nearest pixel with double-confirmed MV could be PF. To adjust the MV of P0, the MV0 of P0 could be the sum of the MV1 of PF and the delta MV of PF. The delta MV of PF can be calculated as the MV0 of PF plus the MV1 of P0. At 508, the double-confirmed 3D pixel MV0 can be output, and at 510, the double-confirmed 3D pixel MV1 can be output (e.g., to the algorithm SDK). Using the double-confirmed 3D pixel MV0, a ​​2.5D block MV0 is generated at 512, and the 2.5D block MV0 is output at 514. Similarly, at 513, the double-confirmed 3D pixel MV1 is converted to block MV1, and at 516, block MV1 is output. To convert 3D pixel MV1 to 2.5D block MV1, for each of a plurality of pixel blocks, the dominant pixel type in the pixel block is identified. If the dominant pixel type is a background pixel, an average MV1 is calculated for all pixels in the pixel set. Otherwise, an average MV1 is calculated for only foreground pixels in the pixel set. The average MV1 is then set as the MV1 of the pixel block, where the set of MV1 blocks includes the average MV1 of each pixel block. A similar process can be performed for the conversion from 3D pixel MV0 to 2.5D block MV0. Furthermore, instead of including depth variations in the MV, the depth variations are replaced with the depth values ​​of the pixels. 3D MV represents the movement of an object in three dimensions between frames. 2.5D MV represents the movement of an object within a 2D screen, where depth values ​​are added to represent object occlusion. That is, the depth values ​​indicate which objects are in front of other objects, which can help generate interpolated frames.

[0048] At position 518, objects with MV (Modular Value) and those without MV are separated into two channels. In some implementations, the channels are physically separated. When physically separated channels are not feasible, logically separated channels (spatially or temporally) can be used. Separating objects with MV from those without MV eliminates the need to exclude regions from the screen during frame rate conversion. Exclusion algorithms may produce visual artifacts at the boundaries of regions with particle effects. Separating these objects allows them to be processed differently, thus improving the results.

[0049] At 526, various information is sent to the image processing module. The information sent to the image processing module includes blocks MV0 and MV1 (as indicated at 520), image data (as indicated at 522), and frame-level metadata (as indicated at 524). Image data 522 may include the objects discussed above, and therefore image data 522 may be sent through more than one physical or logical channel.

[0050] Figure 6 A method 650 for calculating block MV based on pixel-by-pixel MV is shown. Method 650 occurs during the pixel-to-block MV conversion performed at 512 and 513 of method 500, which can be calculated using one or more processors 110 and instructions in memory 120. At 651, the pixels on the screen are divided into multiple blocks, such as a rectangular set of pixels of finite size. In one example (see...) Figure 7 The block can be a 4x4 pixel square. Typically, the blocks do not need to be of equal size or square. At 652, two depth thresholds (depth threshold 1 and depth threshold 2) are calculated. The depth thresholds can be given, for example, by the following formula:

[0051]

[0052]

[0053] Among them Depth Max It is the maximum depth value of the block, and Depth Min This is the minimum depth value within the block. In this example, a larger depth corresponds to an object that is farther away from the camera or viewer.

[0054] At position 655, create foreground (FG), background (BG), and middle range (MID) bins and assign each bin an initial value of 0. The size of each bin can be stored as N. PG N BG and N MID As indicated at 656, for each pixel in the block, at 660, the pixel's depth value (and therefore the depth value of the 2.5D motion vector) is compared to two thresholds. If the depth is greater than depth threshold 1, the BG bin is incremented at 672. If the depth is less than depth threshold 2, the FG bin is incremented at 674. Otherwise, the MID bin is incremented at 676. It should be noted that only one bin should be incremented for each pixel within the block.

[0055] Once each pixel within a block has been compared, the values ​​of the FG, BG, and MID bins are compared at 680 to identify the distribution of pixel types. Pixel type distribution identification can be performed to determine whether the depth component of the MV within the block exhibits a bimodal distribution; a bimodal distribution indicates the presence of two types of objects within the block: foreground objects and background objects. If a bimodal distribution is not detected, ignoring MVs with extreme depth components (e.g., those determined to be in the foreground or background range) can produce a more stable distribution. In this case, the pixels in the middle range should be averaged. However, since the disclosed method only creates three bins to classify pixels, and the number of pixels in each block may be small (e.g., 16), a bimodal distribution may appear biased towards the foreground bins and the background bins. Either case can indicate the presence of foreground objects. The size of the primary bin (denoted as N in this document) can be given, for example, by a procedure specified by the following pseudocode:

[0056]

[0057] It should be noted that the two constants K1 and K2 can be chosen such that 0 ≤ K1 ≤ 2 and K1 ≤ K2. The two constants can be determined empirically to achieve a stable distribution of the depth component of the block MV. In some implementations, K1 = K2 = 0. In this way, when a bimodal distribution is detected (e.g., where the block includes at least one foreground pixel and at least one background pixel) such that both foreground and background objects are detected in the block, the MV of only the foreground pixels is averaged and set as the block MV (even if there are more background pixels than foreground pixels in the block), which allows for the preservation of foreground objects in one or more interpolated frames that might otherwise be lost. When a bimodal distribution is not detected, only one type of object (whether foreground or background) is detected in the block, and the MV of only the pixels in the intermediate range is averaged and set as the block MV.

[0058] At 682, method 650 includes the step of averaging the MV based on the distribution of pixel type. For example, the block MV can be given by a formula such as the following:

[0059]

[0060] Where N represents which bin is being averaged, as determined in step 680 above. The summation is performed on all pixel-by-pixel MVs (MV0 or MV1) within the bin corresponding to N, for example, if N = N... BG Then, summation is performed on all background pixels, etc. The addition here is performed according to standard vector addition (e.g., (x1, y1, z1) + (x2, y2, z2) = (x1 + x2, y1 + y2, z1 + z2)). Method 650 then returns, using the average MV as its return value.

[0061] Figure 7This illustrates an example of processing a pixel MV into a block MV. In this case, pixel block 740 comprises a set 754 of foreground pixels (for visual clarity, in...). Figure 7 The image shows a set of 750 background pixels (shown as a shaded line drawn in one direction) for visual clarity. Figure 7 The image shows a set of 752 pixels (shown as a vertically shaded line) and a middle range of pixels (for visual clarity, in...). Figure 7 (Shown as cross-shaded lines). Depth is used to determine which range each pixel belongs to; for example, pixels in a first depth range are assigned as background pixels, pixels in a second depth range are assigned as middle range pixels, and pixels in a third depth range are assigned as foreground pixels. Since most pixels within pixel block 740 fall within the foreground depth range, the depth values ​​of the foreground range pixels 754 are averaged to produce a single depth value, which is applied to all pixel MVs in block MV, thereby generating block MV 756. Furthermore, the x and y components (not shown) of the foreground MV are averaged and output in block MV. Thus, block MV consists of three semi-independent textures: an x ​​component, a y component, and a depth component, each of which is independently averaged within the pixels fitted to the depth range. This technique applies to both MV1 and MV0.

[0062] Figure 8 Different methods for transmitting data on two separate channels are illustrated. In a first example 800, a first physical channel 802 and a second physical channel 804 are used to simultaneously transmit two different kinds of data, such as objects with motion vectors and objects without motion vectors. It should be noted that the two channels do not necessarily have to be the same size. A second example 810 is illustrated, having a first time-separated channel 812 and a second time-separated logical channel 814. In this method, a single physical channel 816 alternately transmits objects with motion vectors and objects without motion vectors. In a third example 820, a single physical channel 822 is divided into two spatially separated channels of fixed size. This allows information to be transmitted simultaneously through a single channel 822. By accessing separate information for objects with and without motion vectors, the image processing module does not have to perform exclusion calculations and can utilize greater accuracy when performing visual enhancement algorithms such as frame rate conversion.

[0063] The technical effect of generating the MV1 and MV0 textures for the current frame (where both MV1 and MV0 textures include depth values) is that the depth values ​​can be used to convert the MV1 and MV0 textures into MV1 and MV0 blocks respectively, allowing the resulting motion vectors to more closely match the motion occurring within the frame. Another technical effect is that frame interpolation can be performed using blocks MV1 and MV0 with depth values, thereby improving the frame rate.

[0064] This disclosure also provides support for a method comprising: generating a 1-phase motion vector (MV1) and a 0-phase motion vector (MV0) for each pixel of one or more objects to be rendered in a current frame, each MV1 and MV0 having an associated depth value, thereby forming an MV1 texture and an MV0 texture; converting the MV1 texture into a set of MV1 blocks and converting the MV0 texture into a set of MV0 blocks; and outputting the set of MV1 blocks and the set of MV0 blocks for image processing. In a first instance of the method, the positional change of an object at the pixel from the current frame to a previous frame is represented, and each MV0 represents the positional change of the object at the pixel from the previous frame to the current frame. In a second instance of the method, optionally including the first instance, converting the MV1 texture into the set of MV1 blocks comprises: identifying a dominant pixel type in each of a plurality of pixel blocks; and converting each pixel block into a corresponding MV1 block based on the dominant pixel type. In a third instance of the method, which optionally includes one or both of the first and second instances, converting each pixel block into a corresponding MV1 block based on the dominant pixel type includes: determining that the dominant pixel type of the selected pixel block is a foreground pixel, and in response, calculating an average MV1 for only foreground pixels in the pixel block, and setting the average MV1 as the MV1 value of the MV1 block. In a fourth instance of the method, which optionally includes one or more of the first to third instances, converting the MV0 texture into the set of MV0 blocks includes: identifying the dominant pixel type in each of the plurality of pixel blocks, and converting each pixel block into a corresponding MV0 block based on the dominant pixel type. In a fifth instance of the method, which optionally includes one or more of the first to fourth instances, converting each pixel block into a corresponding MV0 block based on the dominant pixel type includes: determining that the dominant pixel type of the selected pixel block is a foreground pixel, and in response, calculating an average MV0 for only foreground pixels in the pixel block, and setting the average MV0 as the MV0 value of the MV0 block. In a sixth instance of the method, which optionally includes one or more of the first to the fifth instances, outputting the set of MV1 blocks and the set of MV0 blocks includes outputting the set of MV1 blocks and the set of MV0 blocks along with frame-level metadata and image data for image processing. In a seventh instance of the method, which optionally includes one or more of the first to the sixth instances, the image data is separated into objects with motion vectors and objects without motion vectors, and the objects with motion vectors are transmitted on a separate channel from the objects without motion vectors.In an eighth instance of the method, which optionally includes one or more, or each of the first to seventh instances, objects without motion vectors include: overlays, particle effects, scene transitions, GUI elements, gun slingshots, and / or smoke effects. In a ninth instance of the method, which optionally includes one or more, or each of the first to eighth instances, the method further includes: applying a double verification process to each MV0 and each MV1, wherein the double verification process includes verifying each associated depth value. In a tenth instance of the method, which optionally includes one or more, or each of the first to ninth instances, applying the double verification process includes: adjusting one or more unverified MV1 and / or MV0 based on the most recent double verification of MV1 and / or MV0. In an eleventh instance of the method, which optionally includes one or more of the first to the tenth instances, applying the double verification process includes: for a selected pixel in the current frame, in response to a coordinate matching of the selected pixel in the current frame using the MV1 or MV0 of the selected pixel to map back to the coordinates of the pixel in a previous frame, double verifying the MV1 or MV0 of the selected pixel, wherein the coordinates of the selected pixel include depth coordinates.

[0065] This disclosure also provides support for a system for rendering 3D graphics, the system comprising: one or more processors and non-transitory memory allocated to form: a 3D motion vector module configured to receive a 3D model from a game engine and configured to output a plurality of 1-phase 3D motion vectors (3D MV1) and a plurality of 0-phase 3D motion vectors (3D MV0) based on the 3D model, each 3D MV1 including vertical position changes, horizontal position changes, and depth position changes of an object at a corresponding pixel from the current frame to the previous frame, each MV0 including vertical position changes, horizontal position changes, and depth position changes of an object at a corresponding pixel from the previous frame to the current frame; and a block motion vector module configured to, respectively, output a plurality of 3D MV1 and a plurality of 0-phase 3D motion vectors based on the plurality of 3D MV1 and the plurality of 0-phase 3D MV0. MV0 generates 2.5D block MV1 textures and 2.5D block MV0 textures, and outputs the 2.5D block MV1 textures and 2.5D block MV0 textures for image processing to form an image to be displayed on a monitor. The 2.5D block MV1 textures comprise a plurality of MV1 blocks, each MV1 block being formed from a corresponding subset of the plurality of 3D MV1 blocks. Similarly, the 2.5D block MV0 textures comprise a plurality of MV0 blocks, each MV0 block being formed from a corresponding subset of the plurality of 3D MV0 blocks. In a first instance of the system, the block MV1 textures and the 2.5D block MV0 textures can be used to interpolate frames between the current frame and the previous frame. In a second instance of the system, optionally including the first instance, the block MV1 textures are formed from a corresponding subset of double-acknowledged and / or corrected 3D MV1 blocks, and the 2.5D block MV0 textures are formed from a corresponding subset of double-acknowledged and / or corrected 3D MV0 blocks. In a third instance of the system, which optionally includes one or both of the first and second instances, each MV1 block includes: the average horizontal position change of the corresponding subset of the plurality of 3D MV1s, the average vertical position change of the corresponding subset of the plurality of 3D MV1s, and the average depth value of the corresponding subset of the plurality of 3D MV1s, and wherein each MV0 block includes: the average horizontal position change of the corresponding subset of the plurality of 3D MV0s, the average vertical position change of the corresponding subset of the plurality of 3D MV0s, and the average depth value of the corresponding subset of the plurality of 3D MV0s.

[0066] This disclosure also provides support for a method comprising: generating a 1-phase motion vector (MV1) and a 0-phase motion vector (MV0) for each pixel of one or more objects to be rendered in a current frame, each MV1 and MV0 having an associated depth change, thereby forming an MV1 texture and an MV0 texture; applying a double verification process to each MV0 and each MV1, wherein the double verification process includes verifying each associated depth change; correcting each MV0 and MV1 that is not double verified; converting the MV1 texture into a set of MV1 blocks and the MV0 texture into a set of MV0 blocks, each MV1 block and each MV0 block having an associated depth value, the conversion including identifying a dominant pixel type in each of a plurality of pixel blocks, and converting each pixel block into a corresponding MV0 block and a corresponding MV1 based on the dominant pixel type; and outputting the set of MV1 blocks and the set of MV0 blocks for image processing. In a first instance of the method, the positional change of an object at the pixel from the current frame to a previous frame is represented, and wherein each MV0 represents the positional change of the object at the pixel from the previous frame to the current frame. In a second instance of the method, which optionally includes the first instance, converting each pixel block into a corresponding MV0 block and a corresponding MV1 block based on the primary pixel type includes: if the primary pixel type is a background pixel, then calculating an average MV0 and an average MV1 for all pixels in the pixel block; otherwise, calculating the average MV0 and the average MV1 for only foreground pixels in the pixel block; and setting the average MV0 as the MV0 value of the MV0 block and setting the average MV1 as the MV1 value of the MV1 block. In a third instance of the method, which optionally includes one or both of the first and second instances, applying the double-confirmation process includes: for a selected pixel in the current frame, in response to a coordinate matching of the selected pixel in the current frame using the MV1 or MV0 of the selected pixel to map back to the coordinates of the pixel in a previous frame, double-confirming the MV1 or MV0 of the selected pixel, wherein the coordinates of the selected pixel include depth coordinates.

[0067] As used herein, elements or steps described in the singular and beginning with the word "a" or "an" should be understood to not exclude the plural form of the elements or steps, unless such exclusion is explicitly stated. Furthermore, references to "an embodiment" of the invention are not intended to exclude the existence of other embodiments that also incorporate the described features. Moreover, unless explicitly stated otherwise, embodiments that "comprise," "include," or "have" one or more elements having a particular characteristic may include additional such elements that do not have said characteristic. The terms "comprise" and "in which" are used as concise linguistic equivalents to the corresponding terms "comprise" and "wherein." Furthermore, the terms "first," "second," "third," etc., are used merely as labels and are not intended to impose numerical requirements or a particular order of position on their objects.

[0068] This written description uses examples to disclose the invention, including the best mode, and also enables those skilled in the art to practice the invention, including making and using any apparatus or system and performing any incorporated methods. The patentable scope of the invention is defined by the claims and may include other examples that would occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that are indistinguishable from the literal language of the claims, or if they include equivalent structural elements that are not substantially indistinguishable from the literal language of the claims.

Claims

1. A method for motion vectors in 3D graphics, comprising: For each pixel of one or more objects to be rendered in the current frame, generate a 1-phase motion vector (MV1) and a 0-phase motion vector (MV0), each MV1 and MV0 having an associated depth value, thus forming an MV1 texture and an MV0 texture; The MV1 texture is converted into a set of MV1 blocks, and the MV0 texture is converted into a set of MV0 blocks. The conversion includes identifying the distribution of pixel types in each of the multiple pixel blocks, and converting each pixel block into a corresponding MV0 block and a corresponding MV1 block based on the distribution of pixel types. as well as Output the set of MV1 blocks and the set of MV0 blocks for image processing.

2. The method of claim 1, wherein each MV1 represents a positional change of an object at the pixel from the current frame to a previous frame, and wherein each MV0 represents a positional change of the object at the pixel from the previous frame to the current frame.

3. The method of claim 1, wherein converting the MV1 texture into the set of MV1 blocks comprises: For each of a plurality of pixel blocks, identify the distribution of pixel types within the pixel block; as well as Each pixel block is converted into a corresponding MV1 block based on the distribution of the pixel type.

4. The method of claim 3, wherein converting each pixel block into a corresponding MV1 block based on the distribution of the pixel type comprises: The first selected pixel block is determined to have a non-bimodal distribution, such that only or mainly intermediate range pixels exist, and in response, the average MV1 is calculated for only intermediate range pixels in the first selected pixel block; The second selected pixel block is determined to have a bimodal distribution, such that the second selected pixel block includes at least one foreground pixel and at least one background pixel, and in response, the average MV1 is calculated for only the foreground pixels in the second selected pixel block; as well as Set each average MV1 to the MV1 value of the corresponding MV1 block.

5. The method of claim 1, wherein converting the MV0 texture into the set of MV0 blocks comprises: For each of a plurality of pixel blocks, identify the distribution of pixel types within the pixel block; as well as Each pixel block is converted into a corresponding MV0 block based on the distribution of the pixel type.

6. The method of claim 5, wherein converting each pixel block into a corresponding MV0 block based on the distribution of the pixel type comprises: The first selected pixel block is determined to have a non-bimodal distribution, such that only or mainly intermediate range pixels exist, and in response, the average MV0 is calculated for only intermediate range pixels in the first selected pixel block. The second selected pixel block is determined to have a bimodal distribution, such that the second selected pixel block includes at least one foreground pixel and at least one background pixel, and in response, the average MV0 is calculated for only the foreground pixels in the second selected pixel block; as well as Set each average MV0 to the MV0 value of the corresponding MV0 block.

7. The method of claim 1, wherein outputting the set of MV1 blocks and the set of MV0 blocks comprises: Output the set of MV1 blocks and the set of MV0 blocks along with frame-level metadata and image data for image processing.

8. The method of claim 7, wherein the image data is separated into objects having motion vectors and objects not having motion vectors, and the objects having motion vectors are transmitted on a separate channel from the objects not having motion vectors.

9. The method of claim 1, further comprising: A double verification process is applied to each MV0 and each MV1, wherein the double verification process includes verifying each associated depth value.

10. The method of claim 9, wherein applying the dual verification process comprises: Adjust one or more unconfirmed MV1 and / or MV0 based on the most recent double confirmation of MV1 and / or MV0.

11. The method of claim 9, wherein applying the dual verification process comprises: For a selected pixel in the current frame, in response to the coordinate matching of the selected pixel in the current frame using the MV1 or MV0 of the selected pixel to map back to the coordinates of the pixel in the previous frame, the MV1 or MV0 of the selected pixel is double-confirmed, and the coordinates of the selected pixel include depth coordinates.

12. A system for rendering 3D graphics, comprising: One or more processors and non-transitory memory, said one or more processors and non-transitory memory are allocated to form: A 3D motion vector module is configured to receive a 3D model from a game engine and to output multiple 1-phase 3D motion vectors (3D MV1) and multiple 0-phase 3D motion vectors (3D MV0) based on the 3D model. Each 3D MV1 includes the vertical, horizontal, and depth position changes of an object at a corresponding pixel from the current frame to the previous frame, and each MV0 includes the vertical, horizontal, and depth position changes of an object at a corresponding pixel from the previous frame to the current frame. A block motion vector module is configured to generate 2.5D block MV1 textures and 2.5D block MV0 textures based on the plurality of 3D MV1 and the plurality of 3D MV0, respectively, and output the 2.5D block MV1 textures and the 2.5D block MV0 textures for image processing to form an image to be displayed on a display, wherein the 2.5D block MV1 textures include a plurality of MV1 blocks, each MV1 block being formed by a corresponding subset of the plurality of 3D MV1, and wherein the 2.5D block MV0 textures include a plurality of MV0 blocks, each MV0 block being formed by a corresponding subset of the plurality of 3D MV0.

13. The system of claim 12, wherein the 2.5D block MV1 texture and the 2.5D block MV0 texture can be used to interpolate frames between the current frame and the previous frame.

14. The system of claim 12, wherein the 2.5D block MV1 texture is formed from a corresponding subset of double-confirmed and / or corrected 3D MV1, and the 2.5D block MV0 texture is formed from a corresponding subset of double-confirmed and / or corrected 3D MV0.

15. The system of claim 12, wherein each MV1 block comprises: The average horizontal position change of the corresponding subsets of the plurality of 3D MV1, the average vertical position change of the corresponding subsets of the plurality of 3D MV1, and the average depth value of the corresponding subsets of the plurality of 3D MV1, wherein each MV0 block includes: the average horizontal position change of the corresponding subsets of the plurality of 3D MV0, the average vertical position change of the corresponding subsets of the plurality of 3D MV0, and the average depth value of the corresponding subsets of the plurality of 3D MV0.

16. A method for motion vectors in 3D graphics, comprising: For each pixel of one or more objects to be rendered in the current frame, generate a 1-phase motion vector (MV1) and a 0-phase motion vector (MV0), each MV1 and MV0 having an associated depth variation, thus forming an MV1 texture and an MV0 texture; A double verification process is applied to each MV0 and each MV1, wherein the double verification process includes verifying each associated depth change; Correct each MV0 and MV1 that has not been double-confirmed; The MV1 texture is converted into a set of MV1 blocks and the MV0 texture is converted into a set of MV0 blocks, each MV1 block and each MV0 block having an associated depth value. The conversion includes identifying the distribution of pixel types in each of the multiple pixel blocks, and converting each pixel block into a corresponding MV0 block and a corresponding MV1 block based on the distribution of pixel types. as well as Output the set of MV1 blocks and the set of MV0 blocks for image processing.

17. The method of claim 16, wherein each MV1 represents a positional change of an object at the pixel from the current frame to a previous frame, and wherein each MV0 represents a positional change of the object at the pixel from the previous frame to the current frame.

18. The method of claim 16, wherein converting each pixel block into a corresponding MV0 block and a corresponding MV1 block based on the distribution of the pixel types comprises: If the distribution is a bimodal distribution, such that there is at least one background pixel and at least one foreground pixel, then the average MV0 and average MV1 are calculated for only the foreground pixels in the pixel block; otherwise, the average MV0 and average MV1 are calculated for only the pixels in the middle range of the pixel block. as well as The average MV0 is set as the MV0 value of the MV0 block, and the average MV1 is set as the MV1 value of the MV1 block.

19. The method of claim 16, wherein applying the dual verification process comprises: For a selected pixel in the current frame, in response to the coordinate matching of the selected pixel in the current frame using the MV1 or MV0 of the selected pixel to map back to the coordinates of the pixel in the previous frame, the MV1 or MV0 of the selected pixel is double-confirmed, and the coordinates of the selected pixel include depth coordinates.

Images