Information processing device and information processing method

Abstract

This information processing device comprises a control unit. This control unit acquires resolution data representing the relationship between the input resolution and the output resolution of a series of video frames. The control unit also renders a current frame of the video frames on the basis of the input resolution. The control unit also acquires jitter pattern data representing a jitter pattern to be used for temporal anti-aliasing. The control unit also acquires adjustment data representing an adjustment range in which sampling based on the jitter pattern data is restricted, on the basis of the magnification ratio of the output resolution to the input resolution. The control unit also performs the temporal anti-aliasing on the current frame on the basis of a history frame of the video frames, the jitter pattern data, and the adjustment data.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method.

[0002] Aliasing in CG rendering is often a problem when producing CG (Computer Graphics) images for movies, computer games, etc. Therefore, in the production of such CG images, techniques called TAA (Temporal Anti-Aliasing) and TAAU (Temporal Anti-Aliasing Upsampling) are sometimes adopted for the purpose of anti-aliasing.

[0003] TAA is a technique for suppressing aliasing by shifting the sampling position of each pixel in each input video frame at the subpixel level and integrating the sampling information of each pixel in each frame in the time direction. Shifting the sampling position at the subpixel level is called "jittering," and the sampling position determined by jittering is called "jitter offset." Jitter offset is expressed, for example, as the amount of offset from the center position of the pixel.

[0004] TAAU is a technique that adds upscaling to TAA and performs upscaling on a video frame to be processed. TAAU may use recursive information from past video frames to infer a super-resolution result using a deep neural network (DNN) (see, for example, Patent Document 1). This super-resolution process using TAAU (hereinafter referred to as "SR process") may be called multi-frame super resolution (MFSR).

[0005] Special Publication No. 2022-547517

[0006] However, the above-described conventional techniques have room for further improvement in terms of suppressing temporal fluctuations in the amount of aliasing when temporal anti-aliasing is used.

[0007] For example, when TAA is used, jittering is performed using a jitter pattern that represents the sampling position and its sequence for each time point corresponding to each frame. In addition to a method of obtaining the sampling position each time using random numbers, the jitter pattern generally uses a pseudo-random number sequence such as a Halton sequence.

[0008] However, depending on the combination of this jitter pattern and the magnification ratio in upscaling, there may be a point at which the positional relationship (hereinafter referred to as "phase") between the input pixel and the output pixel becomes aligned, resulting in a local increase in the amount of aliasing at that point.

[0009] For example, to make it easier to understand, if the magnification is 1.0, then in jittering, if there is a point in time when the jitter offset is ±0.0 from the center position of the pixel, then at that point in time, the phase of all pixels in the input image will be aligned with that of all pixels in the output image.

[0010] The above-mentioned conventional techniques do not pay attention to the phase of the input pixel and the output pixel, nor do they mention the jitter pattern itself.Furthermore, the above-mentioned conventional techniques do not have a mechanism for suppressing fluctuations in the amount of aliasing in the output image after upscaling.

[0011] Therefore, the present disclosure proposes an information processing device and an information processing method that can suppress the time fluctuation of the amount of aliasing when temporal anti-aliasing is used.

[0012] In order to solve the above-described problems, an information processing device according to one aspect of the present disclosure includes a control unit. The control unit acquires resolution data representing a relationship between an input resolution and an output resolution of a series of video frames. The control unit renders a current frame of the video frames based on the input resolution. The control unit acquires jitter pattern data representing a jitter pattern used for temporal anti-aliasing. The control unit acquires adjustment data representing an adjustment range within which sampling by the jitter pattern data is limited, based on an expansion factor of the output resolution relative to the input resolution. The control unit performs the temporal anti-aliasing on the current frame based on history frames of the video frames, the jitter pattern data, and the adjustment data.

[0013] FIG. 1 is a schematic explanatory diagram of an information processing device according to an embodiment of the present disclosure. FIG. 2 is an explanatory diagram of a rendering pipeline in a rendering device according to an embodiment of the present disclosure. FIG. 3 is an explanatory diagram of an MFSR. FIG. 4 is an explanatory diagram of a jitter pattern. FIG. 5 is an explanatory diagram of the phase of an input pixel and an output pixel when an enlargement factor is 1.0. FIG. 6 is an explanatory diagram of the phase of an input pixel and an output pixel when an enlargement factor is 2.0. FIG. 7 is an explanatory diagram of the phase of an input pixel and an output pixel when an enlargement factor is 1.5. FIG. 8 is a schematic explanatory diagram of a rendering method according to an embodiment of the present disclosure. FIG. 9 is a block diagram showing an example configuration of a rendering device according to an embodiment of the present disclosure. FIG. 10 is a diagram showing an example of jitter offset adjustment when an enlargement factor is 1.0. FIG. 11 is a diagram showing an example of jitter offset adjustment when an enlargement factor is 2.0. FIG. 12 is a diagram showing an example of jitter offset adjustment when an enlargement factor is 1.5. FIG. 13 is a block diagram showing an example configuration of an SR processing unit. FIG. 14 is a flowchart showing the processing procedure of a jitter offset adjustment process. FIG. 15 is a hardware configuration diagram showing an example of a computer that realizes the functions of a rendering device.

[0014] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0015] In the following description, an information processing device according to an embodiment of the present disclosure (hereinafter referred to as "the present embodiment") is a rendering device 10. An information processing method according to the present embodiment is a rendering method executed by the rendering device 10. In the following description, a video frame to be processed among video frames input to the rendering device 10 is referred to as a "current frame" or simply "current" as appropriate. Past video frames are referred to as "history frames" or simply "history" as appropriate.

[0016] Furthermore, in the following, "temporal anti-aliasing" will be abbreviated as "TAA" where appropriate.

[0017] The present disclosure will be described in the following order: 1. Overview 1-1. Regarding embodiments of the present disclosure 1-2. Regarding issues with existing technology 1-3. Overview of a rendering method according to an embodiment of the present disclosure 2. Example configuration of a rendering device 3. Processing procedure 4. Modification 5. Hardware configuration 6. Conclusion

[0018] <<1. Overview>> <1-1. Regarding an embodiment of the present disclosure> First, the premise for explaining this embodiment will be described. Fig. 1 is an explanatory diagram of an outline of a rendering apparatus 10 according to an embodiment of the present disclosure. Fig. 2 is an explanatory diagram of a rendering pipeline in the rendering apparatus 10 according to an embodiment of the present disclosure. Fig. 3 is an explanatory diagram of an MFSR. Fig. 4 is an explanatory diagram of a jitter pattern.

[0019] 1, the rendering device 10 functions as a rendering engine that renders high-definition game images for various games such as third-person shooters (TPS), first-person shooters (FPS), and online role-playing games (RPGs). The term "rendering engine" may also be read as "game engine."

[0020] The rendering device 10 stores game assets 11b, which are various resource information for expressing the game world for each game title. The game assets 11b include a 3D (three dimensions) model 11ba and various information such as game sound sources. The 3D model 11ba includes various objects in the 3DCG space of the game world.

[0021] The rendering device 10 generates frames of a 3DCG space in a game world including various objects, captured by a virtual camera VC from an arbitrary viewpoint based on input from an operation unit 3 such as a game controller. ｔ , ｔ－１ , ｔ－２ , ... are input as video frames. Of course, the input order is the oldest frame order.

[0022] The rendering device 10 then executes each step of the rendering pipeline described below based on the input video frame, and displays on the display unit 5 an output image that has been upscaled, including super-resolution, according to the resolution of the display unit 5, which is a display device such as a monitor.

[0023] As shown in FIG. 2, the rendering pipeline includes geometry processing, lighting processing, pre-effect processing, SR (Super Resolution) processing, post-effect processing, and UI (User Interface) processing.

[0024] In the rendering pipeline, first, the objects of the 3D model 11ba are rendered by geometry processing, then shading is applied by lighting processing, and then pre-effect processing is performed. In this embodiment, the geometry processing is performed using TAA. In the pre-effect processing, various effects (e.g., sparks) that would take time to render after SR processing, for example, are rendered.

[0025] After the pre-effect processing, anti-aliasing and upscaling and super-resolution according to the magnification are performed in the SR processing. In this embodiment, the SR processing is performed using MFSR. After the SR processing, post-effect processing is performed to render effects that can be applied after the SR processing, and UI processing is performed to superimpose various UIs, resulting in an output image as a final rendering image.

[0026] In the geometry processing and the SR processing, a jitter offset is input from the jitter pattern, i.e., the sampling position of the input pixel at each time point corresponding to each frame of the video frame input to the rendering pipeline is obtained from the jitter pattern and input to the geometry processing and the SR processing.

[0027] To facilitate understanding of the jitter pattern, an overview of the SR process according to this embodiment will be described. The SR process according to this embodiment is performed using MFSR as shown in Fig. 3. For ease of understanding, an example in which the magnification is 1.0 is used.

[0028] In this SR process, a frame that is an input image to be processed ｔ The (current) pixel value is calculated as a statistical value of color data acquired at multiple points in time while shifting the sampling position. For example, in this embodiment, the pixel value at time t is calculated as a weighted average of 32 pieces of color data from time t, time (t-1), ..., time (t-31).

[0029] The frame at this time ｔ , frame ｔ－１ , …, frame ｔ－３１ The jitter offset indicates the sub-pixel sampling position for each pixel in each frame. The jitter offset is the same for all pixels in each frame.

[0030] An example of a jitter pattern is shown in Figure 4. The example in Figure 4 is a jitter pattern with 32 samples per period using a Halton sequence. In this embodiment, the 32 samples correspond to the use of statistical values ​​for 32 frames, as described above. In this way, it is desirable to use pseudo-random numbers or the like to create a jitter pattern that makes the distribution of jitter offset as uniform as possible.

[0031] More specifically, as shown in Fig. 4, the jitter pattern is information in which each jitter offset is mapped onto an XY Cartesian coordinate system that defines a predetermined offset range (-0.50 to +0.50) from the pixel center position CP as the origin. Each jitter offset corresponds to a white circle in the figure. Each jitter offset corresponds to a sampling position at each point in time corresponding to each frame of the input video frame.

[0032] The physical quantities indicated by the X and Y axes are offset amounts at the sub-pixel level when the length and width of one pixel are each set to 1. The jitter pattern is associated with a sequence of jitter offsets: "1st," "2nd," "3rd," .... The jitter offsets are referenced and acquired in this sequence.

[0033] Returning to the explanation of Fig. 3, the MFSR performs upscaling on the input image in which pixel values ​​have been calculated as a result of TAA performed using such a jitter pattern, and also performs inference of a super-resolution result using a DNN. The difference between the inferred super-resolution result and a ground truth (GT) image is then compared, and the difference is fed back to the DNN as recursive information.

[0034] <1-2. Issues with Existing Technology> However, with the existing technology using jittering described above, depending on the combination of the jitter pattern and the magnification ratio in upscaling, there will be a point in time when the phases of the input pixel and the output pixel become aligned, resulting in a local increase in the amount of aliasing at that point in time.

[0035] Fig. 5 is a diagram illustrating the phases of input pixels and output pixels when the enlargement factor is 1.0, Fig. 6 is a diagram illustrating the phases of input pixels and output pixels when the enlargement factor is 2.0, and Fig. 7 is a diagram illustrating the phases of input pixels and output pixels when the enlargement factor is 1.5.

[0036] Although an actual image is two-dimensional, for ease of understanding, the phases of the input image and output image are shown one-dimensionally in Figures 5 to 7. The dashed double-headed arrow indicates the possible range of jitter offset.

[0037] As shown in FIG. 5, when the enlargement magnification is 1.0, if the jitter offset is ±0, the input pixel Px ｉ is the output pixel Px ｏ As described above, since the jitter offset is the same value for all pixels, all input pixels will be in phase with all output pixels. This causes a local increase in the amount of aliasing at the point in time corresponding to the jitter offset.

[0038] Furthermore, as shown in FIG. 6, when the enlargement magnification is 2.0 times and the jitter offset is −0.25, the input pixel Px ｉ and output pixel Px ｏ１ Furthermore, when the jitter offset is +0.25, the input pixel Px ｉ and output pixel Px ｏ２ Therefore, when the magnification is 2.0, 50% of the input pixels will be in phase with the output pixels. This causes a local increase in the amount of aliasing at the point in time corresponding to the jitter offset.

[0039] Also, as shown in FIG. 7, when the enlargement magnification is 1.5 times and the jitter offset is −0.5, the input pixel Px ｉ１ and output pixel Px ｏ１ Furthermore, when the jitter offset is +0.25, the input pixel Px ｉ１ and output pixel Px ｏ２ The phases are aligned.

[0040] Furthermore, when the jitter offset becomes −0.25, the input pixel Px ｉ２ and output pixel Px ｏ３ Furthermore, when the jitter offset is +0.5, the input pixel Px ｉ２ and output pixel Px ｏ４ Therefore, when the magnification is 1.5, 25% of the input pixels will be in phase with the output pixels, which will locally increase the amount of aliasing at the point corresponding to the jitter offset.

[0041] It is possible that such temporal fluctuations in the amount of aliasing may not become apparent depending on the magnification ratio, but at least in the cases where the magnification ratios are 1.0, 2.0, and 1.5, it is desirable to suppress such temporal fluctuations in the amount of aliasing.

[0042] <1-3. Overview of Rendering Method According to an Embodiment of the Present Disclosure> In the rendering method according to the present embodiment, the rendering device 10 acquires resolution data representing the relationship between the input resolution and the output resolution of a series of video frames. The rendering device 10 then renders a current frame of the video frames based on the input resolution. The rendering device 10 also acquires a jitter offset representing a jitter pattern used for TAA. The rendering device 10 also acquires an adjustment range R (see FIGS. 10 to 12 ) within which sampling due to the jitter offset is limited, based on the magnification factor of the output resolution relative to the input resolution. The control unit 12 then performs TAA on the current frame based on the history frames of the video frames, the jitter offset, and the adjustment range R.

[0043] Fig. 8 is an explanatory diagram illustrating an overview of a rendering method according to an embodiment of the present disclosure. Note that Fig. 8 is an explanatory diagram corresponding to Fig. 2. As shown in Fig. 8, in the rendering method according to this embodiment, the rendering device 10 executes a jitter offset adjustment process in a rendering pipeline, which adjusts the jitter offset acquired from the jitter pattern in accordance with the enlargement factor.

[0044] In the jitter offset adjustment process, the rendering apparatus 10 sets an adjustment range for the jitter pattern shown in FIG. 4, which is a range in which the mapping of the jitter offset (i.e., sampling by the jitter offset) is limited according to the enlargement factor.

[0045] This adjustment range is set and acquired according to each magnification ratio based on the jitter offset at which the phases of the input pixel and output pixel are aligned, as described with reference to Figures 5 to 7. Then, in the jitter offset adjustment process, the rendering device 10 adjusts the jitter offset by adding a predetermined offset amount to the jitter offset included in this adjustment range so that it falls outside the adjustment range. The adjusted jitter offset is then input to the geometry process and the SR process.

[0046] This eliminates the time point at which the phases of the input and output pixels are aligned, and suppresses the temporal fluctuation of the amount of aliasing when TAA is used, thereby enabling the output of an image with high stability in the time direction.

[0047] As described above, in the rendering method according to this embodiment, the rendering device 10 acquires resolution data representing the relationship between the input resolution and the output resolution of a series of video frames. The rendering device 10 then renders a current frame of the video frames based on the input resolution. The rendering device 10 also acquires a jitter offset representing a jitter pattern used for TAA. The rendering device 10 also acquires an adjustment range R within which sampling due to the jitter offset is limited, based on the magnification factor of the output resolution relative to the input resolution. The control unit 12 then performs TAA on the current frame based on the history frames of the video frames, the jitter offset, and the adjustment range R.

[0048] Therefore, according to the rendering method of this embodiment, it is possible to suppress the time fluctuation of the amount of aliasing when TAA is used. Below, a more specific description will be given of an example configuration of a rendering device 10 to which the rendering method of this embodiment is applied.

[0049] <<2. Configuration Example of Rendering Device>> Fig. 9 is a block diagram showing a configuration example of a rendering device 10 according to an embodiment of the present disclosure. Note that in Fig. 9 and Fig. 13 shown later, only components necessary for explaining this embodiment are shown in functional blocks, and descriptions of general components are omitted.

[0050] In addition, in the description using FIG. 9 and FIG. 13, the description of components that have already been described will be appropriately simplified or omitted.

[0051] 9 , the rendering device 10 includes a storage unit 11 and a control unit 12. The rendering device 10 is also connected to an operation unit 3 and a display unit 5. The operation unit 3 is a component that accepts input operations from a user. As described above, the operation unit 3 is realized by, for example, a game controller, but may also be realized by a keyboard, a pointing device, or the like.

[0052] The display unit 5 is a component that displays visual information such as an output image output from the rendering device 10. The display unit 5 is realized by a display or the like. Note that the operation unit 3 and the display unit 5 may be integrated into one unit, for example, by a touch panel display or the like.

[0053] The storage unit 11 is realized by a storage device such as a random access memory (RAM), a read only memory (ROM), a flash memory, or a hard disk drive (HDD).

[0054] In the example of Figure 9, the memory unit 11 stores a game engine program 11a, game assets 11b, pre-adjustment jitter pattern information 11c, resolution information 11d, post-adjustment jitter pattern information 11e, DNN model information 11f, and history information 11g.

[0055] The game engine program 11a stores a game engine program including a rendering program corresponding to the program according to this embodiment. The control unit 12, which will be described later, reads and executes this game engine program 11a, thereby functioning as at least a rendering engine.

[0056] As described above, the game assets 11b are various resource information for expressing the game world for each game title. The game assets 11b include the 3D model 11ba and various information such as game sound sources. As described above, the 3D model 11ba includes various objects in the 3DCG space of the game world.

[0057] The pre-adjustment jitter pattern information 11c is information that includes the jitter pattern shown in Fig. 4. The jitter pattern in the pre-adjustment jitter pattern information 11c is, for example, set in advance. Note that the jitter pattern in the pre-adjustment jitter pattern information 11c may be linked to the game engine program 11a or to the game asset 11b for each game title.

[0058] The resolution information 11d is information relating to the resolution according to the specifications of the display unit 5, and corresponds to an example of resolution data representing the relationship between the input resolution and the output resolution. The resolution information 11d includes an enlargement factor of the output resolution relative to the input resolution. The resolution information 11d is set based on information relating to the specifications of the display unit 5 input from the display unit 5, etc.

[0059] The adjusted jitter pattern information 11e is information including a jitter pattern adjusted by a jitter offset adjustment processing unit 12c (described later) based on the resolution information 11d. A geometry processing unit 12d and an SR processing unit 12g (described later) perform jittering based on the adjusted jitter offset included in the adjusted jitter pattern information 11e.

[0060] The DNN model information 11f is information including a DNN model used in the SR processing unit 12g, which will be described later. This DNN model is a deep learning model that learns and infers super-resolution of the current image. This DNN model is preferably realized as a deep learning model trained using a super-resolution image, which is the output when the current image and history are input, and high-definition training data paired with the super-resolution image, so as to minimize the loss between them by a steepest re-descent method or the like.

[0061] The history information 11g stores various information related to the history. For example, the history information 11g stores the frame to be processed. ｔ corresponding frame ｔ－１ ~Frame ｔ－３１ Frame containing ｔ－１ ~Frame ｔ－３１ History information about the

[0062] The control unit 12 controls each unit of the rendering device 10. The control unit 12 is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphical processing unit (GPU), or the like executing a game engine program 11a, which is a program according to this embodiment stored in the storage unit 11, using RAM as a working area. The control unit 12 can also be realized by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0063] The control unit 12 includes a game progress control unit 12a, an acquisition unit 12b, a jitter offset adjustment processing unit 12c, and a geometry processing unit 12d. The control unit 12 also includes a lighting processing unit 12e, a pre-effect processing unit 12f, an SR processing unit 12g, a post-effect processing unit 12h, a UI processing unit 12i, and a display control unit 12j.

[0064] The game progress control unit 12a controls the progress of the game based on input from the operation unit 3. The acquisition unit 12b acquires the input from the operation unit 3. The acquisition unit 12b also acquires frames obtained by capturing a 3DCG space in the game world including various objects using a virtual camera VC from an arbitrary viewpoint position based on the input from the operation unit 3. ｔ , ｔ－１ , ｔ－２ , ... and inputs them sequentially to the rendering pipeline as video frames. The acquisition unit 12b also acquires the pre-adjustment jitter pattern included in the pre-adjustment jitter pattern information 11c.

[0065] The jitter offset adjustment processing unit 12c executes the jitter offset adjustment process described above in accordance with the pre-adjustment jitter pattern included in the pre-adjustment jitter pattern information 11c and the enlargement ratio included in the resolution information 11d.

[0066] Here, Fig. 10 is a diagram showing an example of jitter offset adjustment when the enlargement factor is 1.0, Fig. 11 is a diagram showing an example of jitter offset adjustment when the enlargement factor is 2.0, and Fig. 12 is a diagram showing an example of jitter offset adjustment when the enlargement factor is 1.5.

[0067] When the enlargement magnification is 1.0, the phases of the input and output pixels become more aligned as the jitter offset in the X-axis or Y-axis direction approaches ±0.0 (see FIG. 5). １ , R ２ is set as shown in FIG.

[0068] That is, as shown in FIG. 10, when the magnification is 1.0, the jitter offset adjustment processing unit 12c adjusts the jitter pattern included in the pre-adjustment jitter pattern information 11c within the adjustment range R １ , R ２ Set the adjustment range R １ corresponds to an example of the "first region." ２ corresponds to an example of the “second region.” Each of the adjustment ranges R has a width of 0.10.

[0069] Adjustment range R in the X-axis direction １ is set in the range of −0.05 to +0.05 in the X-axis direction shown in FIG. 10. The adjustment range R ２ is set in the range of −0.05 to +0.05 in the Y-axis direction shown in FIG.

[0070] The jitter offset adjustment processing unit 12c then adjusts the jitter offset within the adjustment range R １ , R ２ After adding an offset of −0.05 or +0.05 to each jitter offset included in １ , R ２ Adjust so that it is not included in

[0071] Furthermore, when the enlargement magnification is 2.0, the phases of the input and output pixels become more aligned as the jitter offset in the X-axis or Y-axis direction approaches −0.25 or +0.25 (see FIG. 6). Therefore, the jitter offset adjustment processing unit 12c adjusts the adjustment range R １１ ~R １４ is set as shown in FIG.

[0072] That is, as shown in FIG. 11, when the enlargement magnification is 2.0, the jitter offset adjustment processing unit 12c adjusts the jitter pattern included in the pre-adjustment jitter pattern information 11c within the adjustment range R １１ ~R １４ Set the adjustment range R １１ ~R １４ corresponds to an example of a "third region."

[0073] Adjustment range R in the X-axis direction １１ is set in the range of −0.30 to −0.20 in the X-axis direction shown in FIG. １２ is set in the range of +0.20 to +0.30 in the X-axis direction shown in FIG. 11. Also, the adjustment range R １３ is set in the range of +0.20 to +0.30 in the Y-axis direction shown in FIG. 11. Similarly, the adjustment range R １４ is set in the range of −0.30 to −0.20 in the Y-axis direction shown in FIG.

[0074] The jitter offset adjustment processing unit 12c then adjusts the jitter offset within the adjustment range R １１ ~R １４ After adding an offset of −0.05 or +0.05 to each jitter offset included in １１ ~R １４ Adjust so that it is not included in

[0075] Furthermore, when the enlargement magnification is 1.5 times, the jitter offset adjustment processing unit 12c adjusts the adjustment range R ２１ ~R ２８ is set as shown in FIG.

[0076] That is, as shown in FIG. 12, when the enlargement factor is 1.5, the jitter offset adjustment processing unit 12c adjusts the jitter pattern included in the pre-adjustment jitter pattern information 11c within the adjustment range R ２１ ~R ２８ Set the adjustment range R ２２ , R ２３ , R ２６ , R ２７ corresponds to an example of the "third region." ２１ , R ２４ , R ２５ , R ２８ corresponds to an example of the "fourth region."

[0077] Adjustment range R in the X-axis direction ２１ is set to a range of −0.45 or less in the X-axis direction shown in FIG. 12. Also, the adjustment range R ２２ is set in the range of −0.30 to −0.20 in the X-axis direction shown in FIG. 12. Similarly, the adjustment range R ２３ is set in the range of +0.20 to +0.30 in the X-axis direction shown in FIG. 12. Similarly, the adjustment range R ２４ is set in the range of +0.45 or more in the X-axis direction shown in FIG.

[0078] In addition, the adjustment range R ２５ is set to a range of +0.45 or more in the X-axis direction shown in FIG. 12. Also, the adjustment range R ２６ is set in the range of +0.20 to +0.30 in the X-axis direction shown in FIG. 12. Similarly, the adjustment range R ２７is set in the range of −0.30 to −0.20 in the X-axis direction shown in FIG. 12. Similarly, the adjustment range R ２８ is set to a range of −0.45 or less in the Y-axis direction shown in FIG.

[0079] The jitter offset adjustment processing unit 12c then adjusts the jitter offset within the adjustment range R ２１ ~R ２８ An offset amount of +0.05 or −0.05 is added to each jitter offset included in the above, and each jitter offset is adjusted so that it does not fall within the adjustment ranges R1 to R4 after adjustment.

[0080] Returning to the description of Fig. 9, the geometry processing unit 12d performs the geometry processing described above. The lighting processing unit 12e performs the lighting processing described above. The pre-effect processing unit 12f performs the pre-effect processing described above. The SR processing unit 12g performs the SR processing described above.

[0081] 13 is a block diagram showing an example of the configuration of the SR processing unit 12g. As shown in FIG. 13, the SR processing unit 12g includes an upscaling unit 12ga, a motion compensation unit 12gb, a DNN 12gc, and a delay unit 12gd.

[0082] The upscaling unit 12ga scales the current to be processed to the output image size in accordance with the adjusted jitter offset. The motion compensation unit 12gb performs motion compensation based on the history and motion vectors.

[0083] The DNN 12gc operates when the DNN model included in the DNN model information 11f is read into the SR processing unit 12g. The DNN 12gc receives the upscaling result from the upscaling unit 12ga and the motion compensation result from the motion compensation unit 12gb as inputs, infers the current super-resolution result, and outputs it. The delay unit 12gd delays the history to synchronize with the next current input.

[0084] Returning to the description of Fig. 9, the post-effect processing unit 12h executes the post-effect processing described above. The UI processing unit 12i executes the UI processing described above. The display control unit 12j displays an output image, which is a rendering image output from the rendering pipeline, on the display unit 5.

[0085] <<3. Processing Procedure>> Next, the processing procedure of the jitter offset adjustment process executed by the rendering device 10 will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the processing procedure of the jitter offset adjustment process.

[0086] 14, the jitter offset adjustment processing unit 12c receives the jitter pattern before adjustment and the enlargement factor included in the resolution information 11d (step S101). Then, the jitter offset adjustment processing unit 12c sets an adjustment range R according to the enlargement factor (step S102). That is, when the enlargement factor is 1.0, the jitter offset adjustment processing unit 12c sets the adjustment range R shown in FIG. １ , R ２ When the enlargement magnification is 2.0, the jitter offset adjustment processing unit 12c sets the adjustment range R １１ ~R １４ When the enlargement magnification is 1.5, the jitter offset adjustment processing unit 12c sets the adjustment range R ２１ ~R ２８ Set.

[0087] Then, the jitter offset adjustment processing unit 12c executes a loop process that is repeated the number of times equal to the jitter offset included in the jitter pattern (step S103).

[0088] In this loop process, the jitter offset adjustment processing unit 12c determines whether the jitter offset to be processed in the loop process is included in the adjustment range R set in step S102 (step S104).

[0089] If the jitter offset is within the adjustment range R (step S104, Yes), the jitter offset adjustment processing unit 12c adjusts the jitter offset (step S105) so that the jitter offset is outside the adjustment range R. If the jitter offset is not within the adjustment range R (step S104, No), the jitter offset adjustment processing unit 12c does nothing and repeats the loop processing.

[0090] When the loop process of step S103 ends, the jitter offset adjustment processing unit 12c outputs an adjusted jitter pattern including each jitter offset adjusted in this loop process (step S106) and stores it in the adjusted jitter pattern information 11e.Then, the jitter offset adjustment processing unit 12c ends the jitter offset adjustment process.

[0091] <<4. Modifications>> Although the embodiment of the present disclosure has been described up to this point, the information processing method according to the present embodiment can be modified in several other ways.

[0092] For example, in this embodiment, a pre-adjusted jitter pattern is adjusted to obtain an adjusted jitter pattern. However, the present invention is not limited to this, and a new jitter pattern may be generated by, for example, generating random numbers by imposing a restriction corresponding to the adjustment range on the range of values ​​that the jitter offset can take.

[0093] Furthermore, in this embodiment, the jitter pattern includes a sequence of each jitter offset, but the sequence may be determined randomly using random numbers or the like.

[0094] Furthermore, in this embodiment, an example has been given in which the rendering device 10 functions as a game engine, but this is merely an example, and it is sufficient if the rendering device 10 functions as a CG renderer that executes at least a rendering pipeline based on input video frames.

[0095] Therefore, the video frames input to the rendering pipeline are not limited to those captured by a virtual camera VC from an arbitrary viewpoint position of the 3DCG space in the game world, but may be any series of consecutive arbitrary viewpoint images of the 3DCG space observed from an arbitrary viewpoint position.

[0096] Furthermore, among the processes described in the above-described embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0097] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0098] The above-described embodiments of the present disclosure can be combined as appropriate within the scope of the present disclosure without causing any contradiction in the processing content. The order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate.

[0099] <<5. Hardware Configuration>> The rendering device 10 according to the embodiment of the present disclosure described above is realized by a computer 1000 configured as shown in Fig. 15, for example. Fig. 15 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the rendering device 10. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0100] The CPU 1100 operates and controls each component based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0101] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0102] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the CPU 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records the programs according to this embodiment.

[0103] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0104] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0105] For example, when the computer 1000 functions as the rendering device 10, the CPU 1100 of the computer 1000 executes a program loaded onto the RAM 1200 to realize the functions of the control unit 12. The secondary storage device 1400 stores the program according to the present disclosure and data in the storage unit 11. The CPU 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0106] <<6. Conclusion>> As described above, according to one embodiment of the present disclosure, the rendering device 10 (corresponding to an example of an "information processing device") includes a control unit 12. The control unit 12 acquires resolution information 11d (corresponding to an example of "resolution data") representing the relationship between the input resolution and the output resolution of a series of video frames. The control unit 12 also renders a current frame of the video frames based on the input resolution. The control unit 12 also acquires a jitter offset (corresponding to an example of "jitter pattern data") representing a jitter pattern used for TAA. The control unit 12 also acquires an adjustment range R (corresponding to an example of "adjustment data representing an adjustment range") within which sampling by the jitter offset is limited, based on the magnification factor of the output resolution relative to the input resolution. The control unit 12 also performs TAA on the current frame based on history frames of the video frame, the jitter offset, and the adjustment range R. This makes it possible to suppress temporal fluctuations in the amount of aliasing when TAA is used.

[0107] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure. Furthermore, components of different embodiments and modifications may be combined as appropriate.

[0108] Furthermore, the effects of each embodiment described in this specification are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0109] The present technology can also be configured as follows. (1) An information processing device comprising: a control unit that acquires resolution data representing a relationship between input resolution and output resolution of a series of video frames; renders a current frame of the video frames based on the input resolution; acquires jitter pattern data representing a jitter pattern used for temporal anti-aliasing; acquires adjustment data representing an adjustment range in which sampling using the jitter pattern data is limited based on an expansion factor of the output resolution relative to the input resolution; and performs the temporal anti-aliasing on the current frame based on history frames of the video frames, the jitter pattern data, and the adjustment data. (2) The information processing device described in (1), wherein the jitter pattern data indicates a sampling position for an input pixel in the current frame, and the control unit performs the temporal anti-aliasing by sampling each pixel of the current frame based on the jitter pattern data that varies in the time direction. (3) The information processing device described in (2), wherein, when the jitter pattern data is included in the adjustment range, the control unit adjusts the jitter pattern data so that the jitter pattern data is not included in the adjustment range. (4) The information processing device according to (2) or (3), wherein the jitter pattern data is a jitter offset, and the jitter pattern is information obtained by mapping the jitter offset at each time point corresponding to each frame of the video frame onto a two-dimensional orthogonal coordinate system that defines a predetermined offset amount range from a center position of the input pixel as an origin. (5) The information processing device according to (4), wherein the adjustment range includes at least one of the following that corresponds to the enlargement magnification: a first region having a predetermined width along a vertical axis of the two-dimensional orthogonal coordinate system, a second region having a predetermined width along a horizontal axis of the two-dimensional orthogonal coordinate system, a third region that passes through the center of each quadrant of the two-dimensional orthogonal coordinate system, is parallel to the vertical axis or the horizontal axis, and has the width; and a fourth region along an outer edge of the two-dimensional orthogonal coordinate system.(6) The information processing device according to (5), wherein the offset amount range is a range of offset amounts at a sub-pixel level when the length of a side of the input pixel is 1, and is -0.50 to +0.50, and the width is 0.10. (7) The information processing device according to (5) or (6), wherein the adjustment range includes the first region and the second region when the enlargement magnification is 1.0. (8) The information processing device according to (7), wherein the adjustment range is -0.05 to +0.05 in each of the first region and the second region when the enlargement magnification is 1.0. (9) The information processing device according to any one of (5) to (8), wherein the adjustment range includes the third region when the enlargement magnification is 2.0. (10) The information processing device according to (9), wherein the adjustment range is -0.30 to -0.20 and +0.20 to +0.30 in the third region when the enlargement magnification is 2.0. (11) The information processing device according to any one of (5) to (10), wherein the adjustment range includes the third region and the fourth region when the enlargement ratio is 1.5. (12) The information processing device according to (11), wherein the adjustment range is −0.30 to −0.20 and +0.20 to +0.30 in the third region and is −0.45 or less and +0.45 or more in the fourth region when the enlargement ratio is 1.5. (13) The information processing device according to any one of (3) to (12), wherein the control unit adjusts the jitter pattern data included in the adjustment range in the existing jitter pattern so that it is not included in the adjustment range. (14) The information processing device according to any one of (3) to (12), wherein the control unit generates the jitter pattern by generating random numbers while imposing a limit corresponding to the adjustment range on a range of values ​​that the jitter pattern data can take.(15) The information processing device according to any one of (1) to (14), wherein the control unit executes a rendering pipeline that receives the current frame as an input, the rendering pipeline including at least geometry processing, lighting processing, effect processing, and super-resolution processing, and the control unit performs the temporal anti-aliasing during the geometry processing and the super-resolution processing. (16) An information processing method comprising: acquiring resolution data representing a relationship between an input resolution and an output resolution of a series of video frames; rendering a current frame of the video frames based on the input resolution; acquiring jitter pattern data representing a jitter pattern used for temporal anti-aliasing; acquiring adjustment data representing an adjustment range within which sampling by the jitter pattern data is limited, based on a magnification factor of the output resolution relative to the input resolution; and performing the temporal anti-aliasing on the current frame based on history frames of the video frames, the jitter pattern data, and the adjustment data. (17) A computer-readable non-transitory storage medium storing a program that causes a computer to perform the following operations: acquire resolution data representing the relationship between input resolution and output resolution of a series of video frames; render a current frame of the video frames based on the input resolution; acquire jitter pattern data representing a jitter pattern used for temporal anti-aliasing; acquire adjustment data representing an adjustment range within which sampling by the jitter pattern data is limited based on an expansion factor of the output resolution relative to the input resolution; and perform the temporal anti-aliasing on the current frame based on history frames of the video frames, the jitter pattern data, and the adjustment data.

[0110] 3 Operation unit 5 Display unit 10 Rendering device 11 Memory unit 11a Game engine program 11b Game asset 11ba 3D model 11c Pre-adjusted jitter pattern information 11d Resolution information 11e Adjusted jitter pattern information 11f DNN model information 11g History information 12 Control unit 12a Game progress control unit 12b Acquisition unit 12c Jitter offset adjustment processing unit 12d Geometry processing unit 12e Lighting processing unit 12f Pre-effect processing unit 12g SR processing unit 12ga Upscaling unit 12gb Motion compensation unit 12gc DNN 12gd Delay unit 12h Post-effect processing unit 12i UI processing unit 12j Display control unit

Claims

1. An information processing device comprising: a control unit that acquires resolution data representing the relationship between input resolution and output resolution of a series of video frames; renders a current frame of the video frames based on the input resolution; acquires jitter pattern data representing a jitter pattern used for temporal anti-aliasing; acquires adjustment data representing an adjustment range in which sampling by the jitter pattern data is limited based on an expansion factor of the output resolution relative to the input resolution; and performs the temporal anti-aliasing on the current frame based on history frames of the video frames, the jitter pattern data, and the adjustment data.

2. The information processing device according to claim 1, wherein the jitter pattern data indicates sampling positions for input pixels in the current frame, and the control unit performs the temporal anti-aliasing by sampling each pixel of the current frame based on the jitter pattern data that varies in the time direction.

3. The information processing device according to claim 2, wherein, when the jitter pattern data is included in the adjustment range, the control unit adjusts the jitter pattern data so that the jitter pattern data is not included in the adjustment range.

4. The information processing device of claim 2, wherein the jitter pattern data is a jitter offset, and the jitter pattern is information obtained by mapping the jitter offset at each time point corresponding to each frame of the video frame onto a two-dimensional orthogonal coordinate system that defines a predetermined offset range from the center position of the input pixel as the origin.

5. The information processing device of claim 4, wherein the adjustment range includes at least one of the following, corresponding to the magnification ratio: a first region having a predetermined width along the vertical axis of the two-dimensional orthogonal coordinate system, a second region having the width along the horizontal axis of the two-dimensional orthogonal coordinate system, a third region passing through the center of each quadrant in the two-dimensional orthogonal coordinate system, being parallel to the vertical axis or the horizontal axis, and having the width; and a fourth region along the outer edge of the two-dimensional orthogonal coordinate system.

6. The information processing device according to claim 5, wherein the offset amount range is a range of offset amount at a sub-pixel level when the length of a side of the input pixel is 1, and is -0.50 to +0.50, and the width is 0.

10.

7. The information processing device according to claim 5, wherein the adjustment range includes the first region and the second region when the enlargement ratio is 1.

0.

8. The information processing device according to claim 7, wherein the adjustment range is −0.05 to +0.05 in each of the first area and the second area when the enlargement ratio is 1.

0.

9. The information processing device according to claim 5, wherein the adjustment range includes the third region when the enlargement ratio is 2.

0.

10. The information processing device according to claim 9, wherein the adjustment ranges are -0.30 to -0.20 and +0.20 to +0.30 in the third area when the enlargement ratio is 2.

0.

11. The information processing device according to claim 5, wherein the adjustment range includes the third region and the fourth region when the enlargement ratio is 1.5 times.

12. The information processing device according to claim 11, wherein, when the magnification is 1.5, the adjustment ranges are -0.30 to -0.20 and +0.20 to +0.30 in the third region, and are -0.45 or less and +0.45 or more in the fourth region.

13. The information processing device according to claim 3, wherein the control unit adjusts the jitter pattern data that is included in the adjustment range in the existing jitter pattern so that it is not included in the adjustment range.

14. The information processing device according to claim 3, wherein the control unit generates the jitter pattern by generating random numbers while imposing a restriction corresponding to the adjustment range on the range of values ​​that the jitter pattern data can take.

15. The information processing device according to claim 1, wherein the control unit executes a rendering pipeline that receives the current frame as input, the rendering pipeline including at least geometry processing, lighting processing, effect processing, and super-resolution processing, and the control unit performs the temporal anti-aliasing in the geometry processing and the super-resolution processing.

16. An information processing method comprising: obtaining resolution data representing the relationship between input resolution and output resolution of a series of video frames; rendering a current frame of the video frames based on the input resolution; obtaining jitter pattern data representing a jitter pattern used for temporal anti-aliasing; obtaining adjustment data representing an adjustment range within which sampling by the jitter pattern data is limited based on an expansion factor of the output resolution relative to the input resolution; and performing the temporal anti-aliasing on the current frame based on history frames of the video frames, the jitter pattern data, and the adjustment data.

Images