Nerve supersampling method and apparatus

Abstract

A supersampling method includes generating a current rendered image frame by performing dither sampling on a three-dimensional (3D) scene based on sub-pixels of low-resolution pixels of the current rendered image frame; generating a current distorted image frame by distorting the previous output image frame based on a motion vector diagram corresponding to a difference between the current rendered image frame and the previous rendered image frame; generating a current shifted image frame by shifting pixels of the current distorted image frame based on a change in a sampling position according to the dither sampling; and generating a current output image frame based on the current rendered image frame and the current shifted image frame.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority of Korean Patent Application No. 10-2023-0105035 filed in the Korean Intellectual Property Office on August 10, 2023, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to image processing, and more particularly, to a supersampling method and apparatus for neural network-based image processing. Background Art

[0004] Three-dimensional (3D) rendering may refer to the field of computer graphics for rendering 3D scenes into two-dimensional (2D) images. For example, 3D rendering may be used in various application areas, such as but not limited to 3D games, virtual reality (VR), animation, movies, etc. Neural networks may be trained based on deep learning to perform reasoning for a desired purpose by mapping input data and output data that may be nonlinearly related to each other. The ability to be trained to generate such mappings may be referred to as the learning ability of a neural network. Neural networks may be used in various technical fields related to image processing. Summary of the invention

[0005] One or more example embodiments of the present disclosure may solve at least some of the above problems and / or disadvantages and / or other disadvantages not described above. In addition, the example embodiments of the present disclosure may not need to overcome the above disadvantages. For example, the example embodiments may not overcome any of the above problems.

[0006] One or more example embodiments of the present disclosure provide a supersampling method and apparatus for neural network-based image processing.

[0007] According to one aspect of the present disclosure, a supersampling method includes: generating a current rendered image frame by performing jitter sampling on a three-dimensional (3D) scene based on sub-pixels of low-resolution pixels of a current rendered image frame; generating a current warped image frame by warping a previous output image frame based on a motion vector map corresponding to a difference between the current rendered image frame and a previous rendered image frame; generating a current shifted image frame by shifting pixels of the current warped image frame based on a change in a sampling position according to the jitter sampling; and generating a current output image frame based on the current rendered image frame and the current shifted image frame.

[0008] According to one aspect of the present disclosure, an electronic device includes a memory storing instructions, a processor communicatively coupled to the memory, and an output device configured to display a previous output image frame and a current output image frame. The processor is configured to execute the instructions to: generate a current rendered image frame by performing jitter sampling on a three-dimensional (3D) scene based on sub-pixels of low-resolution pixels of a current rendered image frame; generate a current warped image frame by warping the previous output image frame based on a motion vector map corresponding to a difference between the current rendered image frame and the previous rendered image frame; generate a current shifted image frame by shifting pixels of the current warped image frame based on a change in a sampling position according to the jitter sampling; and generate a current output image frame based on the current rendered image frame and the current shifted image frame.

[0009] Additional aspects of example embodiments may be set forth in part in the description which follows and, in part, may be obvious from the description, and / or may be learned by practice of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and / or other aspects, features and advantages of certain embodiments of the present disclosure may be more clearly understood from the following description in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a diagram schematically illustrating a neural supersampling process according to an embodiment;

[0012] Figure 2 is a diagram showing an example of a relationship between low-resolution pixels and high-resolution pixels according to an embodiment;

[0013] Figure 3 is a diagram showing an example of performing jitter sampling on sub-pixels of a rendered image frame according to an embodiment;

[0014] Figure 4 is a diagram showing an example of using sampling positions of sub-pixels of a rendered image frame according to an embodiment;

[0015] Figure 5 is a diagram showing an example of an operation of generating a shifted image frame in response to jitter sampling according to an embodiment;

[0016] Figure 6 is a diagram showing an example of a relationship between a change in a sampling position and a shift pattern according to an embodiment;

[0017] Figure 7 is a diagram showing an example of an operation of performing pixel replacement in response to jitter sampling according to an embodiment;

[0018] Figure 8 is a diagram showing an example of an operation of pixel replacement using a pixel set according to an embodiment;

[0019] Fig. 9 is a diagram showing an example of a jitter sampling process using a double period according to an embodiment;

[0020] Fig.10 is a flow chart illustrating a neural supersampling method according to an embodiment; and

[0021] Fig.11 is a diagram showing an example of a configuration of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0022] The following detailed structural and / or functional description is provided only as an example, and various changes and modifications may be made to the examples. Here, the embodiments may not be interpreted as being limited to the present disclosure, and should be understood to include all changes, equivalents, and replacements within the conceptual and technical scope of the present disclosure.

[0023] Although the terms "first", "second", etc. can be used to explain various components, the components are not limited to these terms. These terms can only be used to distinguish one component from another component. For example, within the scope of the present disclosure, a first component can be referred to as a second component, or similarly, a second component can be referred to as a first component.

[0024] It should be noted that if a first component is described as being “connected,” “coupled,” or “engaged” to a second component, a third component may be “connected,” “coupled,” or “engaged” between the first and second components, even though the first component may be directly connected, coupled, or engaged to the second component.

[0025] The singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms "include", "have" and / or "comprises" when used herein indicate the presence of stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0026] References throughout this disclosure to "one embodiment," "an embodiment," "an example embodiment," or similar language may indicate that a particular feature, structure, or characteristic described in connection with the indicated embodiment is included in at least one embodiment of the present solution. Thus, throughout this disclosure, the phrases "in one embodiment," "in an embodiment," "in an example embodiment," and similar language may, but do not necessarily, all refer to the same embodiment.

[0027] As used herein, "at least one of A and B", "at least one of A, B, or C", etc., may include any one or all possible combinations of the items listed together in the corresponding phrase.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. Terms (such as those defined in commonly used dictionaries) may be interpreted as having a meaning consistent with that in the context of the relevant technology, and may not be interpreted as an ideal or overly formal meaning, unless explicitly defined as such herein.

[0029] It should be understood that the specific order or hierarchy of the blocks in the disclosed process / flowchart is an illustration of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of the blocks in the process / flowchart can be rearranged. In addition, some blocks can be combined or omitted. The attached claims present the elements of each block in a sample order and are not intended to be limited to the specific order or hierarchy presented.

[0030] The embodiments of the present invention may be described and illustrated in terms of blocks as shown in the figures, which perform one or more functions described. These blocks may be referred to herein as units or modules, etc., or may be named as devices, logic, circuits, counters, comparators, generators, converters, etc., and may be physically implemented by analog and / or digital circuits including one or more of logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, etc., and may also be implemented or driven by software and / or firmware (configured to perform the functions or operations described herein).

[0031] Hereinafter, embodiments are described with reference to the accompanying drawings. When describing the embodiments with reference to the accompanying drawings, the same reference numerals may refer to the same elements, and repeated descriptions thereof may be omitted for the sake of brevity.

[0032] Figure 1 Schematically illustrates a neural supersampling process according to an embodiment. Figure 1 , the renderer 110 may include a motion vector rendering module 111 and a low-resolution image rendering module 112. The low-resolution image rendering module 112 may render a three-dimensional (3D) virtual scene into a two-dimensional (2D) image. The 2D image may correspond to a video including a plurality of image frames. The 2D rendered image frame at the current time point may be referred to as a current rendered image frame, and the 2D rendered image frame at the previous time point may be referred to as a previous rendered image frame.

[0033] The motion vector rendering module 111 may generate a motion vector diagram showing changes between rendered image frames flowing over time. For example, the motion vector rendering module 111 may generate a motion vector diagram corresponding to the difference between a current rendered image frame and a previous rendered image frame. The motion vector rendering module 111 may generate a motion vector diagram to exclude the effects of jitter sampling. Figure 3 Describes jitter sampling.

[0034] According to an embodiment, the motion vector rendering module 111 may enlarge the motion vector map according to the resolution of the output image frame. For example, the motion vector rendering module 111 may enlarge the motion vector map corresponding to the difference between the current rendered image frame and the previous rendered image frame according to the resolution of the previous output image frame.

[0035] The warping module 120 may warp the output image frame of the neural supersampling model 150 based on the motion vector map. As used herein, the warping result may refer to a warped image frame. The output image frame may include information corresponding to a next time point according to the warping using the motion vector map. For example, when a previous output image frame of the neural supersampling model 150 is warped based on a motion vector map corresponding to a difference between a current rendered image frame and a previous rendered image frame, the previous output image frame may include information corresponding to the current time point. In this way, the warping result may be referred to as a current warped image frame.

[0036] The input module 140 can generate input data for the neural supersampling model 150 based on the processing results of the output image frame of the neural supersampling model 150 by the warping module 120 and the shifting module 130 and the rendered image frame output from the low-resolution image rendering module 112. For example, the input module 140 can generate input data by combining (e.g., in series) the processing results of the output image frame and the rendered image frame. For example, the processing result can be converted from space to depth. The rendered image frame and the conversion result can be combined to generate the input data. For example, according to the space-to-depth conversion, the processing result can be divided into a pixel set corresponding to the low-resolution image, and the rendered image frame and the pixel set can be combined to generate the input data. The space-to-depth conversion module can be used to perform the space-to-depth conversion. The space-to-depth conversion can convert the data into a structure suitable for parallel processing.

[0037] The neural supersampling model 150 may generate an output image frame by performing supersampling on the input data. For example, the neural supersampling model 150 may generate a next output image frame based on input data including a processing result of a previous output image frame and a current rendered image frame.

[0038] In an embodiment, upscaling and anti-aliasing may be achieved through supersampling. Upscaling may refer to an image processing technique for enhancing resolution, which may also be referred to as super-resolution. Aliasing may refer to the phenomenon that when a signal is reconstructed from a sample, the result may be distorted differently from the continuous form of the original signal.

[0039] The neural supersampling model may include a neural network. The neural supersampling model may be pre-trained to generate a high-resolution supersampling result from a low-resolution input image.

[0040] The neural network may be and / or may include a deep neural network (DNN) including a plurality of layers. The DNN may include at least one of a fully connected network (FCN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc. For example, at least some layers included in the neural network may correspond to a CNN, while other layers may correspond to an FCN. In an embodiment, a CNN may be referred to as a convolutional layer and an FCN may be referred to as a fully connected layer. According to an embodiment, the neural supersampling model 150 may include a neural autoencoder including a neural encoder and a neural decoder.

[0041] Neural networks can be trained based on deep learning, and can be configured to perform reasoning for training purposes by mapping input data and output data that may be nonlinearly related to each other. Deep learning may refer to a machine learning technique for solving problems such as, but not limited to, image and / or speech recognition from large data sets. Deep learning can be interpreted as an optimization problem-solving process of finding a point where energy can be minimized while training a neural network using prepared training data. Through supervised and / or unsupervised learning of deep learning, the structure of the neural network and / or weights corresponding to the model can be obtained, and the input data and output data can be mapped to each other through the weights. If the width and depth of the neural network are large enough, the neural network can have a capacity sufficient to achieve a predetermined function. The neural network can learn a sufficiently large amount of training data through an appropriate training process to achieve optimized performance.

[0042] The low-resolution image rendering module 112 may generate a rendered image frame by performing jitter sampling on the 3D scene based on sub-pixels of low-resolution pixels of the rendered image frame. A low-resolution pixel may refer to a pixel of a low-resolution image, and a high-resolution pixel may refer to a pixel of a high-resolution image. A low-resolution pixel may be divided into a plurality of sub-pixels to have a size corresponding to a high-resolution pixel.

[0043] Jitter sampling may refer to a sampling method that can selectively sample sampling points of a 3D scene corresponding to a sub-pixel of each low-resolution pixel. For example, according to jitter sampling, a sampling point of a 3D scene corresponding to a first sub-pixel of a first low-resolution pixel may be sampled at a first time point, and a sampling point of a 3D scene corresponding to a second sub-pixel of the first low-resolution pixel may be sampled at a second time point. As another example, selective sampling may include periodic sampling, non-periodic sampling, random sampling, etc. According to an embodiment, the sampling points may be sampled alternately according to a predetermined period.

[0044] The shift module 130 may generate a shifted image frame by shifting pixels of the distorted image frame based on a change in the sampling position due to jitter sampling. For example, the shift module 130 may generate a current shifted image frame by shifting pixels of the current distorted image frame based on the current sampling position of the jitter sampling. The shift module 130 may generate a shifted image frame by shifting pixels of the distorted image frame according to a shift pattern synchronized with the change in the sampling position due to the jitter sampling. There may be a shift operation corresponding to the position of a corresponding sub-pixel of each low-resolution pixel of the rendered image frame. When one of the sub-pixels is selected as a sampling target according to the jitter sampling, the current shifted image frame may be generated based on a shift operation corresponding to the position of the sampling target in the shift operation.

[0045] The sampling position of the sub-pixel used to determine the pixel value of the low-resolution pixel of the rendered image frame may be changed according to the jitter sampling. In an embodiment, if the shift module 130 is not provided, the processing result (e.g., the distorted image frame) based on the output image frame that does not reflect the change of the sampling position may be input to the neural supersampling model 150. In such an embodiment, in order to improve the supersampling performance of the neural supersampling model 150, the neural supersampling model 150 may learn such changes in the sampling position. By adjusting the processing result (e.g., the distorted image frame) based on the output image frame according to the change in the sampling position, the shift module 130 can improve the performance of the neural supersampling model 150 without such learning.

[0046] The renderer 110, the motion vector rendering module 111, the low-resolution image rendering module 112, the warping module 120, the shifting module 130, and the input module 140 may be implemented by hardware modules and / or software modules. Fig.11 The electronic device 1100) and / or the processor of the electronic device (e.g., Fig.11The processor 1110 of the present invention may perform a supersampling operation according to an embodiment by using the renderer 110, the motion vector rendering module 111, the low-resolution image rendering module 112, the warping module 120, the shifting module 130, the input module 140, and the neural supersampling model 150. The operations of the renderer 110, the motion vector rendering module 111, the low-resolution image rendering module 112, the warping module 120, the shifting module 130, and the input module 140 may be described as operations of an electronic device and / or a processor of the electronic device.

[0047] Figure 2 is a diagram showing an example of a relationship between low-resolution pixels and high-resolution pixels according to an embodiment. Figure 2 , the low-resolution pixel 211 of the low-resolution image 210 may include sub-pixels (e.g., a first sub-pixel 2111, a second sub-pixel 2112, a third sub-pixel 2113, and a fourth sub-pixel 2114). The sizes of the first sub-pixel 2111 to the fourth sub-pixel 2114 may correspond to the size of the high-resolution pixel 221 of the high-resolution image 220. The length ratio in the horizontal or vertical direction may be referred to as a scaling factor. For example, Figure 2 The scaling factor in may be two (2). However, the present disclosure is not limited thereto, and the scaling factor may be set to another value. The size ratio of the area of ​​the low-resolution pixel 211 to the area of ​​the high-resolution pixel 221 may be proportional to the square of the scaling factor. When the scaling factor is two (2), the size ratio of the area of ​​the low-resolution pixel 211 to the area of ​​the high-resolution pixel 221 may be four (4).

[0048] During the rendering process, a 3D scene may be projected onto the low-resolution image 210. For example, a point of the 3D scene may be projected onto a low-resolution pixel (e.g., low-resolution pixel 211) of the low-resolution image 210. The point of the 3D scene projected onto the low-resolution pixel may be referred to as a sampling point. When selecting the sampling point, the center of the first sub-pixel 2111 to the fourth sub-pixel 2114 may be used instead of the center of the low-resolution pixel 211. As described below, other points may also be used instead of the center of the first sub-pixel 2111 to the fourth sub-pixel 2114 according to the dual period.

[0049] The centers of the first to fourth sub-pixels 2111 to 2114 may be selectively used according to jitter sampling. For example, a sampling point of a 3D scene corresponding to the first sub-pixel 2111 may be sampled when a first rendered image frame is generated, a sampling point of a 3D scene corresponding to the second sub-pixel 2112 may be sampled when a second rendered image frame after the first rendered image frame is generated, and a sampling point of a 3D scene corresponding to the third sub-pixel 2113 may be sampled when a third rendered image frame after the second rendered image frame is generated. As described above, jittering in which the sampling point is changed may be performed for the same low-resolution pixel 211.

[0050] Figure 3 is a diagram showing an example of performing jitter sampling on sub-pixels of a rendered image frame according to an embodiment. Figure 3 , the rendered image frame 300 may include low-resolution pixels (e.g., a first low-resolution pixel 310, a second low-resolution pixel 320, and a third low-resolution pixel 330). The first low-resolution pixel 310 may include sub-pixels (e.g., a first sub-pixel 311, a second sub-pixel 312, a third sub-pixel 313, and a fourth sub-pixel 314), the second low-resolution pixel 320 may include sub-pixels (e.g., a first sub-pixel 321, a second sub-pixel 322, a third sub-pixel 323, and a fourth sub-pixel 324), and the third low-resolution pixel 330 may include sub-pixels (e.g., a first sub-pixel 331, a second sub-pixel 332, a third sub-pixel 333, and a fourth sub-pixel 334). Sub-pixels belonging to a certain low-resolution pixel may be referred to as corresponding sub-pixels. For example, the first to fourth sub-pixels 311 to 314 may be referred to as corresponding sub-pixels of the first low-resolution pixel 310, the first to fourth sub-pixels 321 to 324 may be referred to as corresponding sub-pixels of the second low-resolution pixel 320, and the first to fourth sub-pixels 331 to 334 may be referred to as corresponding sub-pixels of the third low-resolution pixel 330. According to the resolution ratio between the high-resolution image and the low-resolution image (e.g., four (4) times), one low-resolution pixel may include a plurality of sub-pixels. The resolution ratio may correspond to the square of the scaling factor.

[0051] The first to fourth sub-pixels 311 to 314, 321 to 324, and 331 to 334 may be divided according to their positions. For example, the first to fourth sub-pixels 311 to 314, 321 to 324, and 331 to 334 may be divided into positions A, B, C, and D, as shown in FIG. Figure 3 As the sampling position changes according to the jitter sampling, sampling of the sub-pixel at the corresponding position may be performed.

[0052] According to an embodiment, the sampling points may be sampled alternately according to a predetermined period and according to jitter sampling. For example, the jitter sampling may be performed in the order of the first sub-pixels 311, 321, and 331 at position A, the fourth sub-pixels 314, 324, and 334 at position D, the third sub-pixels 313, 323, and 333 at position C, the second sub-pixels 312, 322, and 332 at position B, the first sub-pixels 311, 321, and 331 at position A, the fourth sub-pixels 314, 324, and 334 at position D, the third sub-pixels 313, 323, and 333 at position C, and the second sub-pixels 312, 322, and 332 at position B, etc. According to an embodiment, the sampling points may be sampled aperiodically according to jitter sampling. For example, jitter sampling may be performed in the order of the first subpixels 311, 321 and 331 at position A, the fourth subpixels 314, 324 and 334 at position D, the third subpixels 313, 323 and 333 at position C, the second subpixels 312, 322 and 332 at position B, the fourth subpixels 314, 324 and 334 at position D, the third subpixels 313, 323 and 333 at position C, the second subpixels 312, 322 and 332 at position B, and the first subpixels 311, 321 and 331 at position A, and so on.

[0053] The sampling position according to the periodic sampling may be determined based on the frame progress. For example, a frame progress such as i=0, 1, 2, ... may occur, where i may represent a frame number and is an integer greater than or equal to zero (0). In such an example, positions A to D may be assigned to the frame according to the frame number i. For example, the frame number i may be modulo-divided by the number of sub-pixels belonging to each low-resolution pixel (e.g., four (4)), and positions A to D may be assigned according to the result of the modulo operation. The corresponding result may refer to a resolution ratio between the low-resolution image and the high-resolution image. For example, position A may be assigned to a frame having a modulo result of zero (0) (e.g., i%4=0) for frame number i, position D may be assigned to a frame having a modulo result of one (1) (e.g., i%4=1) for frame number i, position C may be assigned to a frame having a modulo result of two (2) (e.g., i%4=2) for frame number i, and position B may be assigned to a frame having a modulo result of three (3) (e.g., i%4=3) for frame number i. That is, sampling of the first sub-pixels 311, 321 and 331 at position A can be performed in frames with frame number i = {0, 4, 8, 12, ...}, sampling of the fourth sub-pixels 314, 324 and 334 at position D can be performed in frames with frame number i = {1, 5, 9, 13, ...}, sampling of the third sub-pixels 313, 323 and 333 at position C can be performed in frames with frame number i = {2, 6, 10, 14, ...}, and sampling of the second sub-pixels 312, 322 and 332 at position B can be performed in frames with frame number i = {3, 7, 11, 15, ...}.

[0054] Although Figure 3 Three (3) low-resolution pixels (e.g., 310 to 330) are shown, each of which has four (4) sub-pixels (e.g., 311 to 314, 321 to 324, and 331 to 334), but it should be understood that the present disclosure is not limited to this. That is, each rendered image frame 300 may include more (e.g., greater than three (3)) or fewer (e.g., less than three (3)) low-resolution pixels, and each low-resolution pixel may include more (e.g., greater than four (4)) or fewer (e.g., less than four (4)) sub-pixels without departing from the scope of the present disclosure.

[0055] Figure 4 is a diagram showing an example of sampling positions of sub-pixels of a rendered image frame using an embodiment. Figure 4 , the low-resolution pixel 410 may include sub-pixels (e.g., sub-pixel 430). According to jitter sampling, sub-pixel centers (e.g., first sub-pixel center 421, second sub-pixel center 431, third sub-pixel center 441, and fourth sub-pixel center 451) may be used instead of the pixel center 411.

[0056] Figure 5 is a diagram showing an example of an operation of generating a shifted image frame in response to jitter sampling according to an embodiment. Figure 5 , reference values ​​of 0 to 3 are marked on the high-resolution pixels of the warped image frame 501. Similar to the sub-pixels of the low-resolution pixels, the high-resolution pixels of the warped image frame 501 may be divided into positions A to D. Instead of dividing the corresponding sub-pixels of one low-resolution pixel into positions A to D, a plurality of high-resolution pixels corresponding to the corresponding sub-pixels may be divided into positions A to D. Reference values ​​of 0 to 3 may be assigned to positions A to D. In a manner similar to jitter sampling, the reference values ​​of 0 to 3 may represent the remainder obtained by dividing the frame number i by the number of corresponding sub-pixels belonging to one low-resolution pixel, that is, the resolution ratio between the low-resolution image (e.g., the rendered image frame) and the high-resolution image (e.g., the output image frame).

[0057] The input pixel 502 may correspond to the result of the shift operation and form a shifted image frame. According to the jitter sampling, the position of the input pixel 502 extracted from the warped image frame 501 may be determined according to the position of the sub-pixel used in the sampling. For example, the position of the input pixel 502 extracted from the warped image frame 501 may be determined according to the remainder obtained by dividing the frame number i by the resolution ratio.

[0058] like Figure 5 As shown, an example is shown in which a reference value of 0 is assigned to position A, a reference value of 1 is assigned to position D, a reference value of 2 is assigned to position C, and a reference value of 3 is assigned to position B. In this case, the input pixel 502 may be extracted according to a shift operation based on a position (e.g., one of positions A to D) having the same reference value as the remainder. For example, when the remainder is 0, a shift operation may be performed to include all high-resolution pixels at position A having a reference value of 0, and when the remainder is 1, a shift operation may be performed to include all high-resolution pixels at position D having a reference value of 1.

[0059] The shift operation may generate a blank area 505. The pixel values ​​of the blank area 505 may be filled with 0 according to zero padding, filled with adjacent values, or filled according to extrapolation. However, the processing method of the blank area 505 is not limited thereto.

[0060] Figure 6 is a diagram showing an example of a relationship between a change in a sampling position and a shift pattern according to an embodiment. The change in the sampling position due to jitter sampling may be synchronized with the shift pattern. Figure 6, the sampling positions in the rendered image frames (e.g., the first rendered image frame 611, the second rendered image frame 612, the third rendered image frame 613, and the fourth rendered image frame 614) may be changed according to the jitter sampling. For example, sampling of position A may be performed when the frame number is equal to zero (0) (e.g., i=0), sampling of position D may be performed when the frame number is equal to one (1) (e.g., i=1), sampling of position C may be performed when the frame number is equal to two (2) (e.g., i=2), and sampling of position B may be performed when the frame number is equal to three (3) (e.g., i=3). Therefore, a shifting pattern synchronized with the change of the sampling position may occur. For example, shifting based on position A may be performed when the frame number is equal to zero (0) (e.g., i=0), shifting based on position D may be performed when the frame number is equal to one (1) (e.g., i=1), shifting based on position C may be performed when the frame number is equal to two (2) (e.g., i=2), and shifting based on position B may be performed when the frame number is equal to three (3) (e.g., i=3). That is, when the frame number is equal to zero (0) (for example, i=0), a shift operation based on position A can be performed on the first warped image frame 621 to determine the first shifted image frame 631; when the frame number is equal to one (1) (for example, i=1), a shift operation based on position D can be performed on the second warped image frame 622 to determine the second shifted image frame 632; when the frame number is equal to two (2) (for example, i=2), a shift operation based on position C can be performed on the third warped image frame 623 to determine the third shifted image frame 633; ​​and when the frame number is equal to three (3) (for example, i=3), a shift operation based on position B can be performed on the fourth warped image frame 624 to determine the fourth shifted image frame 634.

[0061] The same frame number as the current rendered image frame may be used for the current warped image frame and the current shifted image frame generated based on the previous output image frame. That is, a motion vector map based on the difference between the rendered image frame of i=j-1 and the rendered image frame of i=j may be applied to the output image frame of i=j-1 to generate a warped image frame of i=j, where i and j are positive integers greater than zero (0). The shifted image frame of i=j may be generated based on the warped image frame of i=j. The shifted image frame of i=j and the rendered image frame of i=j may be composed of the input data of i=j of the neural supersampling model. The neural supersampling model may generate the output image frame of i=j based on the input data of i=j.

[0062] Figure 7 is a diagram showing an example of an operation of performing pixel replacement in response to jitter sampling according to an embodiment. Figure 7, the pixel value of the low-resolution pixel 710 can be determined according to the jitter sampling using the sub-pixels (e.g., sub-pixel 711) of the low-resolution pixel 710. When K is a remainder obtained by dividing the time i by the number of corresponding sub-pixels belonging to one low-resolution pixel (e.g., four (4)), the sub-pixel-based sampling position can be determined according to the value of K, where K is a positive integer greater than or equal to zero (0). For example, when K=0, the sub-pixel 711 can be used for sampling. When K=1 to K=3, other sub-pixels can be used for sampling.

[0063] The low-resolution pixel 710 may correspond to a pixel of a rendered image frame, and the high-resolution pixel 720 may correspond to a pixel of a shifted image frame. According to an embodiment, in response to jitter sampling, the pixel of the shifted image frame may be replaced with the pixel of the rendered image frame. That is, a target pixel corresponding to the sampling position of the jitter sampling may be selected in the shifted image frame. Similar to the sub-pixels of the rendered image frame, the pixels of the shifted image frame may be classified based on the position of the pixel. For example, similar to the sub-pixels of the rendered image frame, the pixels of the shifted image frame may be classified according to the remainders of K=0 to K=3. When the sub-pixels at the corresponding positions of the low-resolution pixels are selected in the rendered image frame according to the jitter sampling, the high-resolution pixels having the corresponding positions with the corresponding sub-pixels may be selected in the shifted image frame as the target pixels. When the target pixel is selected in the shifted image frame, the target pixel may be replaced with the low-resolution pixel in the rendered image frame.

[0064] The rendered image frame may correspond to the current image information, and the shifted image frame may correspond to the warped result of the previous image information. The shifted image frame may also have the characteristics of the current image information through warping, but may not sufficiently reflect the current image information compared to the rendered image frame corresponding to the actual current image information. The current image information may be additionally reflected in the shifted image frame through pixel replacement.

[0065] Figure 8 is a diagram showing an example of an operation of pixel replacement using a pixel set according to an embodiment. Figure 8 , the warping module 120 may generate a warped image frame based on the output image frame 802 and the motion vector map 803 , and the shifting module 130 may generate a shifted image frame based on the warped image frame.

[0066] The space-to-depth conversion module 811 may perform sub-sampling corresponding to the jitter sampling to divide the shifted image frame into pixel sets 812, each pixel set 812 corresponding to the low-resolution image. The shifted image frame may be classified into the pixel sets 812 according to the remainder obtained by dividing the frame number by the resolution ratio. The pixel replacement module 810 may set a pixel of one of the pixel sets 812 as a target pixel for pixel replacement. A pixel set corresponding to the sub-pixel sampled according to the jitter may be selected. The pixel replacement module 810 may replace the target pixel with a pixel of the rendered image frame 801.

[0067] like Figure 8 As shown, the rendered image frame 801, the output image frame 802, and the pixel set 812 may be processed in a state of being classified into a plurality of channels (eg, red-green-blue (RGB) channels). Figure 8 An example is shown in which a rendered image frame 801, an output image frame 802, and a pixel set 812 are classified into three (3) channels. The output image frame 802 may correspond to the (i-1)th frame. The warped image frame, the shifted image frame, and the rendered image frame 801 may correspond to the i-th frame.

[0068] refer to Figure 7 The description provided may additionally apply to pixel replacement. Figure 1 The description provided may additionally apply to the warping module 120 and the shifting module 130. The pixel replacement module 810 and the space-to-depth conversion module 811 may be implemented by hardware modules and / or software modules. Fig.11 The electronic device 1100) and / or the processor of the electronic device (e.g., Fig.11 The processor 1110 of the electronic device may perform a supersampling operation according to an embodiment by using the pixel replacement module 810 and the space-to-depth conversion module 811. The operations of the pixel replacement module 810 and the space-to-depth conversion module 811 may be described as operations of an electronic device and / or a processor of the electronic device.

[0069] Fig. 9 is a diagram showing an example of a jitter sampling process using a double period according to an embodiment. Fig. 9 , the sub-pixel region 912 of the low-resolution pixel region 911 may be divided into a secondary sub-pixel region 913. The secondary sub-pixels belonging to a certain sub-pixel may be referred to as corresponding secondary sub-pixels. According to the resolution ratio between the high-resolution image and the low-resolution image (e.g., four (4) times), one sub-pixel may include a plurality of corresponding secondary sub-pixels. The secondary sub-pixels corresponding to the secondary sub-pixel region 913 may be used to perform double-period jitter sampling. Fig. 9 An example of the first to sixteenth sampling points based on a double cycle is shown.

[0070] That is, the corresponding sub-pixel of each low-resolution pixel of the rendered image frame can be alternately selected according to the first cycle. Figure 3 When describing the positions A to D, Fig. 9 An example is shown in which sub-pixels are selected in the order of ABCD according to the first period.

[0071] In an embodiment, the sampling points of the 3D scene corresponding to the corresponding secondary sub-pixel of each selected sub-pixel may be sampled alternately according to the second cycle. Therefore, the entire cycle may correspond to the product of the first cycle and the second cycle. A certain sub-pixel may be selected in each first cycle, and the corresponding secondary sub-pixel of the corresponding sub-pixel may be selected in each first cycle×second cycle. The shift operation may be synchronized with the first cycle.

[0072] According to an embodiment, the secondary sub-pixel may be selected based on various sequence algorithms (e.g., Hammersley sequence and Halton sequence) in the sub-pixel region based on the second cycle. When such a sequence algorithm is used in the entire sub-pixel region, sampling biased toward one sub-pixel region may be performed. When such a sequence algorithm is used in a dual cycle, balanced sampling may be performed in the entire sub-pixel region.

[0073] Fig.10 is a flowchart illustrating a neural supersampling method according to an embodiment. According to an embodiment, an electronic device (e.g., Fig.11 The electronic device 1100 generates a current rendered image frame in operation 1010 by performing jitter sampling on a 3D scene based on sub-pixels of low-resolution pixels of the current rendered image frame, generates a current warped image frame in operation 1020 by warping a previous output image frame of a neural supersampling model based on a motion vector map corresponding to a difference between the current rendered image frame and the previous rendered image frame, generates a current shifted image frame in operation 1030 by shifting pixels of the current warped image frame based on a change in a sampling position according to the jitter sampling, and generates a current output image frame based on the current rendered image frame and the current shifted image frame in operation 1040.

[0074] Operation 1010 may include performing dithered sampling by selectively sampling sampling points of the 3D scene corresponding to corresponding sub-pixels of each low-resolution pixel of the current rendered image frame.

[0075] The sampling points may be alternately sampled according to a predetermined period.

[0076] Operation 1030 may include generating a current shifted image frame by shifting pixels of the current warped image frame according to a shift pattern synchronized with a change in a sampling position according to the dithering sampling.

[0077] There is a shift operation corresponding to the position of the corresponding sub-pixel of each low-resolution pixel of the current rendered image frame, and when one of the sub-pixels is selected as a sampling target according to jitter sampling, the current shifted image frame can be generated based on the shift operation corresponding to the position of the sampling target in the shift operation.

[0078] Operation 1010 may include alternately selecting a corresponding sub-pixel of each low-resolution pixel of the currently rendered image frame according to a first cycle, and alternately sampling a sampling point of the 3D scene corresponding to a corresponding secondary sub-pixel of each selected sub-pixel according to a second cycle.

[0079] The previous output image frame and the current output image frame may be generated by a neural supersampling model based on a neural network.

[0080] Operation 1040 may include generating input data by combining the current rendered image frame and the current shifted image frame, and generating a current output image frame by inputting the input data into a neural supersampling model.

[0081] The generation of the input data may include dividing the shifted image frame into pixel sets corresponding to the low-resolution image by performing sub-sampling corresponding to the jitter sampling, and generating the input data by combining the current rendering image frame and the pixel sets.

[0082] The electronic device may select a target pixel in the shifted image frame corresponding to a sampling position of the jittered sample and replace the target pixel with a pixel of the currently rendered image frame.

[0083] The selecting of the target pixel may include dividing the shifted image frame into pixel sets corresponding to the low-resolution image by performing sub-sampling corresponding to the jitter sampling, and setting a pixel of one of the pixel sets as the target pixel.

[0084] Sub-pixels of low-resolution pixels of the currently rendered image frame may have a size corresponding to high-resolution pixels, previously rendered image frames and the currently rendered image frame may correspond to low-resolution images based on the low-resolution pixels, and previously output image frames and the currently rendered image frame may correspond to high-resolution images based on the high-resolution pixels.

[0085] The electronic device may enlarge a motion vector map corresponding to a difference between a current rendered image frame and a previously rendered image frame according to a resolution of a previously output image frame.

[0086] The neural supersampling model may include a neural network.

[0087] In addition, refer to Figures 1 to 9 and Fig.11 The description provided can be applied to Fig.10 Neural supersampling method.

[0088] Fig.11 An example of a configuration of an electronic device according to an embodiment is shown. Fig.11 , the electronic device 1100 may include a processor 1110, a memory 1120, a camera 1130, a storage device 1140, an input device 1150, an output device 1160, and a network interface 1170, which may communicate with each other via a communication bus 1180. For example, the electronic device 1100 may be implemented as at least a part of the following items: for example, mobile devices such as mobile phones, smart phones, personal digital assistants (PDAs), netbooks, tablet computers, laptop computers, etc.; wearable devices such as smart watches, smart bracelets, smart glasses, etc.; computing devices such as desktop computers, servers, etc.; game consoles such as game consoles, portable game consoles, wearable game consoles, etc.; home appliances such as televisions (TVs), smart TVs, refrigerators, etc.; security devices such as door locks, etc.; vehicles such as self-driving cars, smart vehicles, etc.

[0089] The processor 1110 may execute functions and instructions to be executed in the electronic device 1100. For example, the processor 1110 may process instructions stored in the memory 1120 and / or the storage device 1140. The processor 1110 may execute the instructions with reference to Figures 1 to 10 For example, the processor 1110 may generate a current rendered image frame by performing jitter sampling on a 3D scene based on sub-pixels of low-resolution pixels of the current rendered image frame, generate a current warped image frame by warping a previous output image frame of a neural supersampling model based on a motion vector map corresponding to a difference between the current rendered image frame and the previous rendered image frame, generate a current shifted image frame by shifting pixels of the current warped image frame based on a change in a sampling position according to the jitter sampling, and generate a current output image frame by executing the neural supersampling model based on the current rendered image frame and the current shifted image frame.

[0090] The memory 1120 may include a computer-readable storage medium and / or a computer-readable storage device. The memory 1120 may store instructions to be executed by the processor 1110 and may store relevant information while the electronic device 1100 executes software and / or applications.

[0091] Camera 1130 can capture photos and / or videos. Storage device 1140 includes computer-readable storage media and / or computer-readable storage devices. Storage device 1140 can store a greater amount of information over a long period of time than memory 1120. For example, storage device 1140 can be and / or can include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, and / or other non-volatile memory.

[0092] The input device 1150 can receive input from the user in a traditional input mode through a keyboard and a mouse and / or in other input modes (such as but not limited to touch input, voice input, and image input). For example, the input device 1150 may include a keyboard, a mouse, a touch screen, a microphone, or any other device that detects input from the user and sends the detected input to the electronic device 1100. The output device 1160 may provide the output of the electronic device 1100 to the user through a visual channel, an auditory channel, or a tactile channel. The output device 1160 may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides output to the user. For example, the output device 1160 may display an output image frame including a previous output image frame and a current output image frame. The network interface 1170 may communicate with an external device through a wired and / or wireless network.

[0093] The embodiments described herein can be implemented using hardware components, software components and / or combinations thereof. The processing device can be implemented using one or more general or special computers, such as processors, controllers and arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or any other device that can respond and execute instructions in a defined manner. The processing device can run an operating system (OS) and one or more software applications running on the OS. The processing device can also access, store, manipulate, process, and create data in response to the execution of the software. For the purpose of simplicity, the processing device is described in singular form. However, it should be understood that the processing device may include multiple processing elements and multiple types of processing elements. For example, the processing device may include multiple processors, or a single processor and a single controller. In addition, different processing configurations may be implemented, such as but not limited to parallel processors. When a processor is described as performing an operation and the processor is referenced to perform additional operations, multiple operations may be performed by any one or combination of a single processor or multiple processors.

[0094] Software may include computer programs, code segments, instructions, or some combination thereof to independently or collectively instruct or configure a processing device to operate as desired. Software and data may be stored in any type of machine, component, physical or virtual device, or computer storage medium or device that is capable of providing instructions or data to or interpreted by a processing device. Software may also be distributed on network-coupled computer systems so that the software is stored and executed in a distributed manner. Software and data may be stored by one or more non-transitory computer-readable recording media.

[0095] The method according to the example can be recorded in a non-transitory computer-readable medium including program instructions for implementing various operations of the example. The medium can also include data files, data structures, etc., alone or in combination with program instructions. The program instructions recorded on the medium can be program instructions specially designed and constructed for the purpose of the embodiment, or they can be well-known and available types to technicians in the field of computer software. Examples of non-transitory computer-readable media include, but are not limited to, magnetic media, such as hard disks, floppy disks, and tapes; optical media, such as, but not limited to, compact disk read-only memory (CD-ROM) disks and digital video disks (DVDs); magneto-optical media, such as, but not limited to, optical disks; and hardware devices specially configured for storing and executing program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, etc. Examples of program instructions include, but are not limited to, machine codes such as those generated by a compiler and files containing high-level codes that can be executed by a computer using an interpreter.

[0096] The above-described devices may be configured to act as one or more software modules in order to perform the operations of the above-described examples, or vice versa.

[0097] As described above, although examples have been described with reference to limited figures, those skilled in the art may apply various technical modifications and variations based thereon. For example, if the described techniques are performed in a different order and / or if the components in the described systems, architectures, devices, or circuits are combined in a different manner and / or replaced or supplemented by other components or their equivalents, suitable results may be achieved.

[0098] Accordingly, other implementations are within the scope of the following claims.

Claims

1. A supersampling method, comprising: Generating the current rendered image frame by performing jitter sampling on the three-dimensional 3D scene based on sub-pixels of low-resolution pixels of the current rendered image frame; generating a current warped image frame by warping a previous output image frame based on a motion vector map corresponding to a difference between the current rendered image frame and a previous rendered image frame; generating a current shifted image frame by shifting pixels of the current warped image frame based on a change in sampling position according to the jittered samples; as well as A current output image frame is generated based on the current rendered image frame and the current shifted image frame.

2. The supersampling method according to claim 1, wherein: Generating the current rendered image frame includes: The jitter sampling is performed by selectively sampling a plurality of sampling points of the 3D scene corresponding to corresponding sub-pixels of each low-resolution pixel of the current rendered image frame.

3. The supersampling method according to claim 2, further comprising: The plurality of sampling points are sampled alternately based on a predetermined period.

4. The supersampling method according to claim 1, wherein: Generating the current shifted image frame comprises: Pixels of the current warped image frame are shifted based on a shifting pattern synchronized with changes in sampling positions according to the dithering samples.

5. The supersampling method according to claim 1, wherein: A plurality of shift operations correspond to sub-pixel positions of each low-resolution pixel of the currently rendered image frame, and Wherein, generating the current shifted image frame comprises: Based on the jitter sampling, selecting a sampling target from sub-pixels of each low-resolution pixel of the current rendered image frame; and The current shifted image frame is generated based on a shift operation corresponding to the position of the sampling target among the plurality of shift operations.

6. The supersampling method according to claim 1, wherein: Generating the current rendered image frame includes: Based on the first cycle, alternately selecting a corresponding sub-pixel of each low-resolution pixel of the current rendered image frame; and Based on the second period, sampling points of the 3D scene corresponding to the corresponding secondary sub-pixel of each selected corresponding sub-pixel are alternately sampled.

7. The supersampling method according to claim 1, wherein: The supersampling method further comprises: generating the previous output image frame using a neural supersampling model based on a neural network, and Wherein, generating the current output image frame includes: using the neural network-based neural supersampling model to generate the current output image frame.

8. The supersampling method according to claim 1, wherein: Generating the current output image frame comprises: generating input data by combining the current rendered image frame and the current shifted image frame; and The input data is input into a neural network based neural supersampling model.

9. The supersampling method according to claim 8, wherein: Generating the input data comprises: Dividing the current shifted image frame into pixel sets corresponding to a low resolution image by performing sub-sampling based on the jittered samples; and The currently rendered image frame and the set of pixels are combined.

10. The supersampling method according to claim 1, further comprising: Selecting a target pixel in the current shifted image frame based on a sampling position of the jittered sampling; as well as The target pixel is replaced by a pixel of the current rendered image frame.

11. The supersampling method according to claim 10, wherein: Selecting the target pixel comprises: Dividing the current shifted image frame into pixel sets corresponding to a low resolution image by performing sub-sampling based on the jittered samples; and A pixel of one of the sets of pixels is set as a target pixel.

12. The supersampling method according to claim 1, wherein: The sub-pixels of the low-resolution pixels of the current rendered image frame have a size corresponding to the high-resolution pixels, wherein the previously rendered image frame and the currently rendered image frame correspond to low-resolution images based on the low-resolution pixels, and The previous output image frame and the current output image frame correspond to high-resolution images based on the high-resolution pixels.

13. The supersampling method according to claim 1, further comprising: The motion vector map corresponding to the difference between the current rendered image frame and the previous rendered image frame is enlarged based on the resolution of the previous output image frame. 14 . A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, causes the processor to perform the supersampling method according to claim 1 .

15. An electronic device, comprising: Memory, which stores instructions; a processor communicatively coupled to the memory, wherein the processor is configured to execute the instructions to: Generating the current rendered image frame by performing jitter sampling on the three-dimensional 3D scene based on sub-pixels of low-resolution pixels of the current rendered image frame; generating a current warped image frame by warping a previous output image frame based on a motion vector map corresponding to a difference between the current rendered image frame and a previous rendered image frame; generating a current shifted image frame by shifting pixels of the current warped image frame based on a change in sampling position according to the jittered samples; and generating a current output image frame based on the current rendered image frame and the current shifted image frame; and An output device is configured to display the previous output image frame and the current output image frame.

16. The electronic device according to claim 15, wherein: The processor is further configured to execute the instructions to: A plurality of sampling points of the 3D scene corresponding to corresponding sub-pixels of each low-resolution pixel of the current rendered image frame are selectively sampled.

17. The electronic device according to claim 16, wherein: The processor is further configured to execute the instructions to: The plurality of sampling points are sampled alternately based on a predetermined period.

18. The electronic device according to claim 15, wherein: The processor is further configured to execute the instructions to: Pixels of the current warped image frame are shifted based on a shift pattern synchronized with changes in sampling positions according to the dithering samples.

19. The electronic device according to claim 15, wherein: A plurality of shift operations correspond to sub-pixel positions of each low-resolution pixel of the currently rendered image frame, and The processor is further configured to execute the instructions to: Based on the jitter sampling, selecting a sampling target from sub-pixels of each low-resolution pixel of the current rendered image frame; and The current shifted image frame is generated based on a shift operation corresponding to the position of the sampling target among the plurality of shift operations.

20. The electronic device according to claim 15, wherein: The processor is further configured to execute the instructions to: Based on the first cycle, alternately selecting a corresponding sub-pixel of each low-resolution pixel of the current rendered image frame; as well as Based on the second period, sampling points of the 3D scene corresponding to the corresponding secondary sub-pixel of each selected corresponding sub-pixel are alternately sampled.

Images