Frame Temporal Supersampling

Abstract

According to an embodiment of the subject matter described herein, a solution for temporal supersampling of frames is provided. According to the solution, pixels of a target frame are classified into a plurality of pixel categories. A blending weight map for a reference frame of the target frame is determined based at least on the results of the classification, the blending weight map indicating the importance of pixels of the reference frame in blending. The target frame is blended with the reference frame based on the blending weight map to obtain a supersampled frame corresponding to the target frame. With this solution, more stable and accurate supersampling results can be achieved.

Description

[Background technology]

[0001] background Supersampling techniques are widely used in video rendering, especially three-dimensional (3D) rendering, to eliminate flicker and improve the resolution of video frames. The basic idea behind supersampling techniques is to use pixel samples in a previous frame to determine the corresponding pixel samples in the current frame, so as to achieve anti-aliasing and improved image quality. Summary of the Invention

[0002] overview According to embodiments of the subject matter described herein, a solution for temporal supersampling of frames is proposed. In this solution, pixels of a target frame are classified into multiple pixel categories. A blending weight map for a reference frame of the target frame is determined based at least on the results of the classification, and the blending weight map indicates the importance of pixels of the reference frame in blending. The target frame is blended with the reference frame based on the blending weight map to obtain a supersampled frame corresponding to the target frame. Through this solution, more stable and accurate supersampling results can be achieved.

[0003] This Summary is provided to introduce a selection of concepts in a concise form that are further described below in the Detailed Description. It is not intended to identify key features or essential features of the subject matter described herein, nor is it intended to be used to limit the scope of the subject matter described herein. [Brief explanation of the drawings]

[0004] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1]1 illustrates a block diagram of an exemplary environment in which various embodiments of the subject matter described herein may be implemented. [Figure 2] 1 shows a schematic block diagram of a supersampling system in accordance with some implementations of the subject matter described herein. [Figure 3] 1 shows a schematic block diagram of a supersampling system in accordance with some other implementations of the subject matter described herein. [Figure 4] 1 shows a flowchart of a process for temporal supersampling according to some implementations of the subject matter described herein. [Figure 5] 1 shows a schematic block diagram of an electronic device in which various embodiments of the subject matter described herein may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0005] Throughout the drawings, the same or similar reference symbols refer to the same or similar elements.

[0006] Detailed Description of the Embodiments The subject matter described herein will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only and are intended to help those skilled in the art better understand and thereby practice the subject matter described herein, without implying any limitation on the scope of the subject matter described herein.

[0007] As used herein, the term "comprises" and variations thereof should be read as an open term meaning "including, but not limited to." The term "based on" should be read as "based at least in part on." The terms "embodiment" and "one embodiment" should be read as "at least one embodiment." The term "another embodiment" should be read as "at least one other embodiment." Terms such as "first," "second," etc. may refer to different objects or the same object. Other definitions, either explicit or implicit, may be included below.

[0008] The term "model" as used herein refers to a model that learns associations between corresponding inputs and outputs from training data, such that after training, a corresponding output can be generated for a given input. The generation of the model can be based on machine learning techniques. Deep learning (DL) is a type of machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. As used herein, a "model" may also be referred to as a "machine learning model," a "learning model," a "machine learning network," or a "learning network," which are used interchangeably herein.

[0009] In general, machine learning may broadly include three stages: a training stage, a testing stage, and an application stage (also called an interference stage). In the training stage, a given model may be trained using large-scale training data, and parameter values ​​are iteratively updated until the model can obtain consistent interference from the training data that meets the expected goal. Through training, the model may be considered to be able to learn associations between inputs and outputs (also called input-to-output mapping) from the training data. Parameter values ​​of the trained model are determined. In the testing stage, a test input is applied to the trained model to test whether the model can provide accurate output, so as to determine the model's performance. In the interference stage, the model may be used to process actual inputs and determine corresponding outputs based on the parameter values ​​obtained from training.

[0010] Supersampling is a technique for eliminating image flicker and increasing the resolution of video frames. For example, video frames rendered in computer games or generated by other computer programs can sometimes exhibit image artifacts such as aliasing or pixelated edges. Supersampling techniques are widely used to eliminate such image artifacts and generate frames with higher resolution than the original frames.

[0011] Figure 1 shows a block diagram of an exemplary environment 100 in which various embodiments of the subject matter described herein may be implemented. In the environment of Figure 1, a sequence of frames rendered by a render engine 110 is supersampled by a supersampling system 120 to obtain supersampled frames corresponding to each frame. As used herein, a "frame" refers to a display unit in a video or frame sequence that corresponds to an image, also referred to as a video frame.

[0012] Specifically, for a current frame to be supersampled, referred to as a target frame 112, the supersampling system 120 is configured to blend the target frame 112 with one or more past frames 115-1, 115-2, ..., 115-N (collectively or separately referred to as "past frames 115" for purposes of explanation) that precede the target frame 112 to obtain a supersampled frame 122 that corresponds to the target frame 112.

[0013] The supersampled frame 122 may have a higher resolution than the target frame 112. For example, the resolution of the target frame 112 may be 1920 x 1080 pixels, while the resolution of the supersampled frame 122 may be 3840 x 2160 pixels. In supersampling, the basic principle is that for a given pixel in the target frame 112, one or more corresponding past pixels are determined from the past frame 115, and the determined past pixels are blended with pixels of the target frame 112 to obtain pixels of the supersampled frame.

[0014] Some current supersampling techniques use handcrafted rules to determine one or more past pixels corresponding to pixels in each current frame to use, and then blend the past pixels with pixels in the current frame to generate a supersampling result. While these techniques can achieve high frame rates, they still have difficulty avoiding image artifacts such as flickering and ghosting. Other techniques propose training neural networks in an end-to-end manner, where multiple consecutive frames are directly mapped to a supersampled frame relative to the current frame. However, these techniques still have difficulty avoiding the flicker problem, and the amount of data to be processed by the neural network is large, resulting in high computational costs. Additionally, such neural networks operate as "black boxes," making it difficult to design loss functions to achieve a good balance between temporal stability and visual quality of the frames.

[0015] In an exemplary implementation of the subject matter described herein, an improved solution for temporal supersampling of frames is proposed. The solution provides pixel classification-guided supersampling. Specifically, pixels of a target frame are classified into multiple pixel categories. The classification results are used to determine a blending weight map for a reference frame of the target frame, where the blending weight map indicates the importance of pixels of the reference frame in blending. In this manner, the target frame may be blended with the reference frame based on the blending weight map to obtain a supersampled frame corresponding to the target frame. According to the solution of the subject matter described herein, by distinguishing between different pixel categories of the target frame, the contribution of the reference frame in the supersampled frame of the target frame can be selectively determined, and pixels of the reference frame can be blended with pixels of the target frame using different blending methods. In this way, it is feasible to achieve a better balance between temporal stability and image quality of the frame and to produce more stable and accurate supersampling results.

[0016] Certain exemplary implementations of the subject matter described herein are described in more detail below with reference to the accompanying drawings.

[0017] Figure 2 shows a schematic block diagram of a supersampling system in accordance with some implementations of the subject matter described herein. The supersampling system of Figure 2 may be implemented, for example, as supersampling system 120 of Figure 1. As shown in Figure 2, supersampling system 120 includes a classification network 210, a blending weight network 220, and a blending network 230. In some implementations, supersampling system 200 also includes an upsampler 240 and a warper 250.

[0018] The various components of the supersampling system 120 may be implemented in hardware, software, firmware, or any combination thereof. Specific examples of some frames or information are shown in Figures 2 and 3 below, but this is for illustrative purposes only and does not imply any limitations on the specific implementation of the subject matter described herein.

[0019] In implementations of the subject matter described herein, classification network 210 is configured to perform classification of pixels of target frame 112. Target frame 112 may be, for example, the current frame of the video sequence being processed from render engine 110. The resolution of target frame 112 may be the resolution at which it will be rendered by render engine 110. In some implementations, the pixels of target frame 112 may exhibit color information after being rendered by render engine 110.

[0020] In some implementations, to facilitate subsequent processing, the target frame 112 is upsampled by the upsampler 240 to provide an upsampled target frame 202 for pixel classification by the classification network 210. The upsampled target frame 202 may have the same resolution as the generated supersampled frame 122. For example, if the resolution of the target frame 112 is 1920 x 1080 pixels and the resolution of the supersampled frame 122 is 3840 x 2160 pixels, upsampling may double the resolution of the target frame 112. Unlike supersampling techniques, upsampling increases resolution by interpolating pixels in the target frame 112. The target frame 112 may be upsampled using various interpolation methods, such as bicubic interpolation.

[0021] The classification network 210 may output a classification result 212 for the target frame 112, where the classification result 212 indicates a classification of the pixels of the target frame 112 (pixels of the upsampled target frame 202) among a plurality of pixel categories. The blending weight network 220 is configured to determine a blending weight map 222 for the reference frame 232 of the target frame 112 based at least on the classification result 212.

[0022] In embodiments of the subject matter described herein, different types of pixels are considered to have different dependencies on past pixel values ​​for supersampling, and therefore, different blending methods need to be applied. In particular, pixels that may cause image ghosting and image aliasing need to be treated differently. Image artifacts within a frame may be caused by some pixels with large shading variations or visibility changes, and some pixels that cover high-frequency components and are therefore insufficiently sampled. Although the two types of pixels typically exhibit similar pixel value changes (e.g., brightness changes) between frames, they do not need to be treated differently overall during supersampling. For pixels in the image ghosting category, corresponding pixels in past frames contribute little to the current frame, and therefore, pixels from past frames need to be selectively considered. For pixels in the image aliasing category, corresponding pixels from past frames can be accumulated to solve aliasing problems in the current frame. If these pixels are not distinguished from each other, the supersampling result will exhibit "ghosting" artifacts for the first category of pixels and flickering for the second category of pixels. Additionally, the dependency on past pixels is different due to different reasons in causing ghosting or aliasing.

[0023] In an implementation of the subject matter described herein, two tasks are introduced into the supersampling process: pixel classification and blending weight determination. The two tasks are performed by classification network 210 and blending weight network 220, respectively. The results of pixel classification can be used to guide the blending weight determination.

[0024] In some implementations, pixels of the target frame 112 may be classified with respect to ghost pixels and / or aliasing pixels. Specifically, pixels of the target frame 112 (or the upsampled target frame 202) may be divided into one or more aliasing pixel categories and one or more ghost pixel categories. In some implementations, considering that a pixel may be both an aliasing pixel and a ghost pixel (e.g., high-frequency shading variations), the classification network 210 may be configured to perform two classification tasks, namely, image aliasing classification and image ghost classification, to determine the probability that a pixel of the target frame 112 (or the upsampled target frame 202) belongs to the aliasing pixel category and / or the ghost pixel category.

[0025] In some implementations, at least one pixel category associated with aliasing pixels may be predefined, including, for example, an aliasing pixel category and a non-aliasing pixel category, and at least one pixel category associated with ghost pixels may be predefined, including a ghost pixel category and a non-ghost pixel category. For a given pixel in target frame 112 or upsampled target frame 202, classification result 212 indicates the probability that the pixel belongs to at least one pixel category associated with aliasing pixels and the probability that the pixel belongs to at least one pixel category associated with ghost pixels.

[0026] In some implementations, when performing pixel classification based on image aliasing, aliased pixel categories may be further subdivided according to the reasons causing the aliasing. Aliased pixels typically occur in areas of a frame with high-frequency spatial variations. The reasons causing aliased pixels may include geometric boundaries, high-frequency pixel normals, texture, etc. For aliased pixels, more past valid pixels are required to accurately reconstruct the pixel values ​​of the aliased pixels. Categories related to aliased pixels may include at least the following three categories: a geometric aliased pixel category, a texture aliased pixel category, and a non-aliased pixel category. Different aliased pixel categories may require different strategies to accumulate past pixels for blending with pixels of the current frame.

[0027] In some implementations, when performing pixel classification based on image ghosting, the ghost pixel category may be further subdivided according to the reasons causing the ghosting. The categories related to ghost pixels may include at least the following three categories: a visibility-induced ghost pixel category, a shadow ghost pixel category, and a non-aliased pixel category.

[0028] Visibility-induced ghosting, also known as visibility-induced pixel value change, refers to the occlusion of an object presented in a target frame, e.g., the motion vector of a pixel in the target frame points to a different object due to occlusion. Because of the visibility change, the information of the corresponding pixel in the past frame becomes unreliable for the pixel in the current frame. Shadow ghosting, also known as shading-induced pixel value change, refers to ghosting caused by shading variations. For pixels related to such artifacts, the corresponding pixels in the past frame are useful, but their contribution needs to be calculated based on the shading variations.

[0029] For parallel classification based on image aliasing and image ghosting, the classification network 210 may output at least six values ​​for a pixel of the target frame 112 (or upsampled target frame 202), indicating the respective probabilities of the pixel belonging to the six pixel categories. Some pixels may have a high probability in one category associated with aliased pixels and one category associated with ghost pixels, while some other pixels may have a high probability only in one category of aliased pixels or ghost pixels.

[0030] In some implementations, classification network 210 and / or blending weight network 220 may perform their respective functions based on machine learning techniques. For example, classification network 210 may include a classification model trained based on a machine learning model, and blending weight network 220 may include a blending weight model trained based on a machine learning model. In some implementations, considering classification of aliased pixels and ghost pixels, the classification model may include two classifiers, one used to perform classification related to aliased pixels and one used to perform classification related to ghost pixels, respectively. The input of the classification model (or each classifier) ​​includes at least target frame 112 or upsampled target frame 202, and the output includes classification result 212 or at least a portion thereof (in the example of two classifiers). The input of the blending weight model includes at least classification result 212 and target frame 112 or upsampled target frame 202, and the output includes blending weight map 222.

[0031] The models used by the classification network 210 and the blending weight network 220 may be constructed based on various appropriate model structures. By way of example, the classification model may be implemented using a convolutional structure or may have an encoder-decoder structure, where the encoder extracts features of the model input and the decoder determines the classification result 212 based on the extracted features. In some examples, the classification result output by the classification model may have a lower resolution than the upsampled target frame 202, such as the same resolution as the target frame 112. For example, the classification result indicates the classification result of each pixel in the target frame 112 for multiple pixel categories. In this example, the classification result may be upsampled (e.g., through bilinear upsampling) to obtain the classification result 212 having the same resolution as the upsampled target frame 202. As a result, the classification result 212 may indicate the classification of each pixel in the upsampled target frame 202.

[0032] For example, the blending weight model may be implemented based on a multilayer perceptron (MLP). In some implementations, multiple consecutive pixel-wise MLPs may be used as a backbone network to process model inputs (e.g., the classification result 212 and the upsampled target frame 202). The pixel-wise MLP may include multiple pixel-wise fully connected (FC) layers. In this way, corresponding blending weights may be determined for each pixel in a pixel space of the same size as the upsampled target frame 202. For example, an activation function of the MLP, such as a ReLU function or a sigmoid function, may be selected as needed. The last layer of the MLP may be selected as an activation function that can output blending weights in [0, 1], such as a sigmoid function.

[0033] Of course, some examples of classification models and blending weight models are presented here. Different models can be configured according to actual application requirements, as long as they can achieve the functions described herein. The training data of the classification network 210 and the blending weight network 220 models can be determined according to their respective model inputs and outputs. The training of the models will be described in more detail below.

[0034] In embodiments of the subject matter described herein, different tasks in supersampling are performed by different machine learning models. Compared with some end-to-end trained supersampling models, the classification network 210 and the blending weight network 220 may each have their own objectives to be trained. Such a two-step design allows for higher performance and more compact models to be used for each step to complete the corresponding functions (e.g., classification and weight determination), thereby reducing the computational overhead and memory I / O overhead of the overall system. In addition, by using different classifiers to perform classification for aliased pixels and ghost pixels, these classifiers can be optimized using different training objective functions and loss functions to improve classification accuracy. In this way, a good balance between greater temporal stability and frame image quality can be achieved.

[0035] The blending weight map 222 output by the blending weights 220 indicates the importance of pixels of the target frame 112 relative to the reference frame 232 in blending the reference frame 232 with the target frame 112. In some implementations, the reference frame 232 for blending may include a supersampled past frame corresponding to a past frame 115 preceding the target frame 112, such as a supersampled past frame corresponding to the last frame preceding the target frame 112. The supersampled past frame may also be output from supersampling of the past frame 115 by the supersampling system 120. In this manner, the reference frame 232 may be considered to store information of past pixels of past frames. The blending weight map 222 may have the same size as the reference frame 232 and may indicate the weights of corresponding past pixels of the reference frame 232 for blending with the current target frame 112.

[0036] In some implementations, the larger the weight in the blending weight map 222, the greater the contribution of the corresponding pixel in the reference frame 232 to the target frame 112. Of course, in some implementations, the blending weight map 222 may be configured in the opposite manner, if desired. For example, a larger weight indicates a greater contribution of the current pixel in the target frame 112 to the blended supersampled frame 122 and a smaller contribution of the reference frame 232.

[0037] In some implementations, the reference frame 232 is warped by the warper 250 before blending to map each pixel of the reference frame 232 into the frame space of the target frame 112 or the upsampled target frame 202. Thus, each pixel in the warped reference frame 252 is spatially aligned with a pixel of the upsampled target frame 202. Thereafter, pixels in the reference frame 232 that correspond to the target frame 112 or the upsampled target frame 202 refer to pixels at the same spatial locations after the reference frame 232 is warped into the frame space of the target frame 112.

[0038] The blending network 230 is configured to blend the target frame 112 with the reference frame 232 based on the blending weight map 222 to obtain a supersampled frame 122 corresponding to the target frame 112. The supersampled frame 122 not only has a higher resolution but can also achieve anti-aliasing and eliminate image flicker between successive frames, thereby having better image quality. Specifically, in blending, the blending network 230 uses the blending weight map 222 to blend the upsampled target frame 202 with the warped reference frame 252. The blending weight map 222 can be used to weight the warped reference frame 252 (when the weights directly indicate the importance of pixels in the reference frame) or to weight the upsampled target frame 202 (when the weights directly indicate the importance of pixels in the upsampled target frame 202).

[0039] In implementations of the subject matter described herein, using pixel classification guidance, information from past frames can be blended in different ways for different types of pixels during supersampling, thereby obtaining temporal supersampling results with higher visual quality, effectively eliminating image aliasing, and eliminating flickering artifacts.

[0040] In some implementations, additional auxiliary information, in addition to the target frame itself, may be utilized to assist in determining pixel classifications and / or blending weights. Figure 3 shows a schematic block diagram of a supersampling system 120 according to some other implementations of the subject matter described herein.

[0041] In the exemplary implementation of Figure 3, depth information 302 and / or motion information 304 of target frame 112 may be considered in one or more stages of pixel classification and blending weight determination. Alternatively, or in addition, in some implementations, auxiliary information 340 related to one or more past frames may be considered. In Figure 3 and the following description, all three types of information are shown to be considered by classification network 210 and blending weight network 220 for illustrative purposes. However, it should be understood that in other implementations, one or more types of information may be omitted in actual applications.

[0042] Depth information 302 indicates the depth of target frame 112, e.g., the distance between an object presented in target frame 112 and a (virtual) filming camera. Motion information 304 may indicate a motion vector of an object in target frame 112. Depth information 302 and motion information 304 may be provided by render engine 110. In some implementations, for example, in rendering gaming video, depth information 302 and motion information 304 are generally available as metadata for the video. In other implementations, one or both of depth information 302 and motion information 304 may be obtained in other ways, or may be used when only one of them is available.

[0043] The depth information 302 and the motion information 304 may have the same resolution as the target frame 112. In some implementations, the depth information 302 and the motion information 304 may be upsampled by the upsampler 240 to obtain upsampled depth information 312 and upsampled motion information 314. The upsampled depth information 312 and upsampled motion information 314 may have the same resolution as the upsampled target frame 202. In some implementations, in upsampling the motion information 304, a motion vector corresponding to one pixel in the target frame 112 may be expanded to a motion vector of multiple pixels based on the depth information (from one pixel to two pixels, in the example of supersampling from 1920x1080 to 3840x2160) to obtain the upsampled motion information 314.

[0044] The upsampled depth information 312 and the upsampled motion information 314 may be provided as inputs to the classification model of the classification network 210 to jointly determine the classification result for pixels of the upsampled target frame 202. Alternatively, or in addition, the upsampled depth information 312 and the upsampled motion information 314 are provided as inputs to the blending weight model of the blending weight network 220 to jointly determine the blending weight map 222.

[0045] In some implementations, the auxiliary information 340 associated with one or more past frames may indicate statistical information of the past frames. The auxiliary information 340 may be buffered in an auxiliary buffer and updated frame by frame. For example, the motion information 304 and / or classification results 212 of the current target frame 112 may also be used by the auxiliary information updater 330 to update the auxiliary information 340. In some examples, the auxiliary information 340 useful for pixel classification and / or blending weight determination may be determined based on past frames.

[0046] In some implementations, the auxiliary information 340 may count the number of valid pixels in the past pixels of the past frame. For example, a valid pixel counter may be configured to record the number. Here, valid pixels refer to past pixels in the past frame that are valid or useful for the target pixel in the target frame 112. This is because past pixels corresponding to visibility-inducing ghost pixels are not useful. In some implementations, the number of valid pixels may be determined based on the classification result 212 of the current target frame 112. If the classification result 212 indicates that a pixel in the upsampled target frame 202 is classified into the visibility-inducing ghost pixel category (e.g., the probability that it corresponds to this category is higher than the probability that it corresponds to other categories related to ghost pixels), the number of valid pixels is reset to 0. This is because the past pixels are no longer valid for the ghost pixel resulting from the current visibility. If the classification result 212 indicates that a pixel in the upsampled target frame 202 is not classified into the visibility-inducing ghost pixel category, the number of valid pixels is incremented by one. In some implementations, the inverse of the number of effective pixels may be provided as an input to classification network 210 and / or blending weight network 220 .

[0047] In some implementations, for a target pixel in the current target frame 112, the color change range of the past pixel in the past frame and the valid pixel in the target pixel may be counted for the auxiliary information 340. The color change range may be accumulated by the color change of a specific pixel point in multiple consecutive frames. In some implementations, the color change range may indicate the maximum and minimum values ​​corresponding to each color channel. Counting the color change of multiple past frames may help distinguish between aliasing and shading variation. In some implementations, a past color range counter may be set to record the color change range of the valid pixel. If the classification result 212 indicates that a pixel in the upsampled target frame 202 falls into the visibility-induced ghost pixel category, the past color range counter is reset to the color value of the pixel in the upsampled target frame 202. This is because the past pixel is no longer valid for the ghost pixel resulting from the current visibility. For the target frame 112, the color change range counted from the past frame may first be warped to the frame space of the target frame 112. However, the color values ​​of each pixel in the upsampled target frame 202 can also be used to update the color variation range information corresponding to the warped pixel to reflect the color variation range of the effective pixel of the past pixel and the target pixel.

[0048] In some implementations, for a target pixel in the current target frame 112, the depth difference between the target pixel and a previous pixel of a previous frame may be counted in the auxiliary information 340. For example, the pixel-by-pixel depth difference between depth information corresponding to a previous warped reference frame (warped to the frame space of the target frame) and the upsampled depth information 312 may be determined.

[0049] In some implementations, for a target pixel in the current target frame 112, classification results of past pixels of multiple predetermined pixel categories may be counted in the auxiliary information 340. For example, classification results of frames prior to the target frame 112 may be recorded. In some examples, the classification results may include probabilities that each pixel of the past frame (or upsampled past frame) belongs to multiple pixel categories.

[0050] In some implementations, the auxiliary information 340 may be in a space corresponding to a past frame. To facilitate pixel classification and / or blending weight determination applied to the target frame 112, the auxiliary information 340 may be warped by the warper 250 into a frame space corresponding to the target frame 112 or the upsampled target frame 202. Thus, the warped auxiliary information 342 may represent various types of auxiliary information for the target pixel in the past pixel space corresponding to the target frame 112 or the upsampled target frame 202.

[0051] By considering different types of auxiliary information, such as depth information, motion information, etc., pixel classification and blending weight determination can be performed in a more accurate manner. Because auxiliary information 340 is blended with sufficient information in past frames, classification network 210 and blending weight network 220 may not need to process past frames directly. In such an implementation, instead of utilizing all past frames as model input, statistical information extracted from past frames is used as auxiliary information to facilitate pixel classification and blending weight determination. In this way, long-term past information can be more efficiently utilized to support long-term rendering applications. In addition, due to the small amount of model input, the classification model and blending weight model can also be designed as lightweight models with low model processing complexity and therefore low computational and memory overhead.

[0052] In some implementations, the blending weight network 220 may utilize auxiliary information other than depth difference to determine the blending weights. Additionally, unlike the classification network 210, the blending weight network 220 may directly utilize the classification result 212 for the current target frame 112 without utilizing the classification result of a past frame.

[0053] In some examples such as those described above, the classification network 210 may use a machine learning model to perform pixel classification, and the blending weight network 220 may also use a machine learning model to determine blending weights. These models need to be trained with training data. In some implementations, high-resolution frames having the same resolution as the output supersampled frames may be rendered as sample frames using a rendering algorithm. In addition, labeling information corresponding to the sequence of sample frames is also determined to indicate a plurality of predetermined pixel categories and classification results of pixels of the sample frames in the blending weight map. In some implementations, because the classification results and blending weights need to be labeled on a pixel-by-pixel basis, the classification results may be determined based on a set of predetermined rules, and corresponding blending weights may be determined for pixels of different pixel categories.

[0054] Methods for defining the classification result for a sample frame are described below: In some embodiments, the determination of the classification result may be based on depth and / or motion information of the sample frame.

[0055] As mentioned above, pixel categories can be divided into two categories: image aliasing and image ghosting. For each sample frame used for training, we have an aliasing pixel map A, a geometric aliasing pixel map A, and a G , Texture Aliasing Pixel Map A T , Ghost pixel map R, Visibility-induced ghost pixel map RG , and shadow ghost pixel map R S can be determined. Each of these maps has the same resolution as the sample frame to indicate whether each pixel belongs to the corresponding pixel category. Pixels that do not belong to aliasing and ghost pixels can be determined from the aliasing pixel map A and ghost pixel map R, respectively.

[0056] In terms of image ghosting, a ghost pixel refers to a pixel whose previous pixel is no longer valid, and thus, when the previous pixel is used incorrectly, it results in ghosting. As mentioned above, the categories of ghost pixels may include at least a visibility-inducing ghost pixel category, a shadow ghost pixel category, and a non-ghost pixel category.

[0057] For a visibility-induced ghost pixel, to compensate for the object motion, we can calculate its frame-space position in the (f-1)th frame for a pixel (x,y) in the current fth frame, and obtain the label information R of the visibility-induced ghost. G may be determined based on the depth difference. However, to improve robustness, the velocity difference is also taken into consideration because different objects may have different view space velocities. In some implementations, if the depth difference between a pixel (x, y) of the current f-th frame and a pixel at a corresponding position in the (f-1)-th frame exceeds a certain depth threshold, and the velocity difference also exceeds a certain velocity threshold, the pixel (x, y) of the f-th frame may be determined to be a visibility-induced ghost pixel. The decision criterion for visibility-induced ghost pixels is expressed as follows:

number

number

number

number

[0058] In some cases, the above criteria may be unstable at object boundaries, and the depths of pixels at object boundaries may vary widely between successive frames. As a result, to avoid errors caused by depth changes at boundaries, if certain conditions are met, the visibility-induced ghost label may be extended from a pixel (x, y) in the fth frame to its neighboring pixel (x', y') within the frame. That is, the neighboring pixel may also be labeled as a visibility-induced ghost pixel. In some implementations, if the depth difference between a pixel (x, y) in the current fth frame and its neighboring pixel (x', y') exceeds a certain depth threshold and the velocity difference also exceeds a certain velocity threshold, the neighboring pixel may also be determined as a visibility-induced ghost pixel, which may be expressed as follows:

number

[0059] Shadow ghosting refers to ghosting caused by shading changes. One method for detecting rapid shading changes is to compare pixel colors. In some implementations, the difference between the pixel color in the current fth frame and the pixel color in the reference frame is greater than or equal to a predetermined color threshold.

number

number

[0060] In some cases, spatially high frequency textures, normal maps, etc. present in the frame can make the above decision criteria for shadow ghost pixels inaccurate. max cannot represent all samples in a pixel. Therefore, in some implementations, to further filter out unstable color changes of a pixel caused by spatial high-frequency textures, it also compares the overlap, e.g., intersection over union (IOU), between the pixel color in the fth frame and the pixel color in the (f-1)th frame. IOU is used to measure the ratio of the intersection and union of two bounding boxes, indicating the overlap of the two bounding boxes. For example, if the overlap is less than a predetermined color overlap threshold, the pixel in the current fth frame is classified as a shadow ghost pixel R. STherefore, the decision criterion for shadow ghost pixels can be expressed as:

number

number

number

[0061] In some implementations, to avoid small regions being marked as shadow ghosts (which are typically prone to flicker), morphological opening (erosion followed by dilation) may be applied to the shadow ghost label map. The erosion operation refers to shrinking a highlighted region in an image so that the highlighted region decreases, with the resulting image having a smaller highlighted area than the original image. In operation, the adapted means is replacing neighboring regions with a minimum value to reduce the highlighted area. The dilation operation refers to expanding a highlighted region in an image so that the highlighted area increases, with the resulting image having a larger highlighted area than the original image. In operation, the adapted means is replacing neighboring regions with a maximum value to increase the highlighted area.

[0062] In terms of image aliasing, aliased pixels refer to pixels in regions of high frequency spatial variation. As mentioned above, for pixel classification based on image aliasing, categories associated with aliased pixels may include at least a geometric aliased pixel category, a texture aliased pixel category, and a non-aliased pixel category.

[0063] In some implementations, aliasing pixels in a sample frame may be determined based on pixel depth, normal, color variation, etc. In some implementations, if the depth gradient of pixel (x, y) in the current fth frame is greater than a certain depth threshold, pixel (x, y) may be considered an aliasing pixel. In some implementations, a depth range of multiple consecutive sample frames may be observed for each pixel in frame space, represented by, for example, a maximum depth value and a minimum depth value. In the current sample frame, the depth gradient of a pixel may be determined based on the difference between the maximum recorded depth value and the minimum recorded depth value. The depth gradient-based determination criterion may be expressed as follows:

number

number

[0064] In some implementations, alternatively or additionally, if the normal variation of pixel (x, y) in the current fth frame is greater than a certain normal threshold, pixel (x, y) may be considered an aliasing pixel. For example, all normal samples in pixel (x, y)

number

number

number

number

number

number

[0065] In some implementations, alternatively or in addition, if the color variation of pixel (x, y) in the current fth frame is greater than a certain color threshold, for example, if the ratio of the difference between the maximum color value and the minimum color value of the samples in the pixel to the maximum color value is greater than a certain color threshold, the pixel (x, y) may be considered an aliasing pixel. The decision criterion based on the color variation may be expressed as follows:

number

number

number

number

[0066] In some implementations, aliasing pixels may also be subdivided into geometric aliasing pixel categories and texture aliasing pixel categories based on the above equations (7), (8), and (9). In some implementations, pixels in the fth frame that satisfy the above equations (7) or (8) are included in the geometric aliasing pixel map A corresponding to the fth frame. gIn addition, pixels in the fth frame that satisfy one of the above equations (7), (8), and (9) can also be labeled as aliasing pixels to obtain a complete aliasing pixel map A corresponding to the fth frame. Thus, the texture aliasing pixel map A corresponding to the fth frame T Geometry Aliasing Pixel Map A T can be determined by subtracting from the complete aliased pixel map A.

[0067] In some implementations, the labeling information of shadow ghost pixels is calculated based on motion information, which means that visible points are marked. When there is a visibility change (e.g., there is a visibility-induced ghost), the motion vector is no longer valid, and therefore the visibility-induced ghost should not be marked as a shadow ghost. In some cases, the labeling of shadow ghosts may also be affected by image aliasing. For example, if the content of a frame has strong aliasing, the shadow ghost label may become unstable. Therefore, if a pixel in a frame is marked as a visibility-induced ghost or aliasing, the pixel may not be marked as a shadow ghost pixel.

[0068] After the pixel categories are marked, in some embodiments, blending weights corresponding to pixels of different categories may be set as preset values ​​to obtain a ground truth blending weight map for training. In some embodiments, for a certain sample frame, blending weights corresponding to pixels labeled as the ghost pixel category (including the visibility-induced ghost pixel category and the shadow ghost pixel category) may be set to 1 (indicating that the pixel is completely dependent on the current sample frame) or 0 (indicating that the pixel is not dependent on past pixels). The value of 0 or 1 may be determined based on the definition of the blending weight in the blending weight map. In some embodiments, the blending weights of the remaining pixels in the sample frame may be set to the inverse of the number of valid pixels in the past frame determined by the auxiliary information; i.e., the greater the number of valid pixels, the smaller the contribution of the pixel in the current sample frame and the larger the contribution of the past pixels. Alternatively, the blending weights of the remaining pixels in the sample frame may be set to 1 minus the inverse of the number of valid pixels; i.e., the greater the number of valid pixels in the past frame, the larger the contribution of the past pixels. The manner in which these two weights are set may also be determined based on the definition of the blending weights in the blending weight map.

[0069] When the training data is determined, during training, the classification model used by the classification network 210 and the blending weight model used by the blending weight network 220 can be trained separately. For example, the classification model can be trained first, and then the blending weight model can be trained using the fixed classification model. Because the supersampled frame based on the blending weight map is used as the past frame, it affects the next classification, and the two models can be trained iteratively.

[0070] In some implementations, for the classification model used by the classification network 210 during the training process, a cross-entropy loss function may be applied to each pixel of a frame for model training. In some implementations, because ghost pixels and aliased pixels in each frame are always fewer, the cross-entropy loss may be rebalanced based on the ratio of the total number of ghost pixels and aliased pixels to the total number of pixels in the frame. In some implementations, if some pixels in an aliased pixel region are labeled as ghost pixels (i.e., false-positive samples of ghost pixels), the mislabeled ghost pixels will cause significant flicker in those aliased pixel regions. To enable the classification model to enhance temporal stability in this case, the same weight as for ghost pixels may be used to amplify the loss calculated in this case.

[0071] Based on different pixel classifications, the loss function of the blending weight model may include at least three parts: a loss based on geometric aliased pixels, where the geometric aliased pixels are correctly segmented by the classification model and the L1 loss can be calculated directly; a loss that is not applied to aliased pixels that cannot be correctly classified or labeled as false positive samples; and a regular L1 image reconstruction loss for the remaining pixels.

[0072]

[0013] Several techniques for determining training data for the models and several examples of model training processes have been described above. In other implementations, the training data for the classification models and blending weight models may be obtained in other ways according to the actual application, and the models may be trained in any other suitable way. Implementations of the subject matter described herein are not limited in this respect.

[0073] 4 shows a flow diagram of a process 400 for supersampling a frame in accordance with some implementations of the subject matter described herein. Process 400 may be implemented in supersampling system 120 of FIG. 2 or FIG. 3.

[0074] In block 410, the supersampling system 120 classifies the pixels of the target frame into multiple pixel categories.

[0075] In block 420, the supersampling system 120 determines a blending weight map for the reference frame of the target frame based at least on the results of the classification, the blending weight map indicating the importance of pixels of the reference frame in blending.

[0076] In block 440, the supersampling system 120 blends the target frame with the reference frame based on the blending weight map to obtain a supersampled frame that corresponds to the target frame.

[0077] In some implementations, determining the result of the classification includes determining the result of the classification based on at least one of depth information of the target frame, motion information of the target frame, or auxiliary information related to at least one previous frame.

[0078] In some implementations, determining the blending weight map includes determining the blending weight map based on at least one of depth information of the target frame, motion information of the target frame, or auxiliary information associated with at least one past frame.

[0079] In some implementations, the auxiliary information indicates, for a past pixel of at least one past frame corresponding to a target pixel of the target frame, at least one of the following: the number of valid pixels among the past pixels, the color change range of the valid pixels among the past pixel and the target pixel, the depth difference between the target pixel and the past pixel, or the classification result of the past pixel in multiple pixel categories.

[0080] In some implementations, the plurality of pixel categories includes at least one of at least one pixel category associated with aliasing pixels or at least one pixel category associated with ghost pixels, and in some implementations, the classification result indicates at least one of a probability that a pixel of the target frame belongs to at least one pixel category associated with aliasing pixels and a probability that a pixel of the target frame belongs to at least one pixel category associated with ghost pixels.

[0081] In some implementations, the at least one pixel category associated with aliasing pixels includes at least a geometry aliasing pixel category, a texture aliasing pixel category, and a non-aliasing pixel category. In some implementations, the at least one pixel category associated with ghost pixels includes at least a visibility-inducing ghost pixel category, a shadow ghost pixel category, and a non-ghost pixel category.

[0082] In some implementations, pixels of the target frame are classified into a plurality of pixel categories using a classification model, and the blending weight map is determined based on the results of the classification using the blending weight model.

[0083] In some implementations, the training data for training the classification model and the blending weight model includes at least a sample frame having the same resolution as the supersampled frame and labeling information for the sample frame, where the labeling information indicates classification results of pixels of the sample frame in multiple pixel categories and a blending weight map for the sample frame.

[0084] In some implementations, classifying pixels of the target frame into a plurality of pixel categories includes upsampling the target frame to obtain an upsampled target frame having the same resolution as the supersampled frame, and classifying pixels of the upsampled target frame into a plurality of pixel categories.

[0085] In some implementations, the reference frame comprises a supersampled past frame corresponding to a past frame preceding the target frame.

[0086] 5 shows a schematic block diagram of an electronic device in which various embodiments of the subject matter described herein may be implemented. It should be understood that the electronic device 500 shown in FIG. 5 is provided merely as an example, without suggesting any limitation on the functionality and scope of embodiments of the subject matter described herein.

[0087] 5, electronic device 500 is a form of general-purpose computing device. Components of electronic device 500 may include one or more processors or processing devices 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560.

[0088] In some implementations, electronic device 500 may be implemented as a device having computational capabilities, such as a computing device, computing system, server, mainframe, or the like.

[0089] The processing device 510 may be a physical or virtual processor and may perform various processes based on programs stored in the memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to expand the parallel processing capabilities of the electronic device 500. The processing device 510 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a controller, and / or a microcontroller.

[0090] The electronic device 500 typically includes a variety of computer storage media. Such media may be any available media accessible by the electronic device 500, including, but not limited to, volatile and nonvolatile media, or removable and non-removable media. The memory 520 may be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or any combination thereof. The storage device 530 may be any removable or non-removable media and may include a computer-readable medium such as a memory, a flash memory drive, a magnetic disk, or any other medium accessible by the electronic device 500 that can be used to store information and / or data.

[0091] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile memory media. Although not shown in Figure 5, a disk drive for reading from or writing to a removable non-volatile disk and an optical disk drive for reading from or writing to a removable non-volatile optical disk may be provided. In such a case, each drive may be connected to a bus (not shown) by one or more data media interfaces.

[0092] The communication unit 540 facilitates communication with another computing device over a communication medium. Additionally, the functionality of the components of the electronic device 500 may be implemented by a single computing cluster or multiple computing machines that may communicate with each other over a communication connection. Thus, the electronic device 500 may operate in a networked environment using logical connections with one or more other servers, network personal computers (PCs), or further general-purpose network nodes.

[0093] The input device(s) 550 may include one or more of a variety of input devices, such as a mouse, a keyboard, a data import device, etc. The output device(s) 560 may be one or more output devices, such as a display, a data export device, etc. Through the communication unit 540, the electronic device 500 may further communicate, as needed, with one or more external devices (not shown), such as a storage device and a display device, one or more devices that allow a user to interact with the electronic device 500, or any device (such as a network card, a modem, etc.) that allows the electronic device 500 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0094] In some embodiments, instead of being integrated onto a single device, some or all of the components of electronic device 500 may be configured in the form of a cloud computing architecture. In a cloud computing architecture, components may be provided remotely and cooperate to perform the functions described in the subject matter described herein. In some embodiments, cloud computing provides computing, software, data access, and storage services that do not require end users to be aware of the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services over a wide area network (such as the Internet) using appropriate protocols. For example, a cloud computing provider may provide applications over a wide area network, which may be accessed through a web browser or any other computing component. Software or components of a cloud computing architecture and corresponding data may be stored on servers in remote locations. Computing resources in a cloud computing environment may be aggregated or distributed at remote data center locations. A cloud computing infrastructure may provide services through a shared data center but act as a single access point for users. Thus, a cloud computing infrastructure may be utilized to provide the components and functions described herein from a remotely located service provider. Alternatively, they may be provided from a conventional server or installed directly or otherwise on the client device.

[0095] The electronic device 500 may be used to perform resource management in accordance with various embodiments of the subject matter described herein. The memory 520 may include one or more modules having one or more program instructions. These modules may be accessed and executed by the processing unit 510 to perform the functions of the various embodiments described herein. For example, the memory 520 may include a resource management module 522 for performing resource management for a particular processing unit. As shown in FIG. 5 , the electronic device 500 may obtain inputs necessary for resource management through an input device 550 and provide output for resource management through an output device 560. In some embodiments, the electronic device 500 may further receive inputs from other devices (not shown) via a communication unit 540.

[0096] Some exemplary implementations of the subject matter described herein are listed below.

[0097] In an aspect, the subject matter described herein provides a computer-implemented method, the method including: classifying pixels of a target frame into a plurality of pixel categories; determining a blending weight map for a reference frame for the target frame based at least on results of the classification, the blending weight map indicating importance of pixels of the reference frame in blending; and blending the target frame with the reference frame based on the blending weight map to obtain a supersampled frame corresponding to the target frame.

[0098] In some example implementations, determining the result of the classification includes determining the result of the classification based on at least one of depth information of the target frame, motion information of the target frame, or auxiliary information related to at least one past frame.

[0099] In some example implementations, determining the blending weight map includes determining the blending weight map based on at least one of depth information of the target frame, motion information of the target frame, or auxiliary information related to at least one past frame.

[0100] In some exemplary implementations, the auxiliary information indicates, for a past pixel of at least one past frame corresponding to a target pixel of the target frame, at least one of the following: the number of valid pixels among the past pixels, the color change range of the valid pixels among the past pixel and the target pixel, the depth difference between the target pixel and the past pixel, or the classification result of the past pixel in multiple pixel categories.

[0101] In some exemplary implementations, the plurality of pixel categories includes at least one of at least one pixel category associated with aliasing pixels or at least one pixel category associated with ghost pixels, and in some exemplary implementations, the classification result indicates at least one of a probability that a pixel of the target frame belongs to at least one pixel category associated with aliasing pixels and a probability that a pixel of the target frame belongs to at least one pixel category associated with ghost pixels.

[0102] In some exemplary implementations, the at least one pixel category associated with aliasing pixels includes at least a geometry aliasing pixel category, a texture aliasing pixel category, and a non-aliasing pixel category. In some exemplary implementations, the at least one pixel category associated with ghost pixels includes at least a visibility-inducing ghost pixel category, a shadow ghost pixel category, and a non-ghost pixel category.

[0103] In some example implementations, pixels of the target frame are classified into a plurality of pixel categories using a classification model, and the blending weight map is determined based on the results of the classification using the blending weight model.

[0104] In some example implementations, the training data for training the classification model and the blending weight model includes at least a sample frame having the same resolution as the supersampled frame and labeling information for the sample frame, where the labeling information indicates classification results of pixels of the sample frame in multiple pixel categories and a blending weight map for the sample frame.

[0105] In some exemplary embodiments, classifying pixels of the target frame into a plurality of pixel categories includes upsampling the target frame to obtain an upsampled target frame having the same resolution as the supersampled frame, and classifying pixels of the upsampled target frame into a plurality of pixel categories.

[0106] In some exemplary implementations, the reference frame comprises a supersampled past frame corresponding to a past frame preceding the target frame.

[0107] In another aspect, the subject matter described herein provides an electronic device comprising: a processor; and a memory coupled to the processor and including stored instructions that, when executed by the processor, cause the device to perform operations including: classifying pixels of a target frame into a plurality of pixel categories; determining a blending weight map for a reference frame for the target frame based at least on results of the classification, the blending weight map indicating importance of pixels of the reference frame in blending; and blending the target frame with the reference frame based on the blending weight map to obtain a supersampled frame corresponding to the target frame.

[0108] In some example implementations, determining the result of the classification includes determining the result of the classification based on at least one of depth information of the target frame, motion information of the target frame, or auxiliary information related to at least one past frame.

[0109] In some example implementations, determining the blending weight map includes determining the blending weight map based on at least one of depth information of the target frame, motion information of the target frame, and auxiliary information related to at least one past frame.

[0110] In some exemplary implementations, the auxiliary information indicates, for a past pixel of at least one past frame corresponding to a target pixel of the target frame, at least one of the following: the number of valid pixels among the past pixels, the color change range of the valid pixels among the past pixel and the target pixel, the depth difference between the target pixel and the past pixel, or the classification result of the past pixel in multiple pixel categories.

[0111] In some exemplary implementations, the plurality of pixel categories includes at least one of at least one pixel category associated with aliasing pixels or at least one pixel category associated with ghost pixels, and in some exemplary implementations, the classification result indicates at least one of a probability that a pixel of the target frame belongs to at least one pixel category associated with aliasing pixels and a probability that a pixel of the target frame belongs to at least one pixel category associated with ghost pixels.

[0112] In some exemplary implementations, the at least one pixel category associated with aliasing pixels includes at least a geometry aliasing pixel category, a texture aliasing pixel category, and a non-aliasing pixel category. In some exemplary implementations, the at least one pixel category associated with ghost pixels includes at least a visibility-inducing ghost pixel category, a shadow ghost pixel category, and a non-ghost pixel category.

[0113] In some example implementations, pixels of the target frame are classified into a plurality of pixel categories using a classification model, and the blending weight map is determined based on the results of the classification using the blending weight model.

[0114] In some example implementations, the training data for training the classification model and the blending weight model includes at least a sample frame having the same resolution as the supersampled frame and labeling information for the sample frame, where the labeling information indicates classification results of pixels of the sample frame in multiple pixel categories and a blending weight map for the sample frame.

[0115] In some exemplary embodiments, classifying pixels of the target frame into a plurality of pixel categories includes upsampling the target frame to obtain an upsampled target frame having the same resolution as the supersampled frame, and classifying pixels of the upsampled target frame into a plurality of pixel categories.

[0116] In some exemplary implementations, the reference frame comprises a supersampled past frame corresponding to a past frame preceding the target frame.

[0117] In yet another aspect, the subject matter described herein provides a computer program product tangibly stored on a computer storage medium and including computer-executable instructions that, when executed on a device, cause the device to perform operations including classifying pixels of a target frame into a plurality of pixel categories; determining a blending weight map for a reference frame of the target frame based at least on the results of the classification, wherein the blending weight map indicates the importance of pixels of the reference frame in blending; and blending the target frame with the reference frame based on the blending weight map to obtain a supersampled frame corresponding to the target frame.

[0118] In some example implementations, determining the result of the classification includes determining the result of the classification based on at least one of depth information of the target frame, motion information of the target frame, or auxiliary information related to at least one past frame.

[0119] In some example implementations, determining the blending weight map includes determining the blending weight map based on at least one of depth information of the target frame, motion information of the target frame, or auxiliary information related to at least one past frame.

[0120] In some exemplary implementations, the auxiliary information indicates, for a past pixel of at least one past frame corresponding to a target pixel of the target frame, at least one of the following: the number of valid pixels among the past pixels, the color change range of the valid pixels among the past pixel and the target pixel, the depth difference between the target pixel and the past pixel, or the classification result of the past pixel in multiple pixel categories.

[0121] In some exemplary implementations, the plurality of pixel categories includes at least one of at least one pixel category associated with aliasing pixels or at least one pixel category associated with ghost pixels, and in some exemplary implementations, the classification result indicates at least one of a probability that a pixel of the target frame belongs to at least one pixel category associated with aliasing pixels and a probability that a pixel of the target frame belongs to at least one pixel category associated with ghost pixels.

[0122] In some exemplary implementations, the at least one pixel category associated with aliasing pixels includes at least a geometry aliasing pixel category, a texture aliasing pixel category, and a non-aliasing pixel category. In some exemplary implementations, the at least one pixel category associated with ghost pixels includes at least a visibility-inducing ghost pixel category, a shadow ghost pixel category, and a non-ghost pixel category.

[0123] In some example implementations, pixels of the target frame are classified into a plurality of pixel categories using a classification model, and the blending weight map is determined based on the results of the classification using the blending weight model.

[0124] In some example implementations, the training data for training the classification model and the blending weight model includes at least a sample frame having the same resolution as the supersampled frame and labeling information for the sample frame, where the labeling information indicates classification results of pixels of the sample frame in multiple pixel categories and a blending weight map for the sample frame.

[0125] In some exemplary embodiments, classifying pixels of the target frame into a plurality of pixel categories includes upsampling the target frame to obtain an upsampled frame having the same resolution as the supersampled frame, and classifying pixels of the upsampled target frame into a plurality of pixel categories.

[0126] In some exemplary implementations, the reference frame comprises a supersampled past frame corresponding to a past frame preceding the target frame.

[0127] In yet another aspect, the subject matter described herein provides a computer-readable medium having stored thereon computer-executable instructions that, when executed by a device, cause the device to perform one or more exemplary implementations of the methods of the above aspects.

[0128] The functions described herein may be performed, at least in part, by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0129] Program code for implementing the subject methods described herein may be written in any combination of one or more programming languages. The program code may be provided to a general-purpose computer, a special-purpose computer, or a processor or controller of other programmable data processing flow charts such that, when executed by the processor or controller, the functions / acts specified in the flowcharts and / or block diagrams are performed. The program code may be executed completely or partially on the machine, partially on the machine, partially on a remote machine, or completely on a remote machine or server as a stand-alone software package.

[0130] In the context of the present disclosure, a machine-readable medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, flowchart, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, flowchart, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] Furthermore, while operations are shown in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequential order shown, or that all illustrated operations be performed to achieve desired results. In some situations, multitasking and parallel processing may be advantageous. Similarly, while the above description includes details of several specific implementations, these should not be construed as limitations on the scope of the subject matter described herein, but rather as descriptions of features that may be specific to particular implementations. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Rather, various features described in a single embodiment may also be implemented in multiple implementations separately or in any suitable subcombination.

[0132] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. 1. A computer-implemented method comprising: classifying pixels of a target frame into a plurality of pixel categories; determining a blending weight map for a reference frame of the target frame based at least on a result of the classification, the blending weight map indicating importance of pixels of the reference frame in blending; and blending the target frame with the reference frame based on the blending weight map to obtain a supersampled frame corresponding to the target frame; A method comprising:

2. determining the result of the classification, depth information of the target frame; motion information of the target frame; or auxiliary information relating to at least one previous frame; The method of claim 1 , further comprising determining the result of the classification based on at least one of:

3. Determining the blending weight map comprises: depth information of the target frame; motion information of the target frame; or auxiliary information relating to at least one previous frame; The method of claim 1 , comprising determining the blending weight map further based on at least one of:

4. The auxiliary information is, for a past pixel of the at least one past frame corresponding to a target pixel of the target frame, the number of valid pixels among the past pixels; a color variation range of the valid pixel among the past pixel and the target pixel; the depth difference between the target pixel and the previous pixel; or a classification result of the past pixel in the plurality of pixel categories; The method according to claim 2 or 3, wherein the method exhibits at least one of the following:

5. the plurality of pixel categories includes at least one of at least one pixel category associated with aliasing pixels or at least one pixel category associated with ghost pixels; 2. The method of claim 1, wherein the result of the classification indicates at least one of a probability that a pixel of the target frame belongs to the at least one pixel category associated with the aliasing pixel and a probability that a pixel of the target frame belongs to the at least one pixel category associated with the ghost pixel.

6. the at least one pixel category associated with the aliasing pixels includes at least a geometry aliasing pixel category, a texture aliasing pixel category, and a non-aliasing pixel category; and / or The method of claim 5 , wherein the at least one pixel category associated with the ghost pixel includes at least a visibility-inducing ghost pixel category, a shadow ghost pixel category, and a non-ghost pixel category.

7. The method of claim 1 , wherein the pixels of the target frame are classified into the plurality of pixel categories using a classification model, and the blending weight map is determined based on the results of the classification using the blending weight model.

8. 8. The method of claim 7, wherein training data for training the classification model and the blending weight model includes at least a sample frame having the same resolution as the supersampled frame and labeling information for the sample frame, the labeling information indicating classification results of pixels of the sample frame in the plurality of pixel categories and a blending weight map for the sample frame.

9. classifying the pixels of the target frame into the plurality of pixel categories; upsampling the target frame to obtain an upsampled target frame having the same resolution as the supersampled frame; classifying pixels of the upsampled target frame into the plurality of pixel categories; The method of claim 1 , comprising:

10. The method of claim 1 , wherein the reference frame comprises a supersampled past frame corresponding to a past frame preceding the target frame.

11. 1. An electronic device comprising: a processor; a memory coupled to the processor and containing stored instructions, the instructions, when executed by the processor, classifying pixels of a target frame into a plurality of pixel categories; determining a blending weight map for a reference frame of the target frame based at least on a result of the classification, the blending weight map indicating importance of pixels of the reference frame in blending; and blending the target frame with the reference frame based on the blending weight map to obtain a supersampled frame corresponding to the target frame; and causing the device to perform operations including:

12. determining the result of the classification, depth information of the target frame; motion information of the target frame; or auxiliary information relating to at least one previous frame; The device of claim 11 , further comprising: determining the result of the classification based on at least one of:

13. Determining the blending weight map comprises: depth information of the target frame; motion information of the target frame; auxiliary information relating to at least one previous frame; The device of claim 11 , further comprising: determining the blending weight map based on at least one of:

14. The auxiliary information is, for a past pixel of the at least one past frame corresponding to a target pixel of the target frame, the number of valid pixels among the past pixels; a color variation range of the valid pixel among the past pixel and the target pixel; the depth difference between the target pixel and the previous pixel; or a classification result of the past pixel in the plurality of pixel categories; 14. The device according to claim 12 or 13, exhibiting at least one of:

15. A computer program product tangibly stored on a computer storage medium and including computer-executable instructions, said instructions, when executed on a device, classifying pixels of a target frame into a plurality of pixel categories; determining a blending weight map for a reference frame of the target frame based at least on a result of the classification, the blending weight map indicating importance of pixels of the reference frame in blending; and blending the target frame with the reference frame based on the blending weight map to obtain a supersampled frame corresponding to the target frame; a computer program product for causing the device to perform operations including:

Images