Upsampling

Abstract

A method and processing module for determining one or more pixel values at respective one or more upsampled pixel positions of a current frame in a sequence of frames. A depth value of a pixel position of a reference frame in the sequence of frames is obtained. For each up-sampled pixel position of the one or more up-sampled pixel positions: (a) obtaining a depth value of the current frame for the up-sampled pixel position; (b) obtaining a motion vector of the up-sampled pixel position to indicate a motion between a reference frame and a current frame of the up-sampled pixel position; (c) identifying one or more of the pixels of the reference frame using the motion vector of the up-sampled pixel positions; (d) determining a weight for each identified pixel of the one or more identified pixels of the reference frame as a function of (i) a depth value of the current frame for the upsampled pixel location, and (ii) a depth value of the location of the identified pixel of the reference frame; (e) determining a pixel value for the up-sampled pixel position using the determined weight for each of the one or more identified pixels.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to UK Patent Application No. 2319652.0, filed on December 20, 2023, the entire disclosure of which is incorporated herein by reference. Technical field

[0003] The present disclosure relates to upsampling. For example, temporal resampling may be used, for instance, to apply upsampling to input pixels of a current frame in a sequence of frames to determine one or more pixel values at corresponding one or more upsampled pixel positions. Upsampling may be used in super - resolution techniques. Background art

[0004] The term'super - resolution' refers to a technique for upsampling an image, for example, by estimating the appearance of a higher - resolution version of the image to enhance the apparent visual quality of the image. When implementing super - resolution, the system will attempt to find a higher - resolution version of the lower - resolution input image that is, to the greatest extent possible, reasonable and consistent with the lower - resolution input image. Super - resolution is a challenging problem because for each patch in the lower - resolution input image, there are a large number of potential higher - resolution patches that could correspond to it. In other words, super - resolution techniques are attempting to solve an ill - posed problem because although solutions exist, they are not unique.

[0005] Super-resolution has important applications. It can be used to increase the resolution of an image, thereby improving the 'quality' of the image perceived by an observer. Additionally, super-resolution can be used as a post-processing step in an image generation process, thereby allowing an image to be generated at a lower resolution (which is typically simpler and faster), while still producing a high-quality, high-resolution image. The image generation process can be, for example, an image capture process using a camera. Alternatively, the image generation process can be an image rendering process in which a computer, such as a graphics processing unit (GPU), renders an image of a virtual scene. Compared to directly rendering a high-resolution image using a GPU, allowing the GPU to render a low-resolution image and then applying super-resolution techniques to upsample the rendered image to produce a high-resolution image has the potential to significantly reduce the GPU's latency, bandwidth, power consumption, silicon area, and / or computational cost. The GPU can implement any suitable rendering technique, such as rasterization or ray tracing. For example, the GPU can render a 960×540 image (i.e., an image having 518,400 pixels arranged in 960 columns and 540 rows), and then the image can be upsampled by a factor of 2 (referred to as '2x upsampling') in both the horizontal and vertical dimensions to produce a 1920×1080 image (i.e., an image having 2,073,600 pixels arranged in 1920 columns and 1080 rows). In this way, to produce a 1920×1080 image, the GPU renders an image with a quarter of the number of pixels. This results in very significant savings during rendering (e.g., in terms of the GPU's latency, power consumption, and / or silicon area), and can, for example, allow a relatively low-performance GPU to render high-quality, high-resolution images within a low power and area budget, provided that an appropriately efficient and high-quality super-resolution implementation is used to perform the upsampling. In other examples, different upsampling factors (other than 2) can be applied. Super-resolution techniques can be applied to a sequence of images (or frames), such as a sequence of frames from a video stream rendered by a graphics processing unit.

[0006] Figure 1An upsampling process for applying upsampling to a sequence of frames is illustrated. An image sequence 102 with a relatively low resolution is processed by a processing module 104 to produce an image series 106 with a relatively high resolution. In some systems, the processing module 104 may be implemented as a neural network to upsample each input image in the input images of the image sequence 102 to produce a corresponding output image in the upsampled image sequence 106. Implementing the processing module 104 as a neural network can produce output images of good quality, but generally requires a high-performance computing system (e.g., having large, powerful processing units and memory) to implement the neural network. Therefore, due to processing time, latency, bandwidth, power consumption, memory usage, silicon area, and computational cost, implementing the processing module 104 as a neural network for performing upsampling of images may not be suitable. These efficiency considerations are particularly important in some devices, such as small battery-powered devices with limited computational and bandwidth resources, such as mobile phones and tablets.

[0007] In some systems, in the case where a sequence of frames from a video stream is available, higher-quality results can be obtained by including samples from multiple input frames when generating each output frame. These methods are referred to as video super-resolution (VSR) and can be implemented using a neural network.

[0008] Some systems do not use a neural network to perform super-resolution on an image (sequence), but instead use a more traditional processing module. For example, some systems divide the problem of upsampling an image into two stages: (i) upsampling and (ii) adaptive sharpening. In these systems, the upsampling stage can be performed inexpensively, for example, using bilinear upsampling, and the adaptive sharpening stage can be used to sharpen the image, i.e., reduce the blur introduced by upsampling. Bilinear upsampling is known in the art and uses linear interpolation of adjacent input pixels in two dimensions to produce output pixels at positions between the input pixels.

[0009] The general objectives of a system implementing super-resolution are: (i) a high-quality output image, i.e., the output image is reasonable to the maximum extent considering the low-resolution input image, (ii) low latency such that the output image is generated quickly, and (iii) a low-cost processing module in terms of resources such as power, bandwidth, and silicon area. SUMMARY OF THE INVENTION

[0010] The present Summary of the Invention is provided to introduce a series of concepts that are further described below in the Detailed Description in a simplified form. The present Summary of the Invention is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0011] A method for determining one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence is provided. The method includes:

[0012] Obtaining depth values of positions of pixels of a reference frame in the frame sequence; and

[0013] For each of the one or more upsampled pixel positions:

[0014] Obtaining a depth value of the current frame for the upsampled pixel position;

[0015] Obtaining a motion vector of the upsampled pixel position to indicate the motion of the upsampled pixel position between the reference frame and the current frame;

[0016] Using the motion vector of the upsampled pixel position to identify one or more pixels among the pixels of the reference frame;

[0017] Determining weights of each of the one or more identified pixels of the reference frame based on: (i) the depth value of the current frame for the upsampled pixel position, and (ii) the depth value of the position of the identified pixel of the reference frame; and

[0018] Using the determined weights for each of the one or more identified pixels to determine the pixel value of the upsampled pixel position.

[0019] The method may further include obtaining pixel values of the one or more identified pixels of the reference frame in the frame sequence. The determining the pixel value of the upsampled pixel position may include performing a weighted sum of the pixel values of the one or more identified pixels of the reference frame, and the performing is carried out using the determined weights for each of the one or more identified pixels in the weighted sum.

[0020] For each of the one or more upsampled pixel positions, the weights of each of the one or more identified pixels of the reference frame may be determined based on the difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame.

[0021] If the difference between the depth value of the current frame at the upsampled pixel position and the depth value of the identified pixel at the position in the reference frame is low, the weight of each of the one or more identified pixels in the reference frame can be high. Similarly, if the difference between the depth value of the current frame at the upsampled pixel position and the depth value of the identified pixel at the position in the reference frame is high, the weight of each of the one or more identified pixels in the reference frame can be low.

[0022] The method may further include, for each upsampled pixel position among the one or more upsampled pixel positions:

[0023] Obtaining a plurality of depth values of the current frame at positions within a region surrounding the upsampled pixel position; and

[0024] Determining a standard deviation of the depth values of the current frame within the region, wherein the weight of each of the one or more identified pixels in the reference frame is further determined based on: (iii) the determined standard deviation of the depth values.

[0025] The determining the weight of each of the one or more identified pixels in the reference frame may include comparing the difference between the depth value of the current frame at the upsampled pixel position and the depth value of the identified pixel at the position in the reference frame with a depth threshold, wherein the depth threshold may be based on the determined standard deviation of the depth values of the current frame within the region.

[0026] In response to determining that the difference between the depth value of the current frame at the upsampled pixel position and the depth value of the identified pixel at the position in the reference frame is greater than the depth threshold, it may be determined that the weight of the identified pixel in the reference image is low.

[0027] The depth threshold may be a hard threshold. The weight w of the identified pixel k in the reference image k may be determined such that w k = w i,k ·(|D ref,k - D curr | ≤ T d ), where T d is the depth threshold, where T d = F depth · σ depth , and where w i,k is the initial weight of the identified pixel in the reference image, D ref,k is the depth value of the identified pixel at the position in the reference frame, D currThe depth value F for the upsampled pixel position for the current frame depth is a predetermined factor, and σ depth is the determined standard deviation of the depth values in the region of the current frame around the upsampled pixel position.

[0028] The depth threshold can be a soft threshold. The weight w of the identified pixel k of the reference image k can be determined such that where T d is the depth threshold, where T d = F depth ·σ depth and where w i,k is the initial weight of the identified pixel of the reference image, D ref,k is the depth value of the position of the identified pixel of the reference frame, D curr is the depth value F for the upsampled pixel position for the current frame depth is a predetermined factor, and σ depth is the determined standard deviation of the depth values in the region of the current frame around the upsampled pixel position.

[0029] Said identifying one or more pixels of the reference frame using the motion vector of the upsampled pixel position may include projecting the upsampled pixel position to a position in the reference frame based on the motion vector, and identifying one or more pixels of the reference frame in the neighborhood of the projected position in the reference frame.

[0030] For each upsampled pixel position among the one or more upsampled pixel positions, said determining the weight of each identified pixel among the one or more identified pixels of the reference frame may include determining an initial weight, and using the initial weight to determine the weight of the identified pixel of the reference frame.

[0031] The initial weight of each identified pixel among the one or more identified pixels of the reference frame can be determined by:

[0032] determining the distance between the projected position and the position of the identified pixel in the reference frame; and

[0033] mapping the distance to an initial weight using a predetermined relationship.

[0034] The predetermined relationship can be a Gaussian relationship or a linear relationship.

[0035] For each of the one or more upsampled pixel positions, the weight of each of the one or more identified pixels in the reference frame can be determined based on the degree to which the identified pixel in the reference frame is an outlier compared to other identified pixels in the reference frame.

[0036] The method may further include, for each of the one or more upsampled pixel positions:

[0037] Obtaining a plurality of input pixel values of the current frame for positions within a region surrounding the upsampled pixel position; and

[0038] Determining a mean value of the input pixel values of the current frame within the region surrounding the upsampled pixel position.

[0039] The determining of the pixel value of the upsampled pixel position may include clamping the determined pixel value such that the pixel value does not differ from the determined mean value of the input pixel values of the current frame within the region surrounding the upsampled pixel position by more than a threshold.

[0040] The method may further include, for each of the one or more upsampled pixel positions:

[0041] Determining a standard deviation of the input pixel values of the current frame within the region surrounding the upsampled pixel position,

[0042] where the threshold is based on the determined standard deviation of the input pixel values of the current frame within the region.

[0043] For each of the one or more upsampled pixel positions, the threshold may be F pixel ·σ pixel where F pixel is a predetermined factor and σ pixel is the determined standard deviation of the input pixel values of the current frame within the region surrounding the upsampled pixel position.

[0044] The clamping can be selectively applied to different regions to different extents. The method may further include:

[0045] Comparing an average value of the pixel values determined at the upsampled pixel positions within the region surrounding the upsampled pixel position with the mean value of the input pixel values of the current frame within the region surrounding the upsampled pixel position; and

[0046] The clamping is performed based on a comparison of the following: (i) the difference between the average of the pixel values determined at the upsampled pixel positions within the region surrounding the upsampled pixel position and the mean of the input pixel values within the region surrounding the upsampled pixel position in the current frame, and (ii) a threshold difference.

[0047] In response to determining that the weights of all the one or more identified pixels of the reference frame are zero, the pixel value at the upsampled pixel position can be determined as the determined mean of the input pixel values within the region surrounding the upsampled pixel position in the current frame.

[0048] The upsampled pixel position can be located between the positions of the diagonally adjacent input pixels of the current frame such that the upsampled pixel position and the positions of the input pixels form a repeating five-point pattern.

[0049] The resolution of the pixels of the reference frame can be the same as the resolution of the input pixels of the current frame.

[0050] A dither pattern can be used on the frame sequence such that different frames in the sequence have pixels at positions corresponding to different upsampled pixel positions.

[0051] The resolution of the pixels of the reference frame can be the same as the resolution of the pixels determined at the upsampled pixel position.

[0052] The pixel values and depth values at the input pixel positions of the current frame and the reference frame can be determined by a graphics rendering process.

[0053] The obtaining of the depth value of the current frame for the upsampled pixel position can include:

[0054] Receiving the depth values at the positions of the input pixels within the region surrounding the upsampled pixel position in the current frame;

[0055] For each pair of input pixels for which depth values are received, determining an interpolated depth value at the upsampled pixel position based on the depth values of the pair of input pixels;

[0056] Determining a depth weight for the pair of input pixels based on the depth gradient between the depth values of the pair of input pixels; and

[0057] Determining the depth value of the current frame for the upsampled pixel position by performing a weighted sum of the determined interpolated depth values using the determined depth weights for the pair of input pixels.

[0058] The determining of the depth weights for the pair of input pixels can include:

[0059] Multiplying the depth gradient of the pair of input pixels by a negative number; and

[0060] Input the result of this multiplication into the softmax function.

[0061] This pixel value can be the pixel value of the Y channel.

[0062] A processing module is provided that is configured to determine one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence. The processing module is configured to:

[0063] Obtain the depth value of the position of the pixels of a reference frame in the frame sequence; and

[0064] For each upsampled pixel position among the one or more upsampled pixel positions:

[0065] Obtain the depth value of the current frame for the upsampled pixel position;

[0066] Obtain the motion vector of the upsampled pixel position to indicate the motion of the upsampled pixel position between the reference frame and the current frame;

[0067] Use the motion vector of the upsampled pixel position to identify one or more pixels among the pixels of the reference frame;

[0068] Determine the weight of each identified pixel among the one or more identified pixels of the reference frame based on the following: (i) the depth value of the current frame for the upsampled pixel position, and (ii) the depth value of the position of the identified pixel of the reference frame; and

[0069] Use the determined weight for each identified pixel among the one or more identified pixels to determine the pixel value of the upsampled pixel position.

[0070] A processing module can be provided that is configured to perform any method described herein.

[0071] The processing module can be embodied in hardware on an integrated circuit.

[0072] A computer-readable code can be provided that is configured to cause any of the methods described herein to be executed when the code runs.

[0073] An integrated circuit definition data set can be provided that, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture a processing module as described herein.

[0074] A method for determining one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence can be provided. The method includes:

[0075] Obtain the pixel values of the pixels of the reference frame in the frame sequence;

[0076] For each upsampled pixel position among the one or more upsampled pixel positions:

[0077] Obtain a plurality of input pixel values of the current frame for positions within a region surrounding the upsampled pixel position;

[0078] Determine the mean value of the input pixel values of the current frame within the region surrounding the upsampled pixel position;

[0079] Obtain the motion vector of the upsampled pixel position to indicate the motion of the upsampled pixel position between the reference frame and the current frame;

[0080] Use the motion vector of the upsampled pixel position to identify one or more pixels among the pixels of the reference frame; and

[0081] Combine the pixel values of the one or more identified pixels of the reference frame to determine the pixel value of the upsampled pixel position;

[0082] Wherein combining the pixel values of the one or more identified pixels of the reference frame to determine the pixel value of the upsampled pixel position includes clamping the determined pixel value such that the pixel value does not differ from the determined mean value of the input pixel values of the current frame within the region surrounding the upsampled pixel position by more than a threshold.

[0083] The method may further include, for each upsampled pixel position among the one or more upsampled pixel positions:

[0084] Determine the standard deviation of the input pixel values of the current frame within the region surrounding the upsampled pixel position,

[0085] Wherein the threshold is based on the determined standard deviation of the input pixel values of the current frame within the region.

[0086] For each upsampled pixel position among the one or more upsampled pixel positions, the threshold may be F pixel ·σ pixel , where F pixel is a predetermined factor, and σ pixel is the determined standard deviation of the input pixel values of the current frame within the region surrounding the upsampled pixel position.

[0087] The clamping may be selectively applied to different regions to different extents.

[0088] The method may further include:

[0089] Compare the average of the pixel values determined at the upsampled pixel positions within the region around the upsampled pixel position with the mean of the input pixel values within the region around the upsampled pixel position in the current frame; and

[0090] Perform the clamping based on the comparison of: (i) the difference between the average of the pixel values determined at the upsampled pixel positions within the region around the upsampled pixel position and the mean of the input pixel values within the region around the upsampled pixel position in the current frame, and (ii) a threshold difference.

[0091] The pixel value can be a Y-channel pixel value.

[0092] For each of the one or more upsampled pixel positions, the pixel value of the one or more identified pixels of the reference frame in the combination may include:

[0093] Determine the weight of each of the identified pixels among the one or more identified pixels of the reference frame; and

[0094] Use the determined weight for each of the one or more identified pixels to determine the pixel value at the upsampled pixel position.

[0095] The determining the pixel value at the upsampled pixel position may include performing a weighted sum of the pixel values of the one or more identified pixels of the reference frame, and the performing is carried out using the determined weight for each of the one or more identified pixels in the weighted sum.

[0096] The method may further include, for each of the one or more upsampled pixel positions:

[0097] Obtain the depth value of the position of the one or more identified pixels of the reference frame; and

[0098] Obtain the depth value for the upsampled pixel position in the current frame,

[0099] wherein the weight of each of the one or more identified pixels of the reference frame is determined based on: (i) the depth value for the upsampled pixel position in the current frame, and (ii) the depth value of the position of the identified pixel of the reference frame.

[0100] For each of the one or more upsampled pixel positions, the weight of each of the one or more identified pixels of the reference frame may be determined based on the difference between the depth value for the upsampled pixel position in the current frame and the depth value of the position of the identified pixel of the reference frame.

[0101] The method may further include, for each of the one or more upsampled pixel positions:

[0102] obtaining a plurality of depth values of the current frame for positions within a region surrounding the upsampled pixel position; and

[0103] determining a standard deviation of the depth values of the current frame within the region, wherein the weight of each of the identified pixels in the identified pixels of the reference frame is further determined based on: (iii) the determined standard deviation of the depth values.

[0104] Determining the weight of each of the identified pixels in the one or more identified pixels of the reference frame may include comparing the difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame with a depth threshold, wherein the depth threshold is based on the determined standard deviation of the depth values of the current frame within the region.

[0105] In response to determining that the difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame is greater than the depth threshold, it may be determined that the weight of the identified pixel of the reference image is lower.

[0106] The depth threshold may be a hard threshold. The weight w of the identified pixel k of the reference image k may be determined such that w k = w i,k ·(|D ref,k - D curr | ≤ T d ), where T d is the depth threshold, where T d = F depth · σ depth , and where w i,k is the initial weight of the identified pixel of the reference image, D ref,k is the depth value of the position of the identified pixel of the reference frame, D curr is the depth value of the current frame for the upsampled pixel position, F depth is a predetermined factor, and σ depth is the determined standard deviation of the depth values of the current frame within the region surrounding the upsampled pixel position.

[0107] The depth threshold may be a soft threshold. The weight w of the identified pixel k of the reference image k may be determined such that where T dis the depth threshold, where T d = F depth ·σ depth , and where w i,k is the initial weight of the identified pixel of the reference image, D ref,k is the depth value of the position of the identified pixel of the reference frame, D curr is the depth value of the current frame for the upsampled pixel position, F depth is a predetermined factor, and σ depth is the determined standard deviation of the depth values of the current frame within the region around the upsampled pixel position.

[0108] Obtaining the depth value of the current frame for the upsampled pixel position may include:

[0109] Receiving the depth value at the position of the input pixel of the current frame within the region around the upsampled pixel position;

[0110] For each pair of input pixels for which the depth value is received, determining an interpolated depth value of the upsampled pixel position based on the depth values of the pair of input pixels;

[0111] Determining the depth weight of the pair of input pixels based on the depth gradient between the depth values of the pair of input pixels; and

[0112] Determining the depth value of the current frame for the upsampled pixel position by performing a weighted sum of the determined interpolated depth values using the determined depth weights for the pair of input pixels.

[0113] Determining the depth weight of the pair of input pixels may include:

[0114] Multiplying the depth gradient of the pair of input pixels by a negative number; and

[0115] Inputting the result of the multiplication into a softmax function.

[0116] In response to determining that the weights of all the identified pixels of the reference frame are zero, the pixel value of the upsampled pixel position is determined to be the determined mean of the input pixel values of the current frame within the region around the upsampled pixel position.

[0117] For each upsampled pixel position among the one or more upsampled pixel positions, the weight of each identified pixel among the one or more identified pixels of the reference frame may be determined according to the degree to which the identified pixel of the reference frame is an outlier compared to other identified pixels of the reference frame.

[0118] Identifying one or more pixels of the reference frame using the motion vector of the upsampled pixel position may include projecting the upsampled pixel position to a position in the reference frame based on the motion vector, and identifying one or more pixels of the reference frame in a neighborhood of the projected position in the reference frame.

[0119] For each of the one or more upsampled pixel positions, determining the weight of each of the one or more identified pixels of the reference frame may include:

[0120] Determining an initial weight by: (i) determining a distance between the projected position and the position of the identified pixel in the reference frame, and (ii) mapping the distance to the initial weight using a predetermined relationship; and

[0121] Using the initial weight to determine the weight of the identified pixel of the reference frame.

[0122] The predetermined relationship may be a Gaussian relationship or a linear relationship.

[0123] The upsampled pixel position may be located between the positions of the diagonally adjacent input pixels of the current frame, such that the upsampled pixel position and the positions of the input pixels form a repeating five-point pattern.

[0124] The resolution of the pixels of the reference frame may be the same as the resolution of the input pixels of the current frame.

[0125] A dither pattern may be used on the frame sequence such that different frames in the sequence have pixels at positions corresponding to different upsampled pixel positions.

[0126] The resolution of the pixels of the reference frame may be the same as the resolution of the pixels determined at the upsampled pixel position.

[0127] A processing module may be provided that is configured to determine one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence, the processing module being configured to:

[0128] Obtain pixel values of pixels of a reference frame in the frame sequence;

[0129] For each of the one or more upsampled pixel positions:

[0130] Obtain a plurality of input pixel values of the current frame for positions within a region surrounding the upsampled pixel position;

[0131] Determine an average of the input pixel values of the current frame within the region surrounding the upsampled pixel position;

[0132] Obtain a motion vector for the upsampled pixel position to indicate the motion of the upsampled pixel position between the reference frame and the current frame;

[0133] Use the motion vector of the upsampled pixel position to identify one or more pixels in the pixel of the reference frame; and

[0134] Combine the pixel values of the one or more identified pixels of the reference frame to determine the pixel value of the upsampled pixel position;

[0135] Wherein combining the pixel values of the one or more identified pixels of the reference frame to determine the pixel value of the upsampled pixel position includes clamping the determined pixel value such that the pixel value does not differ from the determined mean of the input pixel values in the current frame within the region around the upsampled pixel position by more than a threshold.

[0136] A processing module may be provided, the processing module being configured to perform any of the methods described herein.

[0137] The processing module may be embodied in hardware on an integrated circuit.

[0138] A computer-readable code may be provided, the computer-readable code being configured to cause any of the methods described herein to be performed when the code runs.

[0139] An integrated circuit definition data set may be provided, which when processed in an integrated circuit manufacturing system configures the integrated circuit manufacturing system to manufacture a processing module as described herein.

[0140] A method for determining a pixel value at an upsampled pixel position of a current frame in a frame sequence may be provided, the method comprising:

[0141] Use a graphics rendering process to determine the pixel values at a first subset of the upsampled pixel positions of the current frame;

[0142] Determine the pixel values at a second subset of the upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of a reference frame in the frame sequence; and

[0143] Determine the pixel values at a third subset of the upsampled pixel positions of the current frame by applying spatial upsampling to the determined pixel values at the upsampled pixel positions in the first subset and the second subset.

[0144] The upsampled pixel positions in the first subset and the second subset may form a repeating pentagon pattern.

[0145] The upsampled pixel positions in the second subset may be located between diagonally adjacent upsampled pixel positions in the first subset such that:

[0146] For each upsampled pixel position in the first subset that is not on the edge of the current frame, the four nearest upsampled pixel positions of the repeating pentagon pattern are upsampled pixel positions in the second subset, and

[0147] For each upsampled pixel position in the second subset that is not on the edge of the current frame, the four nearest upsampled pixel positions of the repeating pentagon pattern are upsampled pixel positions in the first subset.

[0148] The upsampled pixel positions in the third subset can be located in the gaps of the repeating pentagon pattern.

[0149] Each upsampled pixel position in the third subset that is not on the edge of the current frame can be located at: (i) between two horizontally adjacent upsampled pixel positions in the first subset and between two vertically adjacent upsampled pixel positions in the second subset, or (ii) between two vertically adjacent upsampled pixel positions in the first subset and between two horizontally adjacent upsampled pixel positions in the second subset.

[0150] The first subset, the second subset, and the third subset of upsampled pixel positions can be different such that there is no upsampled pixel position that belongs to more than one of the first subset, the second subset, and the third subset.

[0151] All the upsampled pixel positions of the current frame can belong to one of the first subset, the second subset, and the third subset.

[0152] It may be the case that one quarter of the upsampled pixel positions of the current frame are in the first subset, one quarter of the upsampled pixel positions of the current frame are in the second subset, and half of the upsampled pixel positions of the current frame are in the third subset.

[0153] A dither pattern can be used on the frame sequence such that a graphics rendering process can be used to determine pixel values at different upsampled pixel positions in different frames of the frame sequence.

[0154] The subset of upsampled pixel positions for which the pixel values are determined using the graphics rendering process can alternate between the first subset as the upsampled pixel positions and the second subset as the upsampled pixel positions for consecutive frames in the frame sequence.

[0155] The reference frame can be the previous frame or the next frame in the frame sequence relative to the current frame.

[0156] The graphics rendering process can be a rasterization process or a ray tracing process.

[0157] Determining the pixel value at the second subset of the upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of the reference frame in the frame sequence may include:

[0158] Obtaining the pixel value of the pixel of the reference frame;

[0159] For each upsampled pixel position in the second subset of the upsampled pixel positions:

[0160] Obtaining a motion vector of the upsampled pixel position to indicate the motion of the upsampled pixel position between the reference frame and the current frame;

[0161] Using the motion vector of the upsampled pixel position to identify one or more pixels among the pixels of the reference frame; and

[0162] Combining the pixel values of the one or more identified pixels of the reference frame to determine the pixel value of the upsampled pixel position in the second subset.

[0163] Determining the pixel value at the second subset of the upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of the reference frame in the frame sequence may further include:

[0164] Obtaining the depth value of the position of the pixel of the reference frame; and

[0165] For each upsampled pixel position in the second subset of the upsampled pixel positions, obtaining the depth value of the current frame for the upsampled pixel position;

[0166] Wherein for each upsampled pixel position in the second subset of the upsampled pixel positions, combining the pixel values of the one or more identified pixels of the reference frame may include:

[0167] Determining the weight of each identified pixel among the one or more identified pixels of the reference frame based on: (i) the depth value of the current frame for the upsampled pixel position, and (ii) the depth value of the position of the identified pixel of the reference frame; and

[0168] Using the determined weight for each identified pixel among the identified pixels to determine the pixel value of the upsampled pixel position.

[0169] Determining the pixel value of the upsampled pixel position may include performing a weighted sum of the pixel values of the one or more identified pixels of the reference frame, and the performing is carried out using the determined weight for each identified pixel among the one or more identified pixels in the weighted sum.

[0170] The method may further include, for each upsampled pixel position among the one or more upsampled pixel positions in the second subset:

[0171] obtaining a plurality of depth values of the current frame for positions within a region surrounding the upsampled pixel position; and

[0172] determining a standard deviation of the depth values of the current frame within the region, wherein the weight of each identified pixel among the identified pixels of the reference frame is further determined based on: (iii) the determined standard deviation of the depth values.

[0173] Said using the motion vector of the upsampled pixel position to identify one or more pixels among the pixels of the reference frame may include projecting the upsampled pixel position to a position in the reference frame based on the motion vector, and identifying one or more pixels among the pixels of the reference frame in a neighborhood of the projected position in the reference frame.

[0174] Said determining the pixel value at the second subset of upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of the reference frame in the frame sequence may further include, for each upsampled pixel position among the one or more upsampled pixel positions in the second subset:

[0175] determining a mean of the plurality of pixel values at the first subset of upsampled pixel positions within a region surrounding the upsampled pixel position of the current frame,

[0176] wherein said combining the pixel values of the one or more identified pixels of the reference frame to determine the pixel value at the upsampled pixel position of the second subset may include clamping the determined pixel value such that the pixel value does not differ from the determined mean of the pixel values at the first subset of upsampled pixel positions within the region surrounding the upsampled pixel position of the current frame by more than a threshold.

[0177] Said determining the pixel value at the second subset of upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of the reference frame in the frame sequence may further include, for each upsampled pixel position among the one or more upsampled pixel positions in the second subset:

[0178] determining a standard deviation of the plurality of pixel values at the first subset of upsampled pixel positions within a region surrounding the upsampled pixel position of the current frame,

[0179] wherein the threshold is based on the determined standard deviation of the pixel values at the first subset of upsampled pixel positions within the region of the current frame.

[0180] Determining the pixel values at a third subset of the upsampled pixel positions of the current frame by applying spatial upsampling to the determined pixel values at the upsampled pixel positions in the first subset and the second subset may include performing bilinear interpolation on the determined pixel values at the upsampled pixel positions in the first subset and the second subset.

[0181] Determining the pixel values at a third subset of the upsampled pixel positions of the current frame by applying spatial upsampling to the determined pixel values at the upsampled pixel positions in the first subset and the second subset may include:

[0182] Analyzing the pixel values at the upsampled pixel positions in the first subset and the second subset to determine one or more weighting parameters, the one or more weighting parameters indicating the directionality of the filtering to be applied when applying upsampling to the determined pixel values at the upsampled pixel positions in the first subset and the second subset; and

[0183] Determining the pixel values at the third subset of the upsampled pixel positions by applying one or more kernels to at least some of the pixel values at the upsampled pixel positions in the first subset and the second subset according to the determined one or more weighting parameters.

[0184] Analyzing the pixel values at the upsampled pixel positions in the first subset and the second subset to determine one or more weighting parameters may include processing the pixel values at the upsampled pixel positions in the first subset and the second subset using a specific implementation of a neural network, where the neural network has been trained to output an indication of the one or more weighting parameters to indicate the directionality of the filtering to be applied when applying upsampling to the determined pixel values at the upsampled pixel positions in the first subset and the second subset.

[0185] The pixel values at the third subset of the upsampled pixel positions may be unsharp upsampled pixel values.

[0186] The pixel values at the third subset of the upsampled pixel positions may be sharpened upsampled pixel values.

[0187] The pixel values may be Y-channel pixel values.

[0188] A processing system configured to determine pixel values at upsampled pixel positions of a current frame in a frame sequence may be provided, the processing system including:

[0189] A graphics rendering unit configured to determine pixel values at a first subset of the upsampled pixel positions of the current frame using a graphics rendering process;

[0190] A temporal resampling logic configured to determine pixel values at a second subset of the upsampled pixel positions of the current frame by applying temporal resampling to pixel values of pixels of a reference frame in the sequence of frames; and

[0191] A spatial upsampling logic configured to determine pixel values at a third subset of the upsampled pixel positions of the current frame by applying spatial upsampling to the determined pixel values at the upsampled pixel positions in the first and second subsets.

[0192] The processing system may include a first device and a second device arranged to communicate with each other via a network,

[0193] wherein the graphics rendering unit and the temporal resampling logic may be implemented at the first device, and

[0194] wherein the spatial upsampling logic may be implemented at the second device.

[0195] A processing system may be provided, the processing system being configured to perform any of the methods described herein.

[0196] The processing system may be embodied as hardware on one or more integrated circuits.

[0197] A computer-readable code may be provided, the computer-readable code being configured to cause any of the methods described herein to be performed when the code runs.

[0198] An integrated circuit definition dataset may be provided, which when processed in an integrated circuit manufacturing system configures the integrated circuit manufacturing system to manufacture the processing system described herein.

[0199] The processing module or processing system described herein may be embodied as hardware on an integrated circuit. A method of manufacturing a processing module or processing system at an integrated circuit manufacturing system may be provided. An integrated circuit definition dataset may be provided, which when processed in an integrated circuit manufacturing system configures the system to manufacture a processing module or processing system. A non-transitory computer-readable storage medium may be provided, having stored thereon a computer-readable description of a processing module or processing system, which when processed in an integrated circuit manufacturing system causes the integrated circuit manufacturing system to manufacture an integrated circuit embodying the processing module or processing system.

[0200] An integrated circuit manufacturing system can be provided, the integrated circuit manufacturing system including: a non-transitory computer-readable storage medium having a computer-readable description of a processing module stored thereon; a layout processing system configured to process the computer-readable description to generate a circuit layout description of an integrated circuit embodying the processing module of the processing system; and an integrated circuit generation system configured to manufacture the processing module or the processing system according to the circuit layout description.

[0201] Computer program code for performing any of the methods described herein can be provided. A non-transitory computer-readable storage medium can be provided having computer-readable instructions stored thereon that, when executed in a computer system, cause the computer system to perform any of the methods described herein.

[0202] As will be apparent to those skilled in the art, the above features can be appropriately combined and can be combined with any aspect of the examples described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0203] Examples will now be described in detail with reference to the drawings, in which:

[0204] Figure 1 An upsampling process is shown;

[0205] Figure 2 Input pixels of a current frame and input pixels of a reference frame within a frame sequence are shown;

[0206] Figure 3 A processing module configured to determine one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence is shown;

[0207] Figure 4 is a flowchart of a method for determining one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence;

[0208] Figure 5 Upsampled pixel positions of the current frame are shown, indicating the upsampled pixel positions for which pixel values and / or depth values are obtained;

[0209] Figure 6 Illustrates projecting the upsampled pixel positions of the current frame to positions in the reference frame;

[0210] Figure 7 A graph is shown illustrating a linear relationship and a Gaussian relationship for mapping the distance between a projected position and the position of a pixel in the reference frame to an initial weight for determining the pixel value of the upsampled pixel position;

[0211] Figure 8 A graph is shown that illustrates the clamping of the determined pixel values;

[0212] Figure 9 Three versions of a portion of an upsampled frame are shown: (i) a ground truth version, (ii) a version in which historical correction has been applied to the determined pixel values, and (iii) a version in which historical correction has not been applied to the determined pixel values;

[0213] Figure 10 The neighborhood of a reference pixel within a reference frame is illustrated;

[0214] Figure 11A An example method for determining depth values for upsampled pixel positions is illustrated;

[0215] Figure 11B is according to Figure 10 A flowchart of method steps for determining depth values for upsampled pixel positions according to the example shown;

[0216] Figure 12 A processing system for determining pixel values at upsampled pixel positions as described herein is illustrated;

[0217] Figure 13 A flowchart of a method for determining pixel values at upsampled pixel positions in a current frame of a frame sequence;

[0218] Figure 14 illustrates applying temporal resampling and spatial upsampling to a frame in a frame sequence according to the method shown in the flowchart of Figure 13 ;

[0219] Figure 15 A spatial upsampling logic is shown that is configured to upsample pixel values to determine an upsampled pixel value block;

[0220] Figure 16 A flowchart of a method for applying upsampling to pixel values at upsampled pixel positions in a first subset and a second subset;

[0221] Figure 17 A flowchart of method steps for determining upsampled pixel values;

[0222] Figure 18 A portion of the pixel values in a first subset and a second subset to which spatial upsampling is to be applied is shown;

[0223] Figure 19a A first kernel for applying a first weighting parameter is shown;

[0224] Figure 19b A second kernel for applying a second weighting parameter is shown;

[0225] Figure 20 Illustrates pixel values of a frame sequence according to an example implementing a finite impulse response (FIR) method, the pixel values indicating how to project upsampled pixel positions to positions in a reference frame;

[0226] Figure 21 Illustrates pixel values of a frame sequence according to an example implementing an infinite impulse response (IIR) method, the pixel values indicating how to project upsampled pixel positions to positions in a reference frame;

[0227] Figure 22 Shows a computer system in which a processing module and / or a processing system is implemented; and

[0228] Figure 23 Shows an integrated circuit manufacturing system for generating an integrated circuit embodying a processing module and / or a processing system.

[0229] The drawings illustrate various examples. Those skilled in the art will appreciate that the element boundaries shown in the drawings (e.g., boxes, groups of boxes, or other shapes) represent one example of a boundary. In some examples, it may be the case that one element can be designed as multiple elements, or multiple elements can be designed as one element. Where appropriate, common reference numerals are used throughout the drawings to indicate similar features. Detailed Description

[0230] The following description is presented by way of example to enable those skilled in the art to make and use the invention. The invention is not limited to the embodiments described herein, and various modifications to the disclosed embodiments will be apparent to those skilled in the art.

[0231] Embodiments will now be described solely by way of example. In the examples described herein, a temporal resampling method can be used to determine pixel values at upsampled pixel positions of a current frame in a frame sequence. The frame sequence includes frames at respective time instances, e.g., the frame sequence includes a current frame at a current time instance and one or more reference frames at respective reference time instances. Temporal resampling can be applied to pixel values of pixels of the reference frames in the frame sequence to determine pixel values at upsampled pixel positions of the current frame. A "reference frame" can be a previous or a subsequent frame relative to the current frame in the frame sequence. In many of the examples described herein, the reference frame is the frame immediately preceding the current frame in the frame sequence. In some examples, there can be a single reference frame, while in other examples, there can be multiple reference frames.

[0232] Figure 2 Shows input pixels of a current frame 202 and input pixels of a reference frame 204 (e.g., the previous frame) within a frame sequence. In Figure 2In it, the (low-resolution) input pixels are shown with diagonal hatching, e.g., at the upsampled pixel position 206. Figure 2 The squares not shown with hatching in it represent the upsampled pixel positions for which pixel values are to be determined (e.g., the upsampled pixel position 208), i.e., the upsampled pixel positions for which input pixels are not provided. An "upsampled pixel position" is a pixel position in the upsampled output image in the output frame sequence. In Figure 2 As can be seen in the example shown, upsampling will double the resolution, i.e., will double the number of rows of pixels and will double the number of columns of pixels, such that each 2×2 block of upsampled pixel positions includes the position of one input pixel in the current frame 202. Figure 2 An example is shown in which a dither pattern is used on the frame sequence such that the current frame and the reference frame have pixels at positions corresponding to different upsampled pixel positions. Specifically, the current frame 202 has input pixels (shown with diagonal hatching) in the upper left upsampled pixel positions and then has input pixels in alternating rows and alternating columns starting from the upper left position; while the reference frame 204 has input pixels (shown with diagonal hatching) at the upsampled pixel position located in the second row from the top and in the second column from the left and then has input pixels in alternating rows and alternating columns starting from that position. In this way, the upsampled pixel positions of the input pixels in consecutive frames in the sequence are shifted relative to each other. For example, the dither pattern can alternate, i.e., the positions of the shaded pixels in 202 and 204 can switch from one frame to the next. Generally, the content represented by the frames of a frame sequence (e.g., a video stream) does not change significantly from one frame to the next. For example, the pixel value at the upsampled pixel position 210 in the current frame 202 may be similar to the input pixel value at the corresponding position (i.e., in the second row from the top and in the second column from the left) in the reference frame 204, which is the position for which an input pixel is included in the reference frame 204. This may be the case, for example, when most of the camera and the scene are stationary. As described in more detail below, the upsampled pixel position 210 from the current frame 202 can be projected to the projected position 214 in the reference frame 204 using the motion vector 212. In this example, the motion vector has a nearly zero displacement in both the horizontal and vertical directions relative to the position 210. The pixel value at the upsampled pixel position 210 can be estimated using the pixel value of the reference frame, e.g., according to the projected position 214. This estimation process is "temporal resampling", and examples for performing temporal resampling are described herein.

[0233] Generally speaking, there may be a single reference frame, or there may be multiple reference frames, and each reference frame can be a previous frame or a subsequent frame relative to the current frame in the frame sequence.

[0234] Figure 3A processing module 302 is shown, which is configured to apply temporal resampling of reference frames in a frame sequence 304 to obtain pixel values at upsampled pixel positions in a current frame. The frame sequence 304 includes a current frame 202 and a reference frame 204. As described above, in Figure 3 the example shown, the reference frame 204 is the frame immediately preceding the current frame 202 in the frame sequence 304, but in other examples, the reference frame can be a different frame before or after the current frame in the frame sequence. The processing module 302 includes temporal resampling logic 306, which is configured to perform the temporal resampling described herein. The temporal resampling logic 306 can be implemented in software, hardware, or a combination thereof.

[0235] In different examples, the format of the pixels may be different. For example, the pixels can be in YUV format (where each pixel has a value in each of the Y, U, and V channels, where Y represents the luminance value and U and V represent the chrominance values), and upsampling can be applied separately to each of the Y, U, and V channels. The upsampling described herein can be applied only to the Y channel (i.e., the pixel values can be Y-channel pixel values), while upsampling of the U and V channels is performed in a simpler manner (e.g., using bilinear interpolation). In other examples, the upsampling described herein can be applied to each of the Y, U, and V channels. The human visual system is less sensitive to the perceived spatial resolution in the U and V channels than in the Y channel, so using a simpler upsampling technique for the U and V channels (e.g., only bilinear upsampling on the current frame) can be beneficial (e.g., lower power and area costs in a hardware implementation), while a more complex upsampling technique described herein can be used for the Y channel (which can provide an upsampled image with less blur and / or other artifacts). Thus, the temporal resampling method described herein can be applied only to the Y channel, relying on the current frame to provide the U and V channels, which can increase the robustness to chromaticity errors and reduce implementation costs (e.g., reduced bandwidth and execution time in a software implementation of temporal resampling on a GPU). In other examples, the same temporal resampling technique can be applied to all three channels. If the input pixel data is in RGB format, it can be converted to YUV format (e.g., using known color space conversion techniques) and then processed as data in the Y, U, and V channels. Alternatively, if the input pixel data is in RGB format (where each pixel has a value in each of the R, G, and B channels corresponding to red, green, and blue, respectively), the techniques described herein can be implemented on the R, G, and B channels as described herein, where the G channel can be considered a proxy for the Y channel. If the input data includes an alpha channel, upsampling can be applied separately to the alpha channel (e.g., using bilinear interpolation).

[0236] Figure 4Flowchart of a method for determining corresponding pixel values at one or more upsampled pixel positions of a current frame of a frame sequence. Figure 4 The method shown implements temporal resampling. That is, the method applies temporal resampling to one or more reference frames to determine corresponding pixel values at one or more upsampled pixel positions corresponding to a current time instance.

[0237] In step S402, the temporal resampling logic 306 obtains pixel values and depth values of pixels of the reference frame 204. In step S404, the temporal resampling logic 306 obtains pixel values and depth values of the current frame 202. For example, the pixel values and depth values of the current frame 202 and the reference frame 204 can be determined by a graphics rendering process. The graphics rendering process can be any suitable known type of graphics rendering process, such as a rasterization process or a ray tracing process. Note that steps S402 and S404 can be executed in sequence or simultaneously.

[0238] In step S402, for each pixel of the reference frame 204 (shown with diagonal hatching in Figure 2 ), pixel values and depth values are obtained. Similarly, in step S404, for each pixel of the current frame 202 (shown with diagonal hatching in Figure 2 ), pixel values and depth values are obtained. Additionally, in step S404, depth values are obtained for each upsampled pixel position for which pixel values are to be determined. For example, pixel values can be determined at upsampled pixel positions between diagonally adjacent input pixels of the current frame 202. In this way, the input pixels and the determined pixel values are located at positions forming a repeating five-pointed star (or "checkerboard") pattern. The pixel values at the upsampled pixel positions determined by temporal resampling can be referred to as 'temporally resampled pixel values'. Additionally, an 'upsampled pixel position' can be referred to as a 'temporally resampled pixel position'. One of these upsampled pixel positions is labeled with reference numeral 210 in Figure 2 . Figure 3 Shows that for a 2×2 block of the upsampled pixel position 308, the current frame 202 has a pixel value in the upper left of the 2×2 block 308 in the example shown in Figure 3 , and the processing module 302 outputs a block 310 of pixel values, the block of pixel values including the temporally resampled pixel value 312 of the lower right upsampled pixel position. The depth value of a pixel represents the distance from the viewpoint (or "camera center") of the frame to the visible surface in the scene represented by the pixel in the frame.

[0239] Figure 5Illustrates the upsampled pixel positions of the current frame 202, which indicate the upsampled pixel positions for which pixel values and / or depth values are obtained. Specifically, the solid circles indicate the upsampled pixel positions for which pixel values and depth values are obtained in step S404; and the hollow circles represent the upsampled pixel positions for which depth values (but not pixel values) are obtained in step S404. In step S404, for the upsampled pixel positions indicated by the squares without circles in Figure 5 neither pixel values nor depth values are obtained. As described above, the pixel values and depth values obtained in step S404 (and step S402) can be provided, for example, by a graphics rendering process implemented by a graphics processing unit and sent to the processing module 302. Alternatively, as described in more detail below with reference to Figure 11A and Figure 11B in step S404, the graphics rendering process can provide depth values only for the positions of the pixels (i.e., only for the positions shown by solid circles in Figure 5 and not for the positions shown by hollow circles). In these examples, the processing module 302 determines the depth values for the positions shown by hollow circles, as described in more detail below. This can be simpler and less expensive to implement in the graphics rendering process, but may sacrifice some quality of the resulting resampled pixel values.

[0240] In step S406, one or more moments (i.e., statistics) are determined for the positions of the current frame in the region surrounding the upsampled pixel position. These moments can include the mean value and / or the standard deviation, and can be moments related to the depth values and / or pixel values of the positions of the current frame in the region surrounding the upsampled pixel position. In other examples, instead of or in addition to the standard deviation, the moments can include the variance and / or the range. In the example shown in Figure 5 the region 510 (shown by the dashed line) surrounds the upsampled pixel position 504. Within the region 510, there are four pixels (5061, 5062, 5063, and 5064) of the current frame for which pixel values are obtained in step S404. Within the region 510, there are thirteen positions for which depth values are obtained for the current frame: the positions 504, 5061, 5062, 5063, 5064, 5081, 5082, 5083, 5084, 5085, 5086, 5087, and 5088.

[0241] The mean value (μ depth ) of the depth values can be calculated as where D i is the depth value obtained within the region 510, and N D is the number of depth values obtained within the region 510. The standard deviation (σ depth ) of the depth values can be calculated as In an alternative example, the standard deviation (σ depth ) of the depth values can be calculated as Referring to Figure 5 the example shown, N D = 13 because there are thirteen positions within region 510 for which depth values are obtained. In other examples, if region 510 includes a different number of positions for which depth values are obtained, then N D can be different.

[0242] The mean value (μ pixel ) of the pixel values can be calculated as where x i is the pixel value (e.g., Y-channel value) obtained within region 510, and N pixel is the number of pixel values obtained within region 510. The standard deviation (σ pixel ) of the pixel values can be calculated as In an alternative example, the standard deviation (σ pixel ) of the pixel values can be calculated as Referring to Figure 5 the example shown, N pixel = 4 because there are four positions within region 510 for which pixel values are obtained. In other examples, if region 510 includes a different number of positions for which pixel values are obtained, then N pixel can be different.

[0243] Figure 6 Illustrates the projection of the upsampled pixel position 504 of the current frame 202 to the position 604 in the reference frame 204. In step S408, the processing module 302 obtains the motion vector 602 of the upsampled pixel position 504 to indicate the motion of the upsampled pixel position 504 between the reference frame 204 and the current frame 202. The motion vector 602 can be a forward or backward motion vector (or a combination of forward and backward motion vectors, such as an average). A forward motion vector represents motion from an earlier frame (e.g., the reference frame 204) to a later frame (e.g., the current frame 202); while a backward motion vector represents motion from a later frame (e.g., the current frame 202) to an earlier frame (e.g., the reference frame 204).

[0244] In this document, the term "obtain" is used such that an "obtained" value may refer to a "determined" value or a "received" value. As an example, the motion vector 602 may be determined during a graphics rendering process that provides pixel values and depth values and is executed by a graphics processing unit, and step S408 may include the processing module 302 receiving the motion vector 602 from the graphics processing unit. In an alternative example, the processing module 302 may itself determine the motion vector 602 based on the pixel values (and optionally depth values) of the reference frame 204 and the current frame 202. Techniques for determining motion vectors are known in the art, and any suitable technique (such as optical flow or block matching) may be used in the examples described herein. The motion vector 602 may represent the (apparent) displacement of a point imaged at a corresponding time instance from the current frame 202 to the reference frame 204. In some cases, the motion vector 602 does not represent the actual motion of an object in the scene, but rather may point to a position in the reference frame 204 that provides the best match (according to any suitable metric) to the upsampled pixel position 504 in the current frame 202, regardless of whether that position corresponds to any actual motion of an object in the scene.

[0245] In step S410, the processing module 302 uses the motion vector 602 of the upsampled pixel position 504 to identify a plurality of pixels in the pixels of the reference frame 204. Specifically, based on the motion vector 602, the upsampled pixel position 504 is projected onto a position 604 in the reference frame 204, and a plurality of pixels of the reference frame are identified in the neighborhood of the projected position in the reference frame. For example, the four pixels (6061, 6062, 6063, and 6064) closest to the projected position 604 in the reference frame 204 may be identified. As another example, the pixels of the reference frame may be identified as those pixels enclosed by a box, the upper left corner of which is identified by subtracting 0.75 from both the X and Y coordinates of the projected position 602, and the lower right corner of which is identified by adding 0.75 to both the X and Y coordinates of the projected pixel position (assuming that in a high-resolution image space, the reference pixel centers are separated by 1). In other examples, values other than 0.75 may be used, such as values in the range from 0.75 to 1 (but not including 1). Using a value less than 1 tends to exclude pixels in the reference frame that are far from the projected position 604, thereby eliminating their influence on the pixel value being determined. For example, if the projected position 604 is very close to a pixel position in the reference frame, it may be the case that only the closest pixel is considered when determining the pixel value. In the case where the value is in the range from 0.75 to 1 (but not including 1), the area of the identified pixels will be 1×1, 1×2, 2×1, or 2×2 in size.

[0246] In other examples, more than four pixels of the reference frame may be identified. For example, a 3×3 or 4×4 block of pixels in the reference frame around the projected position may be identified.

[0247] Figure 4 A dashed box representing step S411 is shown, where the processing module 302 combines the pixel values of the identified pixels 606 of the reference frame 204 to determine the pixel value at the upsampled pixel location 504. In a simple example, step S411 may include performing bilinear interpolation on the identified pixels 606 of the reference frame. However, in other examples, such as Figure 4 the example shown, step S411 includes steps S412 and S414.

[0248] In step S412, the processing module 302 determines the weight of each of the identified pixels among the identified pixels 606 of the reference frame 204; and in step S414, the processing module 302 uses the determined weights of each of the identified pixels among the identified pixels 606 to determine the pixel value at the upsampled pixel location 504. For example, step S414 may include performing a weighted sum of the pixel values of the identified pixels 606 of the reference frame 204, and this performance is carried out using the determined weights for each of the identified pixels in the weighted sum.

[0249] The determination of the weights of the identified pixels 606 in step S412 can be performed in multiple steps. For example, an initial weight of the identified pixels can be determined, and then this initial weight can be used (or'refined') to determine the (final) weights of the identified pixels of the reference frame. For example, the initial weights of each of the identified pixels (6061 to 6064) of the reference frame 204 can be determined by: determining the distance between the projection position 604 and the position of the identified pixel 606 in the reference frame 204, and then mapping this distance to the initial weight using a predetermined relationship. The distance is Figure 6 shown by a dashed line. The distance can be any suitable distance metric, such as the L2 distance, the squared L2 distance, or the L1 distance. Generally, a linear or non - linear function can be used or a machine - learning method (e.g., using a neural network for calculating weights) can be utilized to determine the initial weights.

[0250] The predetermined relationship for mapping the distance to the initial weight can be any suitable relationship, such as a relationship defined by a function that is monotonically decreasing with distance and provides positive values within the distance range from 0 to such as a Gaussian relationship, a linear relationship, or a relationship defined by a suitable cosine function. Figure 7Illustrates a linear relationship (using dashed line 702) and a Gaussian relationship (using solid line 704) for mapping the distance between the projected position in the reference frame and the position of the pixel to the initial weight of the pixel value to be used to determine the upsampled pixel position. Using the Gaussian relationship to define the initial weight can be beneficial in reducing the influence of more distant pixels. For example, the influence of the pixel (6064) closest to the projected position 604 can be enhanced relative to other identified pixels (6061, 6062, and 6063). The initial weight (w i,k ) of the identified pixel k in the reference frame 204 can be determined using the distance (d) according to the Gaussian relationship as In different embodiments, the variance σ of the Gaussian function w 2 can be different. As an example, the variance σ of the Gaussian function w 2 can be set to 0.4. The initial weights can then be used to determine the (final) weights of the identified pixels 606 in the reference frame 204.

[0251] In the examples described herein, the weight of each identified pixel in the identified pixels 606 of the reference frame 204 can be determined based on: (i) the upsampled depth value of the current frame 202 at the pixel position 504, and (ii) the depth value of the position of the identified pixels 606 of the reference frame 204. By taking the depth value into account when determining the weight, the temporal resampling process can mitigate the blurring effect that might otherwise be introduced when applying temporal resampling near the edges of the objects represented in the frame. For example, if the edge of an object in the scene passes through the region represented by the identified pixels 606 in the reference frame 204, and if all the identified pixels are equally weighted, the effect would be to introduce blurring into the pixel values across the edge of the object. Since only some of the pixel values of the current frame are determined by temporal resampling, the presence of blurring in these pixel values while not in others would result in blocky artifacts, such as jaggedness, which would be very noticeable to the observer of the image. Additionally, by taking the depth value into account when determining the weight, the temporal resampling process can exclude occlusions. Rejecting hidden / misprojected samples can improve edge sharpness and handle occlusions. If all pixels are rejected in this way, a history correction process (described below) can be used to fill in the missing pixel values. Generally, the depth of the objects in the scene does not vary significantly between the frames of the frame sequence. Therefore, if the depth value of the identified pixels 606 of the reference frame 204 is similar enough to the upsampled depth value of the pixel position 504 of the current frame 202, the identified pixel 606 can be considered to represent an adjacent point on the same surface as the upsampled pixel position 504 of the current frame, and thus can be given a relatively high weight. Conversely, if the depth value of the identified pixels 606 of the reference frame 204 is not similar enough to the upsampled depth value of the pixel position 504 of the current frame 202, the identified pixel 606 can be considered to represent a point that is not adjacent to the point represented by the upsampled pixel position 504 of the current frame, which indicates crossing an occlusion boundary, and thus can be given a relatively low weight.

[0252] Specifically, the weight of each identified pixel in the identified pixels 606 of the reference frame 204 can be determined based on the difference between the depth value of the current frame at the upsampled pixel position 504 and the depth value of the position of the identified pixels 606 of the reference frame. Additionally, the weight of each identified pixel in the identified pixels 606 of the reference frame 204 can be determined based on the standard deviation σ depth , of the depth values determined in step S406 d . For example, the difference between the depth value of the current frame at the upsampled pixel position 504 and the depth value of the position of the identified pixels 606 of the reference frame can be compared with a depth threshold T depth。The tolerance of the depth test (i.e., the value of T d can be adaptive. For the following reasons, it is useful to make the tolerance of the depth test (i.e., the value of T d adaptive: (i) If the current frame includes an oblique view of a surface, there will be a higher depth error when comparing the depth value of the current frame at that position with the depth of the corresponding pixel in the reference frame, which means that a larger tolerance can be useful for avoiding rejecting valid pixels; (ii) The processing system usually does not control the scale of the depth. For example, some scenes can be rendered with distances in meters, while other scenes can be rendered with distances in millimeters. Therefore, the value of T d can be adjusted to correct for the scale in some way to have a robust depth test; and (iii) The performance of the depth test for nearby objects and distant objects should be similar. Note that non - adaptive methods (i.e., methods where the value of T d is not adaptive) will only consider the single pixel being compared. A typical non - adaptive method would be to determine a threshold (i.e., T d ) for that position based on the depth value of the current position, e.g., + / - 10%. Such non - adaptive methods will assign a larger acceptable depth range to more distant positions (with large depth values) and a smaller acceptable depth range to positions closer to the camera (with small depth values). In contrast, in the examples described herein, each position is similarly processed by using an adaptive method that, for example, considers the depths of the pixels surrounding the position being compared based on the standard deviation of the depth values of the surrounding pixels.

[0253] If the depth of the identified pixel 606 from the reference frame 204 differs from the depth of the upsampled pixel position 504 in the current frame by more than a threshold amount T d , then the final weight of the identified pixel of the reference frame can be set low, e.g., zero. In other words, in response to determining that the difference between the depth value of the current frame for the upsampled pixel position 504 and the depth value of the position of the identified pixel 606 of the reference frame is greater than the depth threshold T d , the weight of the identified pixel 606 of the reference image can be determined to be lower (e.g., zero). The depth threshold T d can be a hard (binary) threshold or a soft threshold. In the case where the depth threshold T d is a soft threshold, then the weight of the identified pixel 606 of the reference image depends on the difference between the depth value of the current frame for the upsampled pixel position 504 and the depth value of the position of the identified pixel 606 of the reference frame, such that as the difference increases, the weight of the identified pixel 606 decreases.

[0254] In an example using a hard depth threshold, this can be expressed in a more mathematical way as: the weight k of the identified pixel w of the reference image 204 can be determined such that w k = w k ·(|D i,k - D ref,k | ≤ T curr ), where T d is the depth threshold, where T d = F d · σ depth , and where w depth is the initial weight of the identified pixel of the reference image (e.g., determined according to the distance to the projection position 604 and using the predetermined relationship described above), D i,k is the depth value of the position of the identified pixel 606 of the reference frame, D ref,k is the depth value of the current frame for the upsampled pixel position 504, F curr is a predetermined factor, and σ depth is the determined standard deviation of the depth values within the region 510 around the upsampled pixel position 504 of the current frame. In different specific implementations, the predetermined factor F depth can be set by the developer to have different values, but for example, F depth can be 2. In some examples, the predetermined factor F depth can be a trainable parameter that can be pre-trained for a specific application. In the equation given above, if |D depth - D ref,k | ≤ T curr , then (|D d - D ref,k | ≤ T curr ) = 1, and if |D d - D ref,k | > T curr , then (|D d - D ref,k | ≤ T curr ) = 0. Therefore, if the difference between the depth value of the current frame for the upsampled pixel position 504 and the depth value of the position of the identified pixel 606 of the reference frame is not greater than the depth threshold T d , then w d = w k ; and if the difference between the depth value of the current frame for the upsampled pixel position 504 and the depth value of the position of the identified pixel 606 of the reference frame is greater than the depth threshold T i,k , then w d = 0, where w k= 0. In this way, pixels 606 identified from the previous frame 204 that have a depth significantly different from the upsampled pixel position 504 in the current frame 202 are rejected, which avoids (or at least reduces) artifacts that may be caused by blurring at object edges. The standard deviation σ depth makes the threshold T d adaptive. The predetermined factor F depth defines the confidence interval of the region. For example, having F depth = 2 corresponds to 95% coverage of the depth of region 510.

[0255] As an example of a soft threshold, Gaussian weighting can be used The advantage of using a soft threshold instead of a hard threshold is that it helps avoid sudden transitions between included pixels and rejected pixels, which may manifest as temporal artifacts. Additionally, using a soft threshold also makes the algorithm continuously differentiable, which is useful in terms of being able to train the F depth factor.

[0256] As can be seen from the above description, determining the weights of the identified pixels involves per-identified-pixel depth testing. The use of per-pixel depth testing does not blur object edges and classifies relevant parts of the reference frame for temporal resampling. Thus, the quality of the temporal resampling of the method described herein is better than that achievable with other known techniques (such as bilinear sampling methods). Additionally, by taking into account the regions of depth values in the current frame and the regions of depth values in the reference frame (instead of comparing single pixels with single pixels), the depth testing described herein is made robust and adaptive in a way that results in a much higher quality depth test and thus improved temporal resampling.

[0257] In some specific implementations, an S-shaped function can be used instead of the threshold comparison, which will make the solution differentiable and thus known training algorithms (such as error-based backpropagation) can be used for training.

[0258] Note that in the exceptional case where the weights of all identified pixels 606 in the reference frame 204 are determined to be zero, the pixel value at the upsampled pixel position 504 can be determined to be equal to the average μ of the input pixel values within the region 510 around the upsampled pixel position 504 in the current frame 202 pixel . This may occur frequently in unoccluded regions of the current frame. As an alternative to using the average of the (current frame) input pixels, a historical correction process (described below) can be relied upon in this case.

[0259] As described above, in step S414, when the weights of the identified pixels have been determined, the weights can be used to determine the pixel value at the upsampled pixel position by performing a weighted sum, for example. Before the optional history correction (which will now be described), the weights (w) are normalized to make their sum equal to 1, and then the normalized weights (w′) are multiplied by their respective reference input pixels and summed to produce a temporally resampled result. A process referred to herein as "history correction" can be implemented to ensure that the determined pixel value does not differ from the determined average μ pixel of the input pixel values in the current frame 202 within the region 510 surrounding the upsampled pixel position 504 (determined in step S406) by more than a threshold T p , thereby preventing significant errors. For example, step S414 can include clamping the determined pixel value such that it does not differ from the determined average μ pixel of the input pixel values in the current frame within the region 510 surrounding the upsampled pixel position 504 by more than a threshold T p . The threshold T p can be based on the standard deviation σ pixel of the input pixel values in the current frame within the region 510, as determined in step S406. Specifically, the threshold T p can be determined as T p = F pixel ·σ pixel , where F pixel is a threshold factor that can be fixed or variable. The threshold factor F pixel is a predetermined factor that can be pre-trained. In different embodiments, the threshold factor F pixel can have different values and can be set by the developer. For example, F pixel can be 2. Figure 8 shows a graph illustrating the clamping of the determined pixel value at the upsampled pixel position 504. The dashed line 802 represents the case where no history correction is applied (i.e., no clamping) so that the pixel value remains unchanged. The solid line 804 represents the result of applying history correction (e.g., clamping) to the pixel value. If the (unclamped) pixel value is within the range from (μ pixel - T p ) to (μ pixel + T p ), then the history correction (i.e., clamping) does not change the pixel value. However, if the (unclamped) pixel value is less than (μ pixel - T p ), then the clamped pixel value 804 is set equal to (μ pixel - T p ); and if the (unclamped) pixel value is greater than (μ pixel + T p) then set the clamped pixel value 804 to be equal to (μ pixel + T p ).

[0260] The historical correction process described in the previous paragraph ensures that the resampled pixel values do not differ too much from the neighboring pixel values of the current low - resolution image 202. Historical correction is useful when the appearance at the projected position 604 indicated by the motion vector 602 in the reference frame 204 is not a good match for the appearance at the corresponding position 504 in the current frame 202. For example, historical correction is useful when the motion vector does not represent the actual motion between frames (e.g., for transparent objects, for transparent overlays, or for objects such as fire or mirrors). The historical correction method can be applied only to a single channel (the Y channel), and the colors can be filled from the known correct U and V values from the current frame, for example, using simple spatial upsampling (such as bilinear upsampling) and relying on the low sensitivity of human vision to chrominance (UV) spatial resolution. This is simple, effective, and inexpensive (e.g., in terms of power, bandwidth, and computation) compared to other techniques that operate in a 3D color space, and can produce improved visual quality compared to such techniques because the chrominance values at the upsampled pixel positions are derived from nearby true values in the current frame (i.e., the possibility of severe chrominance errors caused by incorrect temporal resampling is completely avoided).

[0261] Figure 9 Three versions of a portion of the upsampled frame are shown: (i) the ground - truth version 902, (ii) the version 904 to which historical correction has been applied to the pixel values, and (iii) the version 906 to which historical correction has not been applied to the pixel values. This portion of the upsampled frame includes a rendering of fire. Region 908 of version 906 to which historical correction has not been applied NHR includes an image of fire, and it can be seen that when historical correction is not applied, significant blocky artifacts are introduced (compared to the corresponding region 908 of the ground - truth version 902 GT ). These blocky artifacts are due to the motion vector not representing the motion of the fire well because the visual effect is not achieved by moving geometric structures. In contrast, the clamping applied by implementing historical correction greatly reduces the significance of these blocky artifacts, which can be seen in the corresponding region 908 of version 904 to which historical correction has been applied HR .

[0262] However, historical correction may not always be beneficial. For example, historical correction sometimes incorrectly removes small image features (e.g., lines with a thickness approximately corresponding to the size of one upsampled pixel). For example, region 910 of the ground - truth version 902 GT includes a thin dark horizontal line near the top of the region, and it can be seen that the corresponding region 910 of version 904 to which historical correction has been appliedHR This dark line does not exist in [it]. In contrast, the corresponding region 910 in version 906 without applying historical correction to its application history NHR includes this thin dark line.

[0263] Thus, in some examples, historical correction can be selectively applied to some regions of the image but not to other regions. Generally, for the entire region of the image rather than isolated pixels, the motion vectors will be incorrect or unreliable, allowing a method based on local neighborhood statistics to be used to selectively enable or disable the method, or alternatively adjust the threshold T p . For example, pixel values can be determined within region 510 surrounding the upsampled pixel position 504 without performing historical correction. The processing module 302 can compare the average value of the pixel values determined within region 510 with the average value μ of the input pixel values within region 510 of the current frame 202 pixel . If the difference between the average value μ of the pixel values determined within region 510 resampled and the average value μ of the input pixel values within this region 510 pixel is greater than the threshold difference, historical correction (i.e., clamping) is performed; while if the difference between the average value of the pixel values determined within region 510 and the average value μ of the input pixel values within this region 510 pixel is not greater than the threshold difference, historical correction (i.e., clamping) is not performed. For example, the difference between the average value of the pixel values within region 908 of version 906 NHR and the average value μ of the input pixel values within this region pixel (which will look similar to 908 of the ground truth version 902 GT ) will be large, for example greater than the threshold difference (if a suitable threshold difference is used), such that historical correction will be applied to this region so that this region of the upsampled image will look like region 908 of version 904 HR . As another example, the difference between the average value of the pixel values within region 910 of version 906 NHR and the average value μ of the input pixel values within this region pixel (which will look similar to 910 of the ground truth version 902 GT ) will be small, for example less than the threshold difference (if a suitable threshold difference is used), such that historical correction will not be applied to this region so that this region of the upsampled image will look like region 910 of version 906 NHR .

[0264] In some examples, different levels of historical correction (i.e., different levels of clamping) can be applied to different regions of the image, for example by changing the value of F in different regions. For example, the threshold T used for historical correction pixel can be...p Determined as In this way, if the average value μ of the pixel values determined within region 510 resampled is equal to the mean value μ of the input pixel values pixel (i.e., if μ resampled = μ pixel ), then the threshold T p is the same as the threshold given above, i.e., T p = F pixel ·σ pixel ; and as the difference between μ resampled and μ pixel increases in magnitude, the threshold T p decreases, which means that historical correction is more likely to be applied. Generally, the threshold can be a soft threshold. Additionally, generally, different degrees of clamping can be selectively applied to different regions. Specifically, the average value of the pixel values determined at the upsampled pixel positions within the region surrounding the upsampled pixel position can be compared with the mean value of the input pixel values in the current frame within the region surrounding the upsampled pixel position, and clamping can be performed based on the comparison of the following items: (i) the difference between the average value of the pixel values determined at the upsampled pixel positions within the region surrounding the upsampled pixel position and the mean value of the input pixel values in the current frame within the region surrounding the upsampled pixel position, and (ii) the threshold difference.

[0265] When pixel values have been determined for the upsampled pixel position 504, then in step S416, the processing module 302 determines whether there is another upsampled pixel position for which pixel values are to be determined. If there is another upsampled pixel position for which pixel values are to be determined, the method returns from step S416 to step S406, and steps S406 to S416 are performed to determine the pixel values of the next upsampled pixel position. Each of the determined pixel values represents the value of the upsampled pixel at the corresponding upsampled pixel position that does not correspond to the position of any input pixel of the current frame. As described above, the upsampled pixel positions (or "temporally resampled pixel positions") are located between the positions of the diagonally adjacent input pixels of the current frame, such that the upsampled pixel positions and the positions of the input pixels form a repeating five-point pattern. As described in more detail below, spatial upsampling can be performed on the resampled pixels to determine the further upsampled pixel values of the further upsampled pixels at the further upsampled pixel positions between adjacent positions of the repeating five-point pattern. In some examples, the pixel values at the upsampled pixel positions can be determined in parallel, i.e., Figure 4 the loops in the method shown do not necessarily imply sequential processing.

[0266] If, in step S416, it is determined that there is no other upsampled pixel position for which to determine a pixel value, the method proceeds from step S416 to step S418. In step S418, the processing module 302 outputs the determined pixel value of the current frame, for example for implementing a super-resolution technique. The output pixel value can be used in any suitable manner, such as being displayed on a display, stored in a memory, or transmitted over a network (such as the Internet) to another device. The processing module 302 can continue to determine the pixel values of all frames in the frame sequence 304.

[0267] Now refer to Figure 10 A more general example method for determining weights is described, which are used to combine pixel values 606 from the reference frame 204 for determining the pixel value at the upsampled pixel position 504 of the current frame 202. As described above, the pixel position 504 of the current frame is projected onto the position 604 in the reference frame 204. A set of reference frame pixels (e.g., four pixels) (e.g., pixels 6061, 6062, 6063, and 6064 surrounding the projection position 604) defines a neighborhood Ω within the reference frame (labeled 1002 in Figure 10 ). In other examples, the neighborhood Ω can include a different number of pixels of the reference frame. For example, the neighborhood Ω can include a set of 1×1, 2×1, 1×2, or 2×2 pixels within a box centered on the projected pixel position 604. Generally, the temporally resampled pixel value at the upsampled pixel position 504 can be determined as a linear combination of the pixel values 606 within the neighborhood Ω of the re-projected pixel position 604 in the reference frame 204. This can be achieved according to a weighted sum, where the normalized weight w i ′ is multiplied by the value x i of the corresponding reference pixel i, and the resulting values are summed to produce the re-projected pixel value at the upsampled pixel position 504 of the current frame 202y. Expressing this mathematically is:

[0268]

[0269] In the exceptional case where all weights within the neighborhood Ω are zero (which can occur when all reference pixels are occluded), a fallback spatial interpolation algorithm can be used. For example, the upsampled pixel value can be determined to be equal to the average of the neighboring input pixel values of the current frame. When determining the re-projected pixel value, the weights can be considered as an indication of the reliability of a given reference pixel 606. Aspects that can reduce the reliability of the reference pixel 606 (which can result in a smaller corresponding weight for using that reference pixel) will include:

[0270] i. A large spatial distance between the re-projected sample position 604 and the pixel position 606 in the reference frame. This has been described above.

[0271] ii. A large depth difference between the upsampled pixel position 504 in the current frame 202 and the pixel 606 in the reference frame 204, which may indicate lack of visibility due to occlusion. This has been described above.

[0272] iii. Pixel values that are outliers in the context of the neighborhood Ω. This is described below.

[0273] iv. Depth values that are outliers in the context of the neighborhood Ω. This is described below.

[0274] The reliability of the reference pixel 606 can be determined individually based on some or all of these cues, for example as a separate weight for each reference pixel. These weights can be determined in any order and then combined, for example by multiplying and then normalizing to a sum of one, such that a total weight w i ′ is obtained that represents the confidence placed in the reference pixel i.

[0275] Aspects iii and iv in the list given in the previous paragraph determine the weight of the corresponding reference pixel 606 based on whether the reference pixel is an outlier compared to other pixels in the neighborhood Ω of the reference frame. In other words, the weight of each identified pixel in the identified pixels of the reference frame (i.e., each reference pixel 606 in the neighborhood Ω) is determined based on the degree to which the identified pixel of the reference frame is an outlier compared to other identified pixels of the reference frame. The reference pixel 606 may be an outlier in terms of its pixel value or depth value. In other words, the influence of the reference pixels in the neighborhood of the reprojection sample position 604 in the reference frame 204 on the reconstructed pixel value at the upsampled pixel position 504 of the current frame 202 depends on the similarity of the reference pixel to its neighboring pixels, rather than on the value at the upsampled pixel position 504. This technique is independent of the position of the projection position 604 within the neighborhood Ω of the reference frame 204 and is also independent of the pixel values and depth values in the current frame 202. This provides a simple technique for improving the quality of temporal resampling. Reference pixels that are similar to the reference pixels in the neighborhood Ω have higher weights; and reference pixels that are not similar to the reference pixels in the neighborhood Ω have lower weights. This can be implemented in different ways in different embodiments, but some examples are given below. The same method can be applied to depth values or pixel values (e.g., Y-channel values) or both. In the examples given below, the values are labeled as P = [P1, P2, P3, P4] for the corresponding pixels 6061, 6062, 6063, and 6064, where the P values can be depth values or pixel values (e.g., Y-channel values).

[0276] As a first example, the "distance" from the mean in the neighborhood can be used. This is a simple and efficient technique. The mean of these values can be determined as each value P at pixel position ii The "distance" (difference) from the mean is determined as |P i - μ|. This is converted into weights that assign low values to less similar pixels (i.e., those with a higher "distance" from the mean). One approach is to use the softmax function with a temperature constant τ. The value of this constant can be chosen during implementation. For example, τ can be 0.1. To express this more mathematically, the contribution of the distance from the mean (w mean ) to the weights can be determined as:

[0277]

[0278] The weight vector w mean returned by this function, for its i-th component w mean,i corresponds to the i 第 -th element P i of the value vector P.

[0279] As a second example, outliers can be determined based on the gradients between the reference pixels 606 in the neighborhood Ω of the reference frame 204. In the Figure 10 example shown, six gradients (d1 to d6) are defined as follows:

[0280] d1 = |P1 - P2|

[0281] d2 = |P2 - P4|

[0282] d3 = |P3 - P4|

[0283] d4 = |P3 - P1|

[0284] d5 = |P3 - P2|

[0285] d6 = |P4 - P1|

[0286] Then, the gradients can be accumulated along the edges to the neighboring nodes in an amount Q i corresponding to the pixel position i such that:

[0287] Q1 = d1 + d4 + d6 Q2 = d1 + d2 + d5 Q3 = d3 + d4 + d5 Q4 = d2 + d3 + d6

[0288] Relatively large values of Q i compared to the other values in Q indicate that the corresponding value P i is an outlier in P. The values of Q i can be converted, for example using the softmax method, into weights based on the gradients (w gradient ):

[0289]

[0290] In the above example, the graphics rendering process determines the pixel positions of the current frame 202 at each pixel position in the current frame (e.g., Figure 5 ) and between diagonally adjacent input pixel positions in the current frame. Figure 5 , and the depth value of each position indicated as 504 and 508 in . Thus, in these examples, the graphics rendering process determines depth values ​​at twice as many positions as it determines pixel values. For example, in a first rendering process, the graphics processing unit may determine a pixel value and a depth value at input pixel position 506 of the current frame 202, and then in a second rendering process, the graphics processing unit may determine depth values ​​at positions 504 and 508. The pixel values ​​and depth values ​​determined by the graphics processing unit are provided to the processing module 302 (e.g., in steps S402 and S404 of the above method). Determining only the depth value is much simpler for the graphics processing unit than determining both the depth value and the pixel value, so the second rendering process can be performed with reduced latency and / or reduced power consumption compared to performing the first rendering process. However, performing two rendering processes for the current frame is still more expensive (e.g., in terms of latency and power consumption) than performing only one rendering process.

[0291] Another possibility is to modify the GPU software (e.g., its driver) to allow the GPU software to render all the five-point depth values ​​in one rendering process. In other examples, the graphics processing unit may perform a single rendering process (equivalent to the first rendering process mentioned in the previous paragraph) for the current frame, and may determine the pixel value and the depth value only at the input pixel position 506 of the current frame 202. In these examples, the graphics processing unit does not determine the depth values ​​at positions 504 and 508. Therefore, in these examples, the processing module 302 receives the pixel value and the depth value of the input pixel position 506 of the current frame 202 from the graphics processing unit, and then the processing module 302 determines the depth values ​​at positions 504 and 508 (in step S404). For example, the depth value at position 504 may be determined to be equal to the depth value at one pixel position of the four neighboring input pixel positions 506 (e.g., the upper left neighboring input pixel position 5061). As another example, the depth value at position 504 may be determined as the average value (e.g., mean or median) of the depth values ​​at the four neighboring input pixel positions 506. Using the mean can be error-prone and may destroy the appearance of edges, whereas using the median may be more robust to edges. Figure 11A and Figure 11B Examples of more complex methods by which processing module 302 may determine depth values ​​at locations 504 and 508 are described. Figure 11A An example method for determining a depth value for an upsampled pixel position 504 of a current frame 202 is illustrated. Figure 11B is based onFigure 11A Flowchart of the method steps for determining the depth value of the upsampled pixel position 504 in the illustrated example.

[0292] In step S1102, the processing module 302 receives the depth values of the current frame 202 at the positions of the input pixels 5061, 5062, 5063, and 5064 surrounding the upsampled pixel position 504. As described above, these depth values can be received from a graphics processing unit that determines the depth values by performing a graphics rendering process.

[0293] In step S1104, for each pair of input pixels for which depth values are received, the processing module 302 determines an interpolated depth value for the upsampled pixel position 504 based on the depth values of the pair of input pixels. In this way, six interpolated depth values are determined for the upsampled pixel position 504. Specifically:

[0294] A first interpolated depth value (d int,1 ) of the upsampled pixel position 504 is determined for the pair of input pixels 5061 and 5062. For example,

[0295] A second interpolated depth value (d int,2 ) of the upsampled pixel position 504 is determined for the pair of input pixels 5062 and 5064. For example,

[0296] A third interpolated depth value (d int,3 ) of the upsampled pixel position 504 is determined for the pair of input pixels 5063 and 5064. For example,

[0297] A fourth interpolated depth value (d int,4 ) of the upsampled pixel position 504 is determined for the pair of input pixels 5061 and 5063. For example,

[0298] A fifth interpolated depth value (d int,5 ) of the upsampled pixel position 504 is determined for the pair of input pixels 5062 and 5063. For example, And

[0299] A sixth interpolated depth value (d int，6 ) of the upsampled pixel position 504 is determined for the pair of input pixels 5061 and 5064. For example,

[0300] where d1, d2, d3, and d4 are the depth values of the input pixels 5061, 5062, 5063, and 5064, respectively.

[0301] In step S1106, the processing module 302 determines the depth weight of the input pixel pair based on the depth gradient between the depth values of the input pixel pair. For example, in step S1106, the processing module 302 can determine the depth gradient for each input pixel pair of the input pixels for which it receives depth values, based on the depth values of the input pixel pair. In this way, six depth gradients can be determined. Specifically:

[0302] A first depth gradient (δ1) of the upsampled pixel position 504 can be determined for the input pixel pair 5061 and 5062. For example, δ1 = |d2 - d1|;

[0303] A second depth gradient (δ2) of the upsampled pixel position 504 can be determined for the input pixel pair 5062 and 5064. For example, δ2 = |d4 - d2|;

[0304] A third depth gradient (δ3) of the upsampled pixel position 504 can be determined for the input pixel pair 5063 and 5064. For example, δ3 = |d4 - d3|;

[0305] A fourth depth gradient (δ4) of the upsampled pixel position 504 can be determined for the input pixel pair 5061 and 5063. For example, δ4 = |d3 - d1|;

[0306] A fifth depth gradient (δ5) of the upsampled pixel position 504 can be determined for the input pixel pair 5062 and 5063. For example, δ5 = |d2 - d3|; and

[0307] A sixth depth gradient (δ6) of the upsampled pixel position 504 can be determined for the input pixel pair 5061 and 5064. For example, δ6 = |d4 - d1|.

[0308] Then the depth gradient can be converted into a depth weight. For example, the depth weight of the input pixel pair can be determined by the following operations: (i) multiplying the depth gradient (δ i ) of the input pixel pair by a negative number β, and (ii) inputting the result of the multiplication into the softmax function. β is a negative number, i.e., β < 0, and as an example, β = -2, but it can be different in other examples.

[0309] The softmax function is known to those skilled in the art. For example, the input (z i ) to the softmax function is the result of multiplying the depth gradient by a negative number (i.e., z i = βδ i ), and the output of the softmax function (i.e., the depth weight w d,i ) can be determined as The softmax function normalizes the input between 0 and 1 using an exponent. Changing the value of β changes the distribution of the determined depth weights. By setting β to a negative number, the softmax function maps larger depth gradients to smaller depth weights and smaller depth gradients to larger depth weights. This is beneficial such that if an input pixel pair has similar depths (i.e., a low depth gradient between their depth values), then that input pixel pair is weighted more strongly than an input pixel pair with very different depths (i.e., a high depth gradient between their depth values) when determining the depth value of the upsampled pixel position 504.

[0310] The processing module 302 may include logic 1100 for performing a weighted sum. In step S1108, the logic 1100 determines the depth value (D) of the current frame at the upsampled pixel position 504 by performing a weighted sum of the determined interpolated depth values (d d,i ) using the determined depth weights (w int,i ) for the input pixel pair i. That is, where P is the number of input pixel pairs (P = 6 in the example described in detail herein). Once the depth value of the upsampled pixel position 504 has been determined, the method may proceed as described above with reference to Figure 4 as described.

[0311] Figure 12 Illustrates a processing system 1202 for determining pixel values at upsampled pixel positions in a current frame of a frame sequence as described herein. The processing system 1202 includes a graphics rendering unit 1204, temporal resampling logic 1206, and spatial upsampling logic 1208. Note that the graphics rendering unit 1204 may be referred to as a graphics processing unit. The temporal resampling logic 1206 may be the same as the temporal resampling logic 306 of the processing module 302 described above. The spatial upsampling logic 1208 may or may not be implemented on the same processing module as the temporal resampling logic 1206. Each of the graphics rendering unit 1204, the temporal resampling logic 1206, and the spatial upsampling logic 1208 may be implemented in software, hardware, or a combination thereof.

[0312] Figure 13 Is a flowchart of a method for determining pixel values at upsampled pixel positions in a current frame of a frame sequence. Additionally, Figure 14 Illustrates applying temporal resampling and spatial upsampling to frames in a frame sequence according to the method shown in the flowchart of Figure 13 . The current frame is labeled 1402 in Figure 14 and this may be the same as the current frame 202 in the example described above. Additionally, the reference frame is in Figure 14It is labeled as 1404, and this can be the same as reference frame 204 in the above example. Reference frame 1404 can be the previous frame or the next frame relative to the current frame 1402 in the frame sequence. For example, reference frame 1404 can be the frame immediately preceding the current frame 1402 in the frame sequence.

[0313] In step S1302, the graphics rendering unit 1204 determines the pixel values at the first subset of the upsampled pixel positions of the current frame 1402 using a graphics rendering process. Techniques for rendering pixel values in the graphics processing unit 1204 are known in the art and thus will not be described in detail herein. For example, the graphics rendering process can be a rasterization process or a ray tracing process. As described above, the pixel value can be a Y-channel value.

[0314] Figure 14 The pixel values (e.g., pixel 1406 in the current frame 1402 and pixel 1408 in the reference frame 1404) rendered by the graphics processing unit 1204 with diagonal hatching are shown. Figure 14 The squares of the current frame 1402 and the reference frame 1404 not shown with hatching in are the upsampled pixel positions for which the graphics processing unit 1204 does not render pixel values. Figure 14 It can be seen that for each frame, the graphics processing unit 1204 renders the pixel values of one quarter of the upsampled pixel positions, and the rendered pixel values are evenly distributed throughout the frame such that for each 2×2 block of upsampled pixel positions, there is one rendered pixel value and three upsampled pixel positions for which the graphics processing unit does not render pixel values.

[0315] In step S1304, the temporal resampling logic 1206 determines the pixel values at the second subset of the upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels in the reference frame 1404 in the frame sequence. Reference can be made, for example, to Figure 4 the method shown, and temporal resampling is performed as described above. Specifically, for each upsampled pixel position in the second subset of upsampled pixel positions, the pixel value can be determined by: using the motion vector of the upsampled pixel position to identify a plurality of pixels in the pixels of the reference frame, and combining the pixel values of the identified pixels in the reference frame to determine the pixel value of the upsampled pixel position in the second subset. The result of the temporal resampling is labeled as 1410 in Figure 14 where the pixel values at the first subset of upsampled pixel positions (e.g., position 1406) are shown with upward diagonal hatching, and the pixel values at the second subset of upsampled pixel positions (e.g., position 1412) are shown with downward diagonal hatching.

[0316] In Figure 14In the example shown, the upsampled pixel positions (at 1410) in the first subset and the second subset form a repeating five-point pattern, i.e., a checkerboard pattern. That is, the upsampled pixel positions in the second subset (e.g., position 1412) are located between diagonally adjacent upsampled pixel positions in the first subset (e.g., position 1406). Thus, for each upsampled pixel position in the first subset that is not on the edge of the current frame, the four nearest upsampled pixel positions in the repeating five-point pattern are the upsampled pixel positions in the second subset; and for each upsampled pixel position in the second subset that is not on the edge of the current frame, the four nearest upsampled pixel positions in the repeating five-point pattern are the upsampled pixel positions in the first subset.

[0317] A dither pattern can be used over a sequence of frames such that a graphics rendering process is used to determine pixel values at different upsampled pixel positions in different frames of the sequence of frames. In Figure 14 the example shown, the subset of upsampled pixel positions for which pixel values are determined using the graphics rendering process alternates between a first subset of upsampled pixel positions and a second subset of upsampled pixel positions for consecutive frames in the sequence of frames.

[0318] In step S1306, the spatial upsampling logic 1208 determines the pixel values at a third subset of upsampled pixel positions of the current frame 1402 by applying spatial upsampling to the determined pixel values at the upsampled pixel positions in the first subset and the second subset. Techniques for applying spatial upsampling are known in the art and some examples of how to apply spatial upsampling are described below with reference to Figures 15 to 19b The upsampled pixel positions in the third subset are located in the gaps of the repeating five-point pattern. In other words, the upsampled pixel positions in the third subset fill the gaps of the repeating five-point pattern shown in 1410. The result of the spatial upsampling is labeled 1414 in Figure 14 where the pixel values at the first subset of upsampled pixel positions (e.g., position 1406) are shown with upward diagonal hatching, the pixel values at the second subset of upsampled pixel positions (e.g., position 1412) are shown with downward diagonal hatching, and the pixel values at the third subset of upsampled pixel positions (e.g., position 1416) are shown with diagonal cross-hatching. As can be seen in 1414, each upsampled pixel position in the third subset that is not on the edge of the current frame is located between: (i) two horizontally adjacent upsampled pixel positions in the first subset and two vertically adjacent upsampled pixel positions in the second subset, or (ii) two vertically adjacent upsampled pixel positions in the first subset and two horizontally adjacent upsampled pixel positions in the second subset.

[0319] It can be seen that 1414 has pixel values for each upsampled pixel position in the upsampled pixel positions, i.e., there are no spaces in 1414 (i.e., no hatched squares). The first subset, second subset, and third subset of upsampled pixel positions are different such that there is no upsampled pixel position that belongs to more than one of the first subset, second subset, and third subset. Further, all upsampled pixel positions of the current frame belong to one of the first subset, second subset, and third subset. As Figure 14 shown by 1414 in the example of

[0320] , for a 2×2 block of upsampled pixel positions, there is one position from the first subset (shown with upward diagonal hatching), one position from the second subset (shown with downward diagonal hatching), and two positions from the third subset (shown with diagonal cross hatching). The two positions from the third subset are diagonally opposite each other in the 2×2 block. Thus, one quarter of the upsampled pixel positions of the current frame are in the first subset, one quarter of the upsampled pixel positions of the current frame are in the second subset, and half of the upsampled pixel positions of the current frame are in the third subset.

[0321] Note that in the examples described herein, the combination of temporal resampling and spatial upsampling means that 2x upsampling can be applied to the pixel values rendered by the graphics rendering unit 1204, i.e., the graphics rendering unit 1204 only needs to render one quarter of the pixel values in the final upsampled image. In addition, due to the dither pattern of the pixel positions rendered by the graphics rendering unit 1204 for different frames in the frame sequence, temporal resampling can be used to determine the pixel values at the upsampled positions of the pixel values for which the graphics rendering unit 1204 has rendered the reference frames. The pixel values at the same positions in consecutive frames are not likely to vary greatly, so temporal resampling at these positions can provide very accurate pixel values. Due to the repeating five-point pattern of the positions in the first and second subsets, i.e., due to all the positions for which spatial upsampling is used to determine the upsampled pixel values (i.e., all the upsampled pixel positions in the third subset) being horizontally and vertically adjacent to at least one position in the first or second subset for which the pixel values have been determined (usually two, unless it is a position on the frame edge), spatial upsampling is also accurate in this system. Thus, compared to using only one of temporal resampling and spatial upsampling, the combination of temporal resampling and spatial upsampling can provide better quality upsampled pixel values (i.e., upsampled pixel values with fewer visible artifacts), and this is achieved without the graphics rendering unit 1204 having to render more pixel values. For example, the graphics rendering unit 1204 does not need to render more than one quarter of the pixel values. The temporal resampling logic and the spatial upsampling logic can be implemented mainly in hardware (e.g., fixed function circuitry), and they can process pixel values faster (i.e., reduced latency) compared to performing twice as many rendering passes through the graphics rendering unit 1204. Thus, the performance of the processing system 1202 (e.g., in terms of the number of frames output per second) is better than the case where the graphics rendering unit 1204 directly renders more pixel values. In addition, the power consumption and silicon area of the temporal resampling logic 1206 and the spatial upsampling logic 1208 are small, so adding these logic blocks to the processing system 1202 is not expensive. Note that spatial upsampling from 2 out of 4 pixels allows the final pixel values to be close to the ground truth image quality, which is usually not possible when upsampling from 1 out of 4 pixels. In addition, compared to spatial upsampling, temporal resampling is relatively expensive in terms of execution time, power, and bandwidth on resource-constrained devices, especially when attempting to use it to fill all the missing pixels. The combination of temporal resampling and spatial upsampling as described herein provides a good compromise that gives high image quality and low cost (e.g., in terms of execution time, power consumption, bandwidth, and silicon area).

[0322] In some examples, the spatial upsampling logic 1208 may determine pixel values at a third subset of upsampled pixel positions by performing bilinear interpolation on the determined pixel values at the upsampled pixel positions in the first and second subsets. Bilinear interpolation is a simple method for performing spatial upsampling.

[0323] In other examples, the spatial upsampling logic 1208 may use a more complex method to perform spatial upsampling, which avoids or reduces artifacts such as blurring that may be introduced by bilinear interpolation. Refer to Figures 15 to 19b for descriptions of these examples.

[0324] Figure 15 FIG. shows the spatial upsampling logic 1208, which is configured to apply upsampling to pixel values 1410 representing an image region to determine an upsampled pixel value block 1504, for example for implementing super-resolution techniques. The upsampled pixel value block 1504 represents Figure 15 positions within the image region indicated by square 1510. In this example, the pixel values 1410 received by the spatial upsampling logic 1208 have positions corresponding to a 10×10 block of upsampled pixel positions, and the upsampled pixel value block 1504 is a 2×2 upsampled pixel block representing the central 2×2 portion of the 10×10 block of upsampled pixel positions. However, note that in other examples, the shape and / or size of the image regions represented by the pixel values 1410 and the upsampled pixel value block may be different. The spatial upsampling logic 1208 includes pixel determination logic 1506 and weighted parameter determination logic 1508.

[0325] Rather than using bilinear upsampling, the spatial upsampling logic 1208 may perform upsampling that depends on the relative horizontal and vertical variations of the pixel values 1410 within the image region. In this way, compared to the case of using bilinear upsampling, the upsampling takes into account anisotropic features (e.g., edges) in the image to reduce "jaggies" and "aliasing" artifacts and blurring that may occur near anisotropic features (e.g., edges, especially diagonal edges of computer-generated images).

[0326] Refer to Figure 16 for a flowchart that describes a method of applying upsampling to pixel values 1410 using the spatial upsampling logic 1208 to determine the upsampled pixel value block 1504, for example for implementing super-resolution techniques.

[0327] In step S1602, the spatially sampled logic 1208 receives the pixel values 1410 at the upsampled pixel positions in the first subset and the second subset. As described above, the pixel values 1410 of the first subset and the second subset of upsampled pixel positions have positions corresponding to the repeated five-point arrangement (or "checkerboard pattern") of the upsampled pixel positions. The first subset of upsampled pixel positions corresponds to positions within the odd rows of the repeated five-point arrangement of upsampled pixel positions, and the second subset of upsampled pixel positions corresponds to positions within the even rows of the repeated five-point arrangement of upsampled pixel positions.

[0328] In step S1604, the weighting parameter determination logic 1508 analyzes the pixel values 1410 at the upsampled pixel positions in the first subset and the second subset to determine one or more weighting parameters. The one or more weighting parameters indicate the directionality of the filtering to be applied when upsampling the determined pixel values at the upsampled pixel positions in the first subset and the second subset. For example, the one or more weighting parameters may indicate the relative horizontal and vertical variations of the pixel values 1410 at the upsampled pixel positions in the first subset and the second subset. Thus, the weighting parameters may be referred to as directional weighting parameters. For example, two weighting parameters (a and b) may be determined in step S1604. The weighting parameters may be normalized such that a + b = 1. This means that as one of a or b increases, the other decreases. It will be apparent from the following description that if the parameters are set such that a = b = 0.5, the system will give the same output as a bilinear upsampler. However, in the system described herein, a and b may be different, i.e., a ≠ b. Additionally, since b = 1 - a, the weighting parameter determination logic 1508 may output an indication of a single weighting parameter, such as a, and this may be used to determine the second weighting parameter b as 1 - a. An indication of the one or more weighting parameters is provided to the pixel determination logic 1506.

[0329] In step S1606, the pixel determination logic 1506 determines one or more of the upsampled pixel values in the upsampled pixel value block 1504 based on the relative horizontal and vertical variations of the pixel values 1410 at the upsampled pixel positions in the first subset and the second subset indicated by the determined one or more weighting parameters.

[0330] In step S1608, the upsampled pixel value block 1504 is output from the pixel determination logic 1506. In some systems, this may be the end of the processing of the upsampled pixel value block 1504, and it may be output from the spatially sampled logic 1208 (and may be output from the processing system 1202), as Figure 15 shown. In other examples, adaptive sharpening (e.g., by blending with sharpened upsampled pixel values) may be applied to the upsampled pixel values before outputting from the spatially sampled logic 1208.

[0331] Figure 17 It is a flowchart showing how the pixel determination logic 1506 determines the (unsharpened) upsampled pixel values based on the relative horizontal and vertical changes of the pixel values 1410 at the upsampled pixel positions in the first subset and the second subset in step S1606. Figure 18 It shows a part of the input pixel values 1802 and illustrates how the positions of the upsampled pixel value blocks 1806 are related to the positions of the input pixel values 1804. In step S1606, each of the upsampled pixel values determined based on the relative horizontal and vertical changes of the pixel values 1410 at the upsampled pixel positions in the first subset and the second subset is located at the corresponding upsampled pixel position in the third subset (rather than the first subset or the second subset). Specifically, in step S1606, based on the relative horizontal and vertical changes of the pixel values 1410 at the upsampled pixel positions in the first subset and the second subset, the upper-right and lower-left upsampled pixel values in the upsampled pixel value block (labeled "TR" and "BL" in Figure 18 ) are determined. If no sharpening is applied, the input pixel values at the positions (in the first subset and the second subset) having a repeated five-point arrangement corresponding to the upsampled pixel positions are used as the upsampled pixel values at these upsampled pixel positions of the upsampled pixel value block (for example, the pixel values at the upper-left and lower-right positions of the upsampled pixel value block 1806 are "passed" to the corresponding positions in the output block). If sharpening is applied, some processing may be performed on the input pixel values to determine all the upsampled pixel values in the upsampled pixel value block.

[0332] As described above, the input pixel values 1804 are located at the positions in the first subset and the second subset. In Figure 18 , the input pixel values 1804 1,1 , 1804 1,2 , 1804 1,3 and 1804 1,4 (which are shown with upward diagonal hatching) are located at the upsampled pixel positions in the first subset; while the input pixel values 1804 2,1 , 1804 2,2 , 1804 2,3 and 1804 2,4 (which are shown with downward diagonal hatching) are located at the upsampled pixel positions in the second subset.

[0333] As Figure 17As shown, step S1606 of determining each of the one or more upsampled pixel values ​​at the third subset of upsampled pixel positions includes applying one or more kernels to at least some of the pixel values ​​at the upsampled pixel positions in the first subset and the second subset according to the determined one or more weighting parameters. Specifically, in step S1702, the pixel determination logic 1506 references one or more first kernels (which may be referred to as horizontal kernels) to at least a first subset of the input pixel values ​​1804 to determine a horizontal component. For example, a kernel of [0.5, 0.5] may be applied to horizontally adjacent input pixel values ​​and to either side of the upsampled pixel position for which the upsampled pixel value is being determined. For example, when determining the horizontal component for Figure 18 When the upsampled pixel value marked as "TR" in step S1702 is executed, the pixel value 1804 at the upsampled pixel position in the first subset is 1,1 and 1804 1,2 Apply the horizontal kernel. Figure 18 When the up-sampled pixel value marked as "BL" in the second subset is executed in step S1702, the pixel value 1804 at the up-sampled pixel position in the second subset is 2,3 and 1804 2,4 Apply a horizontal kernel.

[0334] In step S1704, the pixel determination logic 1506 applies one or more second kernels (which may be referred to as vertical kernels) to at least a second subset of the input pixel values ​​1804 to determine the vertical component. For example, the second kernels may be applied to vertically adjacent input pixel values ​​and to either side of the upsampled pixel position for which the upsampled pixel value is being determined. For example, when targeting Figure 18 When the upsampled pixel value marked as "TR" in step S1704 is executed, the pixel value 1804 at the upsampled pixel position in the second subset is 2,2 and 1804 2,4 Apply a vertical kernel. Figure 18 When the up-sampled pixel value marked as "BL" in the first subset is executed in step S1704, the pixel value 1804 at the up-sampled pixel position in the first subset is 1,1 and 1804 1,3 Apply a vertical kernel.

[0335] Note that for each upsampled pixel position in one or more of the third subsets for which step S1606 is performed, each upsampled pixel position has two horizontally adjacent pixel values, both of which are located at positions in the same subset (from the first subset or the second subset), and has two vertically adjacent pixels, both of which are located at positions in the other of the first subset and the second subset (from the second subset or the first subset). Note also that steps S1702 and S1704 can be performed in any order (e.g., sequentially), or can be performed in parallel.

[0336] In steps S1706 to S1710, the pixel determination logic 1506 combines the determined horizontal and vertical components such that each of the one or more upsampled pixel values in the upsampled pixel values is determined based on the relative horizontal and vertical variations of the pixel values 1410 at the upsampled pixel positions in the first subset and the second subset as indicated by one or more weighting parameters determined as in step S1604. Specifically, in step S1706, the pixel determination logic 1506 multiplies the horizontal component (determined in step S1702) by a first weighting parameter a in the weighting parameters to determine a weighted horizontal component.

[0337] In step S1708, the pixel determination logic 1506 multiplies the vertical component (determined in step S1704) by a second weighting parameter b in the weighting parameters to determine a weighted vertical component. Steps S1706 and S1708 can be performed in any order (e.g., sequentially), or can be performed in parallel.

[0338] In step S1710, the pixel determination logic 1506 sums the weighted horizontal component and the weighted vertical component to determine the upsampled pixel value. In some examples, the kernels applied in S1702 and S1704 can be multiplied by the first weighting parameter and the second weighting parameter respectively before or during application to the input pixel values. Compared to the method described with reference Figure 17 it is expected to be less power and area efficient in hardware because Figure 19a and Figure 19b the application of the kernels of can be obtained using only inexpensive fixed displacements and additions, and the number of variable multiplications required is relatively low.

[0339] Figure 19a shows a 3×3 kernel 1902 (which can be implemented as a 1×3 kernel) that can be applied to the input pixel values 1802 to represent the effect of performing steps S1702 and S1706 for the upsampled pixel position for which the upsampled pixel value is being determined. For example, the kernel 1902 can be centered on the upsampled pixel position for which the value is being determined and applied to the input pixel values 1802 such that the result will be given by the dot product is given, where h is a vector of horizontally adjacent pixel values. For example, if determining the upsampled pixel value labeled "TR", then h = [p 1,1 , p 1,2 , where p 1,1 is the value of input pixel 1804 1,1 and p 1,2 is the value of input pixel 1804 1,2 . Similarly, if determining the upsampled pixel labeled "BL", then h = [p 2,3 , p 2,4 , where p 2,3 is the value of input pixel 1804 2,3 and p 2,4 is the value of input pixel 1804 2,4 .

[0340] Similarly, Figure 19b FIG. shows a 3×3 kernel 1904 (which can be implemented as a 3×1 kernel) that can be applied to input pixel 1802 to represent the effect of performing steps S1704 and S1708 for the upsampled pixel position for which the upsampled pixel value is being determined. For example, the kernel 1904 can be centered on the upsampled pixel position for which the value is being determined and applied to the input pixel 1802 such that the result will be given by the dot product is given, where v is a vector of vertically adjacent pixel values. For example, if determining the upsampled pixel value labeled "TR", then v = [p 2,2 , p 2,4 , where p 2,2 is the value of input pixel 1804 2,2 and p 2,4 is the value of input pixel 1804 2,4 . Similarly, if determining the upsampled pixel value labeled "BL", then v = [p 1,1 , p 1,3 , where p 1,1 is the value of input pixel 1804 1,1 and p 1,3 is the value of input pixel 1804 1,3 .

[0341] Thus, steps S1702 to S1710 can be summarized as determining the upsampled pixel values labeled TR and BL as the sum of dot products where h and v are the vectors of adjacent pixel values as described above for the TR and BL pixels, respectively.

[0342] The values of the weighting parameters a and b can be set according to the local context such that in step S1606, the upsampled pixel values are determined based on the determined weighting parameters according to the relative horizontal and vertical variations of the pixel values in the first and second subsets. In this way, the weighting parameters can be used to reduce artifacts and blurring on anisotropic features (e.g., edges) in the image, which might otherwise be introduced by the upsampling process. As described above, the weighting parameter determination logic 408 analyzes the input pixels in step S1604 to determine one or more weighting parameters. In this way, one or more weighting parameters are determined for the specific image region being processed, such that different weighting parameters can be used during the upsampling process for different image regions. In other words, the weighting parameters can be adjusted to adapt to the specific local context of the image region for which the upsampled pixel values are being determined. This allows for the use of appropriate weighting parameters for different image regions based on the different anisotropic image features present in those regions.

[0343] For example, the weighting parameter determination logic 1508 of the spatial upsampling logic 1208 can include a specific implementation of a neural network for determining the weighting parameters. The specific implementation of the neural network can be implemented on any suitable hardware (e.g., GPU or neural network accelerator (NNA)), or as fixed-function hardware with predetermined fixed weights. The neural network can be trained, for example, using quantization-aware training (QAT) to output an indication of one or more weighting parameters that indicate the directionality of the filtering to be applied when upsampling the determined pixel values at the upsampled pixel positions in the first and second subsets.

[0344] The spatial upsampling logic 1208 can be configured or not configured to apply sharpening as well as spatial upsampling. Thus, in some examples, the pixel values at the third subset of the upsampled pixel positions are unsharpened upsampled pixel values; while in some other examples, the pixel values at the third subset of the upsampled pixel positions are sharpened upsampled pixel values. Techniques for applying sharpening are known in the art and thus are not described in detail herein.

[0345] The description of the example given above operates according to the "finite impulse response (FIR)" technique. In other words, the reference frame is a low-resolution image with pixel values of the same resolution as the input pixel values of the current frame. However, in other examples, these techniques may be adapted to operate according to the "infinite impulse response (IIR)" technique. In other words, the reference frame can be a high-resolution image, which is the result of applying an upsampling technique to the reference frame and has pixel values of the same resolution as the output (i.e., upsampled) pixel values of the current frame. To implement the technique according to the IIR method, a feedback loop can be introduced so that the output of the spatial algorithm of the previous frame can be used as the reference frame for the temporal resampling process performed on the current frame. Including this feedback loop may make the system more complex and expensive to implement (e.g., in terms of power, bandwidth, and GPU SRAM usage), but it may improve the quality of the output image.

[0346] The components of the processing system 1202 (i.e., the graphics rendering unit 1204, the temporal resampling logic 1206, and the spatial upsampling logic 1208) may or may not be implemented within the same device. In some examples, the processing system 1202 may be implemented as a distributed system (e.g., a "client / server system"). For example, the processing system may include a first device and a second device arranged to communicate with each other over a network. The graphics rendering unit 1204 and the temporal resampling logic 1206 may be implemented at the first device (e.g., a server), and the spatial upsampling logic 1208 may be implemented at the second device (e.g., a client, which may be a mobile device, for example). In this way, the server side is responsible for generating the rendered and temporally resampled pixels, while the client is responsible for spatial upsampling. The client and the server may be located in physically separate locations but may communicate with each other over a network (e.g., via the Internet, via a wired connection, and / or via a wireless connection such as a WiFi or Bluetooth connection). A client that can be implemented on a mobile device (e.g., a handheld battery-powered device) typically has more stringent constraints (in terms of factors such as power consumption, physical size, etc.) than a server that is not typically implemented as a mobile device. Implementing different components of the processing system 1202 at different devices (e.g., in a client / server system as described above) may be advantageous for one or more of the following reasons:

[0347] Compared to a system that implements the entire processing system on the server, the transmission bandwidth between the server and the client is halved because only the data corresponding to the pixels in the first subset and the second subset is transmitted.

[0348] The server performs the most expensive parts of the processing (e.g., in terms of power consumption and device bandwidth), such as temporal resampling. This is beneficial because it avoids consuming system resources on the client and allows for less capable hardware (e.g., a smaller GPU) to be implemented at the client device without affecting the quality of the generated output image. This is in line with the general benefits of cloud rendering.

[0349] The client may have dedicated, fixed-function hardware for very efficient, low-latency spatial upsampling, so it is beneficial to implement the spatial upsampling logic 1208 at the client.

[0350] In some examples, rather than implementing a processing system that includes both temporal resampling and spatial upsampling, only temporal resampling may be implemented to process the pixel values output from the graphics rendering unit to determine the pixel values at all upsampled pixel positions. In these examples, spatial upsampling is not implemented. Figure 20 Illustrated are pixel values of a frame sequence according to an example implementing a "finite impulse response" (FIR) method, which indicates how upsampled pixel positions are projected to positions in a reference frame. The term FIR is similar to FIR filtering in signal processing applications that non-recursively combine signal values from multiple time instances to produce a filtered output at the current time instance. In this example, the current frame 2002 is Figure 20 labeled as "frame t" in, and there are three reference frames: "frame t-1" 2004, "frame t-2" 2006, and "frame t-3" 2008. For each frame, the graphics rendering unit renders one quarter of the upsampled pixel values, e.g., as described above, the graphics rendering unit renders the pixel values at a first subset of the upsampled pixel positions. The rendered pixel values are Figure 20 represented by solid circles in. In this example, temporal resampling is used to determine the other three quarters of the upsampled pixel values at the upsampled pixel positions of the current frame 2002 (shown as open circles or hatched circles in Figure 20 ). It can be seen that the set of positions at which the graphics rendering unit renders pixel values jitters across the frame sequence, and specifically, such that over a four-frame sequence, the graphics rendering unit renders pixel values at all upsampled pixel positions. In other words, a dither pattern is used over the frame sequence such that different frames in the sequence have pixels at positions corresponding to different upsampled pixel positions. Thus, in this example, if the camera (i.e., the viewpoint) is static, then all high-resolution pixel positions are rendered once during the four-frame sequence.

[0351] Figure 20 Illustrated is how temporal resampling is used to determine three upsampled pixel values (2010, 2012, and 2014). In Figure 20In the example shown, for each upsampled pixel position among the upsampled pixel positions, a motion vector is obtained, which can be used to project the upsampled pixel position to a projected position in a reference frame (specifically, the reference frame for which the graphics rendering unit renders the pixel value at the corresponding upsampled pixel position). Specifically, the motion vector 2016 is used to project the upsampled pixel position 2010 of the current frame 2002 to the projected position 2018 of the reference frame 2004; the motion vector 2020 is used to project the upsampled pixel position 2012 of the current frame 2002 to the projected position 2022 of the reference frame 2006; and the motion vector 2024 is used to project the upsampled pixel position 2014 of the current frame 2002 to the projected position 2026 of the reference frame 2008. In this FIR method, the pixel resolution of the reference frame is the same as the input pixel resolution of the current frame. Then, as referred to above Figure 4 As described, using the pixel values of the corresponding reference frame to which the position of the upsampled pixel position is projected, a temporal resampling technique can be performed for each upsampled pixel position among the upsampled pixel positions (2010, 2012, and 2014). In this way, the temporal resampling logic can determine the upsampled pixel values at positions 2010, 2012, and 2014 based on three low-resolution reference frames.

[0352] Figure 21 Illustrates the pixel values of a frame sequence according to an example implementing the "Infinite Impulse Response" (IIR) method, which indicates how to project an upsampled pixel position to a position in a reference frame. The term IIR is similar to IIR filtering in signal processing applications, which recursively reuse previously calculated results to produce the filtered output at the current time instance. In this example, the current frame 2102 is Figure 21 labeled as "frame t", and there is a reference frame: "frame t-1" 2104. In this IIR method, a high-resolution reference frame 2104 is used instead of the low-resolution frame 204. The high-resolution reference frame 2104 is the result of applying upsampling to the low-resolution reference frame, such that the high-resolution reference frame 2104 has the pixel values of all upsampled pixel positions. In other words, in this IIR method, the pixel resolution of the reference frame 2104 is the same as the resolution of the upsampled pixels. The situation remains that for each frame, the graphics rendering unit renders one quarter of the upsampled pixel values, for example, as described above, the graphics rendering unit renders the pixel values at the first subset of the upsampled pixel positions. The rendered pixel values of the current frame 2102 and the determined pixel values of the reference frame 2104 are Figure 21 represented by solid circles. In this example, temporal resampling is used to determine the pixel values of the other three quarters of the upsampled pixel positions of the current frame 2102 (at Figure 21shown as hollow circles or hatched circles). As in the previous examples described, the set of positions of pixel values rendered by the graphics rendering unit can be jittered over a frame sequence, e.g., such that over a four-frame sequence, pixel values are rendered by the graphics rendering unit at all upsampled pixel positions. Thus, in this example, if the camera (i.e., the viewpoint) and the scene are static, then all high-resolution pixel positions are rendered once during the four-frame sequence.

[0353] Figure 21 illustrates how to use temporal resampling to determine three pixel values (2110, 2112, and 2114). In Figure 21 the example shown, for each upsampled pixel value among the upsampled pixel values, a motion vector is obtained, which can be used to project the position of the upsampled pixel value to a projected position in the reference frame 2104. Specifically, the motion vector 2116 is used to project the upsampled pixel position 2110 of the current frame 2102 to the projected position 2118 in the reference frame 2104; the motion vector 2120 is used to project the upsampled pixel position 2112 of the current frame 2102 to the projected position 2122 in the reference frame 2104; and the motion vector 2124 is used to project the upsampled pixel position 2114 of the current frame 2102 to the projected position 2126 in the reference frame 2104. To avoid rendering a high-resolution motion vector field, some of these motion vectors can be upsampled / interpolated from a low-resolution motion vector field. Then, as described above with reference to Figure 4 using the pixel values of the reference frame 2104, temporal resampling techniques can be performed for each of the upsampled pixel positions (2110, 2112, and 2114). In this way, the temporal resampling logic can determine the upsampled pixel values at positions 2110, 2112, and 2114 based on one high-resolution reference frame.

[0354] The IIR method (referenced in Figure 21 described) has some advantages over the FIR method (referenced in Figure 20 described). For example, in the IIR method, the pixels of the reference frame are closer together, and thus the distance between the projected position and the pixel identified in the reference frame (used to determine the weights of the combined identified pixels to determine the pixel value at the upsampled pixel position) is less than that of the FIR method. Having a smaller distance generally means that the identified pixel values are better approximations of the pixel value at the upsampled pixel position. However, the IIR method also has some disadvantages compared to the FIR method. For example, the error determined when upsampling a frame will propagate to the next frame and may persist for multiple frames in the IIR method, while in the FIR method, the error when upsampling a frame does not necessarily propagate to the next frame. Therefore, the choice of using the FIR method or the IIR method depends on the specific implementation.

[0355] exist Figure 4 In the temporal resampling method shown, in steps S402 and S404, depth values ​​and pixel values ​​for the reference frame and the current frame are obtained. In other examples, less than all four of these values ​​may be obtained. For example, if history correction is not implemented, or if depth values ​​are not used to determine weights, the temporal resampling logic may not need to obtain all of these values. Similarly, in some examples, in step S406, it may not be necessary to determine the mean and standard deviation of the pixel values ​​and depth values ​​for region 510 of the current frame.

[0356] Figure 22 A computer system is shown in which the processing modules or processing systems described herein may be implemented. The computer system includes a CPU 2202, a GPU 2204, a memory 2206, a neural network accelerator (NNA) 2208, and other devices 2214, such as a display 2216, a speaker 2218, and a camera 2222. A processing block 2210 (corresponding to a processing module described herein) is implemented on the GPU 2204. In other examples, one or more of the depicted components may be omitted from the system, and / or the processing block 2210 may be implemented on the CPU 2202 or within the NNA 2208 or in a separate block in the computer system. The components of the computer system may communicate with each other via a communication bus 2220. The GPU 2204 may correspond to the graphics rendering unit 1204.

[0357] The processing modules and processing systems described herein are shown as including multiple functional blocks. This is only schematic and is not intended to define a strict division between different logical elements of such entities. Each functional block can be provided in any suitable manner. It should be understood that the intermediate values ​​described herein as being formed by the processing module or processing system do not have to be physically generated by the processing module or processing system at any point, and can only represent logical values ​​that conveniently describe the processing performed by the processing module or processing system between its input and output.

[0358] The processing modules and processing systems described herein may be embodied as hardware on an integrated circuit. The processing modules or processing systems described herein may be configured to perform any of the methods described herein. Generally, any of the functions, methods, techniques, or components described above may be implemented in software, firmware, hardware (e.g., fixed logic circuitry), or any combination thereof. Terms such as "module," "function," "component," "element," "unit," "block," and "logic" may be used herein generically to represent software, firmware, hardware, or any combination thereof. In the case of a software implementation, a module, function, component, element, unit, block, or logic represents program code that, when executed on a processor, performs the specified task. The algorithms and methods described herein may be executed by one or more processors executing code that causes the processors to perform the algorithm / method. Examples of computer-readable storage media include random access memory (RAM), read-only memory (ROM), optical discs, flash memory, hard disk memory, and other memory devices that can store instructions or other data using magnetic, optical, and other technologies and that are accessible by a machine.

[0359] As used herein, the terms computer program code and computer-readable instructions refer to any kind of executable code for a processor, including code expressed in machine language, interpreted language, or scripting language. Executable code includes binary code, machine code, byte code, code defining an integrated circuit (e.g., a hardware description language or netlist), and code expressed in a programming language such as C, Java, or OpenCL. Executable code can be, for example, any kind of software, firmware, script, module, or library that, when properly executed, processed, interpreted, compiled, or run in a virtual machine or other software environment, causes a processor of a computer system supporting the executable code to perform the tasks specified by the code.

[0360] A processor, computer, or computer system can be any kind of device, machine, or dedicated circuit, or a collection or part thereof, having processing capabilities such that it can execute instructions. A processor can be or include any kind of general-purpose or special-purpose processor, such as a CPU, GPU, NNA, system-on-chip, state machine, media processor, application-specific integrated circuit (ASIC), programmable logic array, field-programmable gate array (FPGA), etc. A computer or computer system can include one or more processors.

[0361] The present invention also intends to cover software that defines the configuration of the hardware as described herein, such as hardware description language (HDL) software, for designing integrated circuits or for configuring programmable chips to perform the desired functions. That is, a computer-readable storage medium may be provided, on which is encoded computer-readable program code in the form of an integrated circuit definition dataset, which, when processed (i.e., run) in an integrated circuit manufacturing system, configures the system to manufacture a processing module or processing system configured to perform any of the methods described herein, or to manufacture a processing module or processing system including any of the devices described herein. The integrated circuit definition dataset may be, for example, an integrated circuit description.

[0362] Accordingly, a method of manufacturing a processing module or processing system as described herein at an integrated circuit manufacturing system may be provided. Additionally, an integrated circuit definition dataset may be provided, which, when processed in an integrated circuit manufacturing system, causes the method of manufacturing a processing module or processing system to be executed.

[0363] The integrated circuit definition dataset may be in the form of computer code, such as a netlist, code for configuring a programmable chip, a hardware description language defining hardware suitable for manufacturing at any level in an integrated circuit, including as register transfer level (RTL) code, as a high-level circuit representation (such as Verilog or VHDL), and as a low-level circuit representation (such as OASIS(RTM) and GDSII). A higher-level representation (such as RTL) that logically defines hardware suitable for manufacturing in an integrated circuit may be processed at a computer system configured to generate a manufacturing definition of an integrated circuit in the context of a software environment that includes definitions of circuit elements and rules for combining those elements in order to generate a manufacturing definition of the integrated circuit so defined by the representation. As is typically the case where software is executed at a computer system in order to define a machine, one or more intermediate user steps (such as providing commands, variables, etc.) may be required in order to configure the computer system to generate a manufacturing definition of an integrated circuit to execute code that defines the integrated circuit in order to generate a manufacturing definition of the integrated circuit.

[0364] Reference will now be made to Figure 23 an example of processing an integrated circuit definition dataset at an integrated circuit manufacturing system in order to configure the system to manufacture a processing module or processing system.

[0365] Figure 23An example of an integrated circuit (IC) manufacturing system 2302 is shown, which is configured to manufacture a processing module or a processing system as described in any example herein. In particular, the IC manufacturing system 2302 includes a layout processing system 2304 and an integrated circuit generation system 2306. The IC manufacturing system 2302 is configured to receive an IC definition data set (e.g., defining a processing module or a processing system as described in any example herein), process the IC definition data set, and generate an IC according to the IC definition data set (e.g., which embodies a processing module or a processing system as described in any example herein). The processing of the IC definition data set configures the IC manufacturing system 2302 to manufacture an integrated circuit that embodies a processing module or a processing system as described in any example herein.

[0366] The layout processing system 2304 is configured to receive and process an IC definition data set to determine a circuit layout. Methods for determining a circuit layout from an IC definition data set are known in the art and may, for example, involve synthesizing RTL code to determine a gate-level representation of the circuit to be generated, e.g., in terms of logic components (such as NAND, NOR, AND, OR, MUX, and FLIP-FLOP components). By determining the location information of the logic components, the circuit layout can be determined from the gate-level representation of the circuit. This can be done automatically or with user participation to optimize the circuit layout. When the layout processing system 2304 has determined the circuit layout, it may output a circuit layout definition to the IC generation system 2306. The circuit layout definition may be, for example, a circuit layout description.

[0367] As is known in the art, the IC generation system 2306 generates an IC according to the circuit layout definition. For example, the IC generation system 2306 may implement a semiconductor device manufacturing process for generating an IC, which may involve a multi-step sequence of lithography and chemical processing steps during which an electronic circuit is gradually formed on a wafer made of semiconductor material. The circuit layout definition may be in the form of a mask, which may be used in the lithography process to generate an IC according to the circuit definition. Alternatively, the circuit layout definition provided to the IC generation system 2306 may be in the form of computer-readable code, and the IC generation system 2306 may use the computer-readable code to form a suitable mask for generating the IC.

[0368] The different processes performed by the IC manufacturing system 2302 can all be implemented in one location, e.g., by one party. Alternatively, the IC manufacturing system 2302 can be a distributed system such that some processes can be performed at different locations and by different parties. For example, some of the following stages can be performed at different locations and / or by different parties: (i) synthesizing RTL code representing an IC definition data set to form a gate-level representation of the circuit to be generated; (ii) generating a circuit layout based on the gate-level representation; (iii) forming a mask according to the circuit layout; and (iv) manufacturing an integrated circuit using the mask.

[0369] In other examples, the processing of an integrated circuit definition data set at an integrated circuit manufacturing system can configure the system to manufacture a processing module or a processing system without processing the IC definition data set to determine a circuit layout. For example, the integrated circuit definition data set can define the configuration of a reconfigurable processor such as an FPGA, and the processing of the data set can configure the IC manufacturing system to generate (e.g., by loading configuration data into the FPGA) a reconfigurable processor with the defined configuration.

[0370] In some embodiments, when processed in an integrated circuit manufacturing system, the integrated circuit manufacturing definition data set can cause the integrated circuit manufacturing system to generate the devices described herein. For example, by configuring the integrated circuit manufacturing system in the manner described above with reference to Figure 23 the devices described herein can be manufactured.

[0371] In some examples, the integrated circuit definition data set can include software that runs on hardware defined at the data set, or software that runs in combination with the hardware defined at the data set. In the Figure 23 example shown, the IC production system can also be further configured by the integrated circuit definition data set to load firmware onto the integrated circuit according to program code defined in the integrated circuit definition data set during the manufacture of the integrated circuit, or otherwise provide program code for use with the integrated circuit.

[0372] Compared with known implementations, the implementation of the concepts set forth in this application in devices, apparatuses, modules, and / or systems (and in the methods implemented herein) can improve performance. Performance improvements can include one or more of increased computing performance, reduced latency, increased throughput, and / or reduced power consumption. During the manufacture of such devices, apparatuses, modules, and systems (e.g., in integrated circuits), a trade-off can be made between performance improvements and the physical implementation, thereby improving the manufacturing method. For example, a trade-off can be made between performance improvements and layout area to match the performance of known implementations but use less silicon. For example, this can be done by reusing functional blocks in series or sharing functional blocks among the elements of a device, apparatus, module, and / or system. Conversely, the concepts set forth in this application that result in improvements in the physical implementation of devices, apparatuses, modules, and systems (e.g., reduced silicon area) can be traded off against performance improvements. This can be done, for example, by manufacturing multiple instances of a module within a predefined area budget.

[0373] The applicant hereby independently discloses each individual feature described herein and any combination of two or more such features to the extent that such features or combinations are capable of being implemented based on the entire specification in view of the common general knowledge of a person skilled in the art, regardless of whether such features or combinations of features solve any of the problems disclosed herein. In view of the foregoing description, it will be apparent to a person skilled in the art that various modifications can be made within the scope of the present invention.

[0374] Appendix

[0375] The following numbered clauses are provided and form a part of this disclosure:

[0376] 101. A method for determining one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence, the method comprising:

[0377] obtaining depth values of positions of pixels of a reference frame in the frame sequence; and

[0378] for each upsampled pixel position among the one or more upsampled pixel positions:

[0379] obtaining a depth value of the current frame for the upsampled pixel position;

[0380] obtaining a motion vector of the upsampled pixel position to indicate the motion of the upsampled pixel position between the reference frame and the current frame;

[0381] using the motion vector of the upsampled pixel position to identify one or more pixels among the pixels of the reference frame;

[0382] Determine the weight of each identified pixel among one or more identified pixels of a reference frame based on the following items: (i) the depth value of the current frame for the upsampled pixel position, and (ii) the depth value of the position of the identified pixel of the reference frame; and

[0383] Use the determined weights for each identified pixel among one or more identified pixels to determine the pixel value of the upsampled pixel position.

[0384] 102. The method as described in clause 101, further comprising obtaining the pixel values of one or more identified pixels of a reference frame in a frame sequence, wherein determining the pixel value of the upsampled pixel position includes performing a weighted sum of the pixel values of one or more identified pixels of the reference frame, and the performing is carried out using the determined weights for each identified pixel among one or more identified pixels in the weighted sum.

[0385] 103. The method as described in clause 101 or 102, wherein for each upsampled pixel position among one or more upsampled pixel positions, determine the weight of each identified pixel among one or more identified pixels of the reference frame based on the difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame.

[0386] 104. The method as described in any one of clauses 101 to 103, further comprising for each upsampled pixel position among one or more upsampled pixel positions:

[0387] Obtain a plurality of depth values of the current frame for positions within a region surrounding the upsampled pixel position; and

[0388] Determine the standard deviation of the depth values of the current frame within the region, wherein further determine the weight of each identified pixel among one or more identified pixels of the reference frame based on the following item: (iii) the determined standard deviation of the depth values.

[0389] 105. The method as described in clause 104, when subordinate to clause 103, wherein determining the weight of each identified pixel among one or more identified pixels of the reference frame includes comparing the difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame with a depth threshold, and the depth threshold is based on the determined standard deviation of the depth values of the current frame within the region.

[0390] 106. The method as described in clause 105, wherein in response to determining that the difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame is greater than the depth threshold, determine that the weight of the identified pixel of the reference image is lower.

[0391] 107. The method as described in clause 106, wherein the depth threshold is a hard threshold, and wherein the weight w of the identified pixel k of the reference image k is determined such that w k = w i,k ·(|D ref,k - D curr | ≤ T d ), where T d is the depth threshold, where T d = F depth · σ depth , and where w i,k is the initial weight of the identified pixel of the reference image, D ref,k is the depth value of the position of the identified pixel of the reference frame, D curr is the depth value of the current frame for the upsampled pixel position, F depth is a predetermined factor, and σ depth is the determined standard deviation of the depth values in the region around the upsampled pixel position of the current frame.

[0392] 108. The method as described in clause 106, wherein the depth threshold is a soft threshold, and wherein the weight w of the identified pixel k of the reference image k is determined such that where T d is the depth threshold, where T d = F depth · σ depth , and where w i,k is the initial weight of the identified pixel of the reference image, D ref,k is the depth value of the position of the identified pixel of the reference frame, D curr is the depth value of the current frame for the upsampled pixel position, F depth is a predetermined factor, and σ depth is the determined standard deviation of the depth values in the region around the upsampled pixel position of the current frame.

[0393] 109. The method as described in any one of clauses 101 to 108, wherein the using the motion vector of the upsampled pixel position to identify one or more pixels of the pixels of the reference frame includes projecting the upsampled pixel position to a position in the reference frame based on the motion vector, and identifying one or more pixels of the pixels of the reference frame in a neighborhood of the projected position in the reference frame.

[0394] 110. The method according to any one of clauses 101 to 109, wherein for each upsampled pixel position among one or more upsampled pixel positions, determining the weight of each identified pixel among the one or more identified pixels of the reference frame includes determining an initial weight and using the initial weight to determine the weight of the identified pixels of the reference frame.

[0395] 111. The method according to clause 110, when subordinate to clause 109, wherein the initial weight of each identified pixel among the one or more identified pixels of the reference frame is determined by:

[0396] Determining the distance between the projection position and the position of the identified pixel in the reference frame; and

[0397] Mapping the distance to the initial weight using a predetermined relationship.

[0398] 112. The method according to clause 111, wherein the predetermined relationship is a Gaussian relationship or a linear relationship.

[0399] 113. The method according to any one of clauses 101 to 112, wherein for each upsampled pixel position among one or more upsampled pixel positions, the weight of each identified pixel among the one or more identified pixels of the reference frame is determined according to the degree to which the identified pixel of the reference frame is an outlier compared to other identified pixels of the reference frame.

[0400] 114. The method according to any one of clauses 101 to 113, further comprising, for each upsampled pixel position among one or more upsampled pixel positions:

[0401] Obtaining a plurality of input pixel values of the current frame for positions within a region surrounding the upsampled pixel position; and

[0402] Determining the mean of the input pixel values of the current frame within the region surrounding the upsampled pixel position.

[0403] 115. The method according to clause 114, wherein determining the pixel value of the upsampled pixel position includes clamping the determined pixel value such that the pixel value does not differ from the determined mean of the input pixel values of the current frame within the region surrounding the upsampled pixel position by more than a threshold.

[0404] 116. The method according to clause 115, further comprising, for each upsampled pixel position among one or more upsampled pixel positions:

[0405] Determining the standard deviation of the input pixel values of the current frame within the region surrounding the upsampled pixel position,

[0406] wherein the threshold is based on a determined standard deviation of input pixel values of a current frame within a region.

[0407] 117. The method according to clause 116, wherein for each upsampled pixel position among one or more upsampled pixel positions, the threshold is F pixel ·σ pixel , where F pixel is a predetermined factor, and σ pixel is the determined standard deviation of input pixel values of the current frame within a region surrounding the upsampled pixel position.

[0408] 118. The method according to any one of clauses 115 to 117, wherein clamping is selectively applied to different regions to different extents, and the method further includes:

[0409] comparing an average value of pixel values determined at an upsampled pixel position within a region surrounding the upsampled pixel position with an average value of input pixel values of the current frame within a region surrounding the upsampled pixel position; and

[0410] performing clamping based on a comparison of: (i) a difference between the average value of pixel values determined at an upsampled pixel position within a region surrounding the upsampled pixel position and the average value of input pixel values of the current frame within a region surrounding the upsampled pixel position, and (ii) a threshold difference.

[0411] 119. The method according to any one of clauses 114 to 118, wherein in response to determining that the weights of all one or more identified pixels of a reference frame are zero, the pixel value of the upsampled pixel position is determined as the determined average value of input pixel values of the current frame within a region of the upsampled pixel position.

[0412] 120. The method according to any one of clauses 101 to 119, wherein the upsampled pixel position is between positions of diagonally adjacent input pixels of the current frame, such that the positions of the upsampled pixel position and the input pixels form a repeating five-point pattern.

[0413] 121. The method according to any one of clauses 101 to 120, wherein the resolution of pixels of the reference frame is the same as the resolution of input pixels of the current frame.

[0414] 122. The method according to clause 121, wherein a dither pattern is used on a frame sequence such that different frames in the sequence have pixels at positions corresponding to different upsampled pixel positions.

[0415] 123. The method according to any one of clauses 101 to 120, wherein the resolution of pixels of the reference frame is the same as the resolution of pixels determined at the upsampled pixel position.

[0416] 124. The method according to any one of clauses 101 to 123, wherein the pixel values and depth values of the current frame and the reference frame at the input pixel positions are determined by a graphics rendering process.

[0417] 125. The method according to any one of clauses 101 to 123, wherein obtaining the depth value of the current frame for the upsampled pixel position includes:

[0418] Receiving the depth value of the current frame at the positions of the input pixels surrounding the upsampled pixel position;

[0419] For each pair of input pixels of the input pixels for which the depth value is received, determining an interpolated depth value of the upsampled pixel position based on the depth values of the pair of input pixels;

[0420] Determining the depth weights of the pair of input pixels based on the depth gradient between the depth values of the pair of input pixels; and

[0421] Determining the depth value of the current frame for the upsampled pixel position by performing a weighted sum of the determined interpolated depth values using the determined depth weights for the pair of input pixels.

[0422] 126. The method according to clause 125, wherein determining the depth weights of the pair of input pixels includes:

[0423] Multiplying the depth gradient of the pair of input pixels by a negative number; and

[0424] Inputting the result of the multiplication into a softmax function.

[0425] 127. The method according to any one of clauses 101 to 126, wherein the pixel value is a Y-channel pixel value.

[0426] 128. A processing module configured to determine one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence, the processing module being configured to:

[0427] Obtaining the depth values of the positions of the pixels of the reference frame in the frame sequence; and

[0428] For each upsampled pixel position among the one or more upsampled pixel positions:

[0429] Obtaining the depth value of the current frame for the upsampled pixel position;

[0430] Obtaining a motion vector of the upsampled pixel position to indicate the motion of the upsampled pixel position between the reference frame and the current frame;

[0431] Using the motion vector of the upsampled pixel position to identify one or more pixels among the pixels of the reference frame;

[0432] Determine the weight of each identified pixel among one or more identified pixels of a reference frame based on the following items: (i) the depth value of the current frame at the upsampled pixel position, and (ii) the depth value of the position of the identified pixel of the reference frame; and

[0433] Use the determined weights for each of the one or more identified pixels to determine the pixel value at the upsampled pixel position.

[0434] 129. A processing module, the processor being configured to execute the method according to any one of clauses 101 to 127.

[0435] 130. The processing module according to clause 128 or 129, wherein the processing module is embodied as hardware on an integrated circuit.

[0436] 131. A computer-readable code, the computer-readable code being configured to cause the method according to any one of clauses 101 to 127 to be executed when the code is run.

[0437] 132. An integrated circuit definition data set, which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture the processing module according to any one of clauses 128 to 130.

[0438] 201. A method for determining one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence, the method comprising:

[0439] Obtain the pixel values of the pixels of a reference frame in the frame sequence;

[0440] For each upsampled pixel position among the one or more upsampled pixel positions:

[0441] Obtain a plurality of input pixel values of the current frame at positions within a region surrounding the upsampled pixel position;

[0442] Determine the mean of the input pixel values of the current frame within the region surrounding the upsampled pixel position;

[0443] Obtain a motion vector of the upsampled pixel position to indicate the motion of the upsampled pixel position between the reference frame and the current frame;

[0444] Use the motion vector of the upsampled pixel position to identify one or more pixels among the pixels of the reference frame; and

[0445] Combine the pixel values of the one or more identified pixels of the reference frame to determine the pixel value at the upsampled pixel position;

[0446] Determining the pixel value of the upsampled pixel position by using the pixel values of one or more identified pixels of the combined reference frame includes clamping the determined pixel value such that the pixel value does not differ from the determined mean of the input pixel values of the current frame within the region surrounding the upsampled pixel position by more than a threshold value.

[0447] 202. The method according to clause 201, further comprising, for each upsampled pixel position among the one or more upsampled pixel positions:

[0448] Determining the standard deviation of the input pixel values of the current frame within the region surrounding the upsampled pixel position,

[0449] wherein the threshold value is based on the determined standard deviation of the input pixel values of the current frame within the region.

[0450] 203. The method according to clause 202, wherein for each upsampled pixel position among the one or more upsampled pixel positions, the threshold value is F pixel ·σ pixel where F pixel is a predetermined factor, and σ pixel is the determined standard deviation of the input pixel values of the current frame within the region surrounding the upsampled pixel position.

[0451] 204. The method according to any one of clauses 201 to 203, wherein clamping is selectively applied to different regions to different extents.

[0452] 205. The method according to clause 204, wherein the method further comprises:

[0453] Comparing the average of the pixel values determined at the upsampled pixel positions within the region surrounding the upsampled pixel position with the mean of the input pixel values of the current frame within the region surrounding the upsampled pixel position; and

[0454] Performing clamping based on the comparison of: (i) the difference between the average of the pixel values determined at the upsampled pixel positions within the region surrounding the upsampled pixel position and the mean of the input pixel values of the current frame within the region surrounding the upsampled pixel position, and (ii) the threshold difference.

[0455] 206. The method according to any one of clauses 201 to 205, wherein the pixel value is a Y-channel pixel value.

[0456] 207. The method according to any one of clauses 201 to 206, wherein for each upsampled pixel position among the one or more upsampled pixel positions, the pixel values of one or more identified pixels of the combined reference frame include:

[0457] Determine the weight of each identified pixel among one or more identified pixels of a reference frame; and determine the pixel value of an upsampled pixel position using the determined weights for each identified pixel among the one or more identified pixels.

[0458] 208. The method as recited in clause 207, wherein determining the pixel value of the upsampled pixel position includes performing a weighted sum of the pixel values of one or more identified pixels of the reference frame, and the performing is carried out using the determined weights for each identified pixel among the one or more identified pixels in the weighted sum.

[0459] 209. The method as recited in clause 207 or 208, further comprising, for each upsampled pixel position among one or more upsampled pixel positions:

[0460] Obtain the depth value of the position of one or more identified pixels of the reference frame; and

[0461] Obtain the depth value of the current frame for the upsampled pixel position,

[0462] wherein the weight of each identified pixel among one or more identified pixels of the reference frame is determined based on: (i) the depth value of the current frame for the upsampled pixel position, and (ii) the depth value of the position of the identified pixel of the reference frame.

[0463] 210. The method as recited in clause 209, wherein for each upsampled pixel position among one or more upsampled pixel positions, the weight of each identified pixel among one or more identified pixels of the reference frame is determined based on the difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame.

[0464] 211. The method as recited in clause 209 or 210, further comprising, for each upsampled pixel position among one or more upsampled pixel positions:

[0465] Obtain a plurality of depth values of the current frame for positions within a region surrounding the upsampled pixel position; and

[0466] Determine the standard deviation of the depth values of the current frame within the region, wherein the weight of each identified pixel of the reference frame is further determined based on: (iii) the determined standard deviation of the depth values.

[0467] 212. The method as described in clause 211, when subordinate to clause 210, wherein the weight of each of the identified pixels among the one or more identified pixels of the reference frame includes comparing the difference between the depth value of the current frame at the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame with a depth threshold, wherein the depth threshold is based on the determined standard deviation of the depth values of the current frame within the region.

[0468] 213. The method as described in clause 212, wherein in response to determining that the difference between the depth value of the current frame at the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame is greater than the depth threshold, it is determined that the weight of the identified pixel of the reference image is lower.

[0469] 214. The method as described in clause 213, wherein the depth threshold is a hard threshold, and wherein the weight w of the identified pixel k of the reference image k is determined such that w k = w i,k ·(|D ref,k - D curr | ≤ T d ), where T d is the depth threshold, where T d = F depth · σ depth , and where w i,k is the initial weight of the identified pixel of the reference image, D ref,k is the depth value of the position of the identified pixel of the reference frame, D curr is the depth value of the current frame at the upsampled pixel position, F depth is a predetermined factor, and σ depth is the determined standard deviation of the depth values of the current frame within the region surrounding the upsampled pixel position.

[0470] 215. The method as described in clause 213, wherein the depth threshold is a soft threshold, and wherein the weight w of the identified pixel k of the reference image k is determined such that where T d is the depth threshold, where T d = F depth · σ depth , and where w i,k is the initial weight of the identified pixel of the reference image, D ref,k is the depth value of the position of the identified pixel of the reference frame, D curr is the depth value of the current frame at the upsampled pixel position, F depth is a predetermined factor, and σ depthThe determined standard deviation of the depth values of the current frame within the region surrounding the upsampled pixel position.

[0471] 216. The method according to any one of clauses 209 to 215, wherein obtaining the depth value of the current frame for the upsampled pixel position comprises:

[0472] Receiving the depth value of the current frame at the position of the input pixel surrounding the upsampled pixel position;

[0473] For each pair of input pixels for which the depth value is received, determining an interpolated depth value of the upsampled pixel position based on the depth values of the pair of input pixels;

[0474] Determining the depth weights of the pair of input pixels based on the depth gradient between the depth values of the pair of input pixels; and

[0475] Determining the depth value of the current frame for the upsampled pixel position by performing a weighted sum of the determined interpolated depth values using the determined depth weights for the pair of input pixels.

[0476] 217. The method according to clause 216, wherein determining the depth weights of the pair of input pixels comprises:

[0477] Multiplying the depth gradient of the pair of input pixels by a negative number; and

[0478] Inputting the result of the multiplication into a softmax function.

[0479] 218. The method according to any one of clauses 207 to 217, wherein in response to determining that the weights of all the identified pixels of the reference frame are zero, the pixel value of the upsampled pixel position is determined as the determined mean of the input pixel values of the current frame within the region surrounding the upsampled pixel position.

[0480] 219. The method according to any one of clauses 207 to 218, wherein for each upsampled pixel position among one or more upsampled pixel positions, the weight of each identified pixel among the one or more identified pixels of the reference frame is determined based on the degree to which the identified pixel of the reference frame is an outlier compared to the other identified pixels of the reference frame.

[0481] 220. The method according to any one of clauses 201 to 219, wherein using the motion vector of the upsampled pixel position to identify one or more pixels of the reference frame comprises projecting the upsampled pixel position to a position in the reference frame based on the motion vector, and identifying one or more pixels of the reference frame in the neighborhood of the projected position in the reference frame.

[0482] 221. The method as described in clause 220, when subordinate to clause 207, wherein for each upsampled pixel position among one or more upsampled pixel positions, determining the weight of each identified pixel among one or more identified pixels of the reference frame includes:

[0483] Determining an initial weight by: (i) determining the distance between the projection position and the position of the identified pixel in the reference frame, and (ii) mapping the distance to the initial weight using a predetermined relationship; and

[0484] Using the initial weight to determine the weight of the identified pixel of the reference frame.

[0485] 222. The method as described in clause 221, wherein the predetermined relationship is a Gaussian relationship or a linear relationship.

[0486] 223. The method as described in any one of clauses 201 to 222, wherein the upsampled pixel positions are located between the positions of diagonally adjacent input pixels of the current frame, such that the upsampled pixel positions and the positions of the input pixels form a repeating five-point pattern.

[0487] 224. The method as described in any one of clauses 201 to 223, wherein the resolution of the pixels of the reference frame is the same as the resolution of the input pixels of the current frame.

[0488] 225. The method as described in clause 224, wherein a dither pattern is used on the frame sequence such that different frames in the sequence have pixels at positions corresponding to different upsampled pixel positions.

[0489] 226. The method as described in any one of clauses 201 to 223, wherein the resolution of the pixels of the reference frame is the same as the resolution of the pixels determined at the upsampled pixel positions.

[0490] 227. A processing module configured to determine one or more pixel values at corresponding one or more upsampled pixel positions of a current frame in a frame sequence, the processing module being configured to:

[0491] Obtain the pixel values of the pixels of a reference frame in the frame sequence;

[0492] For each of the one or more upsampled pixel positions:

[0493] Obtain a plurality of input pixel values of the current frame for positions within a region surrounding the upsampled pixel position;

[0494] Determine the mean of the input pixel values of the current frame within the region surrounding the upsampled pixel position;

[0495] Obtain a motion vector for an upsampled pixel position to indicate the motion of the upsampled pixel position between a reference frame and a current frame;

[0496] Use the motion vector for the upsampled pixel position to identify one or more pixels among the pixels of the reference frame; and

[0497] Combine the pixel values of the one or more identified pixels of the reference frame to determine the pixel value of the upsampled pixel position;

[0498] Wherein combining the pixel values of the one or more identified pixels of the reference frame to determine the pixel value of the upsampled pixel position includes clamping the determined pixel value such that the pixel value does not differ from the determined mean of the input pixel values of the current frame within a region surrounding the upsampled pixel position by more than a threshold.

[0499] 228. A processing module, the processor being configured to execute the method according to any one of clauses 201 to 226.

[0500] 229. The processing module according to clause 227 or 228, wherein the processing module is embodied as hardware on an integrated circuit.

[0501] 230. A computer-readable code, the computer-readable code being configured to cause the method according to any one of clauses 201 to 226 to be executed when the code is run.

[0502] 231. An integrated circuit definition data set, which when processed in an integrated circuit manufacturing system configures the integrated circuit manufacturing system to manufacture the processing module according to any one of clauses 227 to 229.

[0503] 301. A method for determining the pixel value at an upsampled pixel position of a current frame in a frame sequence, the method comprising:

[0504] Use a graphics rendering process to determine the pixel values at a first subset of the upsampled pixel positions of the current frame;

[0505] Determine the pixel values at a second subset of the upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of a reference frame in the frame sequence; and

[0506] Determine the pixel values at a third subset of the upsampled pixel positions of the current frame by applying spatial upsampling to the determined pixel values at the upsampled pixel positions in the first subset and the second subset.

[0507] 302. The method according to clause 301, wherein the upsampled pixel positions in the first subset and the second subset form a repeating pentagon pattern.

[0508] 303. The method as described in clause 302, wherein the upsampled pixel positions in the second subset are located between the diagonally adjacent upsampled pixel positions in the first subset, such that:

[0509] For each upsampled pixel position in the upsampled pixel positions in the first subset that is not on the edge of the current frame, the four nearest upsampled pixel positions of the five-point pattern are the upsampled pixel positions in the second subset, and

[0510] For each upsampled pixel position in the upsampled pixel positions in the second subset that is not on the edge of the current frame, the four nearest upsampled pixel positions of the five-point pattern are the upsampled pixel positions in the first subset.

[0511] 304. The method as described in clause 302 or 303, wherein the upsampled pixel positions in the third subset are located in the gaps of the repeated five-point pattern.

[0512] 305. The method as described in clause 304, wherein each upsampled pixel position in the upsampled pixel positions in the third subset that is not on the edge of the current frame is located at: (i) between two horizontally adjacent upsampled pixel positions in the first subset and between two vertically adjacent upsampled pixel positions in the second subset, or (ii) between two vertically adjacent upsampled pixel positions in the first subset and between two horizontally adjacent upsampled pixel positions in the second subset.

[0513] 306. The method as described in any one of clauses 301 to 305, wherein the first subset, the second subset, and the third subset of the upsampled pixel positions are different, such that there is no upsampled pixel position that belongs to more than one of the first subset, the second subset, and the third subset.

[0514] 307. The method as described in any one of clauses 301 to 306, wherein all the upsampled pixel positions of the current frame belong to one of the first subset, the second subset, and the third subset.

[0515] 308. The method as described in any one of clauses 301 to 307, wherein:

[0516] One quarter of the upsampled pixel positions of the current frame is located in the first subset,

[0517] One quarter of the upsampled pixel positions of the current frame is located in the second subset, and

[0518] Half of the upsampled pixel positions of the current frame is located in the third subset.

[0519] 309. The method according to any one of clauses 301 to 308, wherein a dither pattern is used on the frame sequence such that a graphics rendering process is used to determine pixel values at different upsampled pixel positions of different frames in the frame sequence.

[0520] 310. The method according to any one of clauses 301 to 309, wherein a subset of the upsampled pixel positions for which pixel values are determined using the graphics rendering process alternates between a first subset of upsampled pixel positions and a second subset of upsampled pixel positions for consecutive frames in the frame sequence.

[0521] 311. The method according to any one of clauses 301 to 310, wherein the reference frame is a previous frame or a subsequent frame in the frame sequence relative to the current frame.

[0522] 312. The method according to any one of clauses 301 to 311, wherein the graphics rendering process is a rasterization process or a ray tracing process.

[0523] 313. The method according to any one of clauses 301 to 312, wherein determining the pixel values at the second subset of upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of the reference frame in the frame sequence includes:

[0524] Obtaining the pixel values of the pixels of the reference frame;

[0525] For each upsampled pixel position in the second subset of upsampled pixel positions:

[0526] Obtaining a motion vector of the upsampled pixel position to indicate the motion of the upsampled pixel position between the reference frame and the current frame;

[0527] Using the motion vector of the upsampled pixel position to identify one or more pixels in the pixels of the reference frame; and

[0528] Combining the pixel values of the one or more identified pixels of the reference frame to determine the pixel value of the upsampled pixel position in the second subset.

[0529] 314. The method according to clause 313, wherein determining the pixel values at the second subset of upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of the reference frame in the frame sequence includes:

[0530] Obtaining the depth value of the position of the pixels of the reference frame; and

[0531] For each upsampled pixel position in the second subset of upsampled pixel positions, obtaining the depth value of the current frame for the upsampled pixel position;

[0532] For each upsampled pixel position in the second subset, the pixel values of one or more identified pixels of the combined reference frame include:

[0533] determining the weight of each identified pixel among one or more identified pixels of the reference frame based on: (i) the depth value of the current frame for the upsampled pixel position, and (ii) the depth value of the position of the identified pixel of the reference frame; and

[0534] using the determined weight for each identified pixel among the identified pixels to determine the pixel value of the upsampled pixel position.

[0535] 315. The method according to clause 314, wherein determining the pixel value of the upsampled pixel position includes performing a weighted sum of the pixel values of one or more identified pixels of the reference frame, and the performing is carried out using the determined weight for each identified pixel among the one or more identified pixels in the weighted sum.

[0536] 316. The method according to clause 314 or 315, further comprising, for each upsampled pixel position among one or more upsampled pixel positions in the second subset:

[0537] obtaining a plurality of depth values of the current frame for positions within a region surrounding the upsampled pixel position; and

[0538] determining the standard deviation of the depth values of the current frame within the region, wherein the weight of each identified pixel of the identified pixels of the reference frame is further determined based on: (iii) the determined standard deviation of the depth values.

[0539] 317. The method according to any one of clauses 313 to 316, wherein using the motion vector of the upsampled pixel position to identify one or more pixels of the reference frame includes projecting the upsampled pixel position to a position in the reference frame based on the motion vector, and identifying one or more pixels of the reference frame in the neighborhood of the projected position in the reference frame.

[0540] 318. The method according to any one of clauses 313 to 317, wherein determining the pixel value at the second subset of upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of the reference frame in the frame sequence further comprises, for each upsampled pixel position among one or more upsampled pixel positions in the second subset:

[0541] determining the mean of a plurality of pixel values at a first subset of upsampled pixel positions within a region surrounding the upsampled pixel position of the current frame,

[0542] Determining the pixel values of the pixels of one or more identified pixels of the combined reference frame to determine the pixel values of the upsampled pixel positions in the second subset includes clamping the determined pixel values such that the pixel values do not differ from the determined mean of the pixel values at the first subset of the upsampled pixel positions in the region around the upsampled pixel position in the current frame by more than a threshold value.

[0543] 319. The method according to clause 318, wherein determining the pixel values at the second subset of the upsampled pixel positions of the current frame by applying temporal resampling to the pixel values of the pixels of the reference frames in the frame sequence further includes, for each of the one or more upsampled pixel positions in the second subset:

[0544] Determining the standard deviation of the plurality of pixel values at the first subset of the upsampled pixel positions in the region around the upsampled pixel position in the current frame,

[0545] wherein the threshold value is based on the determined standard deviation of the pixel values at the first subset of the upsampled pixel positions in the region.

[0546] 320. The method according to any one of clauses 301 to 319, wherein determining the pixel values at the third subset of the upsampled pixel positions of the current frame by applying spatial upsampling to the determined pixel values at the upsampled pixel positions in the first and second subsets includes performing bilinear interpolation on the determined pixel values at the upsampled pixel positions in the first and second subsets.

[0547] 321. The method according to any one of clauses 301 to 319, wherein determining the pixel values at the third subset of the upsampled pixel positions of the current frame by applying spatial upsampling to the determined pixel values at the upsampled pixel positions in the first and second subsets includes:

[0548] Analyzing the pixel values at the upsampled pixel positions in the first and second subsets to determine one or more weighting parameters, the one or more weighting parameters indicating the directionality of the filtering to be applied when applying upsampling to the determined pixel values at the upsampled pixel positions in the first and second subsets; and

[0549] Determining the pixel values at the third subset of the upsampled pixel positions by applying one or more kernels to at least some of the pixel values at the upsampled pixel positions in the first and second subsets according to the determined one or more weighting parameters.

[0550] 322. The method as described in clause 321, wherein analyzing the pixel values at the upsampled pixel positions in the first subset and the second subset to determine one or more weighting parameters includes processing the pixel values at the upsampled pixel positions in the first subset and the second subset using a specific implementation of a neural network, where the neural network has been trained to output an indication of one or more weighting parameters to indicate the directionality of the filtering to be applied when upsampling the determined pixel values at the upsampled pixel positions in the first subset and the second subset.

[0551] 323. The method as described in clause 321 or 322, wherein the pixel values at the third subset of upsampled pixel positions are unsharpened upsampled pixel values.

[0552] 324. The method as described in clause 321 or 322, wherein the pixel values at the third subset of upsampled pixel positions are sharpened upsampled pixel values.

[0553] 325. The method as described in any one of clauses 301 to 324, wherein the pixel values are Y-channel pixel values.

[0554] 326. A processing system configured to determine pixel values at upsampled pixel positions of a current frame in a frame sequence, the processing system comprising:

[0555] A graphics rendering unit configured to determine pixel values at a first subset of upsampled pixel positions of the current frame using a graphics rendering process;

[0556] Temporal resampling logic configured to determine pixel values at a second subset of upsampled pixel positions of the current frame by applying temporal resampling to pixel values of pixels of a reference frame in the frame sequence; and

[0557] Spatial upsampling logic configured to determine pixel values at a third subset of upsampled pixel positions of the current frame by applying spatial upsampling to the determined pixel values at the upsampled pixel positions in the first subset and the second subset.

[0558] 327. The processing system as described in clause 326, wherein the processing system includes a first device and a second device arranged to communicate with each other over a network,

[0559] wherein the graphics rendering unit and the temporal resampling logic are implemented at the first device, and

[0560] wherein the spatial upsampling logic is implemented at the second device.

[0561] 328. A processing system configured to perform the method as described in any one of clauses 301 to 325.

[0562] 329. A processing system as described in any one of clauses 326 to 328, wherein the processing system is embodied as hardware on one or more integrated circuits.

[0563] 330. A computer-readable code configured to cause a method as described in any one of clauses 301 to 325 to be performed when the code is run.

[0564] 331. An integrated circuit definition dataset which, when processed in an integrated circuit manufacturing system, configures the integrated circuit manufacturing system to manufacture a processing system as described in any one of clauses 326 to 329.

Claims

1. A method for determining one or more pixel values ​​at corresponding one or more upsampled pixel positions of a current frame in a sequence of frames, the method comprising: Obtaining a depth value of a pixel position of a reference frame in the frame sequence; as well as For each upsampled pixel position of the one or more upsampled pixel positions: Obtaining a depth value of the current frame for the upsampled pixel position; obtaining a motion vector for the upsampled pixel position to indicate motion of the upsampled pixel position between the reference frame and the current frame; identifying one or more of the pixels of the reference frame using the motion vectors of the upsampled pixel positions; determining a weight for each of the one or more identified pixels of the reference frame based on: (i) the depth value of the current frame for the upsampled pixel position, and (ii) the depth value of the position of the identified pixel of the reference frame; as well as The pixel value of the upsampled pixel location is determined using the determined weight for each identified pixel of the one or more identified pixels.

2. The method of claim 1 , further comprising obtaining pixel values ​​of the one or more identified pixels of the reference frame in the frame sequence, wherein the determining the pixel value of the upsampled pixel position comprises performing a weighted sum of the pixel values ​​of the one or more identified pixels of the reference frame, the performing being performed using a determined weight in the weighted sum for each of the one or more identified pixels.

3. The method of claim 1 or 2, wherein for each of the one or more upsampled pixel positions, the weight of each of the one or more identified pixels of the reference frame is determined based on a difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame.

4. A method as claimed in any preceding claim, further comprising, For each upsampled pixel position of the one or more upsampled pixel positions: obtaining a plurality of depth values ​​of the current frame for positions within a region surrounding the upsampled pixel position; as well as Determining a standard deviation of the depth values ​​of the current frame within the region, wherein the weight of each of the one or more identified pixels of the reference frame is further determined based on: (iii) the determined standard deviation of the depth values.

5. A method as claimed in claim 4, when dependent on claim 3, wherein the determining of the weight of each of the one or more identified pixels of the reference frame comprises comparing the difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame to a depth threshold, wherein the depth threshold is based on a determined standard deviation of the depth values ​​of the current frame within the region.

6. The method of claim 5 , wherein in response to determining that the difference between the depth value of the current frame for the upsampled pixel position and the depth value of the position of the identified pixel of the reference frame is greater than the depth threshold, determining that the weight of the identified pixel of the reference image is lower.

7. The method of claim 6, wherein the depth threshold is a hard threshold, and wherein the weight w of the identified pixel k of the reference image k is determined so that w k =w i,k ·(|D ref,k -D curr |≤T d ), where T d is the depth threshold, where T d =F depth ·σ depth , and where w i,k is the initial weight of the identified pixel of the reference image, D ref,k is the depth value of the position of the identified pixel of the reference frame, D curr is the depth value of the current frame for the upsampled pixel position, F depth is a predetermined factor, and σ depth is the determined standard deviation of the depth values ​​of the current frame within the region surrounding the upsampled pixel position.

8. The method of claim 6, wherein the depth threshold is a soft threshold, and wherein the weight w of the identified pixel k of the reference image k is determined so that Where T d is the depth threshold, where T d =F depth ·σ depth , and where w i,k is the initial weight of the identified pixel of the reference image, D ref,k is the depth value of the position of the identified pixel of the reference frame, D curr is the depth value of the current frame for the upsampled pixel position, F depth is a predetermined factor, and σ depth is the determined standard deviation of the depth values ​​of the current frame within the region surrounding the upsampled pixel position.

9. A method as claimed in any preceding claim, wherein the use of the motion vector of the upsampled pixel position to identify one or more of the pixels of the reference frame comprises projecting the upsampled pixel position to a position in the reference frame based on the motion vector, and identifying one or more of the pixels of the reference frame in a neighbourhood of the projected position in the reference frame.

10. A method as claimed in any preceding claim, wherein for each of the one or more upsampled pixel positions, determining the weight of each of the one or more identified pixels of the reference frame comprises determining an initial weight, and using the initial weight to determine the weight of the identified pixel of the reference frame.

11. The method of claim 10, when dependent on claim 9, wherein the initial weight of each of the one or more identified pixels of the reference frame is determined by: determining a distance between the projected position and the position of the identified pixel in the reference frame; and The distances are mapped to initial weights using a predetermined relationship.

12. The method of any preceding claim, further comprising, for each of the one or more upsampled pixel positions: obtaining a plurality of input pixel values ​​of the current frame for positions within a region surrounding the upsampled pixel position; and A mean of the input pixel values ​​of the current frame within the region surrounding the upsampled pixel position is determined.

13. The method of claim 12, wherein the determining the pixel value of the upsampled pixel position comprises clamping the determined pixel value so that the pixel value differs from a determined mean of the input pixel values ​​of the current frame within the region surrounding the upsampled pixel position by no more than a threshold.

14. The method of claim 13, further comprising, for each of the one or more upsampled pixel positions: determining a standard deviation of the input pixel values ​​of the current frame within the region surrounding the upsampled pixel position, Wherein the threshold is based on a determined standard deviation of the input pixel values ​​of the current frame within the region.

15. The method of claim 14, wherein for each of the one or more upsampled pixel positions, the threshold is F pixel ·σ pixel , where F pixel is a predetermined factor, and σ pixel The determined standard deviation of the input pixel values ​​of the current frame within the region surrounding the upsampled pixel position.

16. The method of any one of claims 13 to 15, wherein the clamping is selectively applied to different regions to different degrees, wherein the method further comprises: comparing an average of pixel values ​​determined at upsampled pixel positions within the region surrounding the upsampled pixel positions with the mean of the input pixel values ​​of the current frame within the region surrounding the upsampled pixel positions; as well as The clamping is performed based on a comparison of: (i) a difference between the average of pixel values ​​determined at upsampled pixel positions within the region surrounding the upsampled pixel positions and the mean of the input pixel values ​​of the current frame within the region surrounding the upsampled pixel positions, and (ii) a threshold difference.

17. A method as claimed in any preceding claim, wherein the upsampled pixel positions are located between the positions of diagonally adjacent input pixels of the current frame such that the upsampled pixel positions and the positions of the input pixels form a repeating quincuncial pattern.

18. The method of any preceding claim, wherein said obtaining a depth value of said current frame for said upsampled pixel position comprises: receiving depth values ​​of the current frame at positions of input pixels surrounding the upsampled pixel positions; for each input pixel pair of the input pixels for which depth values ​​are received, determining an interpolated depth value for the upsampled pixel position based on the depth value of the input pixel pair; determining a depth weight for the input pixel pair based on a depth gradient between the depth values ​​for the input pixel pair; as well as The depth value of the current frame for the upsampled pixel position is determined by performing a weighted sum of the determined interpolated depth values ​​using the determined depth weights for the input pixel pairs.

19. A processing module configured to determine one or more pixel values ​​at corresponding one or more upsampled pixel positions of a current frame in a sequence of frames, the processing module being configured to: Obtaining a depth value of a pixel position of a reference frame in the frame sequence; as well as For each upsampled pixel position of the one or more upsampled pixel positions: Obtaining a depth value of the current frame for the upsampled pixel position; obtaining a motion vector for the upsampled pixel position to indicate motion of the upsampled pixel position between the reference frame and the current frame; identifying one or more of the pixels of the reference frame using the motion vectors of the upsampled pixel positions; determining a weight for each of the one or more identified pixels of the reference frame based on: (i) the depth value of the current frame for the upsampled pixel position, and (ii) the depth value of the position of the identified pixel of the reference frame; as well as The pixel value of the upsampled pixel location is determined using the determined weight for each identified pixel of the one or more identified pixels.

20. A computer-readable storage medium having computer-readable code stored thereon, wherein the computer-readable code is configured to execute the method according to any one of claims 1 to 18 when the code is executed.

21. A computer readable storage medium having stored thereon an integrated circuit definition data set, which when processed in an integrated circuit manufacturing system configures the integrated circuit manufacturing system to manufacture the processing module of claim 19.

Images