Systems and methods for machine learning based image conversion

Abstract

To provide a system and method for converting a newly improved image.SOLUTION: A method is configured to: (a) acquire a first image of a first resolution; (b) select pixels of a first block from the first image; (c) generate a plurality of first input channels on the basis of the pixels of the first block; (d) insert values from each of the plurality of first input channels into a first activation matrix; (e) apply the first activation matrix to a trained neural network to generate a second activation matrix; (f) generate a second image of a second resolution on the basis of a second activation matrix; and (g) output the second image to a display. The steps (b) to (g) are performed for each of a plurality of pixel blocks selected from the first image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Application Nos. 16 / 829,950 and 16 / 830,032, both of which are incorporated by reference in their entireties. [Background technology]

[0002] Technology overview The techniques described herein relate to machine learning and the use of machine learning to convert one data set or signal to another. More particularly, the techniques described herein relate to applying block transforms to such data sets or signals. Applications of the techniques include converting images of one resolution to another (e.g., higher) resolution, which may be used in real-time applications, for example, from images generated by a video game engine.

[0003] Introduction Machine learning can give computers the ability to "learn" a specific task without explicitly programming the computer for that task. One type of machine learning system is called a convolutional neural network (CNN), which is a class of deep learning neural networks. Such networks (and other forms of machine learning) can be used to help, for example, automatically recognize whether a cat is in a photograph. Learning occurs by "training" a model with thousands or millions of photographs to recognize when a cat is in the photograph. While this can be a powerful tool, the resulting processing from using a trained model (and training the model) can still be computationally expensive when deployed in a real-time environment.

[0004] Image upconversion is a technique that allows an image generated at a first resolution (e.g., 540p resolution or 960×540, which is 0.5 megapixels) to be converted to a higher resolution (e.g., 1080p resolution, 1920×1080, which is 2.1 megapixels). This process allows the first resolution image to be shown on a higher resolution display. Thus, for example, a 540p image can be displayed on a 1080p television and (depending on the nature of the upconversion process) may display the 540p image with increased fidelity compared to displaying the 540p image directly on a 540 television with traditional (e.g., linear) magnification. Different techniques for image upconversion can offer a trade-off between speed (e.g., how long the process takes to convert a given image) and the quality of the upconverted image. For example, if the upconversion process occurs in real time (e.g., during a video game), the resulting upconverted image may be of poor quality.

[0005] It is therefore recognized that there is a continuing need for new and improved techniques, systems, and processes in these technical fields. Summary of the Invention [Means for solving the problem]

[0006] overview In one exemplary embodiment, a computer system is provided for converting an image from a first resolution to a second resolution by using a trained neural network. The source image is divided into blocks, and context data is added to each pixel block. The context blocks are divided into channels, and each channel from the same context block is inserted into the same activation matrix. The activation matrix is ​​then run or applied to a trained neural network to generate a modified (e.g., output) activation matrix. The modified activation matrix is ​​then used to generate output channels to construct an image at a second resolution. These techniques can be performed during runtime and in real time along with the generation of the source image.

[0007] In one exemplary embodiment, a computer system for training a neural network to transform signal data (e.g., an image) is provided. For example, an image at a first resolution is transformed to a second resolution. Target signal data (e.g., a target image) is stored in a database or other non-transitory medium. For images, they may be at a resolution that is the target resolution. The computer system includes a processing system having at least one hardware processor. During training for image transformation, the computer system is configured to divide the first image into a first plurality of pixel blocks. Each one of the first plurality of pixel blocks is divided into a plurality of separate output channels to form target output data. A second image at a second resolution is generated based on one of the plurality of separate output channels. A plurality of context blocks are generated from the second image. The plurality of context blocks are then divided into a plurality of separate input channels and used to train the neural network by using the plurality of separate input channels until the neural network converges to the target output data.

[0008] In one exemplary embodiment, a method for transforming signal data using a neural network is provided. The method includes populating an initial activation matrix with a plurality of values ​​based on data from a plurality of samples from a source signal. A separable block transform is then applied across multiple layers of the neural network. The separable block transform, based on at least one trained matrix of coefficients, is applied to the input activation matrix to generate a corresponding output activation matrix. The initial activation matrix is ​​used as the input activation matrix for a first layer of the multiple layers, and the input activation matrix for each subsequent layer is the output activation matrix of the previous layer. The method outputs the output activation matrix of the final layer of the neural network to generate a transformed signal based on the output activation matrix of the final layer.

[0009] In one exemplary embodiment, the method operates such that at least two of the rows or columns of the initial activation matrix correspond to superimposable data from each of the plurality of samples.

[0010] In one exemplary embodiment, a distributed computer gaming system is provided. The system includes a display device configured to output images (e.g., of a video game or another application) at a target resolution. The system includes a cloud-based computer system including a plurality of processing nodes. The processing nodes of the cloud system are configured to execute a first video game thereon and generate images of the first video game at the first resolution. The processing nodes of the cloud system are configured to transmit image data based on the generated images. The system also includes a client computing device configured to receive the image data. The client computing device includes at least one hardware processor and is configured to execute a neural network based on the received image data to generate a target image. The execution of the neural network on the client device applies a separable block transform to a plurality of activation matrices, each corresponding to a different block of pixel data in an image represented by the image data. The target image is generated at the target resolution and output to a display device at the target resolution for display on the display device during gameplay of the first video game.

[0011] This Summary is provided to introduce various concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Rather, this Summary is intended to provide an overview of the subject matter described in this document. Accordingly, it will be appreciated that the features described above are merely examples, and that other features, aspects, and advantages of the subject matter described herein will become apparent from the following Detailed Description, Figures, and Claims.

[0012] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0013] These and other features and advantages will be more fully and completely understood by reference to the following detailed description of exemplary, non-limiting illustrative embodiments taken in conjunction with the drawings. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram illustrating an example computer system according to an example embodiment. [Figure 2] 2 is a flowchart illustrating a machine-learned upconversion process that may be executed on the computer system of FIG. 1. [Figure 3] 3 is a flowchart illustrating an expanded view of the image preparation portion of the machine-learned upconversion process of FIG. 2. [Figure 4] FIG. 3 is a flowchart showing an expanded view of the matrix reorganization portion of the machine-learned upconversion process of FIG. 2. [Figure 5] FIG. 3 is a flowchart illustrating an expanded view of the neural network implementation portion of the machine-learned upconversion process of FIG. 2. [Figure 6] 3 is a flowchart showing an expanded view of the block reorganization portion of the machine-learned upconversion process of FIG. 2. [Figure 7] FIG. 3 is a flowchart showing an enlarged view of the part of the machine-learned upconversion process of FIG. 2 that reorganizes 1080p output data into a converted image. [Figure 8A] Included are exemplary source images, originally in color but now in black and white, and corresponding transformed images, originally in color but now in black and white, generated according to the techniques described in connection with FIGS. 1 and 2. [Figure 8B] It includes two images, originally in color but now in black and white, that are zoomed versions of the image shown in Figure 8A. [Figure 8C] 1A-1C are block diagrams illustrating different SBT architectures according to an example embodiment. [Figure 8D] 1A-1C are block diagrams illustrating different SBT architectures according to an example embodiment. [Figure 8E] 1A-1C are block diagrams illustrating different SBT architectures according to an example embodiment. [Figure 9] FIG. 1 is a block diagram including an exemplary training computer system according to an exemplary embodiment. [Figure 10] 3 is a flowchart illustrating a process for training a neural network that may be used in connection with an exemplary embodiment, including the process shown in FIG. 2. [Figure 11] FIG. 11 is a flowchart showing an expanded view of how 1080p image data is prepared as part of the neural network training process shown in FIG. [Figure 12] FIG. 11 is a flowchart showing an expanded view of how 540p image data is prepared as part of the neural network training process shown in FIG. [Figure 13] 1 illustrates an exemplary computing device that may be used in some embodiments to implement features described herein. DETAILED DESCRIPTION OF THE INVENTION

[0015] Detailed Description In the following description, for purposes of explanation and not limitation, specific details are set forth, such as of particular nodes, functional elements, techniques, protocols, etc., to provide an understanding of the described technology. It will be apparent to those skilled in the art that other embodiments may be practiced apart from the specific details and examples described below. In some instances, detailed descriptions of well-known methods, systems, devices, techniques, etc. are omitted so as not to obscure the description with unnecessary detail.

[0016] This detailed description uses sections only to orient the reader as to the general subject matter of each section. As will be seen below, the description of many features spans multiple sections, and headings should not be construed as affecting the meaning of the descriptions contained in any section.

[0017] In many places in this document, including but not limited to the description of Figures 1 and 10, we describe software modules, software components, software engines, and / or actions performed by such elements. This is done for ease of description, and whenever this document describes a software module or the like as performing an action, it should be understood that the action is actually performed by underlying hardware elements (such as a processor, hardware circuitry, and / or memory devices) pursuant to instructions comprising the software module or the like. Further details in this regard are provided below, particularly in the description of Figure 13.

[0018] overview Certain exemplary techniques herein relate to converting an input signal (e.g., a digital signal) into an output signal through the use of a neural network. Examples of different types of signals may be image, audio, or other data that may be sampled or otherwise divided and converted into a converted signal according to certain exemplary embodiments discussed herein.

[0019] FIG. 1 shows a block diagram of an exemplary computer system (e.g., a video game system) that a user may use to play a video game. The system is configured to implement the process shown in FIG. 2, which causes images generated by a game engine at a first resolution (e.g., 540p) to be upconverted to a different resolution (e.g., 1080p). FIGS. 3 through 7 show different aspects of the process shown in FIG. 2. FIGS. 8A and 8B show non-limiting examples in accordance with the technique discussed in FIG. 2. FIGS. 8C through 8E are block diagrams illustrating different SBT architectures in accordance with an exemplary embodiment. FIG. 9 shows a block diagram with a computer system used to train a neural network that may be used by the process shown in FIG. 2. FIG. 10 is an exemplary process that may be executed on the computer system of FIG. 9 to generate a trained neural network. FIGS. 11 through 12 are more detailed aspects of the process shown in FIG. 10. FIG. 13 is a block diagram of an exemplary computer system that may be used to implement or execute the processes in FIGS. 1 and / or 9 and / or shown in FIGS. 2 and / or 10.

[0020] Description of Figure 1 FIG. 1 is a block diagram illustrating an exemplary computer system according to an exemplary embodiment.

[0021] The game device 100 is an example of a computer system 1300 shown in FIG. Although the term "gaming" device is used in connection with certain exemplary embodiments herein, this is for ease of use, and any type of computing device may be used. Indeed, a "gaming" device as used herein may be a computing device (e.g., a mobile phone, a tablet, a home computer, etc.) that is currently being used (or will be used) to play video games. A non-limiting, exemplary list of computing devices may include, for example, smart or mobile devices (e.g., smartphones), tablet computers, laptop computers, desktop computers, home console systems, video game systems, home media systems, and other types of computing devices. As described in connection with FIG. 13 , computers may vary in size, shape, functionality, etc. In certain exemplary embodiments, the techniques discussed herein may be used with non-gaming applications. For example, they may be used with real-time video surveillance, web browsing, voice recognition, or other applications in which converting one data set to another may be useful. Additional examples and applications of the techniques herein are discussed below.

[0022] Gaming device 100 may include a CPU 102, a GPU 106, and a DRAM (dynamic random access memory) 104. CPU 102 and GPU 106 are examples of processor 1302 of Figure 13. DRAM 104 is an example of memory device 1304 of Figure 13. Different types of CPUs, GPUs, DSPs, dedicated hardware accelerators (e.g., ASICs), FPGAs, and memory technologies (both volatile and non-volatile) may be used on gaming device 100.

[0023] Examples of different types of CPUs include the Intel CPU architecture (e.g., x86) and the ARM (Advanced Risk Machine) architecture. Examples of different GPUs include discrete GPUs, such as the NVIDIA V100 (which may include hardware support for matrix multiplication or tensor cores / accelerators), and integrated GPUs that may be found on a system-on-chip (SoC). An SoC may combine two or more of the CPU 102, GPU 106, and local memory, such as registers, shared memory, or cache memory (also called static RAM or SRAM), on a single chip. DRAM 104 (also called dynamic RAM) is typically fabricated as a separate semiconductor and connected to the SoC through wiring. For example, the NVIDIA Tegra X1 SoC includes multiple CPUs, GPUs, northbridge controllers, southbridge controllers, and memory controllers all on a single SoC. In some examples, the processing power provided by the CPUs, memory components, GPUs, and / or other hardware components that make up a given gaming device may differ from that on other gaming devices. Some gaming devices may be portable, some may be stationary gaming consoles, or may operate as personal computers (e.g., desktop or laptop computer systems used to play video games).

[0024] A GPU may include many processing cores operating in parallel. Each processing core that is part of the GPU may operate with corresponding hardware registers that store data used by the various processing cores. For example, NVIDIA's GPU architecture includes a number of 32-bit, 16-bit, and / or 8-bit registers that output data to the GPU's processing cores. In some GPU architectures, the highest bandwidth memory may be available in registers, followed by shared memory, then cache memory, and then DRAM. As discussed in more detail below, data for a dataset to be converted (e.g., an image to be upconverted) may be efficiently loaded into these registers to improve the efficiency of converting the dataset to another form (e.g., another resolution). Indeed, by utilizing hardware registers on the GPU for this operation, the exemplary The upconversion process may be performed in real time (e.g., less than 1 second, less than 1 / 30 second, or less than 1 / 60 second) and / or during the runtime of the application or game (e.g., without noticeable delay).

[0025] In an exemplary embodiment, the techniques herein may advantageously utilize NVIDIA tensor cores (or other similar hardware). A tensor core may be a hardware unit that multiplies two 16×16 FP16 matrices (or matrices of other sizes, depending on the hardware) and then adds a third FP16 matrix to the result using a fused multiply-add operation to obtain an FP16 result. In an exemplary embodiment, a tensor core (or other processing hardware) may be used to multiply two 16×16 INT8 matrices (or matrices of other sizes, depending on the hardware) and then adds a third INT32 matrix to the result using a fused multiply-add operation to obtain an INT32 result. The INT32 result may then be converted to INT8 by dividing it by an appropriate normalization amount (which may be calculated, for example, during the training process, as described in connection with FIG. 9 ). Such a conversion may be achieved, for example, using a low-processing-cost integer right shift. Such hardware acceleration for the processes discussed herein (e.g., in the context of separable block transforms) may be advantageous.

[0026] 1, gaming device 100 may also be coupled to input devices 114 and display devices 116. Examples of input devices 114 include video game controllers, keyboards, mice, touch panels, sensors, and other components that may provide input used by a computer system (e.g., gaming device) to execute application programs and / or video games hosted on the computer system.

[0027] Examples of display devices 116 include televisions, monitors, integrated displays (e.g., part of a mobile phone or tablet), etc. In some examples, gaming device 100 may be configured to couple to different types of display devices. For example, gaming device 100 may be coupled to an integrated display (e.g., part of the structure housing gaming device 100) to which images may be output. Gaming device 100 may also be configured to output images to a larger television or other display. In some exemplary embodiments, different display devices may natively display different resolutions. For example, the integrated display of a gaming device may have 0.5 million pixels (e.g., a 540p display), while the separate display may have 2.1 million pixels (e.g., a 1080p display). Using techniques herein, gaming device 100 may be configured to output different images for a game depending on which display device the gaming device is output to. Thus, for example, a 540p image may be output to the integrated display when the integrated display is used, and a 1080p image may be output to the 1080p display when used.

[0028] In an exemplary embodiment, a computer system may dynamically switch the type of image being output based on conditions associated with the computer system. Such switching may occur while a user is playing a game (perhaps with a short pause while switching between the two modes occurs). For example, when the computer system is running on battery power (e.g., not plugged into a socket), the computer system may be configured not to use the exemplary image conversion process using the techniques discussed herein. However, when the computer system is plugged into AC power, the techniques discussed herein for upconverting images to higher resolutions may be used or turned on for video games or other applications because the techniques discussed herein may allow the GPU being used to use a higher percentage of the available processing power (e.g., up to 80, 90, or 95%, or more). This is because the GPU power consumption may increase. Thus, if a computer system were to run solely on the mobile device's battery while using, for example, the process shown in FIG. 2 , it could deplete the battery more quickly. Thus, such techniques may enable a user to play games on their mobile device, for example, when commuting home from work. In this mode, the user uses the device's local display (e.g., 540p) for video games. However, when the user gets home, the user may plug the mobile device into an outlet so that the mobile device is no longer relying on its own battery power. Similarly, a user may couple the mobile device to a larger display (such as a television) that is a 1080p display. Such a connection may be wired (e.g., a DisplayPort or HDMI® cable) or wireless (e.g., Bluetooth® or WiFi®). Upon detecting one (or both) of these scenarios (e.g., a destination display capable of displaying the higher resolution and / or a non-battery power source for the computing system), the system may dynamically initiate the image conversion process discussed with respect to Figure 2 to enable the user to play the game on their 1080p television and view the game at the higher resolution. In an example embodiment, the user may also manually initiate the image upconversion process.

[0029] The techniques herein may advantageously provide performance that is less constrained by limited memory bandwidth than prior approaches. In other words, the architectures for converting images (or, more generally, datasets) discussed herein may not be memory bandwidth-bound. This may be particularly true for real-time inference, which may typically be limited to one batch (e.g., instead of a typical training scenario that typically benefits from a larger batch, such as 256). In other words, the techniques herein may enable near 100% utilization of a matrix multiplication hardware accelerator during the runtime of an application (e.g., a video game), thus increasing (e.g., maximizing) overall performance per dollar spent on hardware used for conversion.

[0030] 1 , gaming device 100 stores and executes video game application program 108. Video game application program includes game engine 110 and neural network 112. Gaming device 100 may also store image data (e.g., textures) and other types of assets (e.g., sounds, text, pre-rendered images, etc.) that video game application program 108 and / or game engine 110 use to generate or create content for the video game (or other application), such as, for example, images for the game. Such assets may be included with the video game application program on a CD, DVD, or other physical medium, or may be downloaded over a network (e.g., the Internet), for example, as part of a download package for video game application program 108.

[0031] The game engine 110 includes a program structure for generating an image to be output to the display 116. For example, the game engine 110 may include a program structure for managing and updating the position of an object in a virtual space based on input provided from the input device 114. The provided data is used to render an image of the virtual space using, for example, a virtual camera. This image may be a source image generated at a first resolution (e.g., 540p). The source image is applied to a neural network 112, which converts the source image into an upconverted image (e.g., the upconverted image is generated based on application of the source image to the neural network 112) at a higher resolution (e.g., 1080p) than the original source image. The upconverted image is then output to and displayed on display device 116. Further explanation of how the neural network is generated is provided in connection with FIG.

[0032] In an example embodiment, it takes less than 1 / 60 of a second to upconvert a source image (e.g., generated by game engine 110). Thus, if the game engine is generating images intended to be displayed on display 116 60 times per second, there may be little or no noticeable visual delay when the upconverted image is output to the display in place of the source image. In this manner, such techniques may enable the generation and display of upconverted images in real time from the original source image. For example, if a video game application is developed to generate images at a first resolution (e.g., 540p), the techniques herein may enable visual enhancement of the video game application such that images may be output from the video game application at a higher resolution (e.g., 1080p) than originally intended.

[0033] While video game application program 108 is used for purposes of illustration, it will be appreciated that other applications that provide visual output may be substituted. Also, while neural network 112 is shown as part of video game application program 108, it may be provided separately. For example, it may be part of an operating system service or the like that modifies or upconverts images as they are being output by the video game application program.

[0034] In one exemplary embodiment, a "gaming device" may be a device hosted within a cloud-based environment (e.g., Amazon's AWS or Microsoft's Azure system). In such a scenario, the game (or other application program) may be hosted on a virtual machine in the cloud computer system, and the input and display devices may be local to the user. The user may also have a "thin" client application or computer that communicates with the cloud-based service (e.g., communicates data from the device and receives and displays incoming images from the cloud on a television). In this type of implementation, user input is passed from the user's computer / input device to a cloud-based computer system running a video game application 108. Images are generated by the game engine, transformed (e.g., upconverted) by a neural network, and then sent to the user's display (or a computer that outputs the images to the display).

[0035] In an example embodiment, a cloud-based system may take advantage of upscaling capabilities on a "thin" client by rendering, compressing, and streaming compressed low-resolution (e.g., 540p) video / images to the client at lower server cost (and bandwidth) and having the client hardware perform the upscaling (e.g., neural network processing 112). In some examples, this may also include having a neural network address or compensate for compression artifacts. Thus, features herein may advantageously reduce bandwidth usage in some cloud-based gaming environments.

[0036] In an example embodiment, the cloud-based system may operate dynamically with respect to the destination display being used by a user. So, for example, a video game may natively output a 540p image. A first user may use the cloud system to receive a 1080p image (e.g., upconverted from 540p), while a second user may use the cloud system to receive an image of a different resolution (e.g., a 720p image, 480p image, etc.). k images, or 1440p images). Each instance of the video game application (and / or neural network) may be hosted within its own virtual machine or virtual container, thus providing the flexibility to provide multiple different users with different options (e.g., outputting images at different resolutions) to "play" the same video game.

[0037] Cloud-based implementations may be useful in contexts where users have access to GPUs capable of executing the techniques discussed herein.

[0038] In an example embodiment, the GPU may instead be (or include) an ASIC or FPGA that operates in a manner similar to a GPU.

[0039] In one exemplary embodiment, gaming device 100 may be two or more computer systems.

[0040] It is also recognized that the type of "application" or program or data source providing the source image is not limited to video games. Indeed, other types of applications may utilize the techniques herein, including real-time image recognition from wildlife cameras, speech, word / sentence translation, images coming from home security cameras, movies and other TV programs, etc.

[0041] For more general applications such as image classification, for example, a traditional implementation of a CNN on GPU processing hardware may involve: 1) loading layer weights in fast memory (e.g., the GPU's registers or shared memory); 2) loading layer inputs into registers from DRAM; 3) multiplying the inputs by the weights using a matrix multiplication implemented on the GPU; 4) applying a nonlinear function; 5) storing the layer outputs in DRAM; and 6) repeating this process for each layer. A drawback of this approach is the movement back and forth to and from DRAM. For example, layer data (e.g., activations) typically cannot fit entirely into the relatively limited amount of fast memory (e.g., registers) used in connection with processing neural network layers. Thus, in some cases, it may be necessary to transfer that data between different memory locations because the layer data (e.g., activations, which may be a matrix of 960 × 540 × 16 values, i.e., corresponding to a 540p source image resolution combined with 16 channels in one example) may not fit into the GPU's registers (or other "fast" memory). Therefore, main memory (DRAM 104) may be used to store such information.

[0042] In one exemplary embodiment, fusion of different layers (e.g., "layer fusion") may be used so that the calculations from one layer and the next layer can be realized through a single processing code (e.g., a CUDA kernel). A potential drawback of this approach is that, because CNNs are translation-invariant, as more layers are fused, a larger number of inputs may be required to compute a single output value. Thus, this type of implementation may provide valuable benefits by increasing the receptive field (which depends on the neural network's ability to "see" the end values / a wide range of inputs), but may also entail performance drawbacks.

[0043] In an exemplary embodiment, an approach to how data may be prepared and processed may be based on the nature of the underlying hardware that will perform the operations (e.g., matrix operations). In an exemplary embodiment, an image may be divided into blocks, the size of which may be based on the underlying hardware. An exemplary embodiment may be implemented on NVidia GPU hardware (e.g., Volta and Turning architectures), where In the example shown, the CUDA API exhibits hardware acceleration for 16x16 matrix multiplication. Due to this (and as discussed below), a 4x4 block size may be used in the image to be transformed (these 16 pixels are mapped to rows of a 16x16 matrix). Such an implementation allows the input to be split into 16 inputs with 16 channels (although, as discussed below, in some examples, fewer than 16 channels may be used), thus fitting into an atomic 16x16 matrix. This may then be stored in registers of the GPU (or other "fast memory" that will handle matrix computations). Of course, the GPU 106 (or even the CPU 102, if so designed) for an exemplary block-based neural network architecture for a particular size may be designed with different atomic matrix sizes, depending on the dimensional nature of the fastest atomic multiplication hardware available.

[0044] If the matrices remain in registers, layers of a given pixel (or other type of data from a signal) can be "fused" together because they remain in registers during processing. This is discussed in more detail below in connection with FIG. 2. In an exemplary embodiment, the activation matrix may remain in the internal memory of the hardware (e.g., GPU, CPU, DSP, FPGA, ASIC, etc.) performing the matrix operations on that activation matrix. In other words, the data for a given activation matrix may remain within the same semiconductor hardware (which may be the same silicon for silicon-based memory, or other materials such as gallium, germanium, etc. for other memory types) while various layers of a neural network apply to that activation matrix—e.g., successively transforming the activation matrix across multiple layers of a neural network.

[0045] Based on such blocks, a general transformation of a layer using block matrices (where each of the blocks W is a generic p×p matrix) can exist as follows:

[0046]

number

[0047] It is recognized that in such block matrix designs, the isolation of each block can prevent receptive field propagation that would otherwise occur (e.g., in the case of a typical CNN). In this manner, the techniques herein can enable the fusion of many layers (e.g., as many as desired) while still maintaining the locality of the data in question. Because the width of the data remains somewhat constant between the input and output of each layer, such fused layers are sometimes referred to as "block towers."

[0048] From an inference perspective, this kind of approach can be advantageous because it can be implemented as a series of atomic-sized matrix multiplications:

[0049]

number

[0050] One potential problem is that by maintaining the data in such a localized manner, the system may not benefit from a wider receptive field, which may be beneficial in some classification applications. Such problems may be at least partially addressed in some instances by introducing "block convolution" and "block pooling" layers, for example:

[0051]

number

[0052] In the formula, W i is a p×p matrix (typically p=16). This can be similar to a traditional CNN matrix formulation, but with the individual CNN filter weights (e.g., single real floating-point numbers) replaced by a block matrix. Or, stated another way, the blocking techniques discussed herein can be considered a generalization of CNN, since with a 1x1 dimensional block matrix, the techniques can be reduced to more traditional CNN formulations.

[0053] In an exemplary embodiment, an input signal (which may be, for example, an image) can be processed by a separable block transform (or block convolution SBT) in a “translation-invariant manner.” Thus, in the context of an image, if a signal S (e.g., a first image) is translated by four pixels horizontally or four pixels vertically into a signal S′ (e.g., a second image), the generated 4×4 blocks of signals S and S′ (corresponding to the activation matrices used as inputs to the SBT network) will most likely coincide (except for the boundaries of the respective images). If the blocks of S and S′ are identical (again, except for the boundaries of the signals), the output blocks generated by applying S and S′ through the SBT will be the same. In other words, the transformed signals will also be the same, despite the translational differences between SBT(S) and SBT(S′). Another way to look at this is that the SBT (and / or the block convolution SBT) is computed on a first block, then again on neighboring blocks using the same weights (the same learned L and R matrices), then again on neighboring blocks, and so on. Thus, the signal is processed in a "convolutive manner" by applying the same calculation while moving (eg, translating) the input position along the input signal.

[0054]

number

[0055] This is sometimes called a tensor product, and the (block) matrix is ​​called the Kronecker product of L and R.

[0056]

number

[0057] A left matrix L of dimension p×p (e.g., the pointwise transform in MobileNet) processes all channels of a given data point in the same way for each data point. This is a general form, meaning that all of its coefficients can be learned completely independently.

[0058] A right-hand matrix R of dimension p×p (e.g., the depthwise convolution transform in MobileNet) processes all pixels of a given channel in the same way for each channel. This is a general form, meaning that all of its coefficients can be learned completely independently.

[0059] The above formulation is symmetric and balanced, and may be generally applicable in several different cases. The format may also be further modified to handle rectangular matrices (e.g., of size p × q) on both the L-side and the R-side. In other words, the input dimensions of a layer may match the output dimensions of the previous layer. However, it is recognized that making the values ​​of p and q multiples of the size of the atomic accelerated hardware matrix multiplication may increase / be more efficient in using hardware resources and may be optimal in some instances for speed.

[0060] Advantageously, the block shape and invariance between data points can be used to process them together in a single matrix multiplication, which in one exemplary embodiment results in an efficient use of resources.

[0061] It is recognized that a 3x3 convolution kernel can be realized by adding nine pointwise 1x1 kernels, so the separate transformations discussed above can also be added together as follows:

[0062]

number

[0063]

number

[0064] A potential further benefit of the summation approach may be the performance of the inference implementation. If the input and output matrices of an LXR product have the same format (e.g., 16x16 FP16 values), the code implementing the inference may be strictly limited to matrix multiplications (e.g., with fused multiply-add) performed one after the other. This type of approach may advantageously allow operations to be performed without having to align data, reorganize such data into other forms, or convert data to other formats. This type of approach may also advantageously avoid adding or using unnecessary instructions because the data is already in the correct format for each portion of the summation. In one example, the number of LXR summations may be configurable as a dynamic parameter because the format of the input and output of the summation does not change (e.g., may be assumed to be 16x16 matrices as discussed in connection with the examples herein). In this way, this may be a way to increase the weights at will, and thus be able to learn / memorize more, while ensuring that the time required to load the weights remains hidden behind the time it takes to do the matrix multiplication (which, for example, depends on each particular hardware memory bandwidth and matrix multiplication speed).

[0065] For training, much more flexibility can be applied to train large networks. The network can then be compacted by pruning the least important elements of each sum, while retaining only the most relevant aspects obtained through the "draw" of the lottery / matrix initialization. This dynamic process can help determine, on a content-based basis for each training run, how many multiplications to allocate in each layer for a given budget of processing time. Such a decision can be based on knowing a simple model of inference time, which is linear in the number of matrix multiplications. Such aspects can then be combined to determine the number of layers (which may be as few as a dozen, typically not a particularly large potential search space).

[0066] In some exemplary embodiments, a larger number of channels may be used, in which several of the separable block towers discussed herein may be computed in parallel from the same inputs (e.g., activation matrices) but with different trained weights (L and R matrices). Such an approach may be similar in some respects to grouping channels in convolutional neural networks.

[0067] In one exemplary embodiment, the channels of each tower are isolated from each other until the edge of the network. To avoid maintaining multiple blocks, the outputs of all block towers can be stored together (e.g., in a memory such as DRAM or cache) and used together as inputs for another group of separable block towers. Such an implementation may rely even less on DRAM bandwidth (e.g., as data is accessed more quickly through cache memory) compared to an equivalent convolutional neural network architecture. In other words, a p*p SBT can use more than p*p activations as inputs by multiplying each of several p*p input activations by a different p*p weight matrix and fusing them by adding (e.g., term to term) all the results into a single p*p matrix that becomes the SBT's input activation matrix. This aspect is described in more detail below in connection with Figures 8C-8E.

[0068] Although GPUs are discussed in certain exemplary embodiments herein, it will be appreciated that ASICs and FPGAs may also be designed and used in place of such GPUs in certain exemplary embodiments.

[0069] Description of Figure 2 Figure 2 is a flowchart illustrating a machine-learned upconversion process for converting a 540p image to a 1080p image, which may be executed on the computer system of Figure 1. Figures 3 through 7 are discussed below to provide additional details regarding certain aspects of the upconversion process shown in Figure 2. While images and pixel data are described in connection with examples herein, it is recognized that other types of signals may be used in connection with the techniques herein. For example, each "pixel" in the images discussed herein may be considered data that has been sampled from the entire signal (e.g., an image). Accordingly, techniques are discussed herein for converting or otherwise transforming a source signal (e.g., an image) into a transformed or converted signal (e.g., a higher resolution image).

[0070] At step 200, a 540p source image 205 is rendered by the game engine 110. In an example embodiment, as discussed herein, the source image may originate from other sources, such as a real-world camera, a movie, a television program, or broadcast television. For example, techniques herein may be used to convert a source 540p signal received for a television program (e.g., a live sporting event) into a 1080p signal and output it for display to a user. Furthermore, while 540p is discussed in connection with the example of FIG. 2 (and elsewhere herein), the techniques may also be applied to images of other sizes. It is recognized that the details (e.g., coefficients or L and R) of the neural network 112 used as part of the upconversion process may change as the details of the source and / or converted images change (e.g., as the resolution of such images is adjusted). For example, a neural network for upconversion from 540p to 1080p will be different from one for upconverting from 1080p to 1440p (e.g., 2560×1440). While the examples shown in Figures 3-7 relate to converting a 540p image to a 1080p image, it is appreciated that the techniques herein may also be applied to other image sizes (e.g., 720p to 1080p; 480p to 1080p; 1080p to 1440p; 1080p to 4k / 3840x2160; 720p to 4k, etc.).

[0071] In an exemplary embodiment, the initial image may be generated by rendering or otherwise using motion vector information and / or depth information (e.g., z-buffer data). This information may be used to improve the resulting converted image quality. In an exemplary embodiment, such information may be added to an activation matrix created based on each pixel block.

[0072] In one exemplary embodiment, the horizontal and vertical directions are not integers (or are not in the same ratio). Upscaling ratios (not included) may also be performed in accordance with the techniques discussed herein. For example, for 720p to 1080p, the output blocks may be 6x6 pixels (with 3 channels, and therefore 108 output values), which can still easily fit into the 16x16=256 output values ​​of the SBT output discussed herein. According to an exemplary embodiment, additional ratios are also contemplated, such as 7 / 3 (e.g., this may correspond to converting from 1920x1080 to 4480x2520). In such an exemplary embodiment, the source image may be divided into 3x3 blocks (with added context data) and trained to output 7x7 blocks (which would still fit into the 16x16 output blocks discussed herein). An exemplary embodiment may modify applications that output images at resolutions that are now less common. The techniques herein may handle upscaling, for example, using alternative ratios. For example, in one exemplary embodiment, a horizontal expansion ratio of 8 / 7 (which may then be multiplied by some integer ratio) may be used to compensate for or account for analog TV anamorphic distortions.

[0073] In any event, in step 200, a 540p image 205 is generated (e.g., rendered) by a game engine or the like. Next, in step 210, the image is prepared. This aspect of the process, described in more detail in connection with FIG. 3, involves splitting the image into separate input channels or input data 215. Advantageously, the input data 215 may be stored in a register (e.g., a 16-bit register) of the GPU 106 at this point. Once the input data 215 is generated, it is stored in a register of the GPU. The input data 215 (or activation matrix 225) may remain in a register (or other internal memory) throughout its application to the neural network. This type of implementation advantageously allows the (relatively) slow DRAM 104 in the system 100 to be bypassed during neural network processing (e.g., multiple activation matrices across an image are processed by the GPU). This is facilitated by shaping the data to fit within registers, thus making more effective use of the massively parallel processing that GPU 106 provides.

[0074] In an example embodiment, other types of hardware besides a GPU may be used to handle the conversion of input data 215 to 1080p output data 245. It is generally preferable to keep such data in on-chip memory (e.g., registers on a GPU or SRAM FPGA handling deep learning applications). Thus, once input data 215 is placed in a register (or similar fast memory), it may remain there until 1080p output data 245 is generated (or a final matrix of activations is generated) and used to construct the final converted image (which may occur in DRAM).

[0075] 2, next, in step 220, the input data 215 is reorganized into a matrix to generate a 16×16 matrix of activations 225. This step is discussed in more detail in connection with FIG.

[0076] In step 230, the initial matrix of activations 225 is run through the trained neural network 112 to generate a 16×16 matrix of activations 235 transformed by the neural network 112. As discussed herein, this may involve applying a separable block transform to the matrix of activations. This aspect of the process is discussed in more detail in FIG. 5.

[0077] Once the matrix of activations has been run through the neural network in step 230, it is reorganized into blocks in step 240 to generate 1080p output data 245. This aspect of the process is discussed in more detail in FIG.

[0078] In step 250, the 1080p output data 245 is then reassembled into a 1080p image 255 and output to the display 116 in step 260. Aspects of this process are described in more detail in FIG. 7. As noted above, the processing illustrated between steps 220 and 250 (including both of these steps) may be performed entirely in the registers of the GPU (or other internal memory) without the need to transfer data to DRAM (or other relatively "slow" memory). Thus, for example, a given activation matrix 225 may remain stored in the same semiconductor hardware (e.g., in the same register or location in memory) while it is running through the neural network. Such processing may be applied to each matrix generated for a corresponding pixel block of an image (or other signal), which may then be executed simultaneously across multiple hardware processors of a GPU (or other hardware resource), for example.

[0079] Description of Figure 3 FIG. 3 is a flow chart illustrating an expanded view of the image preparation portion of the machine learned upconversion process of FIG.

[0080] A 540p image 205 output from the game engine 110 is cut or divided into 4x4 pixel blocks in step 300. Block 302 represents one of the pixel blocks from the image, and 304 is one pixel within that block. Each pixel may be represented by a different RGB color value (described in more detail in step 330). While color values ​​(e.g., RGB values) are discussed in connection with an exemplary embodiment, it is recognized that other types of data may be stored in blocks. For example, the techniques may be used in connection with grayscale images, where each pixel stores the amount of light associated with that pixel. In an exemplary embodiment, color information may be processed / provided by using YUV or YCoCg formats. In an exemplary embodiment, the luminance (Y) channel may be used with the techniques discussed herein, thus processing (e.g., upscaling) it using a neural network.

[0081] In some exemplary embodiments, block sizes other than 4×4 may be used. For example, in some exemplary embodiments, 8×2 pixel blocks may be used. In some examples, the size of the pixel blocks may be advantageously determined based on the dimensions of the hardware being used for the matrix multiplication, or a multiple thereof. Thus, if the hardware acceleration supports 16×16 matrix multiplication, a 4×4 or 8×2 block may be initially selected. Such a size may advantageously allow for separate processing of pixels along one dimension of the matrix while processing channels along the other dimension.

[0082] The selection of block size may also be based on the amount of available high-speed memory (e.g., registers, etc.) in the system. By maintaining blocks of matrices and corresponding data in high-speed memory during neural network processing, increased performance may be advantageously facilitated (e.g., enabling real-time or run-time image conversion). Thus, while a 4x4 block size may be appropriate for some hardware, other block sizes are contemplated and may be used in conjunction with the techniques discussed herein.

[0083] In any event, at 310, each block from the original 540p image 205 is selected. Thus, in one exemplary embodiment, there may be over 30,000 pixel blocks that undergo the processing described in FIG. 3 for a single 540p image. Subsequent processing of all of the pixel blocks may occur in parallel, for example, by using the hardware resources of a GPU or other processor. In some cases (e.g., depending on the number of individual processing units in the overall system), multiple groups may be processed sequentially. For example, For example, one group of pixel blocks may be processed in parallel (say 15,000), then another group (the remaining 15,000), etc. From the user's perspective, processing for all of the blocks may still occur in parallel.

[0084] At 320, context data is added to the 4x4 pixel block to create an 8x8 context block 322. The context data may be based on, derived from, or a function of pixel values ​​of pixels in an image surrounding a given pixel block. In one example, the pixel data used for the context block may remain unchanged from pixels outside the 4x4 pixel block. In an exemplary embodiment, other context block sizes (absolute or relative) may be used. For example, a 12x12 context block may be used for a 4x4 pixel block. In an exemplary embodiment, pixel data along the horizontal and / or vertical axes may be selected without taking into account pixel data along the diagonal axis. Thus, if a pixel block is represented as X1-X4 as shown in the table below, horizontal values ​​(A1-A4) and vertical values ​​(B1-B4) may be added to the content block, while diagonal values ​​(C1-C4) are not used within the context block.

[0085] [Table 1]

[0086] In some implementations, one pixel along the diagonal may be used, while two (or more) along the horizontal or vertical directions may be used within a context block. In some exemplary embodiments, multi-resolution data may be included within a context block to increase the receptive field along the direction of a "slightly slanted line." Aliasing may extend far from the block. For example, one layer may contain a 4x4 block calculated as the average of 8x8 context blocks, then a 4x4 block calculated as the average of 16x16 context blocks, and so on. Such data may help increase the receptive field at a limited cost in terms of the number of inputs.

[0087] At 330, context block 322b is divided into four separate input channels 333, 334, 335, and 336. The number represented by each input channel indicates the configuration of that particular channel. Thus, each 1 shown in 322b of FIG. 3 is used to form input channel 333, each 2 is used to form input channel 334, and so on. Each number represents one of the RGB values ​​of the corresponding pixel. Thus, each context block 332 is repeated or otherwise executed for each red (R), green (G), and blue (B) value (or the context block simply stores three values ​​per pixel). Thus, there are 12 input channels per pixel block created as a result of image preparation step 210. In addition, in this exemplary embodiment, there are three input channels per pixel (one for each of the pixel's R, B, and G values). The 12 input channels created for each pixel block form input data 215. This process is repeated or otherwise performed across all of the pixel blocks of a given image, and is typically accomplished in parallel. As discussed herein, in one exemplary embodiment, multiple pixel blocks ( and / or context blocks) may be processed in parallel.

[0088] In an example embodiment, the signal data of a source signal may be cut or separated into at least two blocks, and in an example, such blocks may then be processed independently by using the SBTs discussed herein.

[0089] Description of Figure 4 FIG. 4 is a flow chart illustrating an expanded view of the matrix reorganization portion of the machine-learned upconversion process of FIG.

[0090] In this flowchart, in step 410, the input data 215 for each pixel block (e.g., 12 input channels) is reorganized into a single 16×16 matrix 225. For example, the value of input channel 333a (e.g., having a red value of a “1” pixel value in context block 322b) is inserted (e.g., added) into row 412 of matrix 225. The value of input channel 333b (the blue value of the same “1” pixel from the context block) is inserted into row 414. And the value of input channel 333c (the green value of the same “1” pixel from the context block) is inserted into row 416. This process is repeated or otherwise performed for all 12 rows, such that the 16×16 matrix of activations 225 is populated with values ​​from sampled pixels of the source image (e.g., source signal). Thus, the resulting 16×16 matrix may contain data for a single pixel in multiple rows. For example, the pattern of data for each of the pixels used to generate rows 412, 414, and 416 can be superimposed from one pixel to the next. It is recognized that data can be inserted into a matrix on a column basis rather than a row basis as shown in Figure 4. Thus, in one exemplary embodiment, columns can be used instead of rows as referred to herein.

[0091] Examples of superimposable patterns may include, for example, two horizontally neighboring blocks of 4x4 pixels (e.g., after a horizontal translation of 4 pixels). As another example, any two rows (e.g., of 4x1 pixels) within a 4x4 block of pixels may be superimposable. Similarly, a row of 4x1 pixels can be superimposed on a column of 1x4 pixels (after a 90° rotation). The following patterns of blocks are superimposable: Specifically, the pattern of X in the following table (taking into account rotation and symmetry) is superimposable with the pattern of samples represented by Y:

[0092] [Table 2]

[0093] Other types of data (e.g., different types of signals) may also be superimposed such that the individual pieces that make up a sample piece of data are separated or separated into separate channels. In other words, depending on the nature of the source signal (e.g., an image or some other data), at least two of the rows (or columns) in the initial activation matrix may correspond to similarly organized or structured data from each sample taken from the underlying source. In the case of an image, the similarly organized or structured data may correspond to individual pixels (e.g., multiple pixels per pixel). A pixel may be a pixel with a common shape (if a channel is used) or a group of pixels that follow the same shape but are at different locations in the image. In an exemplary embodiment, at least two of the rows or columns of the activation matrix may be generated based on a common pattern of data from each sample in the underlying source signal.

[0094] In one exemplary embodiment, there are 12 input channels, so in step 420, the remaining 4 rows of the 16-row matrix are set to zero (or to a value that is ignored during matrix processing) to create the activation matrix 225, which is then subjected to neural network processing.

[0095] In an exemplary embodiment, data may be placed in all 16 (or however many rows there are in the activation matrix that will be used). In an exemplary embodiment, additional information may be placed in four additional rows (or "extra" rows that do not have initial color information). For example, the game engine 110 may provide depth information regarding an object or other aspect of the image in question. This information may be incorporated into the additional rows of the 16x16 matrix. In an exemplary embodiment, motion information regarding an object or other aspect of the image may be provided by the game engine 110 and incorporated into the 16x16 matrix.

[0096] Description of Figure 5 Figure 5 is a flow chart illustrating an expanded view of the neural network implementation portion of the machine-learned upconversion process of Figure 2. The neural network implementation on the matrix of activations may include application of a separable block transform utilizing the LXR operation discussed herein.

[0097] The matrix of activations 225 is run through the neural network 112. An example of how such a neural network may be trained is discussed in connection with Figure 9. The output of such training may be a matrix of coefficients (L and R) that have been "trained" on an exemplary training data set.

[0098] As part of the neural network processing in step 230, the matrix of activations 225 generated from the input channels is run through a separable block transform in step 410. An equation representing this process is shown in Figure 5, where L and R are 16x16 matrices (e.g., each having 256 coefficients in a 16x16 matrix) generated using the training system discussed in Figure 9.

[0099] L is a 16x16 pixel-by-pixel matrix (or other sample-by-sample dependent aspect) that is multiplied on the left. It applies a linear transformation to all channel values ​​of each activation pixel (e.g., each sample data) that may be in each column in the activation matrix, independent of the pixel's position (e.g., the same transformation for each pixel).

[0100] R is a 16x16 channel identity matrix, multiplied on the right: it applies a linear transformation to all pixel values ​​in each activation channel (e.g., each row of the activation matrix), independent of channel position (e.g., the same transformation for each channel).

[0101] The transformation can also be expressed as:

[0102]

number

[0103] where k is between 1 and p for a p*p matrix (e.g., p=16 in the example discussed above). 2 Thus, in one exemplary embodiment, for example, k varies between 16 This may provide more trainable freedom for more semantic-carrying layers (e.g., with respect to weights, coefficients of L and R matrices, etc.). In some examples, it may also allow pruning by removing LXR transforms one by one at training time, reducing complexity while preserving the quality of the final image. Such aspects are discussed in more detail in relation to the training process.

[0104] As part of the neural network's execution, an activation function 420 is applied. This may be a ReLU (rectified linear unit), e.g., if the value is negative, it is set to 0. If the value is positive, it is left as is. Depending on the particular application, other types of activation functions (e.g., linear, tanh, bivariate, sigmoid, leaky, parameter-based, and different versions of ReLU such as ELU, Swish, etc.) may also be used. For example, image processing may use one type of activation function, and natural language processing may use another. In an exemplary embodiment, the type of activation function used for a given layer may distinguish one layer from another. For example, (e.g., in connection with the example discussed in FIGS. 2 and 5 for upconverting an image), a ReLU activation function may be used in layers 1 through n−1 (where n is the number of layers), and a sigmoid activation function may be used in the nth (e.g., last) layer.

[0105] This process generates a transformed matrix of activations 425, which is X n+1 It is expressed as: The process shown in FIG. 5 may be repeated or otherwise performed a predetermined number of times or for a predetermined number of layers (e.g., four). In this manner, the activation matrix is ​​modified from the initial activation matrix 225 to the final activation matrix 235 by applying various trained L and R matrices. In an exemplary embodiment, the number of layers may vary between two and twelve or between three and eight. In an exemplary embodiment, more layers may be used, with the understanding that additional layers may degrade performance. Thus, the number of layers may be selected based on the needs of a particular application and a balance between the resulting quality of the generated converted image and the performance of the upconversion process. Additional layers may be added as hardware becomes faster (or performance becomes less of a controlling factor). In an exemplary embodiment, the number of layers may be dynamically controlled by the neural network 112, the video game application 108 (or another application, such as an operating system, that is handling the conversion process). For example, the system may measure the amount of time it takes to process an image and add or remove layers based on such measurements. (For example, if the conversion process takes too long for real-time processing, a pre-trained network with one or more fewer layers may be used.) Such techniques can be useful to accommodate different types of hardware resources used by a given computing device.

[0106] The following pseudocode may illustrate the 16x16 to 16x16 matrix multiplication (in this example, multiplying matrix "Left" by matrix "Right") process shown in FIG.

[0107] [Table 3]

[0108] where Result[i][j] is the coefficient at the ith column and jth row (initialized to 0 before the loop).

[0109] The use of a separable block transform (SBT) at 410 in an exemplary embodiment may be considered an alternative to using a fully connected / linear layer. A linear layer (e.g., a fully connected layer) is a matrix multiplication of an unstructured vector of input activations given an unstructured vector of output activations. For example, a 256 × 256 linear layer can be represented by a 256 × 256 matrix of independent weights and is applicable to 256 unstructured independent inputs. A potential drawback of this number of coefficients in a layer is that it may have too many coefficients (e.g., degrees of freedom) to train or compute at runtime (e.g., to provide real-time image processing). Therefore, an exemplary embodiment may advantageously replace such a linear layer with, for example, a “low-rank approximation.” One example is SBT. In an exemplary embodiment, an SBT layer may be represented as a sum of L×R products (as shown above) with the 256 inputs and outputs structured into a 16 × 16 matrix. As noted above, a generalized version of this may be expressed as follows:

[0110]

number

[0111] SBT layers may be used to generate special case SBTs that are similar or equivalent to linear layers, specifically:

[0112]

number

[0113] L i,j n The matrix is ​​the coefficient l for each of the coordinates i and j. i,j is set to 1 and all other coefficients are zero It is set to a special format that is set to l i,j = 1 and other coefficients are zero, L n X n The product is the matrix X nThe result is to extract the i-th line of X and reposition it to the j-th line while setting the remainder to zero. n+1 The jth result of Each of the lines and therefore X n is a general linear combination of the coefficients of Stated differently, X n+1 All 256 output values ​​in the matrix are n It is a linear combination of the 256 input values ​​of the matrix, which is equivalent to a linear layer with 256 × 256 coefficients. Thus, this configuration n In a matrix) 16 x 16 x 16 x 16 = 256 x 256 With this in mind, in situations where a linear layer is used (eg, it can be used as a surrogate), separable block transform techniques may be applied.

[0114] Compared to linear layers, SBT may offer one or more of the following advantages: In an exemplary embodiment, the SBT may be gradually pruned by removing individual LXR terms (e.g., those that contribute least to the quality of the results). Each removed LXR term may reduce the complexity of training and run-time calculations, the total number of weights stored and transmitted, and the residual learning cost.

[0115] In one exemplary embodiment, a 16x16 SBT can be trained initially with fewer than 256 LXR terms, which may reduce the number of weights to learn and the number of training and run-time operations.

[0116] In one exemplary embodiment, for a 16x16 SBT, the sum of less than 8 LXR terms requires fewer multiplications than a linear layer. For reference, a 256x256 linear layer (and therefore a multiplication of a 256x256 matrix with a vector of size 256) requires 256x256=2 16 In contrast, a single SBT requires two 16x16 matrix multiplications, hence 2x16x16x16=2 13Therefore, the sum of k LXR terms is multiplied by k*2 13 Therefore, k<2 3 (e.g. 8) is more efficient than a linear layer. There will be fewer things to do.

[0117] Benefits of SBT compared to linear layers may include allowing a reduction in the number of weights (e.g., in some weight reuse scheme). It is recognized that a reduction in the number of weights may affect (e.g., perhaps significantly affect) performance because it may reduce memory traffic for handling the weights. This allows more space in memory to be devoted to activations. Memory pressure, e.g., in the form of external memory bandwidth or internal memory size, may also be alleviated (e.g., reduced).

[0118] In one exemplary embodiment, for a 16x16 SBT, the weights (and therefore memory and training time) on the sum of LXR terms less than 128 are less than in a linear layer. For reference, a 256x256 linear layer has 2 16 weighted, while a single 16×16SBT term has a weight of 2×256=2 9 The weight of 7 Adding =128 will make the weights equal.

[0119] In an exemplary embodiment, SBT may be used to replace larger linear layers (e.g., 1024 to 1024, such as those used in natural language processing) with 32x32 SBT layers, which allows for a smaller number of weights while maintaining an acceptable level of quality. Thus, technical implementations of the SBT techniques discussed herein may be used in a variety of different applications and scenarios to achieve increased efficiency with little or no (e.g., perceived) loss of quality of the transformed data.

[0120] In an exemplary embodiment, the sum size learned by trial and error and / or global pruning can vary for each layer. In an exemplary embodiment, smaller versions of the SBT network can be trained through distillation from a trained larger version of the SBT network.

[0121] Description of Figure 6 FIG. 6 is a flowchart illustrating an expanded view of the block reorganization portion of the machine-learned upconversion process of FIG. 2. Once the 16×16 activation matrix 235 is generated by running it through the neural network 112, it is then reconverted back into a multi-channel form. Specifically, each row (or more specifically, the first 12 rows, since the last 4 are all zeroed out) of the activation matrix 235 is reorganized into a corresponding block of one output channel. Thus, as shown in FIG. 6, the first row of the activation matrix 235 is converted back into the first block 602a (e.g., the red value of the top-left sub-pixel) of the 1080p output data 245. The second row of the activation matrix 235 is converted back into the second block 602b (e.g., the green value of the top-left sub-pixel of that same channel) of the 1080 output data 245, and so on. Thus, all 12 blocks (4 subpixel channels per block * 3 channels per color value) of the corresponding 12 rows of activation matrix 235 create 12 output channels of 1080p output data 245.

[0122] Description of Figure 7 Figure 7 is a flow chart showing an expanded view of the reorganizing 1080p output data into a converted image portion of the machine-learned upconversion process of Figure 2. In step 710, the 1080p output data 245 (e.g., 12 output channels in 4x4 blocks) are combined into a single 8x8 pixel block 712.

[0123] 7 shows an example of how values ​​from a block (e.g., showing highlighted value 713 from block 602) may be used to generate a corresponding pixel value 714 (also highlighted) in pixel block 712. This involves combining color values ​​to create each pixel. Thus, the red, green, and blue values ​​of 713 from each of the red (e.g., from 602a), green (e.g., from 602b), and blue blocks 602 are used to generate the RGB value for pixel 714 in pixel block 712. The remaining 63 pixels in the 8x8 block are generated in a similar manner. The resulting 8x8 pixel block 712 is then positioned within the overall 1080p image 255.

[0124] This process of assembling 8x8 pixel blocks is repeated (e.g., in parallel) for each of the 1080p output data 245 generated for the single (original) 540p image. At 720, a 1080p image 255 is assembled from multiple 8x8 pixel blocks 712. Each of the 8x8 pixel blocks is positioned within the overall image (e.g., based on the order in which the source images were processed). In this manner, output images are constructed in a similar manner when source images are processed from left to right and top to bottom. Alternatively, in an exemplary embodiment, position data for each pixel block may be stored as metadata for each created input channel 215 when it is originally created, for example, to determine where the 8x8 pixel blocks should be positioned.

[0125] Once the 1080p image 255 is created, it may then be output at 260 or otherwise stored (eg, in a frame buffer) and ultimately displayed on the display device 116 .

[0126] Description of Figures 8A to 8B 8A-8B show an example image 802 that is 128x128 pixels. Image 802 has been applied to a neural network 803 that has been trained according to the techniques discussed herein (e.g., in connection with FIG. 10). After applying image 802 to neural network 803, an enlarged image 804 is produced. Image 804 is a version of image 802 that has been enlarged to 256x256 pixels.

[0127] Figure 8B includes a version of the image from Figure 8A that has been "zoomed" to create a side-by-side 512 x 512 pixel version. As shown in Figure 8B, image 822, which is a zoomed version of image 802, includes artifacts that are not seen in image 824, which is a zoomed version of image 804. It will be appreciated that the images shown in Figures 8A and 8B are shown by way of example.

[0128] Description of Figures 8C to 8E Figure 8C shows an example block diagram of a single "block tower" according to an example embodiment. Figures 8D and 8E are example block diagrams showing how several block towers may be used according to an example embodiment.

[0129] Figure 8C shows a block diagram that corresponds in some respects to the example discussed in connection with Figures 2-7. Specifically, a block of pixels 830 is selected from a source image 832. For block 830, a 16x16 activation matrix 836 is prepared at 834 (e.g., as described in connection with Figures 3 and 4). Activation matrix 836 is then run through an SBT network 838 (e.g., as shown in Figure 5) to create an output matrix 840. An output pixel block 844 is then created at 842 (e.g., as shown in Figures 6 and 7) and then placed into a converted image 846.

[0130] It is recognized that using a larger number of channels and / or L&R matrices (e.g., 32x32 or 64x64) may be beneficial, as this may provide more expressiveness during processing. However, a drawback of this approach is that such matrices may not fit into local "fast" memory (e.g., registers), thereby necessitating the use of slower DRAM during processing. While larger sizes of fast memory are possible in the future, the underlying problem of not having "fast enough" memory may still remain.

[0131] In one exemplary embodiment, two 16x16 SBT towers with corresponding activation matrices for 16 channels (e.g., L&R matrices) may be used. Such an implementation may address (at least in part) the need to have increasingly large amounts of local high-speed memory, while still benefiting from the increased expressive power (higher degrees of freedom) that can be provided by using an increased number of channels (e.g., 32 or 64). In such cases, the SBTs may be processed sequentially or in parallel. In such an implementation, a given activation matrix may be run through multiple different SBTs, with the outputs combined or used together in one of several different ways.

[0132] FIG. 8D shows a block diagram of an example summation for using several SBTs. As in the example of FIG. 8C, an activation matrix 836 is created from a block using the source image. However, in this example, that activation matrix is ​​applied to multiple different SBT networks. Specifically, activation matrix 836 is applied to SBTs 852A, 852B, 852C, and 852D. In other words, the same activation matrix (derived from the same underlying pixel block) may be processed by different SBTs (e.g., an L&R matrix). Such processing may occur sequentially, in parallel, or in some combination thereof (e.g., two at a time). Each SBT processes activation matrix 836 differently to create (possibly) four different output matrices 854A, 854B, 854C, and 854D. These four outputs are then summed term-by-term to create a final (e.g., 16×16) output matrix, which is then processed as discussed in connection with FIG. 8C.

[0133] FIG. 8E is a block diagram of an alternative example for using several SBTs. This example is similar to that shown in FIG. 8D, except that instead of summing the results from several SBTs, the resulting outputs may be stacked or merged together into a larger matrix at 860. This type of implementation may be useful, for example, for dealing with larger output blocks 862 in the output image 864, which may of course benefit from a larger number of activations in the output activation matrix.

[0134] Such techniques may be similar to or equivalent to, for example, channel grouping / grouped convolution as used in various CNN architectures (e.g., AlexNet, MobileNet, etc.).

[0135] Description of Figure 9 Figure 9 is a block diagram including an example training computer system 900 according to an example embodiment. Training computer system 900 is an example of computer system 1300 shown in Figure 13. In an example embodiment, computer system 900 and computer system 100 may be the same system (e.g., a system used to play a video game may also be configured to train a neural network for that video game).

[0136] System 900 includes a dataset preparation module 902 that is used to prepare images (e.g., 1080p images) for input from a training set database 906. Once the images are prepared, they are then used to train a neural network via a neural network trainer module 904 (e.g., to determine the L&R coefficients that comprise each layer of the sum of L&R transforms discussed herein). Neural network trainer module 904 generates one or more trained neural networks, which are stored in a database 908. The trained neural networks 908 may then be communicated to various gaming devices 1, 2, 3, 4, 5, etc. (each of which may be an example of gaming device 100) via a network 912 (e.g., the Internet) or via physical media (such as a game cartridge). In one exemplary embodiment, one or more trained neural networks may be distributed along with a game acquired by a user. For example, a user may download a game from an online store or the like, and one of the game's components may be a neural network for processing images generated by the game. Similarly, a game provided on a cartridge or other physical medium may include one or more neural networks that a user can use to transform images generated by the game. In one example, multiple neural networks may be provided with the same instance of a game (e.g., separate downloads or specific physical medium instances) to allow the game to be output on different types of displays (e.g., 1080p in one case, 1440p in another case, 4k in another case, etc.).

[0137] As discussed herein, different types of neural networks may be generated and distributed to various gaming devices. Thus, for example, gaming device 1 may receive and use a different neural network than the neural network received and used by gaming devices 2, 3, 4, and 5. In an exemplary embodiment, each game (or more generally, each application) may have a corresponding neural network (or multiple neural networks) generated (e.g., by system 900) for that game. Thus, for example, a gaming device may store multiple different neural networks and use such different networks based on the game (or type of game) being played on the corresponding gaming device. In an exemplary embodiment, multiple games may share or use the same neural network. For example, one neural network may be generated for first-person shooter games, another neural network may be generated for strategy games, etc. Thus, games may be grouped based on their "type." Such classification may be based on the genre of the game, or on another criteria such as the type of rendering engine the game uses to generate the images therein.

[0138] In one example embodiment, a game engine (or other service providing conversion functionality to a game engine) may dynamically decide to select one of various neural networks depending on the remaining time available to "prepare the current video frame." If a frame is rendered quickly, it may have more time to upscale with a higher-quality, slower neural network (e.g., one with additional layers), but if the frame uses up more of the typically available 16 ms (both for rendering the frame and subsequently upscaling the image at 60 frames per second), the engine may select a faster neural network (e.g., one with fewer layers), even though it may not provide as high image quality as a slower one. Such a decision may be made through a "test" phase of a video game application program (e.g., where the game engine generates a number of example images) and / or during normal gameplay.

[0139] Returning to FIG. 9 , training dataset 906 includes multiple datasets used as "targets." Thus, if a neural network is generated to convert 540p images to 1080p images, this may include different 1080p images that will be used to generate the neural network. In an example embodiment, the type of 1080p image may be selected according to a particular use case. In the case of a video game, the image may be a 1080p image natively generated by the game engine. In an example embodiment, the image may be from the same game engine or game in which the neural network is used. Thus, for example, Game A may include a game engine that has the ability to generate 1080p images. This may be beneficial because another version of Game A may be generated that generates game images at 540p. This may be because, for example, another version of Game A is created for lower-power hardware, such as a mobile device. In this way, the game engine for Game A may be used to populate a training dataset with 1080p images, which are then used to train a neural network that can be used in conjunction with other versions of Game A (e.g., allowing the neural network to output 1080p images even if that version was not originally designed for such images).

[0140] In one exemplary embodiment, the target image (e.g., a 1080p image if the network is being trained to upconvert from 540p to 1080) should be of high visual quality. Such an image may be prepared in advance and would not need to be rendered in "real time" (e.g., at 30 or 60 frames per second). Such an image may be rendered as crisp and clean and with high anti-aliasing settings. Advantageously, the image may be generated from the same game or game engine for which the trained network will be used. In such a scenario, the statistics of the training data may more closely match the statistics of the runtime data, and thus the generated neural network may be better optimized for such a game.

[0141] In an exemplary embodiment, a default or "generic" selection of images may be used. Such an implementation may provide a good cross-section across multiple games. For such an implementation, target images may be selected that are relatively good or high quality and have a relatively good level of diversity and sharpness (e.g., relatively free of visible aliasing). This type of approach may allow the full spectrum of available spatial frequencies to be used.

[0142] In an exemplary embodiment, artificially generated images may be used, in which case such images are rendered in pairs of low-resolution and high-resolution images. In an exemplary embodiment, different types of images (e.g., pixel art) may be selected for scaling (e.g., where such images may be disadvantaged by the lack of available high-resolution images and may not look visually good when scaled through the use of a general-purpose neural network).

[0143] In one exemplary embodiment, the training computer system may be implemented in a cloud-based computing system.

[0144] Description of Figure 10 FIG. 10 is a flowchart illustrating a process that may be implemented on the system shown in FIG. 9 to train a neural network that may be used in connection with certain exemplary embodiments, including the process shown in FIG.

[0145] A number of target or training images are selected from the training dataset 906. If a neural network is trained to upconvert to 1080p, the images may be a collection of 1000 1080p images.

[0146] At 1002, each of the images in this collection is passed to the dataset preparation module 902 to prepare a training dataset used to train the neural network. This has two sub-processes. The first is preparing the 1080p images into 1080p output data 1006. This aspect is discussed in FIG. 11. The second is preparing the 540p images (or other images used as source images) into 540p input data 1004. This aspect is discussed in FIG. 12. The processes discussed in FIGS. 11 and 12 may be repeated or otherwise performed for each image used in the training dataset. In an exemplary embodiment, the images may be streamed (e.g., the preparation process may proceed simultaneously with the training process). In an exemplary embodiment, the image preparation may be batch processed (e.g., 256 or cropped sub-portions of such images may be prepared in a training batch before being used as data for one step in the neural network training process).

[0147] Next, at 1008, the neural network is trained using the 540p input data 1004 until the training results converge at 1010 to a coverage that is sufficiently close to the 1080p output data 1006. In other words, until the set of coefficients (e.g., L&R) converges from the initial 540p input data to an acceptable approximation of the 1080p output data. The training process is repeated until this convergence is reached (e.g., within an error threshold or because the error value has not decreased for more than a threshold number of iterations).

[0148] Once converged, the trained neural network weights (e.g., the coefficients of the L&R matrix, sometimes referred to herein as the trained neural network) 910 may be stored in a database within the system 900 and / or communicated to other computer systems (e.g., game devices 1, 2, 3, 4, 5, etc.).

[0149] In an exemplary embodiment, techniques associated with SBT networks may allow a favorable environment for pruning because each individual sum element (e.g., LXR) can be removed without interfering with the rest of the architecture, and even if a connection remains, other connections do not directly depend on this particular term. In other words, each The LXR term in can be thought of as a single "branch" of the architecture that can be removed without disturbing the rest of the network. This type of approach can be advantageous because each channel is typically used as an input to the next layer downstream, and removing a channel in the rest of the network can have negative consequences in terms of quality and / or performance.

[0150] In an example embodiment, the determination of which LXR terms (e.g., each SBT term) to remove (e.g., prune) may be based on calculating the global loss with and without each LXR term (e.g., the result of calculating L*X*R as an individual term or part of a sum of LXR products), and then removing the terms that have the smallest impact on the global loss. In this way, terms below a certain threshold may be removed, or the bottom x% of terms may be removed (e.g., 1% or 5% may be removed), and then the process can start again until a given size or error target is reached.

[0151] In one exemplary embodiment, the SBT network may be pruned by calculating or otherwise determining the slope of the loss for each SBT term and removing the SBT terms with the lowest slope (or the terms with the lowest percentages).

[0152] Description of Figure 11 FIG. 11 is a flow chart showing an expanded view of how 1080p image data is prepared as part of the neural network training process shown in FIG.

[0153] Each 1080p image 1000 is cut into 8x8 pixel blocks at 1110. Next, at 1120, each pixel block (1122) is selected. The pixel blocks are then separated at 1130. FIG. 11 illustrates the separation of pixel blocks 1122 into separate input channels for step 1130. As shown in FIG. 11, correspondingly numbered pixel values ​​in pixel block 1122 are assigned to corresponding input channels. Each channel contains three separate input channels per RGB value of the source pixel. Thus, 12 input channels are created (e.g., 1132, 1134, 1136, 1138, etc., each with RGB) to be used as 1080p output data. This process is repeated for each 1080p image to create multiple 1080p output data that are used during the neural network training process (e.g., to determine when the neural network has converged).

[0154] Description of Figure 12 FIG. 12 is a flow chart showing an expanded view of how 540p input data is prepared as part of the neural network training process shown in FIG.

[0155] The 540p input data 1004 is prepared from the 1080p output data 1006 that is generated as shown in Figure 11. Specifically, at 1210, one of the output channels from the 1080p output data 1006 is used to create a single 540p image 1212.

[0156] From the created image, the process is similar in some respects to that shown in Figure 3, in which an image to be upconverted 540p is prepared. Specifically, at 1220, each 4x4 pixel block (1214) in the created 540p image (which may correspond, for example, to the color channels 1132 in Figure 11) is selected.

[0157] At 1230, context data is then added around the 4x4 pixel block to create an 8x8 context block 1232a. The context data may be derived in a manner similar to that described above in connection with FIG. 2. At 1240, the context block 1232a is Context block 1232b (which may be the same as a but with altered activation indexing) is split into four separate input channels. Each input channel contains three channels, one for each RGB value of the pixels contained in the channel. As shown in FIG. 12, the input channels are created such that a 1 in 1232b is mapped to channel 1242, a 2 is mapped to channel 1244, and so on. The resulting 12 input channels constitute 540p input data 1004 (e.g., a 16×16 matrix) that is used to train the neural network during the training process discussed in connection with FIG. 10.

[0158] Using the techniques described above, the low-resolution input may be generated by downsampling the high-resolution input through point sampling (e.g., nearest neighbor), although other example implementations may use other downsampling methods.

[0159] In an exemplary embodiment, images rendered at high speeds (e.g., 60 fps) by a real-time game engine may naturally be similar to images resulting from point-sampled downsampling, because each pixel value is calculated independently of its neighboring pixels. Therefore, training a neural network using point-sampled data may be more suitable for scaling up game engine output, which may help the game engine in an exemplary embodiment run faster because it may skip the additional and expensive anti-aliasing effort during the traditional rendering stage. Rather, such anti-aliasing may be handled more efficiently by the exemplary neural network techniques discussed herein.

[0160] Point sampling as part of the downsampling for the training process can provide an additional benefit: a critically sampled signal is a discrete signal that arises from a continuous signal, where the frequency reaches the maximum allowed frequency according to the Shannon-Nyquist sampling theorem (i.e., the signal frequency must not exceed half the sampling frequency f), while still being able to perfectly reconstruct the continuous signal from the discrete signal without any loss.

[0161] For high-resolution images, if such an image is critically sampled along spatial frequency, then computing the spectrum of the entire signal (e.g., using a discrete Fourier transform) uses the entire allowed spectrum (e.g., from 0 to f / 2). If lower-resolution input data is prepared, conventional sampling theorem may lead to first removing the high frequencies of the spectrum (e.g., anything between f / 4 and f / 2) using a low-pass filter, followed by a factor of two reduction using point sampling. The resulting image will then respect the sampling theorem by having frequencies below half the (new) signal spatial sampling frequency f' (which is f / 2).

[0162] If a local spectrum is then calculated (e.g., for a 4x4 or 8x8 pixel block), the useful frequencies of the spectrum may be located primarily in the low part of the spectrum (between 0 and f / 4) or in the high part (between f / 4 and f / 2). If point sampling is used without first using a low-pass filter, the high frequencies (between f / 4 and f / 2) will not be removed, but rather may be "folded" into the low part of the spectrum (between 0 and f / 4, which in the newly downsampled signal becomes between 0 and f' / 2).

[0163] Neural networks can then use the context information to reconstruct the signal in a non-linear (e.g., learned) way. For example, they can determine whether the spectrum comes from the actual low frequencies and therefore the upsampled signal. It learns whether it should be reconstructed as a low frequency or whether it comes from the higher part of the spectrum and therefore as a high frequency in the upsampled signal.

[0164] Thus, in some cases, downsampling using point sampling during the training phase can pack up to twice as much information into the same storage space as compared to traditional sampling techniques. In some cases, high-resolution images used during training may be prepared according to techniques similar to those discussed above (e.g., using frequencies above the sampling limit), assuming the images are not later inappropriately resampled through the display process.

[0165] Additional Exemplary Embodiments The processing discussed above generally relates to two-dimensional (e.g., image) data (e.g., signals). The techniques herein (e.g., using SBT) can also be applied to data or signals of other dimensions, such as 1D (e.g., speech recognition, anomaly detection in time series, etc.) and 3D (e.g., video, 3D texture) signals. The techniques can also be applied in other types of 2D domains, such as image classification, object detection and image segmentation, face recognition, style transfer, pose estimation, etc.

[0166] 2 and 9 relate to upconverting an image from 540p to 1080. However, the techniques discussed herein may be used in other scenarios, including 1) converting to a different resolution than those discussed (e.g., 480p to 720p or 1080p, and variations thereof), 2) downconverting an image to a different resolution, 3) converting an image without changing the resolution, and 4) images with other values ​​for how the image is represented (e.g., grayscale).

[0167] In an example embodiment, the techniques herein may be applied to processing images (e.g., in real time and / or during the runtime of an application / video game) to provide anti-aliasing capabilities. In such an example, the sizes of the before and after images remain the same, but anti-aliasing is applied to the final image. Training for such a process may proceed by obtaining relatively low-quality images (e.g., rendered without anti-aliasing) and ones rendered with high-quality anti-aliasing (or a level of anti-aliasing desired for a given application or use) to train a neural network (e.g., L&R as discussed above).

[0168] Other examples of fixed resolution applications (e.g., converting an image from x resolution to x resolution) may include denoising (e.g., in connection with the ray tracing process used by a rendering engine in a game engine). Another example application of the techniques herein may include deconvolution, for example, in the context of image deblurring.

[0169] During runtime, the source image is used to prepare input channels in a manner similar to that shown in FIG. 3. Specifically, each image is divided into blocks (e.g., 4×4), and context data is added to those blocks to create 8×8 context blocks. The subsequent context blocks are then split into four input channels with three channel colors per channel, creating 12 input channels. These 12 input channels are then reorganized into a 16×16 matrix of activations in a manner similar to that shown in FIG. 4. The matrix of activations is then run through a neural network, where a separable block transform is performed using the L and R matrices developed through the training discussed above.

[0170] Once the activation matrix has been transformed, the first three (or any three that may be based on training) output channels (e.g., RGB values ​​corresponding to "1" pixels) are reorganized into their respective blocks and combined into a single 4x4 block. This process is repeated for each of the original 4x4 blocks obtained from the source image. The transformed blocks are combined together, thereby creating a resulting image, which can then be output.

[0171] In an example embodiment, a classification process (e.g., finding / identifying objects in an image) may be used in combination with the SBT techniques discussed herein. For example, a given image may be divided into 4x4 pixel blocks, and a sliding 3x3 block kernel transform may be applied to all of the image blocks. In some examples, the kernel may have other sizes (e.g., the kernel may have other sizes, such as 2x2, or be separable into 3x1 and then 1x3).

[0172] In this example, the eight blocks surrounding a given block (e.g., 3x3 surrounding blocks) and the block itself are processed with SBT, and the results are summed into a single target block (e.g., corresponding to the location of the selected block). Thus, the 16x16 block values ​​are summed term by term.

[0173] For blocks on the edge of the image, blocks outside the image may be ignored. In an example embodiment, one or more block convolutional layers may be modified into various types of reduction layers, such as using max or average pooling, or downsampling with strides or other similar techniques.

[0174] In an example embodiment, the neural network may include one or more normalization layers. Such layers may be generated by using batch normalization, weight normalization, layer normalization, group normalization, instance normalization, batch instance normalization, etc.

[0175] In an example embodiment, layer fusion can be implemented between consecutive block convolutional layers to further reduce pressure on memory bandwidth (e.g., DRAM).

[0176] In an example embodiment, residual connections (e.g., skip connections) can be added between SBT layers to facilitate training of deeper models.

[0177] For strided implementations, the output image may have twice as many fewer blocks in the horizontal and vertical dimensions. Thus, if block convolutional layers are alternated (e.g., several times) with block strided layers, the final image may end up with only one block of 16x16 activations. In one exemplary embodiment, the output neuron counts can be matched to multiple classes (e.g., for classification applications) and the final block can be used as the input for a traditional fully connected layer.

[0178] For a 16x16 matrix, if the number of classes is 16 or less, the output classes may be placed on the diagonal coefficients of the matrix. This allows SBT training to learn the equivalent of a fully connected layer in the L and / or R matrices (even with a single LxR element with no sum). More generally, for a large number of classes greater than 16 but less than or equal to 256, an SBT with up to 256 sum elements may be used (which would be equivalent to a fully connected network of 256 neurons). For a large number of classes less than 256, a sum of fewer than 256 LxR terms is likely to fit the problem well, and an optimal number of terms may be found. In an exemplary embodiment, finding the optimal number of terms may be achieved by pruning the LxR sum. In an exemplary embodiment, finding the optimal number of terms is achieved by singular value decomposition (or matrix spectral decomposition) of the trained fully connected layer to identify "significant" singular values ​​(e.g., zeros). One may determine the number of significant singular values ​​(those not close to B) and train a corresponding number of LXR terms (e.g., 2 LXR terms for the 32 significant singular values).

[0179] For pooling implementations, each group of 2x2 blocks is combined into a single block by calculating the average (or maximum) of the corresponding terms. Thus, in an exemplary embodiment, block convolutional layers may be alternated (e.g., several times) with block pooling layers, and the final image may end up with only one block of 16x16 activations. As with strided implementations, this final 16x16 activations may be used as the input of a traditional fully connected layer, with output neuron counts matching the desired number of classes (e.g., for classification applications).

[0180] It is recognized that software implementation speed and / or dedicated acceleration hardware cost may be related to activation precision. In other words, FP32 is more costly than FP16, which is more costly than INT8. In an example embodiment, using INT8 may provide an attractive sweet spot in terms of speed / quality and / or cost / quality tradeoffs.

[0181] In some cases, low-resolution and high-resolution outputs from a game engine may be used for training purposes (e.g., instead of downsampling). However, such an approach may introduce inconsistencies and / or hinder training. Images generated in such a manner may mitigate these problems if the rendering engine generating the images is "resolution independent."

[0182] 2 and 9 are provided in the context of converting a 540p image to a 1080p image, it will be appreciated that the techniques discussed herein may be applied to converting other resolutions to new resolutions. For example, whenever 540p is mentioned herein, similar techniques may be applied to a 1080p source image. Also, whenever 1080p is mentioned in connection with a destination image, the techniques discussed herein may also be applied to a 4k image (e.g., 3840x2160).

[0183] In certain exemplary embodiments, the conversion techniques discussed herein may operate in a two-stage process. In one example, a first image (e.g., a 1080p image) may be converted to, for example, an 8k image. Such a process may include first converting the 1080p image to a 4k image and then converting the resulting 4k image to an 8k image according to the techniques discussed herein.

[0184] Description of Figure 13 FIG. 13 is a block diagram of an exemplary computing device 1300 (which may also be referred to as, e.g., a “computing device,” “computer system,” or “computing system”) according to some embodiments. In some embodiments, the computing device 1300 includes one or more of one or more processors 1302, one or more memory devices 1304, one or more network interface devices 1306, one or more display interfaces 1308, and one or more user input adaptors 1310. Additionally, in some embodiments, the computing device 1300 is connected to or includes one or more display devices 1312. Additionally, in some embodiments, the computing device 1300 is connected to or includes one or more input devices 1314. In some embodiments, the computing device 1300 may be connected to one or more external devices 1316. These elements (e.g., the processor 1302, the memory devices 1304, the network interface devices 1306, the display interfaces 1308, the user input adaptors 1310, the display devices) are connected to one or more external devices 1316, as described below. 1312 , input device(s) 1314 , external device(s) 1316 are hardware devices (e.g., electronic circuits or combinations of circuits) configured to perform various different functions for and / or in connection with the computing device 1300 .

[0185] In some embodiments, each or any of the processors 1302 is or includes, for example, a single-core or multi-core processor, a microprocessor (e.g., which may be referred to as a central processing unit or CPU), a digital signal processor (DSP), a microprocessor associated with a DSP core, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, or a system-on-chip (SOC) (e.g., an integrated circuit that includes a CPU, a GPU, and other hardware components such as a memory and / or memory controller (e.g., a northbridge), an I / O controller (e.g., a southbridge), a networking interface, etc.). In some embodiments, each or any of the processors 1302 uses an instruction set architecture such as x86 or advanced RISC machine (ARM). In some embodiments, each or any of the processors 1302 is or includes, for example, a graphics processing unit (GPU), which may be an electronic circuit designed to generate images, etc. One or more of the processors 1302 may also be referred to as a hardware processor, and in some examples, one or more of the processors 1302 may be used to form a processing system.

[0186] In some embodiments, each or any of the memory devices 1304 is or includes random access memory (RAM) (such as dynamic RAM (DRAM) or static RAM (SRAM)), flash memory (e.g., based on NAND or NOR technology), a hard disk, a magneto-optical medium, an optical medium, a cache memory, a register (e.g., holding instructions or data that may be executed by one or more of the processors 1302), or other types of devices that perform volatile or non-volatile storage of data and / or instructions (e.g., software executed on or by the processors 1302). The memory devices 1304 are examples of non-transitory computer-readable storage devices. Memory devices as discussed herein may include memory provided on the same “die” as the processor (e.g., internal to the die on which the processor is located) and memory provided external to the die that includes the processor. Examples of “on-die” memory may include cache and registers, while “off-die” or external memory may include DRAM. As discussed herein, on-die memory in the form of cache or registers may provide faster access at the expense of being more expensive to produce.

[0187] In some embodiments, each or any of the network interface devices 1306 includes one or more circuits (such as a baseband processor and / or a wired or wireless transceiver) implementing layer 1, layer 2, and / or higher layers for one or more wired communication technologies (such as Ethernet (IEEE 802.3)) and / or wireless communication technologies (such as Bluetooth, WiFi (e.g., IEEE 802.11), GSM, CDMA2000, UMTS, LTE, LTE-Advanced (LTE-A), and / or other short-range (e.g., Bluetooth Low Energy, RFID), medium-range, and / or long-range wireless communication technologies). A transceiver may include circuitry for a transmitter and a receiver. The transmitter and receiver may share a common housing and may share some or all of the circuitry within the housing for transmission and reception. In some embodiments, the transmitter and receiver of a transceiver may not share any common circuitry and / or may be in the same or separate housings.

[0188] In some embodiments, each or any of the display interfaces 1308 is or includes one or more circuits that receive data from the processor(s) 1302 (e.g., via a discrete GPU, an integrated GPU, a CPU performing graphical processing, etc.) used to generate corresponding image data based on the received data, and / or output the generated image data to a display device 1312 that displays the image data (e.g., a high-definition multimedia interface (HDMI), a display port interface, a video graphics array (VGA) interface, a digital video interface (DVI), etc.). Alternatively or additionally, in some embodiments, each or any of the display interfaces 1308 is or includes, for example, a video card, a video adapter, or a graphics processing unit (GPU). In other words, each or any of the display interfaces 1308 may include a processor therein that is used to generate image data. The generation of such images may be performed in conjunction with processing performed by one or more of the processors 1302.

[0189] In some embodiments, each or any of user input adaptors 1310 is or includes one or more circuits that receive and process user input data from one or more user input devices (1314) included in, attached to, or otherwise in communication with computing device 1300, and output data to processor 1302 based on the received input data. Alternatively or additionally, in some embodiments, each or any of user input adaptors 1310 is or includes, for example, a PS / 2 interface, a USB interface, a touch screen controller, etc. and / or user input adaptor 1310 facilitates input from user input device(s) 1314.

[0190] In some embodiments, display device 1312 may be a liquid crystal display (LCD) display, a light emitting diode (LED) display, or other types of display devices. In embodiments in which display device 1312 is a component of computing device 1300 (e.g., the computing device and display device are included in a single housing), display device 1312 may be a touchscreen display or a non-touchscreen display. In embodiments in which display device 1312 is connected to computing device 1300 (e.g., is external to computing device 1300 and communicates with computing device 1300 via wired and / or wireless communication techniques), display device 1312 may be, for example, an external monitor, projector, television, display screen, etc.

[0191] In some embodiments, each or any of the input devices 1314 is or includes a mechanical and / or electronic device that generates a signal that is provided to the user input adapter 1310 in response to a physical phenomenon. Examples of input devices 1314 include, for example, a keyboard, a mouse, a trackpad, a touchscreen, a button, a joystick, and a sensor (e.g., an accelerometer, a gyro sensor, a temperature sensor, etc.). In some examples, one or more of the input devices 1314 generate a signal that is provided in response to a user providing an input, for example, by pressing a button or activating a joystick. In other examples, one or more of the input devices generate a signal based on a sensed physical quantity (e.g., force, temperature, etc.). In some embodiments, each or any of the input devices 1314 is a component of a computing device (e.g., a button provided on a housing that includes the processor 1302, the memory device 1304, the network interface device 1306, the display interface 1308, the user input adapter 1310, etc.).

[0192] In some embodiments, each or any of the external devices 1316 includes an additional computing device (e.g., another instance of computing device 1300) that communicates with computing device 1300. Examples include a server computer, a client computer system, a mobile computing device, a cloud computing device, a External device 1316 may include a computer system, a computing node, an Internet of Things (IoT) device, etc., all of which may be in communication with computing device 1300. In general, external device 1316 may include any device that communicates (e.g., electronically) with computing device 1300. By way of example, computing device 1300 may be a gaming device that communicates over the internet with a server computer system, which is an example of external device 1316. Conversely, computing device 1300 may be a server computer system that communicates with a gaming device, which is an example external device 1316.

[0193] In various embodiments, computing device 1300 includes one, two, three, four, or more of each or any of the above-mentioned elements (e.g., processor 1302, memory device 1304, network interface device 1306, display interface 1308, user input adapter 1310, display device 1312, input device 1314). Alternatively or additionally, in some embodiments, computing device 1300 includes one or more of a processing system including processor 1302, a memory or storage system including memory device 1304, and a network interface system including network interface device 1306.

[0194] The computing device 1300 may be arranged in many different ways in various embodiments. By way of example only, the computing device 1300 may be arranged such that the processor 1302 includes a multi- (or single-) core processor, a first network interface device (e.g., implementing WiFi, Bluetooth, NFC, etc.), a second network interface device implementing one or more cellular communication technologies (e.g., 3G, 4G LTE, CDMA, etc.), and memory or storage devices (e.g., RAM, flash memory, or hard disk). The processor, first network interface device, second network interface device, and memory device may be integrated as part of the same SOC (e.g., one integrated circuit chip). As another example, computing device 1300 may be arranged such that processor 1302 includes two, three, four, five, or more multi-core processors, network interface device 1306 includes a first network interface device implementing Ethernet and a second network interface device implementing WiFi and / or Bluetooth, and memory device 1304 includes RAM and flash memory or a hard disk. As another example, computing device 1300 may include an SoC having one or more processors 1302, multiple network interface devices 1306, memory device 1304 including system memory and memory for application programs and other software, a display interface 13068 configured to output video signals, a display device 1312 integrated into the housing and overlaid with a touch screen input device 1314 along with those mentioned, and multiple input devices 1314 such as one or more joysticks, one or more buttons, and one or more sensors.

[0195] As previously noted, whenever a software module or software process is described herein as performing any action, that action is actually performed by the underlying hardware elements in accordance with the instructions comprising the software module. Consistent with the foregoing, in various embodiments, each or any combination of the game device 100, game engine 110, neural network 112, input device 114, video game application 108, neural network trainer 904, dataset preparation module 902, etc., is implemented using the example computing device 1300 of FIG. 13. However, for clarity, the remainder of this paragraph will refer to each individually as a "component." In such embodiments, the following applies to each component: (a) the elements of computing device 1300 (i.e., one or more processors 1302, 1303, 1304, 1306, 1308, 1309, 1310, 1311, 1312, 1313, 1314, 1315, 1316, 1317, 1318, 1319, 1320, 1321, 1322, 1323, 1324, 1325, 1326, 1327, 1328, 1329, 1330, 1331, 1332, 1333, 1334, 1335, 1336, 1337, 1338, 1339, 1340, 1341, 1342, 1343, 1344, 1345, 1346, 1347, 1348, 1349, 1350, 1351, 1352, 1353, 1354, 1355, 1356, 13 , one or more memory devices 1304, one or more network interface devices 1306, one or more display interfaces 1308, and one or more user input adapters 1310), or one or more display devices 1312, one or more input devices 1314, and / or external devices 1316, with or without an appropriate combination or subset of the foregoing, configured, adapted, and / or programmed to implement each or any combination of the features described herein as being performed by the component and / or by any software modules described herein as being included within the component; (b) alternatively or in addition, to the extent one or more software modules are described herein as being present within a component, some implementations may In an embodiment, such software modules (as well as any data described herein as being handled and / or used by the software modules) are stored in memory device 1304 (e.g., in various embodiments, in a volatile memory device such as RAM or instruction registers and / or in a non-volatile memory device such as flash memory or a hard disk), and all actions described herein as being performed by the software modules are performed by processor 1302 in conjunction with other elements in and / or connected to computing device 1300 (e.g., network interface device 1306, display interface 1308, user input adapter 1310, display device 1312, input device 1314, and / or external device 1316), as appropriate;(c) Alternatively or additionally, to the extent that a component is described herein as processing and / or otherwise manipulating data, in some embodiments, such data may be stored in memory device 1304 (e.g., in some embodiments, in a volatile memory device such as RAM and / or in a non-volatile memory device such as flash memory or a hard disk) and / or processed by processor 1302, as appropriate, in conjunction with other elements in and / or connected to computing device 1300 (e.g., network interface device 1306, display interface 1308, user input adapter 1310, display device 512, input device 1314, and / or external device 1316). (d) alternatively or in addition, in some embodiments, the memory device 1302 stores instructions that, when executed by the processor 1302, cause the processor 1302, together with other elements in and / or connected to the computing device 1300, as appropriate (e.g., the memory device 1304, the network interface device 1306, the display interface 1308, the user input adapter 1310, the display device 1312, the input device 1314, and / or the external device 1316), to perform each or any combination of the acts described herein as being performed by any software modules described herein as being included in and / or within the components;

[0196] The above-described hardware configuration shown in FIG. 13 is provided as an example, and the subject matter described herein can be utilized with a variety of different hardware architectures and elements. For example, in many of the diagrams in this document, individual function / act blocks are shown. In various embodiments, the functionality of these blocks can be realized (a) using individual hardware circuits, (b) using application-specific integrated circuits (ASICs) specifically configured to perform the described functions / acts, (c) using one or more digital signal processors (DSPs) specifically configured to perform the described functions / acts, (d) using the hardware configuration described above with reference to FIG. 13, or (e) via other hardware arrangements, architectures, and configurations, and / or via a combination of the techniques described in (a)-(e).

[0197] Technical advantages of the described subject matter In one exemplary embodiment, new techniques are provided for transforming, converting, or otherwise processing data from a source signal. The technique may include processing the data in blocks and applying two separate trained matrices (e.g., a pair per layer of a trained neural network) to an activation matrix based on the blocked signal data, thereby generating an output matrix. One of the trained matrices is applied to the left side of the activation matrix, and the other is applied to the right side. The sizes of the matrices (both the trained matrix and the activation matrix) may be selected to take advantage of hardware acceleration. The technique may also advantageously process superimposable patterns of data (e.g., which may be pixels) from the source signal.

[0198] In certain exemplary embodiments, the arrangement of blocks of data (e.g., pixel data) of a signal may more efficiently use the processing capacity of a processor (e.g., a GPU). For example, a GPU may operate at close to 100% (e.g., at least 90 or 95 percent) instead of leaving excess processing capacity unused (e.g., which may be considered a waste of time and / or resources). In this way, in accordance with certain exemplary embodiments discussed herein (e.g., in connection with using separable block transforms over traditional neural network approaches), closer to theoretical maximum processing throughput may be achieved.

[0199] In an exemplary embodiment, an image may be divided into blocks to improve how transforms are applied during the execution of a neural network. In an exemplary embodiment, the size of the blocks may be determined based on the smallest size matrix that can be used in hardware (e.g., a GPU or ASIC) that handles matrix operations. In an example, atomic operations performed on input data from a 1080p source image may be in a relatively quick time frame, enabling real-time image processing (e.g., exemplary atomic operations may be performed in less than approximately 0.04 ms).

[0200] The techniques herein allow for a flexible approach to training models (e.g., neural networks) that can be tailored to different use cases. As an example, different neural networks may be trained to handle different types of games. One model may handle platformer games, while another model may handle first-person games. Using different models for different use cases (including specific models for specific games) may increase the accuracy of the resulting images.

[0201] The techniques discussed herein may provide processing advantages. For example, the processing may operate on a relatively small grain, e.g., by using 16 x 16 x 16 = 4096 multiplications per matrix multiplication. Thus, 2 x 4096 / 16 = 512 multiplications / pixel for each "atomic operation." And 2 x 16 x 16 = 512 weights, thus 1 Kbyte per atomic operation in FP16. Such processing may be scaled up as needed by increasing the width and depth of the network by a multiple of the atomic operations.

[0202] The techniques herein can advantageously operate with lower overhead on a computer system's DRAM because the data being computed during application of a neural network to the activation matrix remains in registers (e.g., internal memory) of the GPU (or other suitable hardware performing the matrix operations).

[0203] In an example embodiment, the techniques herein may provide for reducing the overall amount of storage space (e.g., file size) required to generate images at higher resolution sizes. For example, an application that generates images at higher resolutions may also require assets (e.g., texture data) that are correspondingly sized for generating such high-resolution images. In this way, an example application may render By reducing image size, the size of the data used for such rendering may likewise be reduced, and thus may occupy less memory or storage space. For example, the size of textures used by a rendering engine may be reduced. Thus, the overall size required to deliver an application (e.g., a video game) may be reduced, may fit onto a relatively smaller physical medium (e.g., in terms of how much storage space is provided), and / or may reduce the amount of bandwidth or data required for download. As an illustrative example, a video game designed to natively output images at 4k may have a total size of 60GB. However, if the size of the images generated by the video game engine is 1080p, the total size required for the video game may be reduced to, for example, 20GB. Even if images are output by the video game engine at 1080p, techniques herein may be used to convert such images to 4k images during runtime.

[0204] In an exemplary embodiment, the nature of how the data is prepared and the use of SBT can further leverage tensor hardware acceleration present in some GPU hardware. For example, in NVIDIA's CUDA architecture, an atomic neural network operation is a 16x16 matrix multiply. Certain exemplary embodiments discussed herein are designed to work with this atomic operation. It is recognized that other types of hardware may have atomic operations of other sizes, and the techniques herein may also be adapted to such processing hardware.

[0205] Selected terms Whenever a given item is described in this document as being present in "some embodiments," "various embodiments," "an embodiment," "an exemplary embodiment," "some exemplary embodiments," "exemplary embodiments," or any other similar language is used, it should be understood that the given item is present in at least one embodiment, but not necessarily in all embodiments. Consistent with the above, whenever a description is provided herein that an action "may," "can," or "could" be performed, it should be understood that a feature, element, or component "may," "can," or "could" be included within, or applicable to, a given context; whenever a given item "may," "can," or "could" have, or any similar phrase involving the terms "may," "can," or "could," it should be understood that the given action, feature, element, component, attribute, etc. is present in at least one embodiment, but not necessarily in all embodiments. Terms and phrases used in this document, and variations thereof, unless expressly stated otherwise, should be construed as open-ended rather than limiting. In the above examples, "and / or" includes any and all combinations of one or more of the associated listed items (e.g., a and / or b means a, b, or a and b). The singular forms "a," "an," and "the" should be construed to mean "at least one," "one or more," etc. The term "examples" is used to provide examples of the subject matter under discussion, but is not an exhaustive or limiting list thereof. The terms "comprise" and "include" (and other conjugations and other variations) specify the presence of the associated listed items, but do not exclude the presence or addition of one or more other items. When an item is described as an "option," such description should not be understood to indicate that the other items are not also optional.

[0206] As used herein, the term "non-transitory computer-readable storage medium" includes registers, cache memory, ROM, semiconductor memory devices (such as D-RAM, S-RAM, cache, or other RAM), magnetic media such as flash memory, hard disk drives, and the like. "non-transitory computer-readable storage medium" includes non-transitory computer-readable storage media, magnetic media, optical media such as CD-ROMs, DVDs, or Blu-ray discs, or other types of devices for non-transitory electronic data storage. The term "non-transitory computer-readable storage medium" does not include transitory, propagating electromagnetic signals.

[0207] Additional Applications of the Described Subject Matter Although process steps, algorithms, and the like, including but not limited to those with reference to Figures 2 through 7 and 10 through 12, may be described or claimed in a particular order, such processes may be configured to work in different orders. In other words, any sequence or order of steps that may be explicitly described or claimed in this document does not necessarily indicate a requirement that the steps be performed in that order. Rather, steps of processes described herein may be performed in any order possible. Furthermore, although some steps are described or implied as occurring non-concurrently (e.g., because one step is described after another), they may be performed simultaneously (or in parallel). Furthermore, the illustration of a process by its descriptive nature in drawings does not imply that the illustrated process is exclusive of other variations and modifications, nor does it imply that the illustrated process or any of its steps are required, or that the illustrated process is preferred.

[0208] Although various embodiments have been shown and described in detail, the claims are not limited to any particular embodiment or example. Nothing in the above description should be construed to imply that any particular element, step, range, or function is essential. All structural and functional equivalents to the elements of the above-described embodiments known to those skilled in the art are expressly incorporated herein by reference and are intended to be encompassed by the present invention. Furthermore, a device or method need not address each and every problem sought to be solved by the present invention, for it to be encompassed by the invention. No embodiment, feature, element, component, or step in this document is intended to be made available to the public.

Claims

1. 1. A computer system for training a neural network to convert an image from a first resolution to a second resolution, the computer system comprising: a computer readable storage device configured to store a plurality of target images, the plurality of target images including a first image at the first resolution; and a processing system including at least one hardware processor, said processing system comprising: (a) dividing the first image into a first plurality of pixel blocks; (b) dividing each one of the first plurality of pixel blocks into a plurality of separate output channels to form target output data; (c) generating a second image at the second resolution based on one of the plurality of separate output channels; (d) generating a plurality of context blocks from the second image; (e) dividing the plurality of context blocks into a plurality of separate input channels; (f) training the neural network using the plurality of separate input channels until the neural network converges to the target output data; A computer system configured to:

2. 2. The computer system of claim 1, wherein each of said plurality of separate channels is comprised of a plurality of sub-channels.

3. 3. The computer system of claim 2, wherein each one of the plurality of sub-channels that make up a channel corresponds to a different color value that makes up an individual pixel in the first image.

4. 4. The computer system of claim 3, wherein each of the plurality of sub-channels corresponds to a different one of red, green, and blue values ​​of an RGB image.

5. 5. The computer system of claim 1, wherein the plurality of separate channels is four separate channels, and one of the four separate channels is used to generate the second image.

6. 6. The computer system of claim 5, wherein each of the four separate channels is comprised of multiple sub-channels, each sub-channel corresponding to a different individual color value.

7. The computer system of claim 1 , wherein the size of each of the first plurality of pixel blocks is the same as the size of each one of the plurality of context blocks.

8. 8. The computer system of claim 1, wherein the processing system is further configured to select a second plurality of pixel blocks from the generated second image, each one of the plurality of pixel blocks including data for a plurality of pixels from the generated second image, and each one of the plurality of context blocks based on a corresponding one of the second plurality of pixel blocks.

9. 9. The computer system of claim 8, wherein context data is added to each pixel block to create a corresponding context block.

10. 10. The computer system of claim 1, wherein each content block is divided into four separate input channels, each one of said plurality of separate input channels including a plurality of sub-channels.

11. The plurality of target images include a plurality of images with different target resolutions, and the processing system (a2) dividing one of the plurality of images into a first plurality of pixel blocks; (b2) dividing each one of the first plurality of pixel blocks into a plurality of separate output channels to form target output data; (c2) generating a second image based on one of the plurality of separate output channels, the second image having a resolution lower than the resolution of the one of the plurality of images; (d2) generating a plurality of context blocks from the second image; (e2) dividing the plurality of context blocks into a plurality of separate input channels; (f2) training another neural network using the plurality of separate input channels until the neural network converges to the target output data; The computer system of any one of claims 1 to 10, further configured to:

12. the plurality of target images includes a plurality of images generated by different game engines of different video games, each of the plurality of images having a resolution, and the processing system: (a2) dividing one of the plurality of images into a first plurality of pixel blocks; (b2) dividing each one of the first plurality of pixel blocks into a plurality of separate output channels to form target output data; (c2) generating a second image based on one of the plurality of separate output channels, the second image having a resolution lower than the resolution of the one of the plurality of images; (d2) generating a plurality of context blocks from the second image; (e2) dividing the plurality of context blocks into a plurality of separate input channels; (f2) training another neural network using the plurality of separate input channels until the neural network converges to the target output data; The computer system of any one of claims 1 to 10, further configured to:

13. further comprising a transceiver configured to receive a plurality of different requests for the video game enabled neural network from a plurality of different computing devices; The processing system includes: For each corresponding request, we select from several different trained neural networks. selecting at least one of the trained neural networks based on data included in the corresponding request; communicating, via the transceiver, the selected at least one of the trained neural networks to a requesting computing device; The computer system of any one of claims 1 to 12, further configured to:

14. 14. The computer system of claim 13, wherein the data included in the corresponding request is an identifier for a particular video game.

15. 14. The computer system of claim 13, wherein the data included in the corresponding request indicates a target resolution.

16. The computer system of any one of claims 1 to 15, wherein the processing system is further configured to generate the first image by using a rendering engine for a video game.

17. 17. The computer system of claim 16, wherein the trained neural network is stored in the computer-readable storage device in association with the video game.

18. 20. The computer system of claim 17, wherein the trained neural network is communicated over a computer network to a gaming device being used to play the video game.

19. 19. The computer system of claim 1, wherein the trained neural network includes multiple separable block transform (SBT) terms across multiple layers of the trained neural network.

20. 20. The computer system of claim 19, wherein the processing system is further configured to prune the trained neural network by removing at least one SBT term from the plurality of SBT terms, and the pruned trained neural network is communicated to and used on a plurality of computing devices.

21. The processing system includes: calculating a first loss value from the trained neural network including a first SBT term; calculating a second loss value from the trained neural network that does not include the first SBT term; calculating a difference between the first loss value and the second loss value; further configured as follows:

21. The computer system of claim 20, wherein the first SBT term is pruned based on the calculated difference.

22. A computer program for execution by a processing system including at least one hardware processor, the computer program, when executed, causing the processing system to: instructions to cause the processing system to: acquire a plurality of target images, the plurality of target images including a first image at a first resolution; and Dividing the first image into a first plurality of pixel blocks; dividing each one of the first plurality of pixel blocks into a plurality of separate output channels to form target output data; generating a second image at a second resolution based on one of the plurality of separate output channels; generating a plurality of context blocks from the second image; dividing the plurality of context blocks into a plurality of separate input channels; training the neural network using the plurality of separate input channels until the neural network converges to the target output data; A computer program containing instructions to perform the following:

23. 1. A method executed on a computer system, the method comprising: processing a plurality of target images, the plurality of target images including a first image at a first resolution; Separating the first image into a first plurality of pixel blocks; dividing each one of the first plurality of pixel blocks into a plurality of separate output channels to form target output data; generating a second image at a second resolution based on one of the plurality of separate output channels; generating a plurality of context blocks from the second image; dividing the plurality of context blocks into a plurality of separate input channels; training the neural network using the plurality of separate input channels until the neural network converges to the target output data.

24. 1. A distributed computer game system, comprising: a display device configured to output an image at a target resolution; and a cloud-based computer system including a plurality of processing nodes, at least one of the processing nodes: executing a first video game on at least one of the processing nodes to generate images for the first video game at a first resolution; transmitting image data based on the generated image; It is configured as follows: a client computing device configured to receive the image data, the client computing device including at least one hardware processor, the at least one hardware processor comprising: and configured to execute a neural network based on the received image data to generate a target image, the execution of the neural network applying a separable block transform to a plurality of activation matrices, each activation matrix corresponding to a different block of pixel data in the image represented by the image data, and the target image is generated at the target resolution, the at least one hardware processor further comprising: a distributed computer gaming system configured to output the target image at the target resolution to the display device for display on the display device during gameplay of the first video game;

25. 25. The distributed computer game system of claim 24, wherein the first resolution is lower than the target resolution.

26. 25. The distributed computer game system of claim 24, wherein the first resolution is the same as the target resolution.

Images