Fusion processing of continuous mathematical operators

Abstract

This invention relates to fused processing of continuous mathematical operators. Systems and methods for fused processing of continuous mathematical operators are disclosed. Fused processing of continuous mathematical operations (e.g., pointwise nonlinear functions) without storing intermediate results to memory improves performance when memory bus bandwidth is limited. In one embodiment, continuous mathematical operations including at least two of convolution, upsampling, pointwise nonlinear function, and downsampling are performed to process input data and generate aliasing-free output data. In one embodiment, input data is spatially blocked for parallel processing such that intermediate results generated during processing of each block of input data can be stored in shared memory within the processor. Storing intermediate data in shared memory improves performance compared to storing and loading intermediate data to and from external memory.

Description

[0001] Claiming priority

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 211,899, filed June 17, 2021, entitled “Alias-Free Generative Adversarial Networks,” and U.S. Provisional Application No. 63 / 244,125, filed September 14, 2021, entitled “Alias-Free Generative Adversarial Neural Networks,” the entire contents of which are incorporated herein by reference. Background Technology

[0003] Continuous mathematical operators can be approximated by processing input data through a sequence of functions, where intermediate data generated by each function in the sequence is stored in memory. The memory is coupled to the processor implementing the functions, allowing intermediate data to be transferred via a memory bus to store intermediate data in memory and to load intermediate data from memory for each function. As one function increases the resolution of its intermediate data, the amount of data transferred via the memory bus also increases. When memory bus bandwidth is limited, increasing the amount of data transferred can degrade processing performance. These and / or other issues associated with the prior art need to be addressed. Summary of the Invention

[0004] The fusion of sequential mathematical operations, such as pointwise nonlinear functions, without storing intermediate results in memory, improves performance when memory bus bandwidth is limited. In one embodiment, sequential mathematical operations including at least two of convolution, upsampling, pointwise nonlinear functions, and downsampling are performed to process input data and generate output data. In another embodiment, the input data is spatially partitioned for parallel processing, such that intermediate results generated during processing of each tile of input data can be stored in shared memory within the processor. Storing intermediate data in shared memory improves performance compared to storing intermediate data in and loading intermediate data from external memory.

[0005] A method, computer-readable medium, and system for fusion processing of continuous mathematical operators are disclosed. The method, computer-readable medium, and system can be used to approximate continuous mathematical operators. In one embodiment, the method includes storing input data in a memory comprising one or more memory devices, and reading the input data from the memory by a processor coupled to the memory via an interface. The processor executes instructions to approximate the continuous mathematical operator by: upsampling an intermediate feature map to produce a resolution-increased feature map approximating the continuous feature map; applying a nonlinear operation to the resolution-increased feature map to produce a processed resolution-increased feature map; downsampling the processed resolution-increased feature map to produce a processed intermediate feature map; and storing the processed intermediate feature map in the memory. Attached Figure Description

[0006] The system and method for fusion processing of continuous mathematical operators are described in detail below with reference to the accompanying drawings, wherein:

[0007] Figure 1A Images generated using a conventional generative neural network and images generated using a non-aliasing generative neural network according to an embodiment are shown.

[0008] Figure 1B The image shows fine details generated using a conventional generative neural network according to existing technology.

[0009] Figure 1C Fine details of an image generated using an alias-free generative neural network according to an embodiment are shown.

[0010] Figure 1D Generative neural networks that can be used to implement certain embodiments of this disclosure are shown, along with conceptual feature maps of hierarchical structures at different levels.

[0011] Figure 1E This is a conceptual diagram illustrating discrete and continuous representations of feature maps used to implement certain embodiments of the present disclosure.

[0012] Figure 2A Block diagrams are shown of example processing layers for conventional generative neural networks and example processing layers for non-aliasing generative neural networks suitable for implementing certain embodiments of this disclosure.

[0013] Figure 2B A flowchart of a method for performing an aliasing-free operation according to an embodiment is shown.

[0014] Figure 2C Feature maps generated by a conventional generative neural network and feature maps generated by a non-aliased generative neural network according to an embodiment are shown.

[0015] Figure 3An example parallel processing unit is shown, suitable for implementing certain embodiments of this disclosure.

[0016] Figure 4A Some embodiments suitable for implementing this disclosure are shown. Figure 3 An example of a general-purpose processing cluster within a parallel processing unit.

[0017] Figure 4B Some embodiments suitable for implementing this disclosure are shown. Figure 3 An example memory partition cell for a parallel processing unit.

[0018] Figure 4C Some embodiments suitable for implementing this disclosure are shown. Figure 4A An example of streaming multiprocessors.

[0019] Figure 5A This is a conceptual diagram of a processing system implemented using the PPU of FIG4, suitable for implementing some embodiments of the present disclosure.

[0020] Figure 5B Exemplary systems in which various architectures and / or functions of various prior embodiments can be implemented are shown.

[0021] Figure 5C Components of an exemplary system that can be used to train and utilize machine learning in at least one embodiment are shown.

[0022] Figure 6 An exemplary streaming system suitable for implementing some embodiments of this disclosure is shown.

[0023] Figure 7A A conceptual diagram of a processing system for fusion processing of continuous mathematical operators suitable for implementing some embodiments of this disclosure is shown.

[0024] Figure 7B The diagram illustrates blocks of intermediate data suitable for implementing some embodiments of this disclosure.

[0025] Figure 7C A flowchart is shown of a method for fusion processing of continuous mathematical operators suitable for implementing some embodiments of this disclosure. Detailed Implementation

[0026] Systems and methods related to alias-free generative neural networks, particularly image generator neural networks, are disclosed. In the real world, details at different scales often undergo hierarchical transformations. For example, moving a person's head causes the nose to move, which in turn causes the pores on the nose to move. Generative neural networks have a similar structure: they refine coarse, low-resolution features hierarchically through upsampling, perform local blending through convolution, and introduce new details through nonlinear operations. Despite the apparent similarity in their hierarchical structures, traditional generative neural network architectures do not synthesize images in a natural hierarchical manner: coarse features seem to primarily control the presence of finer features rather than their precise location. Instead, many fine details appear fixed at pixel coordinates and appear as residual visual artifacts. These disturbing "residuals" disrupt the illusion of solid, coherent objects moving in space. These residual visual artifacts are manifestations of aliasing in the intermediate feature maps of the neural network. Alias-free generative neural networks provide an architecture that exhibits a more natural hierarchical transformation, where the sub-pixel position of each feature is inherited from the underlying coarse features.

[0027] Compared to the traditional systems described above, aliasing-free generative neural networks append details from the output data to features in the intermediate data of each layer, rather than appending details to image coordinates (e.g., pixels). Aliasing-free techniques provide more natural motion in generated videos. For example, when a person rotates their head, their teeth translate with the movement, rather than remaining stationary. Aliasing occurs during feature upsampling and nonlinear operations performed within neural network layers. Modifications have been made to the convolutions, upsampling, downsampling, and pointwise nonlinear operations performed by traditional generative neural networks to achieve aliasing-free performance.

[0028] Image synthesis is used in a variety of applications, including image editing, domain transformation, and video generation. Generative neural networks can also be used to synthesize 3D rolls, audio signals, and more. Anti-aliasing generative neural networks provide an architecture that exhibits a more natural hierarchy of transformations. For example, in one embodiment, the exact subpixel location of each feature is specifically inherited from the underlying coarse features. These coarse features not only control the presence of finer features but also the location of those finer features on surfaces (e.g., faces, heads, etc.). Therefore, anti-aliasing generative neural networks reduce residual visual artifacts.

[0029] Despite the hierarchical convolutional nature of generative neural networks (GNNs), the synthesis process of traditional GNNs depends unhealthily on absolute pixel coordinates. This dependency manifests as, for example, details appearing glued to image coordinates rather than the surface of the object being depicted. The root cause of aliasing can be traced back to careless signal processing. For instance, traditional generator neural networks partially circumvent ideal hierarchical structures by using image boundaries, per-pixel noise input, and positional encoding, and by exploiting unintentional positional references available in intermediate layers for aliasing. Often, neural networks can be quite lazy and, where possible, take shortcuts to bypass rigorous hierarchical refinement.

[0030] Although aliasing is a subtle yet crucial problem, it receives little attention in the neural network literature. Two sources of aliasing can be identified: first, the blurred pixel grid after the image due to non-ideal upsampling filters (e.g., recent, bilinear, strided convolutions); and second, non-linear pointwise applications, such as rectified linear activation functions (RELU). Traditional generator neural networks have the means and motivation to amplify even the slightest amounts of aliasing, and when combined across multiple scales, they can form the basis for texture patterns with finer details defined in screen coordinates. Aliasing can be amplified by most filters commonly used in deep learning, even high-quality filters used in image processing.

[0031] Figure 1A Images generated using a conventional generative neural network and an image generated using a generative neural network with reduced aliasing, according to an embodiment, are shown. Image 105 is synthesized from latent code using a conventional generative neural network, and image 110 is synthesized from latent code using a generative neural network with reduced aliasing. Both images 105 and 110 appear to be of high quality and realistic. Aliasing, or “texture residue,” is illustrated by averaging images of the same region generated in response to small changes (e.g., random perturbations) in the latent code. Without aliasing, the result should be uniformly blurred because fine details should move with the surface rather than be stuck at pixel coordinates. However, as shown in image 115, which represents the average of several images generated by a conventional generative neural network, some fine details, such as fur, are stuck at the same pixel coordinates and appear sharp and defined. In contrast, in image 120, which represents the average of several images generated by a generative neural network with reduced aliasing, fine details appear uniformly blurred.

[0032] Figure 1BFine details of images 121, 122, and 123 generated using a conventional generative neural network according to existing technology are shown. Images 121, 122, and 123 (and additional images not shown) are synthesized sequentially over time via latent spatial interpolation to produce intermediate images that transition smoothly over time, thus creating a morphing effect. Short vertical segments of pixels are extracted from the same location in each of images 121, 122, and 123 to generate pixel regions 125, 126, and 127, respectively. Pixel regions 125, 126, and 127 are horizontally organized (stacked), as shown in magnified region 128. The desired result is that the hair moves in the animation, creating a temporal variation field. Magnified region 128 also includes additional horizontally stacked pixel regions, where additional regions are extracted from additional images synthesized between images 121 and 122 and between images 122 and 123. As shown by the horizontal stripes from pixel region 125 to pixel region 126 and extending to pixel region 127 in magnified region 128, the hair changes almost no time.

[0033] Figure 1C Fine details of images 131, 132, and 133 generated using a generative neural network with reduced aliasing according to an embodiment are shown. Images 131, 132, and 133 (and additional images not shown) are synthesized temporally via latent spatial interpolation to produce intermediate images that transition smoothly over time, thus creating a morphing effect. Short vertical segments of pixels are extracted from the same location in each of images 131, 132, and 133 to generate pixel regions 135, 136, and 137, respectively. Pixel regions 135, 136, and 137 are horizontally organized (stacked), as shown in magnified region 138. Magnified region 138 also includes additional horizontally stacked pixel regions, where additional regions are extracted from additional images synthesized between images 131 and 132 and between images 132 and 133. Compared to the almost unchanging hair in magnified region 128, the hair in magnified region 138 moves over time from pixel region 135 to pixel region 136 and extends into pixel region 137, thus creating a field that varies over time.

[0034] Now, further illustrative information regarding the various optional architectures and features that can implement the aforementioned framework will be provided, based on user needs. It should be strongly noted that the following information is listed for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be optionally combined, excluding or not excluding other features described.

[0035] Figure 1DIllustrations 141, 142, and 143 show a reduced-aliasing generative neural network 100 suitable for implementing certain embodiments of this disclosure, and conceptual feature diagrams of different levels of processing hierarchy. In one embodiment, the reduced-aliasing generative neural network 100 receives input features as input, which are processed according to a latent code to synthesize output data. In one embodiment, the output data is a generated image 140. The output data can be a one-dimensional (1D) signal (e.g., audio data) or higher-dimensional data (e.g., 3D scan), without limitation. In one embodiment, the latent code contains 512 digits. In one embodiment, the latent code is mapped to an intermediate latent space to generate one or more modulated signals, which are provided to at least one layer of the generative neural network 100. In one embodiment, each of the one or more modulated signals is associated with a different layer of the generative neural network 100. In one embodiment, the input features are learning constants. In one embodiment, the input features are Fourier features that naturally define a continuous feature map. Output data is synthesized based on the input features and the latent code.

[0036] As the feature maps generated by each layer are processed by subsequent layers or stages in the generative neural network 100, the resolution of feature maps 141, 142, and 143 increases. In one or more layers, the feature maps are upsampled to increase resolution. For example, feature map 143 has a higher resolution than feature maps 141 and 142. Feature map 142 has a higher resolution than feature map 141. A grid pattern may be visible when upsampling using simple filters (e.g., nearest neighbors). Even with more complex filters, a grid pattern may not be visible, but a conventional generative neural network may actively attempt to extract it.

[0037] When the feature maps are represented in a continuous rather than discrete (e.g., pixelated) form, feature maps 145, 146, and 147 replace feature maps 141, 142, and 143, respectively. Processing feature maps 145, 146, and 147 will produce aliased-free output data, but processing requires higher resolution, so the bit width of each computation is infinite, which is impractical. However, as further described herein, upsampling and other operations can be performed to better approximate the continuous feature maps 145, 146, and 147.

[0038] To reduce or eliminate aliasing, information that bypasses layer refinement should be eliminated, thus preventing the generative neural network 100 from using that information. While information introduced by image boundaries can be reduced by simply operating on a slightly larger image, reducing or preventing aliasing is more difficult. Aliasing is handled most naturally within the classic Shannon-Nyquist signal processing framework, and a solution can be found by shifting the focus to a bandwidth-limiting function over a continuous domain represented only by a discrete grid of samples. In the continuous domain, successfully eliminating sources with all positional references means that details can be generated equally well regardless of pixel coordinates, which is equivalent to enforcing continuous equivariance with subpixel translations (and optional rotations) across all layers of the generative neural network 100.

[0039] Visually, equivalence with subpixel translation means that a portion of the horizontally shifted feature map 141 should cause a shift in each subsequent feature map 142 and 143 and the corresponding portion of the output data, with no other effects. Similarly, equivalence with rotation means that a portion of the rotated feature map 141 should rotate the corresponding portion of each subsequent feature map 142 and 143 and the output data, with no other effects. The translation and rotation of fractions or subpixels are not well-defined in the discrete forms of feature maps 141, 142, and 143, but are well-defined for the continuous forms 145, 146, and 147.

[0040] Reconsidering the signal processing aspects of traditional generative neural networks, conventional upsampling filters are not aggressive enough in suppressing aliasing and may require extremely high-quality filters with stopband attenuation exceeding 100dB. By considering the effects of pointwise nonlinearity in the continuous domain and applying appropriate low-pass filtering to the results, a principled solution to aliasing caused by pointwise nonlinearity can be found. Furthermore, constraining convolution operations to a 1×1 kernel can yield a powerful rotationally equivariant generative neural network 100. Once aliasing is sufficiently suppressed to force the generative neural network 100 to implement a more natural hierarchical refinement, the operation of the generative neural network 100 changes dramatically: details are correctly attached to the underlying surface, such as... Figure 1C As shown.

[0041] Although data can be stored as values ​​in a pixel grid, such as discrete feature maps 141, 142, and 143, it should not be assumed that these values ​​directly represent the signals processed by the layers of the generative neural network 100. According to the Nyquist-Shannon sampling theorem, a periodically sampled signal can represent any continuous signal containing frequencies between zero and half the sampling rate. Consider a two-dimensional, discretely sampled feature map Z[x] composed of a regular grid of Dirac pulses of varying amplitudes, spaced at 1 / s units, where s is the sampling rate. This is analogous to an infinite two-dimensional value grid.

[0042] Given Z[x] and s, the Whittaker-Shannon interpolation formula states that the corresponding continuous representation z(x) is obtained by interpolating the discretely sampled Dirac grid Z[x] with an ideal interpolation filter φ. s It is obtained by convolution, i.e., z(x) = (φ s *Z)(x), where * denotes continuous convolution, and φ s The expression (x) = sinc(sx0)·sinc(sx1) uses the signal processing convention defined as sinc(x) = sin(πx) / (πx). φ s With a bandwidth limit of s / 2 along both the horizontal and vertical dimensions, the resulting continuous signal capture can be expressed in terms of all frequencies using a sampling rate s.

[0043] Figure 1E This is a conceptual diagram illustrating discrete and continuous representations of feature maps used in some embodiments of this disclosure. The discrete representation Z of discrete feature map 150 and the continuous representation z of continuous feature map 155 are obtained by using an ideal interpolation filter φ. s (Sample filter 160) convolution and using Dirac Comb III S The point-by-point multiplications of (Dirac Comb165) are interconnected.

[0044] The transformation from continuous feature map 155 to discrete feature map 150 corresponds to sampling the continuous signal z(x) at sampling points of Z[x], where each sampling point of Z[x] can be defined as a sample interval offset by half to lie at the "pixel center" shown as a circle. Sampling of the continuous feature map can be represented as a two-dimensional Dirac Comb... Point-by-point multiplication.

[0045] The unit square in z(x) is x∈[0,l] 2 (Continuous feature map 155) can be designated as the canvas for the signal of interest. In discrete feature map 150Z[x], there are s in the unit square. 2 A discrete sample, but with a sample filter of 160φ s The convolution of Z[x] means that the value of Z[x] outside the unit square also affects z(x) inside it. Therefore, storing an s×s pixel feature map is insufficient; theoretically, the entire infinite Z[x] would need to be stored. As a practical solution, Z[x] can be stored as a two-dimensional array that covers an area slightly larger than the unit square. For example, each feature map 141, 142, and 143 is expanded to include at least one additional feature value in each dimension. Expanding the feature maps eliminates residual artifacts caused by boundaries. However, aliasing caused by discrete processing operations still exists and can be reduced or eliminated as described further in this paper.

[0046] After establishing the correspondence between the bandwidth-constrained, continuous feature map 155z(x) and the discretely sampled feature map 150Z[x], the typically pixel-centric view of the signal can be ignored to support continuous signals. In the context of the following description, z(x) is interpreted as the actual signal in operation, while the discretely sampled feature map 150Z[x] is merely a convenient encoding of the continuous feature map 155z(x).

[0047] Practical neural networks operate on discretely sampled feature maps, such as discrete feature map 150. Consider an operation F (convolution, nonlinearity, etc.) operating on a discrete feature map: Z' = F(Z). The discrete feature map (e.g., discrete feature map 150) has a corresponding continuous counterpart (e.g., continuous feature map 155), and therefore there is a corresponding mapping in the continuous domain: z' = f(z). Now, we can see that an operation specified in one domain performs a corresponding operation in another domain:

[0048] f(z)=φ s′ *F(III s ⊙z), F(Z)=III s′ ⊙f(φ s *Z), Equation (1)

[0049] Where ⊙ represents pointwise multiplication, and s and s' are the input and output sampling rates.

[0050] If an operation f commutes with a spatial transformation t in a continuous domain, then the operation f is equivariant with respect to the spatial transformation t in the 2D plane: Please note that when the input bandwidth is limited to s / 2, equivariant operations must not generate frequency content higher than the output bandwidth limit of s' / 2; otherwise, a faithful discrete output representation will not exist. For example, as the frequency content of the continuous feature map 155 increases, the sampling rate s may also need to increase to ensure that details in the continuous domain can be represented in the discrete domain. Failure to use a sufficient sampling rate will result in aliasing, which appears as residual artifacts at every level of the discrete feature map and the output. Residual artifacts are not equivariantly translated relative to the input. According to the Nyquist-Shannon sampling theorem, a periodically sampled discrete feature map can represent a continuous feature map containing frequencies between zero and half the sampling rate.

[0051] In the case of rotational equivariance, the spectral constraints are somewhat more stringent—the rotated image corresponds to a rotated spectrum, and to ensure bandwidth limiting in both the horizontal and vertical directions, the spectrum must be confined to a disk with a radius of s' / 2. As further described in this paper, this applies to both the initial network input and the bandwidth-limiting filter used for downsampling.

[0052] Typical primitive operations in generative neural networks include convolution, upsampling, downsampling, and nonlinearity. Without loss of generality, these operations are described as acting on a single feature map: pointwise linear combinations of features have no effect on the analysis. Consider a standard convolution with a discrete kernel K. K can be interpreted as occupying the same grid as the input feature map, with a sampling rate s. Discrete-domain operations are simply F... conv (Z) = K*Z, and the corresponding continuous operation is obtained from equation (1):

[0053] f conv (z)=φ s *(K*(III s ⊙z))=K*(φ s *(III s Equation (2) = K*z

[0054] This is due to the commutativity of convolution and the fact that discretization immediately following convolution with an ideal low-pass filter (both having the same sampling rate s) is an identity operation, i.e., φ s *(III s ⊙z)=z. In other words, convolution operates by continuously sliding a discretized kernel over a continuous representation of the feature map. Convolution does not introduce new frequencies, thus satisfying the bandwidth constraints of translation and rotation.

[0055] Convolution is also commutative with translation in a continuous domain, therefore convolution operations are equivariant with translation. For rotational equivariance, the discrete kernel K needs to be radially symmetric. In one embodiment, a symmetric l×l convolution kernel, although simple, is a feasible choice for rotationally equivariant generative neural networks.

[0056] Ideally, upsampling should not alter the continuous representation, as the purpose of upsampling is to increase the output sampling rate (s' > s) to increase the net space in the spectrum, where subsequent layers of the generative neural network 100 might introduce additional content. Translation and rotation are equivariant directly stemming from the fact that upsampling is an identity operation in the continuous domain. In f up When (z) = z, the discrete operation according to equation (1) is F up (Z)=(III s ′⊙(φ s *Z))._ Choosing s' = ns with integer n, the discrete operation can be performed by first interleaving Z with zeros to increase its sampling rate, and then interleaving the interleaved Z with the discretized downsampling filter III. s′ ⊙φ s This is achieved through convolution.

[0057] In downsampling, z is a low-pass filter to remove frequencies above the output bandwidth limit, so that the signal can be faithfully represented in a coarser discretization. The operation in the continuous domain is f.down (z)=ψ s′ *z, where the ideal low-pass filter ψ s :=s 2 ·φ s It is merely the corresponding interpolation filter normalized to unit quality. The discrete correspondence is F. down (Z)=III s′ ⊙(ψ s′ *(φ s *Z))=1 / s 2 ·III s′ ⊙(ψ s′ *ψ s *Z)=(s′ / s) 2 ·III s′ ⊙(φ s′ *Z). The latter equation comes from ψ s *ψ s′ =ψ min(s,s′) Similar to upsampling, downsampling of integer fractions can be achieved through discrete convolution followed by discarding sample points. Translation equivariance automatically follows f. down (z) has the transformability of translation, but for rotational isomorphism φ s′ It must be replaced with a radially symmetric filter having a disk-shaped frequency response. Ideally, such a filter would be... Given, where J1 is a Bessel function of the first class.

[0058] Applying pointwise nonlinearity σ in the discrete domain does not commutate with fractional translations or rotations. However, in the continuous domain, any pointwise function is independent of geometric transformations and is therefore equivalent to translations and rotations. Satisfying bandwidth constraints is another challenge—for example, applying ReLU in the continuous domain may introduce arbitrary high frequencies that cannot be represented in the discrete output.

[0059] The natural solution to pointwise nonlinearity is to combine the continuous result with an ideal low-pass filter ψ. s Convolution is performed to eliminate harmful high-frequency content. The nonlinear continuous representation then becomes f. σ (z)=ψ s *σ(z)=s 2 ·φ s *σ(z), discrete correspondence is F σ (Z)=s 2 ·III s ⊙(φ s *σ(φ s*Z)). Discrete computation cannot be achieved without temporarily entering a continuous representation. A continuous representation can be approximated by upsampling the signal, applying nonlinearity at a higher resolution, and subsequently downsampling the result. Even though the nonlinearity is still performed in the discrete domain, in one embodiment, a mere 2× temporary resolution increase is sufficient to obtain high-quality equivariance. For rotational equivariance, a radially symmetric low-pass filter... It should be used in the downsampling step, as described above.

[0060] Note that nonlinearity is the only operation capable of generating new frequencies in the generative neural network 100, and the range of these new frequencies can be limited by applying a reconstruction filter with a cutoff value below s / 2 before the final discretization operation. The reconstruction filter provides precise control over how much new information is introduced into each layer of the generative neural network 100. In summary, pointwise nonlinear operations can potentially generate arbitrarily high frequencies in the continuous domain σ(z). φ needs to be used. s Low-pass filtering is performed to ensure that Z' captures the results of pointwise nonlinear operations in the discrete domain. By applying the theoretical ideas of performing convolution, upsampling, downsampling, and pointwise nonlinear operations in practice, generative neural networks can be transformed into fully equivariant networks that achieve the desired level of precision, translation, and rotation, thereby producing generative neural networks with reduced aliasing, such as generative neural network 100.

[0061] In one embodiment, the generative neural network 100 includes a mapping network and a synthesis network G. The mapping network transforms initial, normally distributed latent values ​​into intermediate latent codes w~w. The synthesis network receives learned 4x4x512 constant Z0 (i.e., input features) or Fourier features and applies a series of N layers—including convolutions, nonlinearities, upsampling, and (optionally) per-pixel noise—to generate an output image Z. N = G(Z0; w). In one embodiment, the intermediate latent code w controls the modulation of the convolutional kernels in G. In one embodiment, these layers follow a strict 2× upsampling schedule, where two layers are executed per resolution, with the number of feature maps halved after each upsampling. In one embodiment, upsampling discrete data by 2× approximates continuous data and is sufficient to significantly reduce or prevent aliasing introduced during subsequent pointwise nonlinear operations. In one embodiment, upsampling is greater than 2× to better approximate continuous data.

[0062] Reducing or eliminating aliasing makes each layer of G equivalent to the continuous signal, so that all the finer details are transformed along with the coarse features of the local neighborhood. If successful, the entire generative neural network 100 becomes similarly equivalent. In other words, the continuous operation g of the synthetic network is equivalent to the transformation t (translation and rotation) applied to the continuous input z0: g(t[z0]; w) = t[g(z0; w)].

[0063] Figure 2A Block diagrams are shown of an example processing layer 205 for a conventional generative neural network and an example processing layer 200 for reducing aliasing in a generative neural network suitable for implementing some embodiments of this disclosure. It should be understood that these and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Moreover, those skilled in the art will understand that any system performing the operation of the generative neural network 100 is within the scope and spirit of the embodiments of this disclosure.

[0064] Traditional generative neural networks and generative neural networks 100 both include a mapping network and a synthesis network G. As mentioned earlier, the mapping network transforms the latent code into intermediate latent code. The synthetic network receives input features Z0 and applies a series of N layers—including convolutions, nonlinearities, and upsampling—to produce the output image Z. N =G(Z0; w). The intermediate latent code w controls the modulation of the convolutional kernel in G.

[0065] Layer 205 is included in a conventional synthesis network and receives low-resolution input data, such as feature maps, and produces intermediate data for aliasing processing. Layer 205 includes a convolution operator 210, which also receives modulation control (not shown) from a mapping network and produces intermediate data 212. ReLU 215 processes the intermediate data 212 to produce processed intermediate data 217. The processed intermediate data 217 is optionally processed by a 2x upsampler 220 to produce intermediate data for aliasing processing (e.g., feature maps). Aliasing can be introduced by ReLU 215 and the 2x upsampler 220. Techniques based on sequential analysis can be used to reduce or remove aliasing from each operation in the reduced-aliasing generative neural network 100.

[0066] Layer 200 is modified compared to layer 205 and can be included as one or more stages in the synthetic network portion of the reduced-aliasing generative neural network 100. Layer 200 receives low-resolution input data, such as feature maps, and produces processed intermediate data with reduced aliasing compared to the aliased intermediate data produced by layer 205. Layer 200 includes a convolution operator 230, which also receives modulation control (not shown) from a mapping network and produces intermediate data 232. In one embodiment, the convolution operator 230 can be configured to use an N×N convolution kernel, where N=1 for rotational equivalence.

[0067] Upsampler 235 processes intermediate data 232 to produce resolution-increased intermediate data 237 (e.g., a feature map). In one embodiment, upsampler 235 may be configured to perform 2x or 4x upsampling. The resolution-increased intermediate data 237 is processed by pointwise nonlinear operation 240 to produce processed resolution-increased intermediate data 242. In one embodiment, pointwise nonlinear operator 240 performs a ReLU operation. In one embodiment, upsampling the intermediate data 232 reduces aliasing during the application of pointwise nonlinearity.

[0068] The processed, resolution-enhanced intermediate data 242 is then downsampled by downsampler 245 to produce processed intermediate data. In one embodiment, downsampler 245 downsamples the processed, resolution-enhanced intermediate data 242 by 2x to produce processed intermediate data at the original resolution of intermediate data 232. In one embodiment, intermediate data 232 contains only spatial frequencies at or below half the sampling rate of the original resolution of intermediate data 232. In one embodiment, the upsampling rate is equal to the downsampling rate. In one embodiment, the upsampling rate is twice the downsampling rate. In one embodiment, the downsampling rate is twice the upsampling rate. In one embodiment, the convolution operator 230, upsampler 235, pointwise nonlinear operator 240, and downsampler 245 perform operations equivalent to translation and / or rotation.

[0069] In one embodiment, at least one of the convolution operator 230, upsampler 235, pointwise nonlinear operator 240, and downsampler 245 is implemented on a server or in a data center, and the high-resolution output data is streamed to a user device. In one embodiment, at least one of the convolution operator 230, upsampler 235, pointwise nonlinear operator 240, and downsampler 245 is implemented in a cloud computing environment. In one embodiment, at least one of the convolution operator 230, upsampler 235, pointwise nonlinear operator 240, and downsampler 245 is implemented for training, testing, or validating a neural network used in a machine, robot, or autonomous vehicle. In one embodiment, at least one of the convolution operator 230, upsampler 235, pointwise nonlinear operator 240, and downsampler 245 is implemented on a virtual machine including a portion of a graphics processing unit.

[0070] Figure 2B A flowchart of a method 250 for near-aliasing-free operation according to an embodiment is shown. Each block of the method 250 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or a plug-in to another product, to name a few. Furthermore, as an example, for Figure 2A The system described herein is for method 250. However, this method may be performed additionally or alternatively by any system or any combination of systems, including but not limited to those described herein. Furthermore, those skilled in the art will understand that any system performing method 250 is within the scope and spirit of the embodiments of this disclosure.

[0071] A neural network model receives low-resolution input data processed by a series of stages to produce high-resolution output data. In one embodiment, the neural network model is a generative neural network 100 with reduced aliasing. In one embodiment, for each operation performed in a sequence of stages that applies pointwise nonlinearity to intermediate data at the original resolution, the processing includes upsampling and downsampling. In one embodiment, one or more layers of the neural network model perform method 250. In one embodiment, one or more layers are layers 200. In one embodiment, the low-resolution input data is processed by a convolution operator to produce intermediate data.

[0072] In step 260, the intermediate data is upsampled to produce intermediate data with increased resolution. In step 265, a pointwise nonlinearity is applied to the intermediate data with increased resolution to produce processed intermediate data with increased resolution. In step 270, the processed intermediate data with increased resolution is downsampled to produce processed intermediate data. In one embodiment, the processed intermediate data corresponds to the output data of the reduced aliasing generative neural network 100. In one embodiment, a linear transformation is performed on the processed intermediate data to produce the output data.

[0073] In one embodiment, the intermediate data does not contain spatial frequencies at a sampling rate higher than half the original resolution. In one embodiment, the upsampling rate is equal to the downsampling rate, and the processed intermediate data is at the original resolution. In one embodiment, the upsampling rate is twice the downsampling rate, and the processed intermediate data is at twice the original resolution. In one embodiment, the downsampling rate is twice the upsampling rate. In one embodiment, upsampling approximates continuous data and reduces aliasing during the application of pointwise nonlinearity. In one embodiment, the neural network model is a generator, and the high-resolution output data is an image.

[0074] Figure 2C Feature maps 251 and 252 generated by a conventional generative neural network and feature maps 261, 262, 271, and 272 generated by a reduced-aliasing generative neural network according to an embodiment are shown. The feature maps are internal representations of processed intermediate data generated by a layer of the neural network. Feature maps 251 and 252 are generated by layers of a conventional generative neural network, with feature map 251 being coarser compared to feature map 252. Note that each pair of feature maps (e.g., 251 and 252) is processed to generate images of different people. Even though the resulting images differ, the characteristics of feature maps 251 and 252 are significantly different compared to feature maps 261, 262, 271, and 272. Feature maps 251 and 252 appear to encode signal amplitude.

[0075] Feature maps 261, 262, 271, and 272 were generated by layers of a generative neural network with reduced aliasing, where feature maps 261 and 271 are coarser than feature maps 262 and 272. Feature maps 261, 262, 271, and 272 appear to encode phase information compared to feature maps 251 and 252. Generative neural networks require phase information to synthesize details on a surface because they cannot rely on pixel coordinates but need to learn a coordinate system. The existence of a coordinate system that allows for precise localization on the surface of an object should prove useful in a variety of applications, including advanced image and video editing.

[0076] Compared to feature maps 251 and 252, feature maps 261 and 262 are translationally equivariant. Feature maps 271 and 272 are both translationally and rotationally equivariant. More specifically, the convolution operations used to generate feature maps 271 and 272 are restricted to N=1, and a radially symmetric downsampling filter is used to improve rotational equivariance. Conversely, the convolution operations used to generate feature maps 261 and 262 use N=3. The emergency position encoding patterns appearing in feature maps 271 and 272 appear to be better defined compared to feature maps 261 and 262.

[0077] Compared to conventional systems, the anti-aliasing generative neural network 100 appends details from the output data to features in the intermediate data of each layer, rather than appending details to image coordinates (e.g., pixels). The anti-aliasing generative neural network 100 provides significant improvements for video and animation generation. Interpreting all signals in layer 200 of the anti-aliasing generative neural network 100 as continuous ensures that unwanted information does not leak into the layered synthesis process. Layer 200 of the anti-aliasing generative neural network 100 is equivariant to translation and rotation even at the sub-pixel scale. Compared to conventional implementations of generative neural networks, the convolutions, upsampling, downsampling, and pointwise nonlinear functions in layer 200 provide the generative neural network 100 with reduced or no aliasing operations.

[0078] Parallel processing architecture

[0079] Figure 4 illustrates a parallel processing unit (PPU) 400 according to one embodiment. PPU 400 can be used to implement method 250. PPU 400 can be used to implement generative neural network 100 and / or layer 200. In one embodiment, a processor such as PPU 400 can be configured to implement a neural network model. The neural network model can be implemented as software instructions executed by the processor, or in other embodiments, the processor can include a matrix of hardware elements configured to process a set of inputs (e.g., electrical signals representing values) to generate a set of outputs that can represent activations of the neural network model. In other embodiments, the neural network model can be implemented as a combination of processing and software instructions executed by the hardware element matrix. Implementing a neural network model can include determining a set of parameters for the neural network model through, for example, supervised or unsupervised training of the neural network model, and, alternatively, performing inference using that parameter set to process new input sets.

[0080] In one embodiment, PPU 400 is a multi-threaded processor implemented on one or more integrated circuit devices. PPU 400 is a latency-hiding architecture designed to process many threads in parallel. A thread (e.g., an execution thread) is an instantiation of an instruction set configured to be executed by PPU 400. In one embodiment, PPU 400 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device. In other embodiments, PPU 400 may be used to perform general-purpose computing. While an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that such a processor is illustrated for illustrative purposes only, and any processor may be employed to complement and / or replace this processor.

[0081] One or more PPU 400s can be configured to accelerate thousands of high-performance computing (HPC), data center, cloud computing, and machine learning applications. PPU 400s can be configured to accelerate numerous deep learning systems and applications used in autonomous vehicles, simulations, computational graphics such as ray or path tracing, deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0082] As shown in Figure 4, PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a job allocation unit 425, a hub 430, a crossbar (Xbar) 470, one or more general purpose processing clusters (GPCs) 450, and one or more memory partitioning units 480. PPU 400 can be connected to a host processor or other PPU 400 via one or more high-speed NVLink 410 interconnects. PPU 400 can be connected to a host processor or other peripheral devices via interconnect 402. PPU 400 can also be connected to local memory 404, which includes multiple memory devices. In one embodiment, local memory may include multiple dynamic random access memory (DRAM) devices. The DRAM devices may be configured as a high-bandwidth memory (HBM) subsystem, where multiple DRAM dies are stacked within each device.

[0083] The NVLink 410 interconnect enables the system to expand and include one or more PPUs 400 in conjunction with one or more CPUs, supporting cache coherency between the PPUs 400 and the CPU, as well as the CPU controller. Data and / or commands can be sent from or from the NVLink 410 to other units of the PPU 400 via hub 430, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 5B A more detailed description of the NVLink 410.

[0084] I / O unit 405 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 402. I / O unit 405 may communicate directly with the host processor via interconnect 402, or via one or more intermediate devices such as memory bridges. In one embodiment, I / O unit 405 may communicate with one or more other processors, such as one or more PPUs 400, via interconnect 402. In one embodiment, I / O unit 405 implements a Peripheral Component Interconnect High Speed ​​(PCIe) interface for communication via a PCIe bus, and interconnect 402 is a PCIe bus. In alternative embodiments, I / O unit 405 may implement other types of known interfaces for communication with external devices.

[0085] I / O unit 405 decodes data packets received via interconnect 402. In one embodiment, the data packets represent commands configured to cause PPU 400 to perform various operations. I / O unit 405 transmits the decoded commands to various other units of PPU 400 that these commands may specify. For example, some commands may be transmitted to front-end unit 415. Other commands may be transmitted to hub 430 or other units of PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 405 is configured to route communication between and among the various logical units of PPU 400.

[0086] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. The workload may include instructions and data to be processed by those instructions. The buffer is an area of ​​memory accessible (e.g., read / write) by both the host processor and the PPU 400. For example, I / O unit 405 may be configured to access a buffer in system memory connected to interconnect 402 via a memory request transmitted through interconnect 402. In one embodiment, the host processor writes a command stream to the buffer and then transmits a pointer to the start of the command stream back to the PPU 400. Front-end unit 415 receives pointers to one or more command streams. Front-end unit 415 manages the one or more streams, reads commands from these streams, and forwards the commands to the respective units of the PPU 400.

[0087] Front-end unit 415 is coupled to scheduler unit 420, which configures various GPCs 450 to process tasks defined by the one or more streams. Scheduler unit 420 is configured to track status information related to the various tasks managed by scheduler unit 420. Status can indicate which GPC 450 a task is assigned to, whether the task is active or inactive, the priority associated with the task, etc. Scheduler unit 420 manages the execution of multiple tasks on the one or more GPCs 450.

[0088] Scheduler unit 420 is coupled to job allocation unit 425, which is configured to dispatch tasks for execution on GPC 450. Job allocation unit 425 can track several scheduled tasks received from scheduler unit 420. In one embodiment, job allocation unit 425 manages a pending task pool and an active task pool for each GPC 450. When GPC 450 completes the execution of a task, the task is evicted from the active task pool of GPC 450, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 450. If an active task on GPC 450 is idle, for example, while waiting for data dependencies to be resolved, then the active task can be evicted from GPC 450 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 450.

[0089] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API), enabling the execution of one or more applications on the host processor to schedule operations for execution on the PPU 400. In one embodiment, multiple computing applications are executed concurrently by the PPU 400, and the PPU 400 provides isolation, Quality of Service (QoS), and independent address spaces for the multiple computing applications. Applications can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 400. The driver kernel outputs the tasks to one or more streams being processed by the PPU 400. Each task may include one or more associated thread groups, referred to herein as warps. In one embodiment, a warp includes 32 associated threads that can execute in parallel. Cooperative threads may refer to multiple threads that include instructions for executing tasks and can exchange data via shared memory. Tasks may be assigned to one or more processing units within the GPC 450, and instructions are scheduled for execution by at least one warp.

[0090] The work allocation unit 425 communicates with one or more GPCs 450 via an XBar (crossbar switch) 470. The XBar 470 is an interconnect network that couples a number of units of the PPU 400 to other units of the PPU 400. For example, the XBar 470 can be configured to couple the work allocation unit 425 to a specific GPC 450. Although not explicitly shown, one or more other units of the PPU 400 can also be connected to the XBar 470 via a hub 430.

[0091] Tasks are managed by scheduler unit 420 and dispatched to GPC 450 by work allocation unit 425. GPC 450 is configured to process tasks and generate results. Results may be consumed by other tasks within GPC 450, routed to different GPC 450 via XBar 470, or stored in memory 404. Results may be written to memory 404 via memory partitioning unit 480, which implements a memory interface for reading data from and writing data to memory 404. Results may be transferred to another PPU 400 or CPU via NVLink 410. In one embodiment, PPU 400 includes U memory partitioning units 480, which is equal to the number of independent and distinct memory devices coupled to memory 404 of PPU 400. Each GPC 450 may include a memory management unit to provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit provides one or more translation back buffers (TLBs) for performing translations from virtual addresses to physical addresses in memory 404.

[0092] In one embodiment, memory partitioning unit 480 includes a raster operation (ROP) unit, a secondary (L2) cache, and a memory interface coupled to memory 404. The memory interface can implement 32, 64, 128, or 1024-bit data buses for high-speed data transfer. PPU 400 can connect to up to Y memory devices, such as high-bandwidth memory stacks or graphics dual-data-rate version 5 synchronous dynamic random access memory or other types of persistent memory. In one embodiment, the memory interface implements an HBM2 memory interface, and Y equals half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 400, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies, and Y equals 4, wherein each HBM2 stack includes two 128-bit channels per die, for a total of eight channels and a 1024-bit data bus width.

[0093] In one embodiment, memory 404 supports single-error-corrected double-error-detection (SECDED) error-correcting codes (ECC) to protect data. ECC provides enhanced reliability for computational applications sensitive to data corruption. In large-scale cluster computing environments where the PPU 400 handles very large datasets and / or long-running applications, reliability is particularly critical.

[0094] In one embodiment, PPU 400 implements a multi-level memory hierarchy. In one embodiment, memory partitioning unit 480 supports unified memory to provide a single, unified virtual address space for the CPU and PPU 400 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 400 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 400 that accesses the page more frequently. In one embodiment, NVLink 410 supports an address translation service that allows PPU 400 direct access to the CPU's page tables and provides full access to the CPU's memory by the PPU 400.

[0095] In one embodiment, the replication engine transfers data between multiple PPUs 400 or between a PPU 400 and a CPU. The replication engine can generate page faults for addresses not mapped to page tables. Memory partitioning unit 480 can then service the page faults, mapping the addresses to page tables, after which the replication engine can perform the transfer. In conventional systems, multiple replication engines operating on fixed memory (e.g., non-pageable) across multiple processors significantly reduce available memory. Due to hardware paging faults, addresses can be passed to the replication engine without concern for whether memory pages reside, and the replication process is transparent.

[0096] Data from memory 404 or other system memory can be retrieved by memory partitioning unit 480 and stored in L2 cache 460, which is located on-chip and shared among the various GPCs 450. As shown, each memory partitioning unit 480 includes a portion of the L2 cache 460 associated with the corresponding memory 404. Lower-level caches can then be implemented in multiple units within the GPC 450. For example, each processing unit within the GPC 450 can implement a Level 1 (L1) cache. The L1 cache is a dedicated memory for a specific processing unit. The L2 cache 460 is coupled to memory interface 470 and XBar 470, and data from the L2 cache can be retrieved and stored in each L1 cache for processing.

[0097] In one embodiment, the processing unit within each GPC 450 implements a SIMD (Single Instruction, Multiple Data) architecture, where each thread in a thread group (e.g., a warp) is configured to process different datasets based on the same instruction set. All threads in the thread group execute the same instructions. In another embodiment, the processing unit implements a SIMT (Single Instruction, Multiple Threads) architecture, where each thread in a thread group is configured to process different datasets based on the same instruction set, but where individual threads in the thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each thread bundle, enabling concurrency between the thread bundle and serial execution within the thread bundle when threads within the thread bundle diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, enabling equal concurrency among all threads within and between thread bundles. When maintaining an execution state for each individual thread, threads executing the same instructions can converge and execute in parallel for maximum efficiency.

[0098] Collaboration groups are a programming model for organizing groups of communicating threads. They allow developers to express the granularity at which threads are communicating, enabling richer and more efficient parallel decompositions. The Collaboration Startup API supports synchronization between blocks of threads to execute parallel algorithms. Conventional programming models provide a single, simple structure for synchronizing collaborative threads: a barrier across all threads in a block (e.g., the `syncthreads()` function). However, programmers often want to define thread groups at a granularity smaller than that of thread blocks and synchronize within those groups, enabling higher performance, design flexibility, and software reuse through a collective group-wide function interface.

[0099] Collaboration groups enable programmers to explicitly define thread groups at both the sub-block (e.g., as small as a single thread) and multi-block granularity levels and perform collective operations, such as synchronization on threads within a collaboration group. The programming model supports clean composition across software boundaries, allowing libraries and utility functions to be safely synchronized in their local environments without making assumptions about convergence. Collaboration group primitives enable new modes of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire mesh of thread blocks.

[0100] Each processing unit comprises a large number (e.g., 128, etc.) of different processing cores 442 (e.g., functional units), which may be fully pipelined, single-precision, double-precision, and / or mixed-precision, and include floating-point arithmetic logic units and integer arithmetic logic units. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the cores include 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0101] Tensor cores are configured to perform matrix operations. Specifically, tensor cores are configured to perform deep learning matrix operations, such as GEMM (matrix-matrix multiplication) used for convolution operations during neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs matrix multiplication and accumulation operations D = A×B + C, where A, B, C, and D are 4×4 matrices.

[0102] In one embodiment, matrix multiplication inputs A and B can be integer, fixed-point, or floating-point matrices, while accumulation matrices C and D can be integer, fixed-point, or floating-point matrices of equal or higher bit width. In one embodiment, the Tensor Core operates on one-, four-, or eight-bit integer input data with 32-bit integer accumulation. An 8-bit integer matrix multiplication requires 1024 operations and produces a full-precision product, which is then accumulated using 32-bit integer addition with other intermediate products to achieve an 8x8x16 matrix multiplication. In one embodiment, the Tensor Core operates on 16-bit floating-point input data with 32-bit floating-point accumulation. A 16-bit floating-point multiplication requires 64 operations, produces a full-precision product, and is then accumulated using 32-bit floating-point addition with other intermediate products from a 4×4×4 matrix multiplication. In practice, the Tensor Core is used to perform larger two-dimensional or higher-dimensional matrix operations built from these smaller elements. APIs (such as the CUDA 9 C++ API) expose specialized matrix loading, matrix multiplication and accumulation, and matrix storage operations to efficiently utilize the tensor core from CUDA-C++ programs. At the CUDA level, the thread bundle-level interface assumes that a 16×16 size matrix spans all 32 threads of the thread bundle.

[0103] Each processing unit may also include M Special Function Units (SFUs) that perform special functions (e.g., attribute evaluation, inverse square root, etc.). In one embodiment, an SFU may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, an SFU may include a texture unit configured to perform texture map filtering operations. In one embodiment, a texture unit is configured to load a texture map (e.g., a 2D array of texture pixels) from memory 404 and sample the texture map to produce sampled texture values ​​for use in a shader program executed by the processing unit. In one embodiment, the texture map is stored in shared memory, which may contain or include an L1 cache. The texture unit implements texture operations, such as filtering operations using mip maps (i.e., texture maps at different levels of detail). In one embodiment, each processing unit includes two texture units.

[0104] Each processing unit also includes N Load Memory Units (LSUs) that implement load and store operations between shared memory and the register file. Each processing unit includes an interconnect network that connects each core to the register file and the LSUs to the register file and shared memory. In one embodiment, the interconnect network is a crossbar switch that can be configured to connect any core to any register in the register file and the LSUs to the register file and memory locations in shared memory.

[0105] Shared memory is an on-chip memory array that allows data storage and communication between processing units and between threads within a processing unit. In one embodiment, shared memory includes 128KB of storage capacity and is located on the path from each processing unit to memory partition unit 480. Shared memory can be used for cache reads and writes. One or more of shared memory, L1 cache, L2 cache, and memory 404 are backup storage.

[0106] Combining data caching and shared memory functionality into a single memory block provides optimal overall performance for both types of memory access. This capacity can be used by the program as a cache that does not utilize shared memory. For example, if shared memory is configured to use half its capacity, texture and load / store operations can use the remaining capacity. Integration within shared memory allows it to function as a high-throughput pipeline for streaming data, while simultaneously providing high-bandwidth and low-latency access to frequently reused data.

[0107] When configured for general-purpose parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, fixed-function graphics processing units are bypassed, creating a simpler programming model. In the general-purpose parallel computing configuration, the work allocation unit 425 directly assigns and distributes thread blocks to processing units within the GPC 450. Threads execute the same program, using unique thread IDs in the computation to ensure each thread produces unique results, using one or more processing units to execute programs and perform computations, using shared memory for communication between threads, and using LSUs to read from and write to global memory via shared memory and memory partitioning unit 480. When configured for general-purpose parallel computing, processing units can also write commands to the scheduler unit 420 to start new work on the processing unit.

[0108] Each PPU 400 may include, and / or be configured to perform the functions of one or more processing cores and / or components thereof, such as a tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), ray tracing (RT) core, vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, and so on.

[0109] The PPU 400 can be included in desktop computers, laptop computers, tablet computers, servers, supercomputers, smartphones (e.g., wireless, handheld devices), personal digital assistants (PDAs), digital cameras, vehicles, head-mounted displays, handheld electronic devices, etc. In one embodiment, the PPU 400 is contained on a single semiconductor substrate. In another embodiment, the PPU 400 is included on a system-on-a-chip (SoC) along with one or more other devices, such as an additional PPU 400, memory 404, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.

[0110] In one embodiment, PPU 400 may be included on a graphics card, which includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on a desktop computer's motherboard. In yet another embodiment, PPU 400 may be an integrated graphics processing unit (iGPU) or a parallel processor included in a chipset of the motherboard. In yet another embodiment, PPU 400 may be implemented in reconfigurable hardware. In yet another embodiment, a portion of PPU 400 may be implemented in reconfigurable hardware.

[0111] Exemplary computing system

[0112] Systems with multiple GPUs and CPUs are being used across various industries as developers expose to and leverage greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of compute nodes are being deployed in data centers, research institutions, and supercomputers to tackle larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to be scaled to support the increased bandwidth.

[0113] Figure 5A This is a conceptual diagram of a processing system 500 implemented using the PPU 400 of FIG4 according to one embodiment. The exemplary system 500 can be configured to implement... Figure 2B The method 250 shown is described. The processing system 500 includes a CPU 530, a switch 510, and multiple PPUs 400 and corresponding memory 404.

[0114] The NVLink 410 provides a high-speed communication link between each PPU 400. Although Figure 5B A specific number of NVLink 410 and interconnect 402 connections are shown, but the number of connections to each PPU 400 and CPU 530 can vary. Switch 510 interfaces between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLink 410 can reside on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols that interface between various different connections and / or links.

[0115] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and CPU 530, and switch 510 interfaces between interconnect 402 and each PPU 400. PPU 400, memory 404, and interconnect 402 may reside on a single semiconductor platform to form parallel processing module 525. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 and CPU 530, and switch 510 uses NVLink 410 to interface between each PPU 400 to provide one or more high-speed communication links between PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between PPUs 400 and CPU 530 via switch 510. In yet another embodiment (not shown), interconnect 402 directly provides one or more communication links between each PPU 400. One or more NVLink 410 high-speed communication links can be implemented as physical NVLink interconnects or on-chip or die interconnects using the same protocol as NVLink 410.

[0116] In the context of this specification, a single semiconductor platform can refer to a unique, single semiconductor-based integrated circuit fabricated on a bare die or chip. It should be noted that the term single semiconductor platform can also refer to a multi-chip module with increased connectivity, simulating on-chip operation and substantially improved by utilizing conventional bus implementation methods. Of course, various circuits or devices can also be placed separately or in various combinations of semiconductor platforms, depending on the user's needs. Optionally, the parallel processing module 525 can be implemented as a circuit board substrate, and each of the PPU 400 and / or memory 404 can be a packaged device. In one embodiment, the CPU 530, switch 510, and parallel processing module 525 reside on a single semiconductor platform.

[0117] In one embodiment, the signaling rate of each NVLink 410 is 20 to 25 gigabits per second, and each PPU400 includes six NVLink 410 interfaces (e.g., Figure 5A As shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 gigabits per second in each direction, with six links providing 400 gigabits per second. When the CPU 530 also includes one or more NVLink 410 interfaces, the NVLink 410 can be specifically used for applications such as... Figure 5A The example shows PPU-to-PPU communication, or a combination of PPU-to-PPU and PPU-to-CPU communication.

[0118] In one embodiment, NVLink 410 allows direct load / store / atomic access from CPU 530 to memory 404 of each PPU 400. In one embodiment, NVLink 410 supports coherent operation, allowing data read from memory 404 to be stored in the cache hierarchy of CPU 530, reducing cache access latency of CPU 530. In one embodiment, NVLink 410 includes support for Address Translation Service (ATS), allowing PPU 400 to directly access page tables within CPU 530. One or more NVLink 410s can also be configured to operate in a low-power mode.

[0119] Figure 5B An exemplary system 565 is illustrated, in which various architectures and / or functions of various prior embodiments can be implemented. The exemplary system 565 can be configured to implement... Figure 2B Method 250 is shown.

[0120] As shown in the figure, a system 565 is provided, which includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 may directly or indirectly couple one or more of the following devices: main memory 540, network interface 535, CPU 530, display device 545, input device 560, switch 510, and parallel processing system 525. The communication bus 575 may be implemented using any suitable protocol and may represent one or more links or buses, such as address bus, data bus, control bus, or combinations thereof. The communication bus 575 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect High Speed ​​(PCIe) bus, HyperTransport, and / or another type of bus or link. In some embodiments, direct connections exist between components. As an example, CPU 530 may be directly connected to main memory 540. Furthermore, CPU 530 may be directly connected to parallel processing system 525. In cases where there is a direct or point-to-point connection between components, the communication bus 575 may include a PCIe link that implements the connection. In these examples, the PCI bus need not be included in the system 565.

[0121] Although using lines Figure 5BThe different blocks are shown connected via a communication bus 575, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component such as a display device 545 can be considered an I / O component, such as an input device 560 (e.g., if the display is a touchscreen). As another example, the CPU 530 and / or the parallel processing system 525 may include memory (e.g., main memory 540 may represent storage devices other than the parallel processing system 525, the CPU 530, and / or other components). In other words, Figure 5B The term "computing device" is merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all of these are expected to fall under [the relevant category]. Figure 5B Within the scope of computing devices.

[0122] System 565 also includes main memory 540. Control logic (software) and data are stored in main memory 540, which can take the form of a variety of computer-readable media. Computer-readable media can be any available medium that can be accessed by system 565. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. For example and without limitation, computer-readable media can include computer storage media and communication media.

[0123] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, main memory 540 may store computer-readable instructions such as an operating system (e.g., representing programs and / or program elements). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic cartridges, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that may be used to store desired information and that can be accessed by system 565. When used herein, computer storage media does not include the signal itself.

[0124] Computer storage media may contain computer-readable instructions, data structures, program modules, or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information transport medium. The term "modulated data signal" may refer to a signal whose characteristics are set or altered in a manner that encodes information into that signal. For example and without limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0125] When executed, the computer program enables system 565 to perform various functions. CPU 530 may be configured to execute at least some of the computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Each of CPUs 530 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing numerous software threads simultaneously. Depending on the type of system 565 implemented, CPU 530 may include any type of processor and may include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of system 565, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, system 565 may include one or more CPUs 530.

[0126] In addition to or alternatively to CPU 530, parallel processing module 525 may be configured to execute at least some of the computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Parallel processing module 525 may be used by system 565 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, parallel processing module 525 may be used for general-purpose computing on a GPU (GPGPU). In embodiments, CPU 530 and / or parallel processing module 525 may execute any combination of the methods, processes, and / or portions thereof, discretely or jointly.

[0127] System 565 also includes input device 560, parallel processing system 525, and display device 545. Display device 545 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. Display device 545 may receive data from other components (e.g., parallel processing system 525, CPU 530, etc.) and output that data (e.g., images, video, sound, etc.).

[0128] Network interface 535 enables system 565 to be logically coupled to other devices, including input device 560, display device 545, and / or other components, some of which may be embedded (e.g., integrated into) system 565. Illustrative input device 560 includes microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. Input device 560 can provide a natural user interface (NUI) that processes user-generated air gestures, voice, or other physiological input. In some instances, input can be transmitted to appropriate network elements for further processing. NUI can implement voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head-eye tracking, and touch recognition associated with the display of system 565 (described in more detail below). System 565 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. In addition, system 565 may include an accelerometer or gyroscope that allows motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by system 565 to render immersive augmented reality or virtual reality.

[0129] Furthermore, system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) via network interface 535 for communication purposes. System 565 can be included in a distributed network and / or cloud computing environment.

[0130] Network interface 535 may include one or more receivers, transmitters, and / or transceivers, enabling system 565 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. Network interface 535 may be implemented as a network interface controller (NIC) including one or more data processing units (DPUs) to perform operations such as (e.g., but not limited to) packet parsing and accelerating network processing and communication. Network interface 535 may include components and functions that allow communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0131] System 565 may also include an auxiliary storage device (not shown). The auxiliary storage device includes, for example, a hard disk drive and / or a removable storage drive representing a floppy disk drive, magnetic tape drive, compact disc drive, digital versatile disc (DVD) drive, recording device, Universal Serial Bus (USB) flash memory. The removable storage drive reads from and / or writes to the removable storage unit in a well-known manner. System 565 may also include a hard-wired power supply, a battery power supply, or a combination thereof (not shown). This power supply can supply power to System 565 to enable the components of System 565 to operate.

[0132] Each of the aforementioned modules and / or devices may even reside on a single semiconductor platform to form system 565. Alternatively, various different modules may be placed individually or located in various combinations of semiconductor platforms as desired by the user. Although various different embodiments have been described above, it should be understood that they are given by way of example only and without limitation. Therefore, the breadth and scope of preferred embodiments should not be limited to any of the exemplary embodiments described above, but should be defined only by the following claims and their equivalents.

[0133] Example network environment

[0134] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage devices (NAS), other back-end devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 5A Processing system 500 and / or Figure 5B Implemented on one or more instances of the exemplary system 565, for example, each device may include similar components, features and / or functions of the processing system 500 and / or the exemplary system 565.

[0135] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or a combination of both. A network can include multiple networks or a network of networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks—such as the Internet, and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (along with other components) can provide wireless connectivity.

[0136] A compatible network environment may include one or more peer-to-peer network environments—in which case the server may not be included in the network environment—and one or more client-server network environments—in which case one or more servers may be included in the network environment. In a peer-to-peer network environment, the functionality described herein regarding the server can be implemented on any number of client devices.

[0137] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework of one or more applications supporting a software layer and / or an application layer. The software or application may include web-based service software or applications, respectively. In embodiments, one or more client devices may use the web-based service software or application (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software web application framework type that can be used for large-scale data processing (e.g., "big data").

[0138] A cloud-based network environment can provide cloud computing and / or cloud storage to implement the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., a central or core server in one or more data centers, which may be distributed across states, regions, countries, globally, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, then the core server can assign at least a portion of the functions to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0139] Client devices may include Figure 5A Example processing system 500 and / or Figure 5B At least some of the components, features, and functions of the exemplary system 565. For example and without limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronics device, workstation, edge device, any combination of these defined devices, or any other suitable device.

[0140] Machine Learning

[0141] Deep neural networks (DNNs) developed on processors such as the PPU 400 have been used in a wide variety of use cases, from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technique that models the neural learning process of the human brain, which continuously learns, becomes smarter, and delivers more accurate results faster over time. Just as a child is initially taught by adults to correctly identify and classify various shapes, eventually becoming able to identify shapes without any guidance, a deep learning or neural learning system needs to be trained in object recognition and classification so that it becomes smarter and more efficient at identifying basic objects, occluded objects, and so on, while also attaching context to objects.

[0142] At its simplest level, neurons in the human brain receive various inputs, assigning a level of importance to each of these inputs, and the output is passed to other neurons to make a response. Artificial neurons, or perceptrons, are the most basic model of neural networks. In one example, a perceptron can receive one or more inputs representing various features of objects that the perceptron is being trained to recognize and classify, and each of these features is assigned a weight based on its importance in defining the shape of the object.

[0143] Deep neural network (DNN) models consist of multiple layers of numerous connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) that can be trained on massive amounts of input data to solve complex problems quickly and with high accuracy. In one example, the first layer of a DNN model breaks down an input image of a car into different segments and searches for basic patterns such as lines and angles. The second layer assembles these lines to find higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final few layers generate labels for the input image that identify the model of a specific car brand.

[0144] Once trained, a DNN can be deployed and used to identify and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include identifying handwritten digits on a check deposited into an ATM, identifying images of friends in a photograph, delivering movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human language in real time.

[0145] During training, data flows through the DNN in the forward propagation phase until a prediction is produced indicating the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and the weights are adjusted for each feature during the backpropagation phase until the DNN correctly labels the input as well as other inputs in the training dataset. Training complex neural networks requires significant parallel computing power, including floating-point multiplication and addition supported by a PPU400. Inference is less computationally intensive than training and is a latency-sensitive process where the trained neural network is applied to new inputs it has not seen before for tasks such as image classification, sentiment detection, label recommendation, language recognition and translation, and typically infers new information.

[0146] Neural networks rely heavily on matrix operations, and complex, multi-layered networks require massive amounts of floating-point performance and bandwidth for both efficiency and speed. Leveraging thousands of processing cores optimized for matrix operations and delivering tens to hundreds of TFLOPS of performance, the PPU 400 is a computing platform capable of providing the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0147] Furthermore, images generated using one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for recognizing real-world objects and environments. Such images can include driveways, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or scenes of real-world environments. Such images can be used to train, test, or certify DNNs used in machines or robots to manipulate, process, or modify real-world physical objects. Additionally, such images can be used to train, test, or certify DNNs used in autonomous vehicles to navigate and move vehicles in the real world. Furthermore, images generated using one or more of the techniques disclosed herein can be used to communicate information to users of such machines, robots, and vehicles.

[0148] Figure 5C Components of an example system 555, which can be used to train and utilize machine learning according to at least one embodiment, are illustrated. As will be discussed, various components can be provided by a single computing system or various combinations of computing devices and resources, which may be under the control of a single entity or multiple entities. Furthermore, aspects may be triggered, initiated, or requested by different entities. In at least one embodiment, the training of the neural network may be guided by a vendor associated with vendor environment 506, while in at least one embodiment, training may be requested by a customer or other user who can access the vendor environment through client device 502 or other such resources. In at least one embodiment, training data (or data to be analyzed by the trained neural network) may be provided by a vendor, user, or third-party content provider 524. In at least one embodiment, client device 502 may be, for example, a vehicle or object to be navigated on behalf of a user, who can submit requests and / or receive instructions that aid in device navigation.

[0149] In at least one embodiment, a request can be submitted via at least one network 504 for receipt by a vendor environment 506. In at least one embodiment, the client device can be any suitable electronic and / or computing device that enables a user to generate and send such requests, such as, but not limited to, desktop computers, laptop computers, computer servers, smartphones, tablets, game consoles (portable or otherwise), computer processors, computing logic, and set-top boxes. One or more networks 504 can include any suitable network for transmitting requests or other such data, such as the Internet, intranet, Ethernet, cellular network, local area network (LAN), wide area network (WAN), personal area network (PAN), self-organizing network providing direct wireless connectivity between peers, etc.

[0150] In at least one embodiment, a request may be received at interface layer 508, which in this example may forward data to training and inference manager 532. Training and inference manager 532 may be a system or service including hardware and software for managing services and requests corresponding to data or content. In at least one embodiment, training and inference manager 532 may receive a request to train a neural network and may provide data for the request to training module 512. In at least one embodiment, if the request is not specified, training module 512 may select an appropriate model or neural network to use and may train the model using the associated training data. In at least one embodiment, training data may be a batch of data stored in training data repository 514, received from client device 502, or obtained from third-party vendor 524. In at least one embodiment, training module 512 may be responsible for training the data. The neural network may be any suitable network, such as a recurrent neural network (RNN) or convolutional neural network (CNN). Once the neural network is trained and successfully evaluated, the trained neural network may be stored in, for example, model repository 516, which may store different models or networks for users, applications, or services, etc. In at least one embodiment, there may be multiple models for a single application or entity, which can be utilized based on multiple different factors.

[0151] In at least one embodiment, at a subsequent point in time, a request for content (e.g., path determination) or data that is at least partially determined or influenced by a trained neural network can be received from client device 502 (or another such device). This request may include, for example, input data to be processed using the neural network to obtain one or more inference or other output values, classifications, or predictions. Alternatively, in at least one embodiment, the input data may be received by interface layer 508 and directed to inference module 518, although different systems or services may also be used. In at least one embodiment, if not already locally stored in inference module 518, inference module 518 may obtain a suitably trained network, such as a trained deep neural network (DNN) as discussed herein, from model repository 516. Inference module 518 may provide data as input to the trained network, which may then generate one or more inferences as outputs. This may, for example, include the classification of instances of input data. In at least one embodiment, the inference may then be transmitted to client device 502 for display to a user or for other communication with the user. In at least one embodiment, the user's context data may also be stored in a user context data repository 522, which may include data about the user that can be used as network input to generate inference or determine data to be returned to the user after obtaining an instance. In at least one embodiment, relevant data, which may include at least some of the input or inference data, may also be stored in a local database 534 for processing future requests.

[0152] In at least one embodiment, a user may use account information or other information to access resources or functions of the vendor environment. In at least one embodiment, user data may also be collected and used to further train the model, if permitted and available, to provide more accurate inference for future requests. In at least one embodiment, requests to the machine learning application 526 executed on client device 502 may be received via a user interface, and the results may be displayed via the same interface. The client device may include resources such as a processor 528 and memory 562 for generating requests and processing results or responses, and at least one data storage element 552 for storing data for the machine learning application 526.

[0153] In at least one embodiment, processor 528 (or the processor of training module 512 or inference module 518) will be a central processing unit (CPU). However, as mentioned above, resources in such an environment can utilize GPUs to process data for at least some types of requests. GPUs, such as the PPU 400, have thousands of cores and are designed to handle large amounts of parallel workloads, thus becoming popular in deep learning for training neural networks and generating predictions. While using GPUs for offline building allows for faster training of larger, more complex models, offline prediction generation means that request-time input features cannot be used, or predictions must be generated for all features and stored in a lookup table for real-time service requests. If the deep learning framework supports CPU mode and the model is small and simple enough that the feedforward can be performed on the CPU with reasonable latency, then a service on a CPU instance can host the model. In this case, training can be done offline on the GPU and inference can be performed in real-time on the CPU. If the CPU approach is not feasible, the service can run on a GPU instance. However, due to the different performance and cost characteristics of GPUs compared to CPUs, running a service that offloads runtime algorithms to the GPU may require it to be designed differently from a CPU-based service.

[0154] In at least one embodiment, video data can be provided from client device 502 for enhancement in vendor environment 506. In at least one embodiment, the video data can be processed for enhancement on client device 502. In at least one embodiment, the video data can be streamed from third-party content provider 524 and enhanced by third-party content provider 524, vendor environment 506, or client device 502. In at least one embodiment, video data can be provided from client device 502 for use as training data in vendor environment 506.

[0155] In at least one embodiment, supervised and / or unsupervised training can be performed by client device 502 and / or vendor environment 506. In at least one embodiment, a set of training data 514 (e.g., classified or labeled data) is provided as input for use as training data. In one embodiment, this training dataset can be used to train a generative neural network in a generative adversarial training configuration.

[0156] In at least one embodiment, the training data may include images of at least one human subject, avatar, or character against which the neural network will be trained. In at least one embodiment, the training data may include instances of at least one type of object to which the neural network is to be trained, and information identifying that object type. In at least one embodiment, the training data may include a set of images, each image including a representation of an object of a type, wherein each image also includes, or is associated with, tags, metadata, classifications, or other information identifying or identifying the type of object represented in the corresponding image. Various other types of data may also be used as training data, which may include text data, audio data, video data, and so on. In at least one embodiment, training data 514 is provided as training input to training module 512. In at least one embodiment, training module 512 may be a system or service including hardware and software, such as one or more computing devices executing a training application for training a neural network (or other model or algorithm, etc.). In at least one embodiment, training module 512 receives instructions or requests indicating the type of model to be used for training; in at least one embodiment, the model may be any suitable statistical model, network, or algorithm useful for such a purpose, which may include artificial neural networks, deep learning algorithms, learning classifiers, Bayesian networks, and so on. In at least one embodiment, training module 512 may select an initial model or other untrained model from an appropriate repository and train the model using training data 514 to generate a trained model (e.g., a trained deep neural network) that can be used to classify similar types of data or generate other such inference. In at least one embodiment where training data is not used, an initial model may still be selected to train each training module 512 on its input data.

[0157] In at least one embodiment, the model can be trained in several different ways, which may depend in part on the type of model chosen. In at least one embodiment, a training dataset can be provided to a machine learning algorithm, wherein the model is a model artifact created through a training process. In at least one embodiment, each instance of the training data contains the correct answer (e.g., classification) that may be referred to as the target or target attribute. In at least one embodiment, the learning algorithm finds patterns in the training data that map input data attributes to the target—the answer to be predicted—and the machine learning model is the output that captures these patterns. In at least one embodiment, the machine learning model can then be used to obtain predictions for new data without a specified target.

[0158] In at least one embodiment, the training and inference manager 532 may select from a set of machine learning models, including binary classification, multi-class classification, generative, and regression models. In at least one embodiment, the type of model to be used may depend at least in part on the type of target to be predicted.

[0159] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device may be directly coupled to the system or processor that generates or renders the image. In other embodiments, the display device may be indirectly coupled to the system or processor, for example, via a network. Examples of such networks include the Internet, mobile telecommunications networks, Wi-Fi networks, and any other wired and / or wireless networking systems. When the display device is indirectly coupled, images generated by the system or processor can be streamed to the display device over the network. Such streaming allows, for example, video games or other applications that render images to execute on servers, data centers, or cloud-based computing environments, and the rendered images are transmitted and displayed on one or more user devices (e.g., computers, video game consoles, smartphones, other mobile devices, etc.) physically separate from the server or data center. Therefore, the techniques disclosed herein can be applied to enhance streamed images and services that stream images, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0160] Example Streaming System

[0161] Figure 6 This is an example system diagram of a streaming system 605 according to some embodiments of the present disclosure. Figure 6 Includes server 603 (which may include with Figure 5A Example processing system 500 and / or Figure 5B (Similar components, features and / or functions to exemplary system 565), client 604 (which may include similar ... Figure 5A Example processing system 500 and / or Figure 5B The exemplary system 565 has similar components, features, and / or functions to the network 606 (which may be similar to the network described herein). In some embodiments of this disclosure, system 605 may be implemented.

[0162] In one embodiment, the streaming system 605 is a game streaming system, and the server 604 is a game server. In system 605, for a game session, the client device 604 can simply receive input data in response to input from the input device 626, send the input data to the server 603, receive encoded display data from the server 603, and display the display data on the display 624. In this way, computationally intensive computation and processing are offloaded to the server 603 (e.g., rendering of the game session's graphics output, especially ray or path tracing, is performed by the GPU 615 of the server 603). In other words, the game session is streamed from the server 603 to the client device 604, thereby reducing the demands on the client device 604 for graphics processing and rendering.

[0163] For example, regarding the instantiation of a game session, client device 604 can display frames of the game session on display 624 based on display data received from server 603. Client device 604 can receive input from one of input devices 626 and generate input data in response. Client device 604 can send the input data to server 603 via communication interface 621 and through network 606 (e.g., the Internet), and server 603 can receive the input data via communication interface 618. CPU 608 can receive the input data, process the input data, and send the data to GPU 615, which causes GPU 615 to generate a rendering of the game session. For example, the input data can represent the movement of a user character in the game, such as firing a weapon, reloading, passing a ball, turning a vehicle, etc. Rendering component 612 can render the game session (e.g., representing the result of the input data), and rendering capture component 614 can capture the rendering of the game session as display data (e.g., image data as frames of the captured game session rendering). The rendering of a game session may include lighting and / or shadow effects calculated using one or more parallel processing units of server 603 (e.g., a GPU, which may further employ one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques). Encoder 616 can then encode the display data to generate encoded display data, which can be sent to client device 604 via communication interface 618 through network 606. Client device 604 can receive the encoded display data via communication interface 621, and decoder 622 can decode the encoded display data to generate display data. Client device 604 can then display the display data via display 624.

[0164] Fusion processing of continuous mathematical operators [wy1]

[0165] Systems and methods for fusion processing of sequential mathematical operators are disclosed. Fusion processing of sequential mathematical operations, such as pointwise nonlinear functions, without storing intermediate results in memory, improves performance when memory bus bandwidth is limited. In one embodiment, sequential mathematical operations including at least two of convolution, upsampling, pointwise nonlinear functions, and downsampling are performed to process input data and generate aliasing-free output data. In one embodiment, fusion processing is performed to implement an aliasing-free generative neural network, wherein intermediate results (e.g., upsampled feature maps) are stored in on-chip memory within the processor, such as shared memory / L1 cache 465, instead of external memory 304. Storing intermediate data in shared memory improves performance compared to storing intermediate data in and loading intermediate data from external memory. In one embodiment, input data is spatially partitioned for parallel processing, such that intermediate results generated during processing of each block of input data can be stored in shared memory.

[0166] Using primitive operations available in traditional deep learning frameworks to perform continuous mathematical operations, such as the upsampling-nonlinear-downsampling sequence of layer 200 or method 250, is inefficient because intermediate data is stored in and loaded from external memory. Intermediate data 232, resolution-increased intermediate data 237, and processed resolution-increased intermediate data 242 are typically transferred multiple times between internal (e.g., on-chip) and external (e.g., off-chip) memory. This transfer cost is particularly high because the pointwise nonlinear operator 240 and the downsampling unit 245 operate on the upsampled high-resolution data.

[0167] Instead of using traditional raw operations, layer 200 or method 250 can be implemented as a custom kernel that combines or "fusions" individual operations for processor execution, enabling the transfer of intermediate data from one operation to the next via shared memory within the processor. For example, the custom kernel could be a CUDA kernel executed by one or more SM 440s within the PPU 300. Compared to traditional processing, fusion processing by executing fused kernels improves training performance by reducing external memory traffic and also reduces external memory usage.

[0168] Figure 7A A conceptual diagram of a processing system for fusion processing of consecutive mathematical operators suitable for implementing some embodiments of this disclosure is shown. Layer 700 may include... Figure 2ALayer 200 includes one or more processing elements, wherein upsampler 735, pointwise nonlinear operator 740, and downsampler 745 respectively include upsampler 235, pointwise nonlinear operator 240, and downsampler 245. Similarly, input data 732, resolution-increased intermediate data 737, and processed resolution-increased intermediate data 742 may respectively include intermediate data 232, resolution-increased intermediate data 237, and processed resolution-increased intermediate data 242. In one embodiment, convolution operator 230 is also included in layer 700.

[0169] When the fusion kernel is executed by SM 440 to implement layer 700, each of the input data 732, the resolution-increased intermediate data 737, and the processed resolution-increased intermediate data 742 can be stored in or loaded from shared memory / L1 cache 465. In one embodiment, the input data 732 is loaded directly into a register accessible to the upsampler 735 and is not stored in the shared memory / L1 cache 465. In one embodiment, the convolution operator is fused with the upsampler 735, and the input data 732 output by the convolution operator is directly transferred to the upsampler 735 via a register without being stored in the shared memory / L1 cache 465. In one embodiment, the upsampler 735 is fused with a pointwise nonlinear operator 740, such that the resolution-increased intermediate data 737 is directly transferred to the pointwise nonlinear operator 740 via a register without being stored in the shared memory / L1 cache 465. In one embodiment, the pointwise nonlinear operator 740 is fused with the downsampler 745, such that the processed intermediate data 742 with increased resolution is directly transferred to the downsampler 745 via a register instead of being stored in the shared memory / L1 cache 465. In another embodiment, the downsampler 745 is fused with the storage of the processed intermediate data, such that the processed intermediate data output by the downsampler 745 is directly stored in the shared memory / L1 cache 465 instead of being stored in a register. In yet another embodiment, the downsampler 745 is fused with subsequent operators that process the processed intermediate data, so the processed intermediate data is directly transferred to the subsequent operators via a register instead of being stored in the shared memory / L1 cache 465.

[0170] In one embodiment, when at least one of the input data 732, the resolution-increased intermediate data 737, and the processed resolution-increased intermediate data 742 exceeds the capacity of the shared memory / L1 cache 465, the low-resolution input data can be divided into blocks for processing, such that the intermediate data 732, the resolution-increased intermediate data 737, and / or the processed resolution-increased intermediate data 742 of each block reside in the shared memory / L1 cache 465. In one embodiment, the input data is divided into at least partially overlapping blocks, such that input data within the perimeter of a first block is also included in adjacent blocks.

[0171] Figure 7B The diagram illustrates the segmentation of intermediate data suitable for implementing some embodiments of this disclosure. In one embodiment, the execution of the fusion kernel for approximate continuous mathematical operations is parallelized by subdividing target data 725 (e.g., output or intermediate data) into non-overlapping blocks and having each block of target data 725 computed by different thread blocks. For example, each of the target data tiles 720, 722, 724, and 726 is generated by a different thread block. Note that the size of the target data 725 is adjusted to include some data outside a conceptual perimeter (solid line) aligned with the generated output image.

[0172] Assuming source data 715 is the input data to be processed to generate target data 725, source data blocks 710, 712, 714, and 716 are defined corresponding to target data blocks 720, 722, 724, and 726, respectively. A first thread block executes a fusion kernel, loading source data block 710 into the internal shared memory allocated to the first thread block. In one embodiment, different thread blocks are allocated to execute the fusion kernel to process each of source data blocks 712, 724, and 716 and generate target data blocks 722, 724, and 726, respectively. In one embodiment, at least one thread is allocated to execute the fusion kernel to process two or more of source data blocks 712, 724, and 716 and generate two or more of target data blocks 722, 724, and 726, respectively. Note that due to the spatial range of the filter, for example... Figure 1E The sample filter 160 shown, each of the source data blocks 720, 722, 724, and 726, spatially overlaps with adjacent source data blocks. In one embodiment, the size of the target block is determined based on the amount of shared memory available for allocation to each thread block. The shared memory capacity per thread block should be equal to or greater than the required storage amount for at least one source data block and at least one of the source block input data, resolution-increased intermediate data, and processed resolution-increased intermediate data.

[0173] In one embodiment, when a separable filter is used for upsampler 735 and / or downsampler 745, intermediate results of the filtering operation (e.g., partially upsampled intermediate data) can be shared between different threads. Using a separable filter allows processing in a single dimension (horizontal or vertical), which has the advantage of requiring less computation to produce the same result for the entire block as processing in two dimensions. A number of registers or shared memory storing input data 732 and the single-dimensional upsampled input data can be reused multiple times to upsample the entire block. Similarly, a number of registers or shared memory storing the processed resolution-increased intermediate data 742 can be reused multiple times to downsample the entire block.

[0174] Figure 7C A flowchart of a method 750 for fusion processing of continuous mathematical operators suitable for implementing some embodiments of the present disclosure is shown. In one embodiment, the fusion processing approximates a continuous mathematical operator by processing a discrete representation of data. Each block of the method 750 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or a plug-in to another product, to name a few. Furthermore, as an example, for Figure 7A Layer 700 describes method 750. However, this method may be performed additionally or alternatively by any system or any combination of systems, including but not limited to those described herein. Furthermore, those skilled in the art will understand that any system performing method 750 is within the scope and spirit of the embodiments of this disclosure.

[0175] In step 755, the input data is stored in a memory comprising one or more memory devices. In one embodiment, the memory is external memory coupled to the processor via an interface. For example, in one embodiment, the processor includes a PPU 300 and the memory includes memory 304. The processor executes instructions to approximate continuous mathematical operators. In one embodiment, the processor implements a neural network model including at least one stage of processing the input data to generate output data. In one embodiment, the input data includes feature maps.

[0176] In step 760, the processor reads input data from the memory via an interface coupled to the memory. In one embodiment, the input data is transferred via the interface and stored in shared memory within the processor. In another embodiment, the input data is divided into blocks, and one or more blocks are transferred via the interface and stored in shared memory.

[0177] In step 765, the input data is upsampled to produce resolution-incremented intermediate data that approximates continuously increasing resolution intermediate data. In one embodiment, for 2× upsampling, zeros are interleaved with the input data in each dimension before the input data is filtered to produce resolution-incremented intermediate data. In one embodiment, the one-dimensional (1D) interleaved input data is convolved with a 2n-tap filter, where n non-zero input values ​​are under the 2n-tap filter. If each resolution-incremented intermediate data value (e.g., an output pixel) is computed separately, the convolution requires n multiply-accumulate operations per pixel and the same number of shared memory load instructions, for a total of 2n instructions per output pixel. However, note that the computation of two adjacent output pixels only accesses n+1 input pixels in total. By computed two output pixels at once and avoiding redundant shared memory load instructions, the average cost per pixel is [missing information]. This reduces the number of instructions by nearly 25%. For 4× upsampling, by calculating four output pixels at a time, the number of instructions can be similarly reduced by up to 37.5%. These optimizations can be applied to both 2× and 4× upsampling for separable and non-separable filters.

[0178] For separable filters, vertical and horizontal 1D convolutions can be performed sequentially, while non-separable filters use a single 2D convolution. In one embodiment, the input data is multidimensional and upsampling involves upsampling (using a separable filter) slices of the input data (horizontally or vertically) to produce intermediate results (e.g., partially upsampled intermediate data) that contribute to increased resolution.

[0179] In step 770, a nonlinear operation is applied to the resolution-increased intermediate data to produce processed resolution-increased intermediate data. In one embodiment, at least one of the resolution-increased intermediate data and the processed resolution-increased intermediate data is stored only in shared memory within the processor and is not transferred between the processor and memory.

[0180] In step 775, the processed resolution-increased intermediate data is downsampled to produce processed intermediate data. In one embodiment, the upsampling rate is equal to the downsampling rate. In one embodiment, the upsampling rate is twice the downsampling rate. In one embodiment, the downsampling rate is twice the upsampling rate. In one embodiment, the execution of upsampling and downsampling depends on whether the corresponding filters are separable. In one embodiment, the processed resolution-increased intermediate data is multidimensional, and downsampling involves downsampling slices of the processed resolution-increased intermediate data in one dimension (using separable filters) to produce intermediate results that contribute to the processed intermediate data (e.g., partially downsampled intermediate data).

[0181] In one embodiment, all convolution and nonlinear operations process data stored in internal shared memory, and only the final output of downsampling operations or subsequent operations is stored in external memory. In one embodiment, upsampling, application, and downsampling are equivalent to translation to a certain accuracy level. In another embodiment, upsampling, application, and downsampling are equivalent to translation and rotation to a certain accuracy level. In step 780, the processed intermediate data is stored in memory.

[0182] In one embodiment, the input data is a portion of a larger input data set, which is divided (e.g., tiled) into input data blocks, and the processor executes instructions to approximate continuous mathematical operators by performing steps 765, 770, and 775 on the input data blocks. In another embodiment, upsampling, nonlinear operations, and downsampling are performed simultaneously on at least two of the input data blocks to generate at least two output data blocks.

[0183] In one embodiment, at least one of steps 765, 770, and 775 is performed on a server or in a data center to generate output data streamed to a user device. In one embodiment, at least one of steps 765, 770, and 775 is performed in a cloud computing environment. In one embodiment, at least one of steps 765, 770, and 775 is performed to train, test, or validate a neural network used in a machine, robot, or autonomous vehicle. In one embodiment, at least one of steps 765, 770, and 775 is performed on a virtual machine that includes a portion of a GPU.

[0184] The fusion of sequential mathematical operations, such as pointwise nonlinear functions, without storing intermediate results in memory, improves performance in situations where memory bus bandwidth is limited. In one embodiment, sequential mathematical operations including at least two of convolution, upsampling, pointwise nonlinear functions, and downsampling are performed to process input data and generate aliasing-free output data. In another embodiment, the input data is spatially partitioned for parallel processing, allowing intermediate results generated during processing of each block of input data to be stored in shared memory within the processor. Storing intermediate data in shared memory improves performance compared to storing intermediate data in and loading intermediate data from external memory.

[0185] It should be noted that the techniques described herein can be contained in executable instructions stored in a computer-readable medium for use by or in conjunction with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will appreciate that, for some embodiments, various different types of computer-readable media may be included for storing data. When used herein, “computer-readable medium” includes one or more of any suitable media for storing executable instructions of a computer program, such that an instruction execution machine, system, apparatus, or device can read (or retrieve) the instructions from the computer-readable medium and execute those instructions to implement the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable compact discs (CDs), portable digital video discs (DVDs), and the like.

[0186] It should be understood that the arrangement of components shown in the accompanying drawings is for illustrative purposes, and other arrangements are possible. For example, one or more of the elements described herein may be implemented wholly or partially as electronic hardware components. Other elements may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other elements may be combined, some may be omitted entirely, and additional components may be added while still achieving the functionality described herein. Therefore, the subject matter described herein can be implemented in many different variations, and all such variations are contemplated to be within the scope of the claims.

[0187] To facilitate understanding of the topics described herein, many aspects are described in sequence of actions. Those skilled in the art will recognize that various actions can be performed by dedicated circuitry or circuit systems, by program instructions executed by one or more processors, or by a combination of both. The description of any sequence of actions herein is not intended to imply that a particular order in which the actions described for execution must be followed. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context.

[0188] In the context of describing the subject matter (especially in the context of the claims below), the use of the terms “a,” “an,” “this,” and similar designations should be interpreted to cover both the singular and plural, unless otherwise specified herein or obviously contradicted by the context. The use of the term “at least one” (e.g., at least one of A and B) followed by a list of one or more items should be interpreted to mean one item selected from the listed items (A or B), or any combination of two or more of the listed items (A and B), unless otherwise specified herein or obviously contradicted by the context. Furthermore, the foregoing description is for illustrative purposes only and not for limiting purposes, as the scope of protection sought is defined by the claims set forth thereafter with their equivalents. The use of any and all example or exemplary language provided herein (e.g., “such as”) is intended merely to better illustrate the subject matter and does not constitute a limitation on the scope of the subject matter, unless otherwise stated. The use of “based on,” and other similar phrases indicating conditions leading to the result, in both the claims and the written description, is not intended to exclude any other conditions leading to that result. The language in the description should not be interpreted as indicating that any unclaimed element is essential for the implementation of the claimed invention.

Claims

1. A computer-implemented method for fusion processing of continuous mathematical operators, comprising: The input data is stored in a memory that includes one or more memory devices; as well as The input data is read from the memory by a processor coupled to the memory via an interface, wherein the processor executes instructions to perform the fusion processing of the successive mathematical operators in such a way as follows: The input data is upsampled to produce intermediate data with increased resolution; Nonlinear operations are applied to the intermediate data with increased resolution to produce processed intermediate data with increased resolution; The processed intermediate data with increased resolution is downsampled to produce processed intermediate data, wherein the upsampling, application, and downsampling are equivariant to the rotation to the desired level of accuracy; as well as The processed intermediate data is stored in the memory.

2. The computer-implemented method of claim 1, wherein the processor implements a neural network model comprising at least one stage and the at least one stage processes the input data to generate the processed intermediate data.

3. The computer-implemented method according to claim 2, wherein the input data includes a feature map.

4. The computer-implemented method of claim 1, wherein at least one of the resolution-increased intermediate data and the processed resolution-increased intermediate data is stored only in a shared memory within the processor and is not transferred between the processor and the memory.

5. The computer-implemented method of claim 1, wherein the input data is divided into input data blocks, and the processor executes the instructions to perform the fusion processing of the consecutive mathematical operators in the following manner: Simultaneously, at least two of the input data blocks are upsampled to generate at least two blocks of intermediate data with increased resolution; Simultaneously, the nonlinear operation is applied to at least two blocks of the intermediate data with increased resolution to generate at least two blocks of the processed intermediate data with increased resolution; Simultaneously, at least two blocks of the processed intermediate data with increased resolution are downsampled to generate at least two blocks of the processed intermediate data; as well as At least two blocks of the processed intermediate data are stored in the memory.

6. The computer-implemented method of claim 1, wherein the processed intermediate data is divided into a plurality of output data blocks, and the processor executes the instructions to perform the fusion processing of the consecutive mathematical operators in the following manner: Each output block is assigned to a thread block that processes input blocks from among multiple input blocks, each input block comprising a portion of the input data, in the following manner: The input data block is upsampled to produce a portion of intermediate data with increased resolution for the output block; The nonlinear operation is applied to a portion of the resolution-increased intermediate data to produce the processed resolution-increased intermediate data portion for the output block; A portion of the processed intermediate data with increased resolution is downsampled to produce a portion of the processed intermediate data for the output block; as well as A portion of the processed intermediate data is stored in the memory.

7. The computer-implemented method of claim 1, wherein the input data is multidimensional, and the upsampling includes upsampling the input data in one dimension to produce partially upsampled intermediate data.

8. The computer-implemented method of claim 1, wherein the processed resolution-increased intermediate data is multidimensional, and the downsampling includes downsampling the processed resolution-increased intermediate data in one dimension to produce partially downsampled intermediate data.

9. The computer-implemented method of claim 1, wherein the upsampling, application, and downsampling are equivalent to the translation to the desired accuracy level.

10. The computer-implemented method of claim 1, wherein at least one of the steps of upsampling, applying, and downsampling is performed on a server or in a data center to generate output data to be streamed to a user device.

11. The computer-implemented method of claim 1, wherein at least one of the steps of upsampling, application, and downsampling is performed in a cloud computing environment.

12. The computer-implemented method of claim 1, wherein at least one of the steps of upsampling, applying, and downsampling is performed for training, testing, or validating a neural network used in a machine, robot, or autonomous vehicle.

13. The computer-implemented method of claim 1, wherein at least one of the steps of upsampling, applying, and downsampling is performed on a virtual machine comprising a portion of a graphics processing unit.

14. A computer-implemented system for fusion processing of continuous mathematical operators, comprising: Memory, including one or more memory devices that store input data; as well as A processor, coupled to the memory via an interface, wherein the processor executes instructions to perform the fusion processing of the sequential mathematical operators in such a way as: Read the input data from the memory; The input data is upsampled to produce intermediate data with increased resolution; Nonlinear operations are applied to the intermediate data with increased resolution to produce processed intermediate data with increased resolution; The processed intermediate data with increased resolution is downsampled to produce processed intermediate data, wherein the upsampling, application, and downsampling are equivariant to the rotation to the desired level of accuracy; as well as The processed intermediate data is stored in the memory.

15. The system of claim 14, wherein the processor implements a neural network model including at least one stage and the at least one stage processes the input data to generate the processed intermediate data.

16. The system of claim 15, wherein the input data includes a feature map.

17. The system of claim 14, wherein the processor includes shared memory, and at least one of the resolution-increased intermediate data and the processed resolution-increased intermediate data is stored only in the shared memory and is not transferred between the processor and the memory.

18. The system of claim 14, wherein the upsampling, application, and downsampling are equivalent to the translation to the desired accuracy level.

19. A non-transitory computer-readable medium storing computer instructions for fusion processing of successive mathematical operators, which, when executed by one or more processors, cause the one or more processors to perform the following steps: Input data is read from memory coupled to at least one of the one or more processors via an interface; The input data is upsampled to produce intermediate data with increased resolution; Nonlinear operations are applied to the intermediate data with increased resolution to produce processed intermediate data with increased resolution; The processed intermediate data with increased resolution is downsampled to produce processed intermediate data, wherein the upsampling, application, and downsampling are equivariant to the rotation to the desired level of accuracy; as well as The processed intermediate data is stored in the memory.

20. The non-transitory computer-readable medium of claim 19, wherein at least one of the one or more processors includes shared memory, and at least one of the resolution-increased intermediate data and the processed resolution-increased intermediate data is stored only in the shared memory and is not transferred between the processors and the memory.

Images