SYSTEM AND METHOD FOR OPTIMIZING GRAPHICS PROCESSING FOR MACHINE LEARNING INFERIORITY

Abstract

To provide systems and methods directed to a method for optimizing the utilization of graphics processors for machine learning inference tasks.SOLUTION: The method includes simultaneously rendering a plurality of textures from an input to a machine-learned model by a computing system comprising one or more computing devices. The method includes generating, by the computing system, a plurality of shaders on the basis at least in part of a layout of the plurality of textures, and each of the plurality of shaders corresponds to at least one operator of a plurality of operators of the machine-learned model. The method includes processing, by the computing system using a graphics processing unit (GPU), the plurality of textures with the plurality of shaders to acquire a machine learning output for the machine-learned model.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 331,593, filed April 15, 2022. U.S. Provisional Patent Application No. 63 / 331,593 is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to optimizing system resource usage and, more particularly, to optimizing graphics processing hardware usage via application programming interfaces for machine learning inference. [Background technology]

[0003] Recently, many applications have started to leverage machine learning to significantly optimize the performance of various tasks (e.g., videoconferencing, image recognition services, etc.). However, in some execution environments, such as web browsers, these applications generally lack the capacity to efficiently use the bandwidth of specialized hardware (e.g., graphics processing units) for processing machine learning inference tasks. Summary of the Invention [Means for solving the problem]

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned through practice of the embodiments.

[0005] One exemplary aspect of the present disclosure is directed to a computing system for optimizing the use of a graphics processor for machine learning inference tasks. The computing system includes one or more processors, the one or more processors comprising a graphics processing unit (GPU). The computing system includes one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations include simultaneously rendering a plurality of textures from an input to a machine-learned model. The operations include generating a plurality of shaders based at least in part on a layout of the plurality of textures, each of the plurality of shaders corresponding to at least one operator of a plurality of operators of the machine-learned model. The operations include processing the plurality of textures with the plurality of shaders using the GPU to obtain a machine learning output for the machine-learned model.

[0006] Another exemplary aspect of the present disclosure is directed to a method for optimizing the use of a graphics processor for machine learning inference tasks. The method includes simultaneously rendering, by a computing system having one or more computing devices, a plurality of textures from inputs to a machine learning model. The method includes generating, by the computing system, a plurality of shaders based at least in part on a layout of the plurality of textures, each of the plurality of shaders corresponding to at least one operator of a plurality of operators of the machine learning model. The method includes processing, by the computing system using a graphics processing unit (GPU), the plurality of textures with the plurality of shaders to obtain a machine learning output for the machine learning model.

[0007] Another example aspect of the present disclosure is directed to one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations include simultaneously rendering a plurality of textures from inputs to a machine learning model. The operations include generating a plurality of shaders based at least in part on a layout of the plurality of textures, each of the plurality of shaders corresponding to at least one operator of a plurality of operators of the machine learning model. The operations include processing, with a GPU, the plurality of textures with the plurality of shaders to obtain a machine learning output for the machine learning model.

[0008] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain related principles.

[0010] Detailed descriptions of embodiments directed to persons skilled in the art are set forth herein and refer to the accompanying figures. [Brief description of the drawings]

[0011] [Figure 1A] FIG. 1 is a block diagram of an example computing system that optimizes graphics processor usage for machine learning inference tasks, in accordance with an example embodiment of the present disclosure. [Figure 1B] FIG. 1 is a block diagram of an exemplary computing device that implements the use of a graphics processor for machine learning inference tasks, in accordance with an exemplary embodiment of the present disclosure. [Figure 1C]FIG. 1 is a block diagram of an exemplary computing device that implements the use of a graphics processor for machine learning inference tasks, in accordance with an exemplary embodiment of the present disclosure. [Diagram 2] FIG. 1 illustrates a data flow diagram for optimizing graphics processor usage for machine learning inference tasks, according to an exemplary embodiment of the present disclosure. [Diagram 3] FIG. 2 is a data flow diagram for a decoder block implemented according to some embodiments of the present disclosure. [Figure 4A] 13A-13C illustrate results for inference latency for high-quality segmentation in various implementations of an application programming interface according to some embodiments of the present disclosure. [Figure 4B] FIG. 1 illustrates results for evaluation of high-quality segmentation models, according to some embodiments of the present disclosure. [Figure 5A] FIG. 1 illustrates image segmentation results using two models processed according to some embodiments of the present disclosure. [Figure 5B] FIG. 1 illustrates high quality segmentation results using two models processed according to some embodiments of the present disclosure. [Figure 6] FIG. 2 is a data flow diagram for a method for optimizing graphics processing for machine learning inference, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Reference numbers that are repeated among the figures identify like features in various implementations.

[0013] overview In general, the present disclosure is directed to optimizing the use of system resources. More specifically, the present disclosure relates to optimizing the use of graphics processing hardware via an application programming interface for machine learning inference. Specifically, for example, input (e.g., image input data for an image segmentation task, etc.) may be obtained for a trained machine learning model used by an application (e.g., a videoconferencing application, etc.) running in a web browser. To obtain high-quality results, the application may leverage specialized hardware, such as a graphics processing unit (GPU), for processing the machine learning model via the application programming interface. However, some conventional application programming interfaces (APIs) lack the capacity to optimally use the bandwidth of the GPU (e.g., WebGL API, etc.).

[0014] Therefore, the disclosed system and method propose to optimize the use of the graphics processor for machine learning inference tasks. For example, after obtaining an input, multiple textures may be rendered simultaneously from the input to the machine learning model (e.g., via the Multi-Render Targets (MRT) process of the WebGL API, etc.). Multiple shaders (e.g., fragment shaders, etc.) may then be generated based at least in part on the layout of the multiple textures (e.g., the number of textures, the dimensions of the textures, etc.). Each of the multiple shaders may correspond to at least one operator of the multiple operators of the machine learning model. Using a GPU, the multiple textures may be processed by the multiple shaders to obtain a machine learning output for the machine learning model.

[0015] The systems and methods of the present disclosure provide several technical effects and benefits. As one exemplary technical effect and benefit, shader operations performed by a GPU incur some overhead. Because GPUs are heavily parallelized, inefficiencies may arise when the complexity of shader operations is less than the overhead. Thus, by leveraging existing features of some APIs, such as the MRT feature of WebGL, embodiments of the present disclosure may complicate shader operations by converting model inputs into multiple textures, thus substantially increasing GPU bandwidth usage (e.g., by parallelization, etc.). In contrast, by increasing GPU bandwidth usage, embodiments of the present disclosure facilitate more efficient model usage and the provision of higher quality model output, while reducing or eliminating any processing bottlenecks in the GPU (e.g., thus reducing power usage, computation cycles, memory usage, etc. associated with the bottleneck).

[0016] Referring now to the drawings, exemplary embodiments of the present disclosure will be discussed in further detail.

[0017] Exemplary Devices and Systems 1A illustrates a block diagram of an exemplary computing system 100 for optimizing graphics processor usage for machine learning inference tasks, according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, communicatively coupled via a network 180.

[0018] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0019] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. For example, the processor 112 may include one or more graphics processing units (GPUs). These GPUs may be leveraged through application programming interfaces (APIs) accessed by applications executed by the user computing device (102) and / or other computing devices (e.g., server computing system 130, etc.). The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 executed by the processor 112 to cause the user computing device 102 to perform operations.

[0020] In some implementations, the user computing device 102 can store or include one or more models 120. For example, the models 120 can be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine learning models can leverage attention mechanisms such as self-attention. For example, some exemplary machine learning models can include multi-headed self-attention models (e.g., transformer models).

[0021] In some implementations, one or more models 120 may be received from the server computing system 130 over the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single model 120 (e.g., to perform parallel processing across multiple instances of the model).

[0022] Additionally or alternatively, one or more models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the model 140 may be implemented by the server computing system 130 as part of a web service (e.g., an image segmentation service, an image recognition service, etc.). Thus, one or more models 120 may be stored and implemented at the user computing device 102 and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0023] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be useful for implementing a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0024] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0025] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0026] As described above, the server computing system 130 can store or otherwise include one or more models 140. For example, the models 140 can be or otherwise include various machine learning based models. Exemplary machine learning based models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine learning based models can leverage attention mechanisms such as self-attention. For example, some exemplary machine learning based models can include a multi-headed self-attention model (e.g., a transformer model).

[0027] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 by interacting with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0028] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0029] The training computing system 150 may include a model trainer 160 that trains the machine learning based models 120 and / or 140 stored in the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation. For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters for several training iterations.

[0030] In some implementations, performing the error backpropagation may include performing truncated backpropagation over time. The model trainer 160 can implement several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0031] In some implementations, if the user has given consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 against user-specific data received from the user computing device 102. In some instances, this process may be referred to as individualizing the model.

[0032] The model trainer 160 includes computer logic used to provide the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as a RAM, a hard disk, or an optical or magnetic medium.

[0033] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may be carried over any type of wired and / or wireless connections, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0034] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0035] In some implementations, the input to the machine learning based model of the present disclosure may be image data. The machine learning based model may process the image data to generate an output. As an example, the machine learning based model may process the image data to generate an image recognition output (e.g., recognition of the image data, latent embedding of the image data, encoded representation of the image data, hash of the image data, etc.). As another example, the machine learning based model may process the image data to generate an image segmentation output. As another example, the machine learning based model may process the image data to generate an image classification output. As another example, the machine learning based model may process the image data to generate an image data modification output (e.g., alteration of the image data, etc.). As another example, the machine learning based model may process the image data to generate an encoded image data output (e.g., encoded and / or compressed representation of the image data, etc.). As another example, the machine learning based model may process the image data to generate an upscaled image data output. As another example, the machine learning based model may process the image data to generate a prediction output.

[0036] In some implementations, the input to the machine learning based model of the present disclosure may be text or natural language data. The machine learning based model may process the text or natural language data to generate an output. As an example, the machine learning based model may process the natural language data to generate a language encoding output. As another example, the machine learning based model may process the text or natural language data to generate a latent text embedding output. As another example, the machine learning based model may process the text or natural language data to generate a transformation output. As another example, the machine learning based model may process the text or natural language data to generate a classification output. As another example, the machine learning based model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning based model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning based model may process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is of higher quality than the input text or natural language, etc.). As another example, the machine learning based model may process the text or natural language data to generate a prediction output.

[0037] In some implementations, the input to the machine learning based model of the present disclosure may be audio data. The machine learning based model may process the audio data to generate an output. As an example, the machine learning based model may process the audio data to generate a speech recognition output. As another example, the machine learning based model may process the audio data to generate a speech conversion output. As another example, the machine learning based model may process the audio data to generate a latent embedding output. As another example, the machine learning based model may process the audio data to generate an encoded audio output (e.g., an encoded and / or compressed representation of the audio data, etc.). As another example, the machine learning based model may process the audio data to generate an upscaled audio output (e.g., audio data of higher quality than the input audio data, etc.). As another example, the machine learning based model may process the audio data to generate a text representation output (e.g., a text representation of the input audio data, etc.). As another example, the machine learning based model may process the audio data to generate a predicted output.

[0038] In some implementations, the input to the machine learning based model of the present disclosure may be latent coding data (e.g., a latent space representation of the input, etc.). The machine learning based model may process the latent coding data to generate an output. As an example, the machine learning based model may process the latent coding data to generate a recognition output. As another example, the machine learning based model may process the latent coding data to generate a reconstruction output. As another example, the machine learning based model may process the latent coding data to generate an exploration output. As another example, the machine learning based model may process the latent coding data to generate a reclustering output. As another example, the machine learning based model may process the latent coding data to generate a prediction output.

[0039] In some implementations, the input to the machine learning based model of the present disclosure may be statistical data. The statistical data may be, represent, or otherwise include data calculated and / or derived from some other data source. The machine learning based model may process the statistical data to generate an output. As an example, the machine learning based model may process the statistical data to generate a recognition output. As another example, the machine learning based model may process the statistical data to generate a prediction output. As another example, the machine learning based model may process the statistical data to generate a classification output. As another example, the machine learning based model may process the statistical data to generate a segmentation output. As another example, the machine learning based model may process the statistical data to generate a visualization output. As another example, the machine learning based model may process the statistical data to generate a diagnostic output.

[0040] In some implementations, the input to the machine learning based models of the present disclosure may be sensor data. The machine learning based models may process the sensor data to generate an output. As an example, the machine learning based models may process the sensor data to generate a recognition output. As another example, the machine learning based models may process the sensor data to generate a prediction output. As another example, the machine learning based models may process the sensor data to generate a classification output. As another example, the machine learning based models may process the sensor data to generate a segmentation output. As another example, the machine learning based models may process the sensor data to generate a visualization output. As another example, the machine learning based models may process the sensor data to generate a diagnostic output. As another example, the machine learning based models may process the sensor data to generate a detection output.

[0041] In some cases, the machine learning based model may be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data and the output may include compressed audio data. In another example, the input may include visual data (e.g., one or more images or videos) and the output may include compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or visual data).

[0042] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing a likelihood that one or more images show an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, a likelihood that the region shows an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines, for each pixel in one or more images, a respective depth value. As another example, the image processing task may be motion estimation, where the network input includes multiple images and the image processing output defines, for each pixel of one of the input images, the scene motion represented in pixels between the images in the network input.

[0043] In some cases, the input includes audio data representing speech and the task is a speech recognition task. The output may include text output that is mapped to the speech. In some cases, the task includes encrypting or decrypting input data. In some cases, the task includes a microprocessor-implemented task such as branch prediction or memory address translation.

[0044] 1A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems may be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be both trained and used locally on the user computing device 102. In some of such implementations, the user computing device 102 can implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0045] 1B illustrates a block diagram of an exemplary computing device 10 for implementing an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0046] The computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine learning model. For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0047] 1B, each application may communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0048] 1C illustrates a block diagram of an exemplary computing device 50 for implementing according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0049] Computing device 50 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0050] The central intelligence layer includes several machine learning models. For example, as shown in FIG. 1C, a respective machine learning model may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model to all of the applications. In some implementations, the central intelligence layer is included in or otherwise implemented by the operating system of the computing device 50.

[0051] The central intelligence layer can communicate with a central device data layer, which can be a centralized repository of data for computing devices 50. As shown in FIG. 1C, the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0052] FIG. 2 illustrates a data flow diagram for optimizing the use of graphics processors for machine learning inference tasks, according to an exemplary embodiment of the present disclosure. Specifically, as mentioned above, the importance of some applications, such as videoconferencing, has increased in recent years with the growing number of meetings conducted virtually with remote participants. For videoconferencing applications, one feature that is becoming increasingly important in this setting is background replacement or blurring, which is typically achieved through foreground / background segmentation. In traditional high quality (HQ) segmentation networks, model capacity is generally a significant limitation beyond what central processing unit (CPU) inference can achieve. Thus, embodiments of the present disclosure leverage GPUs for real-time model inference for heavier networks. However, existing engines and / or application programming interfaces (APIs), such as JavaScript inference engines featuring WebGL acceleration, exhibit a significant performance gap compared to others, such as OpenGL inference in native apps.

[0053] To achieve maximum performance, some embodiments of the present disclosure run fully on a GPU, from acquiring input 202 (e.g., image acquisition), through model (e.g., neural network) inference, to providing output 210 (e.g., rendering the segmented results on a screen).

[0054] Traditionally, the basic architecture of a GPU-accelerated neural network inference engine implements neural network operators in the form of GPU shaders. The inference loop essentially queues these shader programs in a topologically sorted order of the neural network into a GPU command queue for asynchronous execution. Authoring such kernel implementations is relatively straightforward, as there are modern GPU APIs that support compute shaders. However, some existing APIs, such as WebGL, lack support for compute shaders. Thus, in some embodiments, kernel implementations for the WebGL inference engine may be written in the form of fragment shaders.

[0055] Turning to FIG. 2, an input 202 (e.g., image input) to a machine learning model can be obtained. Then, in some embodiments, logical and GPU objects can be separated. Logical objects refer to model (e.g., neural network) objects such as tensors, tensor buffers, spatial tensors, weight tensors, etc. GPU objects refer to storage types available in an API (e.g., WebGL), such as 2D textures, 2D texture arrays, frame buffers, etc. By separating these two types of objects, a programmer can utilize the fastest GPU building block for a particular situation and not be tied to a particular GPU object.

[0056] The renderer 204 then processes the input 202 while leveraging features of the API to render multiple textures 206. For example, the renderer 204 may leverage WebGL features such as multiple render targets (MRT), a feature of modern GPUs that allows rendering of an image to multiple textures at once. When a programmer writes a shader that returns an output value for each render target, the shader renders to all render targets in a single draw call, saving a lot of overhead of multiple draw calls. In that way, the input 202 can be rendered by the renderer 204 via MRT to generate the textures 206.

[0057] The shader 208 can be generated based at least in part on the plurality of textures 206. In particular, in some embodiments, flexible tensor layout and shader code generation may be leveraged to generate the shader 208. For example, in its simplest form, a tensor of shape [H, W, C] is generally a tensor of size

[0058]

number

[0059] 206 to a 4-channel 2D texture. However, depending on the parallelism of the workload and its memory access patterns, a different layout with swapped x- and y-axes may be more efficient than a simple mapping. To allow optimization of various use cases, embodiments of the present disclosure facilitate flexible tensor layouts that can be specified by a user or determined in real time. Since this must be accompanied by multiple shader implementations that can accept different axis layouts, embodiments of the present disclosure support on-the-fly shader code generation with respect to the active layout for generating shaders 208. Shaders 208 can process textures 206 to obtain machine learning outputs 210.

[0060] In that way, multi-target rendering can be used to substantially increase the efficiency of GPU processing for inference. To give an illustrative example, a convolution can be performed from 32×32×1 to 32×32×1. Using conventional techniques, 32×32 (1024) threads would be run. For example, reading input values ​​from a source tensor incurs one float per thread (1024 floats total), reading weight values ​​from a weight tensor incurs one float per thread (1024 floats total), and writing output values ​​to a destination tensor incurs one float per thread (1024 floats total). However, using embodiments of the present disclosure, it is possible to run 16×16 threads (256), with each thread processing four times as many values. For example, with MRT, reading input values ​​from a source tensor incurs 4 floats per thread (1024 floats total), reading weight values ​​from a weight tensor incurs 1 float per thread (256 total), and writing output values ​​to a destination tensor incurs 4 floats per thread (1024 floats total). Thus, 1024 requests can be reduced to a total of 256 requests.

[0061] More specifically, in conventional approaches, logical objects such as tensors are mapped one-to-one with GPU objects (e.g., textures, buffers, etc.). For example, a 3D tensor (e.g., logical object) with shape [H,W,3] may be represented as a GL texture (e.g., webGL, etc.) of size [H,W] because GL textures have a depth of 4. Thus, the representation of tensors as textures is hard-coded and implemented.

[0062] Conversely, embodiments of the present disclosure enable a separation between logical and GPU objects, thus facilitating the mapping of logical and GPU objects in other than a hard-coded one-to-one manner. For example, an input tensor 202 may be represented by four GL textures 206. Thus, by separating the logical and physical objects, the execution of the shader 208 can read (e.g., via MRT, etc.) multiple textures that represent the input. When rendering, the compiler can be fully aware of the characteristics of the logical objects and automatically transform them into GPU objects (e.g., textures, etc.). Thus, a single source representation of an ML operation (e.g., a convolution) can use tensors as inputs and tensors as outputs. However, the storage / layout of these tensors can be determined later depending on other requirements (e.g., performance, memory consumption, initialization time, GPU capabilities, etc.).

[0063] As an example, inference time processing for machine learning type models may require an application programming interface. Model processing may include the following operations: input(512×512×3)>conv1>relu1>depth_wise_conv>conv2>relu2>resize>output(512×512×1). Input 202 may be gpu_texture(512×512×3) (e.g., width×height×RGB channels). For the initialization phase, GPU programs may be created for conv1+relu1, depth_wise_conv, conv2+relu2, and resize. In some embodiments, relu1 may be performed in a single pass with convolution to save memory bandwidth.

[0064] To follow the previous example, GPU objects tensor1 (256×256×16), tensor2 (256×256×16), tensor3 (128×128×1), and tensor4 (512×512×1) may be created. In some embodiments, tensors may reuse memory. Starting from the input, the processing may occur as follows: tensor0(input)>conv1+relu1>tensor1>depth_wise_conv>tensor2>conv2+relu2>tensor3>resize>tensor4(output). In this example, tensor1 may use 4 textures for layout / storage to fully utilize MRT in conv1+relu1 operations. tensor2 may use 1 texture to optimize performance for conv2+relu2. tensor3 can contain 4 textures to facilitate MRT use in conv2+relu2 operations, and tensor4 can contain 1 texture to eliminate an extra conversion for the output of gpu_texture (512x512x1). Specifically, since tensor2 is the input for the conv2+relu2 operation and tensor3 is the output of said operation, the combination with the best performance may be asymmetric in the number of textures used in the tensor representation.

[0065] As an example, the input 202 may be a five-dimensional tensor. The textures 206 may include four textures that are rendered in the renderer 204 to represent the input tensor 202.

[0066] It should be noted that the embodiments of the present disclosure are described with respect to the WebGL API simply to illustrate the functionality of the embodiments. In particular, image segmentation is described because CPU inference is not sufficient to run higher capacity image segmentation networks at high frame rates, and therefore inference must be performed using hardware accelerators available through some APIs (e.g., WebGL). While newer accelerators such as digital signal processors (DSPs) and neural processing units (NPUs) are fragmented and do not have web standards, graphics processing units (GPUs) are widely available and have an established API for the web, making them a natural choice to illustrate.

[0067] Similarly, WebGL is a standard API for GPU rendering for applications that run within web browsers and are supported by all major web browsers. Although there are a handful of other ML inference engines for the web that have WebGL acceleration, such as TensorFlow and ONNX Runtime Web, the performance of WebGL acceleration in existing solutions has not met expectations and is 3-4 times slower than native performance. Thus, examples of the present disclosure are illustrated with WebGL acceleration.

[0068] Some aspects of the present disclosure are based on traditional engines, such as TensorFlow. However, the model (e.g., neural network) operations of the present disclosure are implemented by shaders (e.g., fragment shaders) since WebGL does not support compute shaders. MRT is used to rephrase these tasks in rendering terms. MRT allows the rendering of images to multiple textures at once with a single draw command, significantly reducing the overhead of multiple draw calls. In some embodiments, modifications are applied to existing engine designs, such as TFLite GPUs, to leverage MRT. First, logical tensors and physical GPU objects are separated (e.g., have a 1:1 correspondence in TFLite GPUs). Then, tensors are allowed to take flexible shapes rather than hard-coded layouts to efficiently leverage MRT.

[0069] FIG. 3 shows a data flow diagram for a decoder block implemented according to some embodiments of the present disclosure. Specifically, FIG. 3 shows a decoder block for a segmentation head, in which the 1×1 convolution, batch normalization, and activation layers are omitted. As shown, the decoder block 300 takes the low-resolution segmentation result and the high-resolution skip connections and outputs the high-resolution result to the next level. The next level extracts features from the skip connections through per-channel attention and adds them to the segmentation result from the previous level. The enhanced segmentation is further refined with several convolution layers. For foreground / background segmentation, the head only needs to output one channel mask. To show the effectiveness of the segmentation head including the decoder block 300, two models are trained with the same backbone (MobileNetV3-Small) and training configuration, but with different heads, namely, LR-ASPP and the head shown in FIG. 3. As shown in the table below, the segmentation head of FIG. 3 significantly enhances the quality metrics of intersection-to-union ratio (IOU) and boundary F-score, adding only 8% more parameters.

[0070] [Table 1]

[0071] FIG. 4A shows results on inference latency for high-quality segmentation in various implementations of application programming interfaces according to some embodiments of the present disclosure. Specifically, to evaluate embodiments of the present disclosure, the focus is on in-browser applications such as web-based videoconferencing and AR effects, where the ML solution is running in a sandbox environment without direct access to the GPU on the device. FIG. 4A shows inference time using embodiments of the present disclosure (e.g., ML drift) versus other inference engines such as TensorFlow.js (TF.js) WebAssembly (Wasm) and WebGL. The Wasm backend is accelerated by SIMD instructions, and the TF.js WebGL backend uses the GPU to accelerate model inference, and similar implementations are available in other engines with comparable performance.

[0072] FIG. 4B shows results for evaluation of high-quality segmentation models according to some embodiments of the present disclosure. Specifically, using embodiments of the present disclosure, the HQ model can be run in the browser with larger capacity and image sizes. For evaluation, two groups of segmentation models, namely HQ-MS, HQ-ML, and HQ-E0-E4, are trained according to some embodiments of the present disclosure. The first two are based on small and large versions of MobileNetV3. The remaining five are based on EfficientNet-Lite with increasing model size. All models are equipped with our segmentation head and trained with the same data and hyperparameters. The input image size is 512×288.

[0073] As shown in Figure 3, the HQ-MS model exhibits higher quality than HQ-E0 with comparable model size, which demonstrates the effectiveness of SE for small models. As the model size increases, the two groups of HQ models show similar values ​​of the quality metric. For models with comparable inference times, the HQ-E model consistently has higher quality than the HQ-M model by a notable margin. This is due to global pooling in the squeeze and excitation layers, which is a challenge for GPU acceleration.

[0074] FIG. 5A illustrates image segmentation results using two models processed according to some embodiments of the present disclosure. In particular, FIG. 5A illustrates the improvement of image segmentation compared to the prior art. For example, image 502 is an image of a particular input size (e.g., 512×288 pixels, etc.). Image segmentation result 504 is the output of a conventional CPU-based image segmentation model. As illustrated, the segmentation result demonstrates a substantial loss of detail. Image segmentation results 506 and 508 are the output of two machine learning models constructed according to two particular implementations of the present disclosure. As illustrated, unlike image segmentation result 504 generated using a conventional model, image segmentation results 506 and 508 retain substantially more detail.

[0075] FIG. 5B shows high-quality segmentation results using two models processed according to some embodiments of the present disclosure.

[0076] 6 shows a flow chart diagram of an exemplary method performed in accordance with an exemplary embodiment of the present disclosure. Although FIG. 6 shows operations performed in a specific order for purposes of explanation and discussion, the method of the present disclosure is not limited to the order or sequence specifically shown. Various operations of method 600 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0077] In operation 602, the computing system may include a graphics processing unit (GPU). The computing system may simultaneously render multiple textures from inputs to the machine learning model. For example, the computing system may render the textures via a Multi-Render Target (MRT) process of a WebGL application programming interface (API). The machine learning model may be associated with an application that uses the WebGL API. For example, the application may be a gaming application that uses the WebGL API to render textures, and the machine learning model may be a model associated with the gaming application. In some implementations, the application may run in a web browser. Additionally or alternatively, in some implementations, the application may be a videoconferencing application.

[0078] In some implementations, multiple textures can be rendered simultaneously by a computing system using a GPU. For example, an input to a machine learning model can be an image. The computing system can use the rendering software and / or hardware of the GPU to simultaneously render textures that represent the images. In other words, the computing system can, in some implementations, simultaneously render multiple textures that represent a machine learning model (e.g., multiple textures that collectively represent an image input or a tensor derived from an input image, etc.).

[0079] More specifically, the MRT process is a feature of the WebGL API that allows for the rendering of an input, such as an image, into multiple textures at once. It is traditionally used for rendering, but can also be used for machine learning inference. For example, for an image segmentation task, multiple textures can be rendered simultaneously from the image to be segmented. The textures can then be processed with a machine learning model object represented by a shader (e.g., a fragment shader). This allows for the use of a GPU rather than a central processor, and also facilitates parallel processing to substantially increase the performance, efficiency, and accuracy of the machine learning model.

[0080] In some implementations, prior to simultaneously rendering multiple textures, the computing system may obtain an input to a machine learning model. The machine learning model may be trained to perform a task associated with the input. For example, the input to the machine learning model may be or otherwise include image data, and the task that the machine learning model is trained to perform may be an image segmentation task.

[0081] At operation 604, the computing system may generate a plurality of shaders based at least in part on the layout of the plurality of textures. Each of the plurality of shaders may correspond to at least one operator of the plurality of operators of the machine learning model. For example, the plurality of shaders may be a plurality of fragment shaders.

[0082] In some implementations, prior to generating the shaders, the computing system may determine GPU objects indicative of available processing capabilities of the GPU. The computing system may determine a layout of the textures based at least in part on the GPU objects. In other words, the layout of the textures may be determined based on the processing capabilities of the GPU. In some implementations, the layout of the textures may be specified by a user.

[0083] In operation 606, the computing system can process the multiple textures with the multiple shaders to obtain machine learning outputs for the machine learning model. In some implementations, to process the shaders, the computing system can extract the machine learning outputs from the intermediate representation. For example, the machine learning outputs may be or otherwise include intermediate texture outputs that represent the machine learning outputs. The computing system can extract the machine learning outputs from the intermediate texture outputs.

[0084] Additional Disclosures The techniques described herein refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0085] Although the present subject matter has been described in detail with respect to various specific exemplary embodiments thereof, each example is provided as an explanation, not a limitation of the present disclosure. Upon understanding the above, those skilled in the art can easily create modifications, variations, and equivalents of such embodiments. Thus, the present disclosure does not exclude such modifications, variations, and / or additions to the present subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may also be used with another embodiment to produce yet a further embodiment. Thus, it is intended that the present disclosure cover such modifications, variations, and equivalents. [Explanation of symbols]

[0086] 10. Computing Devices 50 Computing Devices 100 Computing Systems 102 User Computing Devices 112 processors 114 Memory, computing device memory 120 models, machine learning models 122 User Input Components 130 Server Computing System 132 processors 134 Memory 140 models, machine learning models 150 Training Computing System 152 processors 154 Memory 160 Model Trainer 180 Network 204 Renderer 208 Shaders

Claims

1. 1. A computing system for optimizing graphics processor usage for machine learning inference tasks, comprising: one or more processors having a graphics processing unit (GPU); one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations; and the operation comprises: Rendering multiple textures simultaneously from inputs to a machine learning model by the computing system via a Multi-Render Target process of a web browser rendering application programming interface; determining, by the computing system, a number of GPU objects indicative of available processing capabilities of the GPU; generating a plurality of shaders based at least in part on a layout of the plurality of textures determined by the computing system based at least in part on the plurality of GPU objects, each of the plurality of shaders corresponding to at least one operator of a plurality of operators of the machine learning model; using the GPU to process the plurality of textures with the plurality of shaders to obtain machine learning outputs for the machine learning model; A computing system comprising:

2. 2. The computing system of claim 1, wherein the machine learning based model is associated with an application that uses the web browser rendering application programming interface.

3. The computing system of claim 2 , wherein the application runs in a web browser.

4. The computing system of claim 3 , wherein the application is a videoconferencing application.

5. The computing system of claim 1 , wherein the multiple textures are rendered simultaneously using the GPU.

6. 2. The computing system of claim 1, wherein the plurality of textures represent the input to the machine learning based model.

7. The computing system of claim 1, wherein the layout of the multiple textures is specified by a user.

8. The computing system of claim 1 , wherein the plurality of shaders includes a plurality of fragment shaders.

9. The input to the machine learning model is Image, or A tensor derived from the image The computing system of claim 1 .

10. the machine learning output includes an intermediate texture output that represents the machine learning output; 2. The computing system of claim 1, wherein processing the plurality of shaders further comprises extracting, by the computing system, the machine learning output from the intermediate texture output.

11. Prior to simultaneously rendering the plurality of textures, the operations include: obtaining, by the computing system, the inputs to the machine learning based model; 11. The computing system of claim 10, wherein the machine learning based model is trained to perform a task associated with the input.

12. the input to the machine learning model includes image data; 12. The computing system of claim 11, wherein the task that the machine learning based model is trained to perform comprises an image segmentation task.

13. 1. A computer-implemented method for optimizing graphics processor usage for machine learning inference tasks, comprising: Rendering, by a computing system having one or more computing devices, multiple textures from inputs to a machine learning model simultaneously via a Multi-Render Target process of a web browser rendering application programming interface; determining, by the computing system, a number of graphics processing unit (GPU) objects indicative of available processing capabilities of a GPU; generating, by the computing system, a plurality of shaders based at least in part on a layout of the plurality of textures determined by the computing system based at least in part on the plurality of GPU objects, each of the plurality of shaders corresponding to at least one operator of a plurality of operators of the machine learning model; processing, by the computing system using the GPU, the plurality of textures with the plurality of shaders to obtain machine learning outputs for the machine learning model; 4. A computer-implemented method comprising:

14. 14. The computer-implemented method of claim 13, wherein the machine learning based model is associated with an application that uses the web browser rendering application programming interface.

15. The computer-implemented method of claim 14 , wherein the application runs in a web browser.

16. The computer-implemented method of claim 15 , wherein the application is a videoconferencing application.

17. One or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, including: Rendering multiple textures simultaneously from inputs to the machine learning model via a Multi-Render Target process of the web browser rendering application programming interface; determining a plurality of graphics processing unit (GPU) objects indicative of available processing capabilities of a GPU; generating a plurality of shaders based at least in part on a layout of the plurality of textures determined based at least in part on the plurality of GPU objects, each of the plurality of shaders corresponding to at least one operator of a plurality of operators of the machine learning model; using the GPU to process the plurality of textures with the plurality of shaders to obtain machine learning outputs for the machine learning model; A non-transitory computer readable medium comprising:

Images