Efficient execution of inference computations for fully convolutional networks for inputs of different sizes.

Abstract

The present invention provides a method, system, and apparatus for performing inference computations on a fully convolutional neural network that receives inputs of different sizes. [Solution] The method includes receiving a new input to be processed by a fully convolutional neural network. The new input has a first size different from the fixed size processed by the fully convolutional neural network. The method also includes determining one or more fixed-size inputs from the new input, obtaining each fixed-size output generated by the fully convolutional neural network performing inference computations for each of the one or more fixed-size inputs, and generating a final output from each fixed-size output, which includes one or more invalid pixel values, that is equivalent to the output generated by processing the new input using the fully convolutional neural network.

Description

[Background technology]

[0001] background This specification relates to neural networks. In particular, this specification relates to the efficient execution of inference computations for fully convolutional networks that receive inputs of different sizes.

[0002] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates an output from a given input according to the current values ​​of each set of network parameters.

[0003] A fully convolutional network is a neural network that consists only of convolutional neural network layers and, selectively, other layers composed only of components that operate only in local input regions, such as pooling layers and element-wise layers, such as those applying element-wise nonlinear activation functions. In particular, unlike other types of convolutional neural networks, a fully convolutional network has no fully connected layers. A fully convolutional network can be configured to make pixel-wise predictions of an input (e.g., an image with multiple pixels). In other words, a fully convolutional network can be used to make individual predictions for each pixel of an input. An example of a task that requires pixel-wise predictions is image segmentation, in which case the neural network is configured to generate individual scores for each of a number of classes for each pixel of the input image. [Overview of the project]

[0004] overview This specification generally describes techniques for performing inference computations on neural networks.

[0005] In one embodiment, the technique described relates to a method performed by one or more computers. The method includes receiving a new input to be processed by a fully convolutional neural network deployed on a hardware accelerator; determining one or more fixed-size inputs from the new input; providing each of the one or more fixed-size inputs to the hardware accelerator to perform inference computations using the fully convolutional neural network; obtaining from the hardware accelerator each fixed-size output generated by the fully convolutional neural network for each of the one or more fixed-size inputs; and generating a final output from each fixed-size output that is equivalent to the output produced by processing the new input using the fully convolutional neural network. The new input has a first size different from the fixed size configured for processing by the fully convolutional neural network when deployed on the hardware accelerator. Each of the one or more fixed-size inputs has a fixed size. Each fixed-size output has one or more inaccurate pixel-by-pixel results.

[0006] The subject matter described herein may be implemented in particular embodiments to achieve one or more of the following advantages:

[0007] The described technology enables a statically compiled fully convolutional network model deployed on a hardware accelerator to process input data having unknown or various sizes. Generally, a fully convolutional neural network can, in principle, process inputs of any arbitrary size, but a statically compiled neural network already deployed on a hardware accelerator cannot process inputs having various sizes. In addition, it is difficult to compile a neural network for deployment on a hardware accelerator that can dynamically process input data having unknown or various sizes. However, the described technology can efficiently tile the input data into a plurality of smaller fixed-size inputs and provide the inputs to perform the inference calculation of the statically compiled fully convolutional network.

[0008] The described technology can also stitch the generated fixed-size outputs to generate a final output for a given input of random size that is equivalent to the output generated by a fully convolutional network that processes the random-size input. Thus, the described technology enables a fully convolutional network compiled only to receive fixed-size inputs when deployed on a hardware accelerator to generate accurate outputs for inputs of different sizes without modifying the operation of the compiled model or the hardware accelerator.

[0009] In addition, the described technology can automatically generate optimized parameters for tiling and stitching the inputs and outputs for the network based on the characteristics of the fully convolutional network. Using these optimized parameters, the described technology can increase the computational efficiency when performing inference calculations for input data having unknown or various sizes.

[0010] The described techniques can execute inference operations on different tiles (e.g., fixed-size inputs) in parallel by leveraging data sharing characteristics between adjacent accelerators to reduce memory usage. For example, the described techniques can optimize data transfers across overlapping regions of adjacent fixed-size inputs according to input or output data of various sizes.

[0011] Furthermore, the described techniques are robust to different input sizes and hardware accelerator architectures. The described techniques can automatically identify hardware constraints or requirements, such as system memory bandwidth. Based on the identified hardware constraints or requirements, the described techniques can efficiently tile inputs of any large size to fit a fully convolutional network deployed on a hardware accelerator. The system can also robustly process inputs having sizes smaller than the fixed size for a fully convolutional network by padding zeros around the input to reach the fixed size.

[0012] For example, in the case of accelerators having up-to-date memory addressing capabilities (e.g., accelerators including a direct memory access (DMA) engine), the described techniques can reduce or eliminate the overhead time associated with data manipulation for tiling inputs and stitching fixed-size outputs. As another example, in the case of accelerators having a simpler architecture or a smaller memory bandwidth, the described techniques can execute operations for one model at a time. In some implementations, the described techniques determine whether an accelerator array exists in a computing system, and in response to determining that an accelerator array exists, the described techniques can reduce memory usage by leveraging data sharing characteristics between adjacent accelerators to execute inference operations on different tiles in parallel.

[0013] Furthermore, the techniques described herein are distinctive and advantageous over conventional data parallelization techniques. Generally, data parallelization techniques can divide input data (e.g., an input image) into a large number of disjoint parts (e.g., segments of the input image), assign a large number of parts to a large number of hardware components (e.g., hardware accelerators), process those parts independently and in parallel, and generate partial outputs. After all parts have been processed by the hardware components, a system configured to perform the data parallelization technique can generate a final output by aggregating the partial outputs. As long as the calculations are performed precisely by each hardware component for the part for which they were designed, the system does not need to consider whether any part of the partial outputs is unsuitable or inaccurate for generating the final output.

[0014] However, fully convolutional networks generally do not utilize data parallelization techniques. This is because the output produced by a fully convolutional network processing a portion of an input image (e.g., a tile of the input image as described herein) may contain one or more incorrect or inaccurate pixel-by-pixel values. This is because the computation of the system processing the input tile may include "adjacent pixels," which may result in some of the output pixels being inaccurate.

[0015] Throughout this specification, the term “adjacent pixels” refers to pixels that surround the boundary of an input to a fully convolutional network model. Adjacent pixels may include pixels added to the boundary of the input by zero-padding specified by one or more layers of the fully convolutional network model. In the case of a fixed-size input to a fully convolutional network model (e.g., tiles extracted from full input data), adjacent pixels may also include pixels that originally surrounded the fixed-size input in the full input data.

[0016] The region surrounding an input to a fully convolutional network model or a fixed-size input and containing adjacent pixels is referred to throughout the specification as the “adjacent pixel region.” The adjacent pixel region may have a width of one or more pixels. In some implementations, the width of the adjacent pixel region can be determined based on the properties of the fully convolutional network model. Adjacent pixels may have or be replaced with zero pixel values ​​during computation, which can lead to inaccurate output from processing adjacent pixels through the fully convolutional network model.

[0017] In some implementations, adjacent pixels initially exist in the complete input data. When a fixed-size input is extracted from the complete input data, the system may require one or more adjacent pixels to process the fixed-size input. However, the system may change the values ​​of one or more non-zero adjacent pixels to zero, leading to inaccurate calculations at some pixel locations.

[0018] For example, the system may include one or more convolutional layers with filter sizes of 2 or more. To handle boundary pixels of a fixed-size input, the system may use one or more neighboring pixels outside the boundary pixels to compute the corresponding per-pixel output. Non-zero neighboring pixels may be replaced with zero values ​​during computation. By using zero-value neighboring pixels instead of the true pixel values ​​associated with neighboring pixels to handle a fixed-size input, one or more pixel values ​​in the fixed-size output may become inaccurate.

[0019] As another example, the system may include one or more transposed convolutional layers with filter sizes of 2 or more. If the calculation for one of the transposed convolutional layers uses zero values ​​to replace non-zero adjacent pixels, the output pixel values ​​may be inaccurate.

[0020] In other words, zero-value adjacent pixels (e.g., non-zero pixels that were originally replaced with zero values) can make one or more pixel values ​​in the output tile inaccurate. Therefore, it is problematic for a system to perform operations in a fully convolutional network to process a fixed-size input to generate a final output by combining fixed-size outputs without determining and discarding inaccurate data. The system needs to determine both accurate data (e.g., valid values) and inaccurate data (e.g., dummy pixel values) based on the characteristics of the network layers in the fully convolutional network when processing a fixed-size input.

[0021] The technique described herein can determine which per-pixel values ​​in a fixed-size output are inaccurate by analyzing the characteristics of the network layers in a fully convolutional network and determining layer or overall alignment information and appropriate fixed sizes to compile a fully convolutional network model and tile input data. The alignment information and appropriate fixed sizes can be used for a system employing the technique described to generate accurate values ​​for each pixel in the final output through the fully convolutional network model. The accurate value for each pixel is generated at least once in at least one fixed-size output, and the system can obtain the accurate value for a pixel from at least one fixed-size output.

[0022] The techniques described herein can further reduce memory traffic by reducing, and even avoiding, the computation of invalid or overlapping pixel values ​​between different fixed-size outputs. In some situations where a fixed size is determined, the techniques can optimize memory traffic between the accelerator and the host by minimizing the exact pixel overlap of different fixed-size output tiles so that a valid final output can be produced based on the minimized overlap. In some situations where a fixed size has not yet been determined, the techniques described can select one of several candidate fixed sizes as the fixed size based on the characteristics of the input data and hardware so that computations to generate inaccurate or overlapping pixel values ​​are minimized or even eliminated.

[0023] Details of one or more embodiments of the subject matter described herein are shown in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0024] [Figure 1] This figure shows an exemplary inference system for performing inference computations on a fully convolutional network for inputs of different sizes. [Figure 2A] This diagram illustrates an example inference process using a convolutional system. [Figure 2B] This figure shows an exemplary inference process using the exemplary inference system shown in Figure 1. [Figure 3A] This figure shows an exemplary fixed-size input with adjacent pixel regions and an exemplary fixed-size output with dummy regions. [Figure 3B] Figure 1 shows an example of the tiling and stitching process performed by the exemplary inference system. [Figure 3C]This figure shows another example of the tiling and stitching process performed by the exemplary inference system in Figure 1. [Figure 3D] This figure shows an example process for generating output using a transposed convolutional layer in an FCN model. [Figure 4] This figure shows an exemplary process for performing inference computations on a fully convolutional network for inputs of different sizes. [Modes for carrying out the invention]

[0025] Detailed explanation A fully convolutional network (FCN) can include at least one or more convolutional neural network layers and optionally pooling layers, transposed convolutional layers, and element-wise layers (e.g., layers that apply element-size activation functions). An FCN can be deployed on a hardware accelerator to generate pixel-wise predictions for an input (e.g., an input image with multiple pixels). In particular, an FCN is configured to produce an output with pixels associated with one or more corresponding pixels in the input image, making predictions for each pixel of the input image. In some implementations, an FCN can also associate output pixels with input pixels and neighboring pixels in a fixed-size neighborhood. Since an FCN model can process pixels of input, an FCN can, in principle, process inputs of arbitrary size.

[0026] While FCNs have an advantage over typical neural networks in that they can generate pixel-by-pixel predictions for input data provided at different sizes, several hardware limitations make it difficult, or even impossible, to dynamically deploy FCNs on hardware accelerators (i.e., to process inputs of varying sizes).

[0027] Dynamically deploying FCNs to process inputs of varying sizes sequentially can introduce computational cost issues. First, network parameters, including data structures (e.g., matrix dimensions for computation, or padding, stride, filter size, and scale factors for network layers), scale with the input size, and changes in input size may require shuffling the current network parameters, which can lead to increased downtime (e.g., overhead) for systems involving numerous hardware accelerators. Furthermore, to accommodate dynamic input sizes, the host needs to send instructions to perform inference computations using more general execution mechanisms. For example, the host might allocate larger memory for data storage (sometimes at the cost of slower processing), perform more checks on vector or tensor sizes, or more frequently and dynamically change the number of computing units used to perform computations during parallel computing. Therefore, in practice, FCNs are typically statically deployed on one or more hardware accelerators (e.g., compiled with fixed network hyperparameters) and configured to accept fixed-size inputs to avoid the problems posed by dynamic deployment.

[0028] The techniques described below can solve the aforementioned problem by enabling statically compiled FCNs, already deployed (or to be deployed) on a hardware accelerator, to effectively handle inputs of different sizes.

[0029] The technique described allows input data of a specific size to be tiled into multiple smaller inputs, each with a fixed size. A statically compiled FCN on a hardware accelerator processes each of the multiple fixed-size inputs and the corresponding fixed size... It is possible to generate an output of size. The technique described can therefore stitch together fixed-size outputs to produce a final output as if the input had been fully processed by an FCN compiled for the input size.

[0030] In general, the described technique can provide a method for determining specific "tiling and stitching" parameters for tiling an input of a specific size into a number of fixed-size inputs and stitching a number of fixed-size outputs generated from the number of fixed-size inputs, in order to produce a final output equivalent to the output produced by processing the input holistically with an FCN compiled for a specific size. More specifically, according to the properties of the FCN model, the described technique can generate fixed-size outputs with different regions by processing fixed-size input tiles through the FCN model. The different regions may include dummy regions and effective regions. By analyzing the properties of the FCN model (e.g., padding, stride, filter size, scale factor, and layer type for all layers in the FCN model), the described technique can determine the effective regions in the fixed-size output where the per-pixel values ​​are accurate, i.e., zero-value neighboring pixels are not used to generate the output pixel values, and the dummy regions in the fixed-size output where the per-pixel values ​​are not at least "completely" accurate, i.e., the per-pixel values ​​are generated by the FCN by utilizing at least one zero-value neighboring pixel. The described technique can combine (e.g., "stitch") precise pixel-wise values ​​from all fixed-size outputs to generate the final output, ensuring that each precise pixel-wise value corresponding to a pixel in the final output is generated and obtained from at least one fixed-size output. This is in contrast to conventional parallelization techniques where input data is easily partitioned to generate output data independently, eliminating the need to consider inaccuracies caused by adjacent pixels.

[0031] The described technique can also determine the fixed size by various means before the FCN is compiled and deployed on a hardware accelerator. First, the described technique can propose multiple candidate sizes based on the FCN model and the characteristics of the hardware accelerator on which the FCN is deployed. The multiple candidate sizes are valid and suitable for the tiling and stitching processes performed by the described technique. For example, if the FCN includes one or more transposed layers, the multiple candidate sizes can be determined based on alignment information of the output tiles. Throughout the specification, the term “alignment information” refers to data that represents constraints or requirements for the arrangement of fixed-size outputs. The alignment information is acquired by the system so that the fixed-size outputs can be properly projected onto fixed-size inputs, or vice versa.

[0032] The system can also determine coordinate shifts between fixed-size outputs and corresponding fixed-size inputs based on alignment information to obtain an appropriate tiling pattern. The tiling pattern may include one or more fixed sizes (e.g., one or more candidate sizes selected automatically or by the user), overlap sizes for fixed-size inputs at a given fixed size, and optionally, coordinates for fixed-size inputs and outputs, in particular, coordinates for dummy and effective areas of fixed-size outputs.

[0033] The determined tiling pattern must satisfy at least two criteria: (i) the alignment information must be correct; that is, the tiling pattern should have fixed-size outputs precisely positioned so that each fixed-size output can be accurately projected onto a fixed-size input, or vice versa; and (ii) each pixel value for the complete output data should be generated and extracted from at least the effective area of ​​one fixed-size output. Selectively, the system will enhance computational performance and optimize computational resource usage. To achieve this, a tiling pattern can be determined that minimizes the overlap area for fixed-size inputs. More details on determining the tiling pattern based on alignment information will be discussed later.

[0034] In some implementations, the described technique can selectively choose one of the candidate sizes included in a suitable tiling pattern as a fixed size, based on performance metrics (e.g., total execution time or overhead). The described technique can also generate a range of candidate sizes for deploying an FCN model on a hardware accelerator and provide a range of candidate sizes for user selection. The user can then select one size from the range of candidate sizes as a fixed size, for example, according to the characteristics of the FCN model, hardware accelerator, or task's specific computational requirements.

[0035] The tiling pattern described above may include a fixed size for tiling the complete input data into one or more fixed-size inputs, and a size for the overlapping region for generating the fixed-size inputs. Generally, the system can tile the fixed-size inputs so that they often overlap each other, thereby ensuring that accurate or correct per-pixel values ​​are obtained for all pixels in the final output, i.e., each accurate value is generated and obtained from at least one fixed-size output. The tiling pattern described above may further include data representing the effective region and dummy region for the fixed-size outputs. A fixed-size output may generally include a dummy region of substantial size, consisting of one or more zero-value neighboring pixels used to generate the output pixel values. The system may employ one or more algorithms to determine alignment information based on the properties of the FCN model, and additional algorithms may be applied to determine the relationship (e.g., mapping) between the coordinates of the fixed-size outputs and the corresponding fixed-size inputs, to determine the effective region for the fixed-size outputs, and to determine the coordinate shift for the fixed-size outputs based on the mapping and effective region described above. Details of these algorithms will be described later.

[0036] After determining the tiling pattern, the system can stitch the fixed-size outputs by combining the effective regions of each fixed-size output. Note that the tiling pattern is generated based on alignment information for the FCN model. Since the system can generate a suitable tiling pattern for fixed-size inputs, the stitching process is quite efficient because the system has coordinate information for all pixels in the effective region of the fixed-size output. In some implementations, the system can take the pixel value in the effective region of the fixed-size output at least once for each pixel placed in the complete output data in order to generate complete output data. Details of the stitching will be discussed later in relation to specific algorithms and Figures 3A-3D.

[0037] Furthermore, the described technique can perform “tiling and stitching” analysis both online and offline. To deploy the compiled FCN on a hardware accelerator in a form similar to that previously deployed using the described technique, the host processor can perform the analysis offline by reusing previously saved parameters for “tiling” the complete input data and “stitching” the output tiles to generate complete output data of unknown or varying sizes. The previously saved parameters may include, at a minimum, tiling patterns such as alignment information for the FCN, fixed sizes for tiling, overlap regions for fixed-size inputs or fixed-size outputs or both, and dummy and effective regions for fixed-size outputs. The system reuses these parameters to process new complete input data, and new complete inputs are processed by new complete inputs. The complete output data can be generated as if it had been processed directly by the compiled FCN model due to the size of the force data. The host processor can generate a new set of "tiling and stitching" parameters for processing the input data in situations where a new FCN is deployed.

[0038] Figure 1 shows an exemplary inference system 100 for performing inference computations of a fully convolutional network for inputs of different sizes. Inference system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, which can implement the systems, components, and techniques described later.

[0039] As shown in Figure 1, system 100 includes a host 130 and an accelerator 110 that communicate with each other. Generally, system 100 receives input data 150 and generates output data 170 using a trained FCN 115 deployed on the hardware accelerator 110. Generally, the output data 170 can be smaller than the size of the input data 150, larger than the size of the input data 150, or equal to the size of the input data 150.

[0040] More specifically, the system can use the expanded FCN115 for tasks such as object detection and classification (e.g., face detection), image segmentation, image generation, image super-resolution, image completion, and image coloring, to name a few examples. For example, if the task is image segmentation, the input data 150 can be an image input, and the system 100 can generate output data 170 having pixel-by-pixel predictions, i.e., each prediction for each pixel of the input image or for each pixel of the output image generated by the FCN115. The output data 170 can also include a score distribution for each pixel, assigning each score to each of a number of categories. In this case, the system 100 can detect the presence, shape, and location of objects from the input image. As another example, if the task is image super-resolution, the system 100 can use the expanded FCN115 to increase the image resolution for the input image by predicting pixels to be added around each input pixel. In that case, the output image 170 may have a higher resolution than the input image 150, and one or more output cells may be associated with each pixel in the input image.

[0041] System 100 can compile an FCN for processing fixed-size inputs in a compilation engine 160 included in the host 130, and deploy the compiled FCN on the hardware accelerator 110. To compile the FCN, the host 130 can receive data 155 representing the trained FCN model in the compilation engine 160, compile the trained FCN model, and generate instructions (e.g., binary data) to deploy the trained FCN model on the hardware accelerator 110. In some implementations, the compilation engine 160 can recompile the trained FCN model to process different input data with varying sizes. Details on recompiling the trained FCN model and deploying the recompiled FCN model on the hardware accelerator 110 will be described later.

[0042] In general, a compiled FCN 115 can process any suitable input of the same size (e.g., a fixed-size input 138). Therefore, the hardware accelerator 110 can use the compiled FCN model 115 to perform inference calculations for the provided fixed-size input 138.

[0043] The compilation engine 160 can apply conventional compilation techniques to compile the FCN on the hardware accelerator 110. Generally, the compilation engine 160 can decode program code, which is encoded into machine-readable binary code on the hardware accelerator, by data written in any suitable high-level language and representing the characteristics of the FCN. The data representing the characteristics of the FCN may include hyperparameters that define the structure of the FCN (e.g., input size, number of layers, number of nodes in each layer, layer type and position, padding, stride, filter size, and scale factor for one or more layers), and layer weights obtained from the training process. During compilation, the system 100 needs to allocate its respective computational resources based on the characteristics of the FCN. For example, the system 100 needs to allocate its respective data structures to provide their respective computations for performing inference calculations. As another example, the system needs to allocate its respective memory to store its respective data structures and associated computation results while performing inference calculations for the unfolded FCN.

[0044] Conventionally, system 100 needs to allocate each data structure and memory according to the input size. For example, the data structures allocated for the layer weight matrix, activation input, and output are at least based on the input size. Each memory allocated to store the layer structure and associated computation results is also based on the input size. Therefore, the FCN being unfolded is configured to receive fixed-size inputs once unfolded. Furthermore, the system (e.g., system 100) often statically compiles the FCN to receive fixed-size inputs, thereby allowing the system to efficiently allocate computational resources all at once during compilation.

[0045] In some implementations, host 130 can perform tiling pattern analysis to determine tiling parameters, such as an appropriate fixed size, for compiling FCN115 based on the characteristics of the FCN model and associated hardware accelerators. It should be noted that one or more hosts different from host 130 (e.g., offline analysis / compilation hosts) can perform tiling pattern analysis offline or before host 130. These one or more hosts can then compile and deploy FCN115 on host 130 (e.g., one or more communicably coupled computers) or deploy FCN115 as an “application” on one or more edge devices (e.g., mobile phones or tablets) to handle random or unknown-sized inputs. Further details on determining the fixed size will be discussed later.

[0046] System 100 can include any suitable type of hardware accelerator 110 for performing inference calculations of the FCN model. For example, the hardware accelerator 110 can be a CPU, GPU, or TPU. The hardware accelerator 110 can include components such as memory for storing the parameters of the FCN model. Furthermore, the hardware accelerator 110 can include one or more computing units for parallel computing.

[0047] The input data 150 can have one or more sizes different from the fixed size in which the compiled FCN 115 is configured to process its input. For example, the input data 150 may include multiple image frames, each having a different size from the fixed size.

[0048] The generated output data 170 is generated by a system 100 that performs inference operations for a trained and statically expanded FCN model 115 for the input data 150. This is the output. Each of the generated output data 170 can have a size related to the size of the corresponding input data.

[0049] As a simple example, if the input data (e.g., an input image) has a size of 500x500 pixels, the generated output 170 can have a size of 50x50 pixels, 500x500 pixels, or 1000x1000 pixels, with each pixel of the output data being associated with a pixel in an 8x8, 10x10, or 20x20 pixel neighborhood of the input image, based on the properties of the FCN model (e.g., filter size, stride size, padding size, and scale factor for each layer of the FCN model). In general, the size of the output generated from the input by a trained FCN is a function of the properties of the FCN model.

[0050] For example, to simplify the explanation, a naive FCN model can include two network layers, each having a 2x2 pixel filter size, a 1-pixel stride size, and 1x1 zero padding, so that each of the two layers can process a 3x3 pixel input to produce a 4x4 pixel output. When a 3x3 input passes through both layers of the network, a 5x5 output is produced, and similarly, a 5x5 input produces a 7x7 output through the naive FCN model.

[0051] Assume a naive FCN model is compiled to receive tiled 5x5 pixel tiles from input data 150. The system can generate a 7x7 pixel fixed-size output by analyzing the properties of the FCN model. The system can then generate the final output 170 by determining the active and dummy regions for the fixed-size output and associating the per-pixel values ​​in the active region with the corresponding pixels in the final output for all fixed-size outputs. Referring to the example above, the active region of the fixed-size output generated by the naive FCN model may have a size of 3x3 pixels, with per-pixel values ​​calculated using the values ​​in the 5x5 pixel input tile (i.e., the per-pixel values ​​in the active region are not generated based on any padded zeros).

[0052] As another example, an FCN model may include two convolutional layers, each having a filter size of 3x3 pixels, a stride size of 1, and no zero padding. For a fixed-size input of 50x50 pixels, the fixed-size output produced by the FCN model processing the fixed-size input may have 46x46 pixels. System 100 can determine that there are no dummy regions in the fixed-size output and that the effective region of the fixed-size output is 46x46 pixels.

[0053] As another example, an FCN model may include two convolutional layers, each having a filter size of 3x3 pixels, a stride size of 1, and a single-pixel zero-padding. A fixed-size output from an FCN model processing a fixed-size input of 50x50 pixels may have a size of 50x50 pixels. System 100 can determine a dummy region with a width of 2 pixels on all sides of the output data (e.g., the output image), so the effective region of the fixed-size output is 46x46 pixels. The process of determining the dummy and effective regions of the fixed-size output is performed by the FirstValidPixelOffset() algorithm. More details regarding rhythm will be discussed later.

[0054] In general, if an FCN model includes one or more convolutional layers with stride sizes of 2 or more, the input and output sizes can be many-to-one mappings, and no longer one-to-one mappings. For example, an FCN model may have a first convolutional layer with a filter size of 3x3 pixels, a stride size of 1, and single-pixel zero padding. The FCN model may include a second convolutional layer having a 5x5 filter size, a stride size of 2, and zero padding of 1. An FCN model can produce an output of the same size (e.g., 24x24 pixels) by processing inputs of different sizes (e.g., 50x50 pixel input and 49x49 pixel input). This is because the stride size of 2 in the second convolutional layer can trigger a rounding process as the input is processed through each network layer.

[0055] In addition, an FCN model may include one or more transposed convolutional layers. For example, an FCN model may include a transposed convolutional layer having a filter size of 5 pixels, no zero padding, and a stride of 2 pixels. The transposed convolutional layer may be added to a second layer having a stride size of 2 pixels. A transposed convolutional layer is generally configured to increase (e.g., blow up) the output size from the input provided by the preceding layer by a factor based on the stride size of the transposed convolutional layer. In the above example, the transposed convolutional layer can produce a 51x51 pixel output by processing a 24x24 pixel output from the second convolutional layer. That is, an FCN model can process a 50x50 pixel or 49x49 pixel input to produce a 51x51 pixel output.

[0056] An inverted convolutional layer can generate or expand dummy regions even if the padding size is zero. The size of the dummy region can be based on the characteristics of the inverted convolutional layer, for example, the relationship between the filter size and the stride size. For example, if the inverted convolutional layer has a stride size smaller than the filter size, the output can include a dummy region because the calculation involves adjacent pixel regions when a fixed-size input is extracted from a complete input. As described above, by extracting a fixed-size input from complete input data, the FCN may include one or more zero-value adjacent pixels instead of true pixel values ​​in one or more calculations, which makes the calculation of boundary pixels in the fixed-size input inaccurate and produces an output with a dummy region containing inaccurate pixel values ​​and an effective region of true pixel values ​​surrounded by the dummy region.

[0057] Referring again to the example above, a dummy region is generated when a fixed-size input is extracted from complete input data (e.g., input data 150). Otherwise, if the FCN is compiled to process the complete input data directly, no dummy region is generated. For example, an FCN model containing one or more transposed convolutional layers can produce an output with a dummy region. This is because when a fixed-size input is extracted from complete input data, it causes the pixel values ​​of one or more non-zero adjacent pixels contributing to the output to be replaced with zero values.

[0058] To determine the effective and dummy regions of a fixed-size output, system 100 can trace one or more pixels in the input image for the output pixels by analyzing the properties of the FCN model and the coordinates of the output pixels, or vice versa. For example, system 100 can arrange 50x50 pixels in the input tile to generate a fixed-size output of 51x51 pixels. As described above, tiles generated from the input data 150 by a host 130 or hardware accelerator 110 with sufficient computing power can overlap each other in one or more pixels, and therefore the system can employ a specific algorithm (e.g., the ProjectBackwards() algorithm, as described in more detail later) to trace back the corresponding input pixels used to generate the output pixels in the fixed-size output through the expanded FCN model.

[0059] Referring again to Figure 1, the host 130 transmits data, or instructions, or both. By reaching this point, communication with the hardware accelerator 110 is possible. The host 130 and the hardware accelerator 110 can communicate via wired or wireless connection and, in some cases, may be located far apart from each other. For example, the host 130 may be a server located in a different physical location from where the accelerator 110 is located.

[0060] The host 130 can receive input data 150 that is larger than a fixed size and generate multiple fixed-size inputs 138, each having a size smaller than the input data 150. The host 130 can provide the fixed-size inputs 138 to the hardware accelerator 110 and receive multiple corresponding fixed-size outputs 133 from the hardware accelerator 110. The received fixed-size outputs 133 are generated by the hardware accelerator 110, which performs inference operations on the expanded FCN 115 for the multiple fixed-size inputs 138 provided.

[0061] In some implementations, as described above, a hardware accelerator including hardware components such as a CPU can perform a tiling process to tile the input data 150 into a number of fixed-size inputs 138.

[0062] If the input data 150 is smaller than a fixed size, the system 100 can pad the input data 150 with zeros to reach the fixed size and provide it to the hardware accelerator 110 for performing inference calculations.

[0063] To generate output data 170, the host 130 may further include a stitching engine 140 configured to combine the received fixed-size outputs 133. The stitching engine 140 performs a stitching process by determining alignment information for each of the fixed-size outputs 133 based on the characteristics of the fixed-size and unfolded FCN models, thereby generating a final output 170 that is equivalent to an output obtained by directly processing the input data 150 using the unfolded FCN model to process the input, which is the same FCN model but has the same size as the input data 150, without tiling.

[0064] In some implementations, a hardware accelerator, which includes a hardware component capable of performing a stitching process, can generate a final output 170 on the hardware accelerator based on a fixed-size output 133 and provide the final output 170 to the host 130 or on the user interface display.

[0065] In some implementations, the tiling and stitching processes described herein do not necessarily have to be performed remotely from the host. For example, any suitable accelerator, including a suitable hardware component such as a CPU, can perform the tiling and stitching processes on the accelerator. Furthermore, the tiling and stitching processes may be performed at a physical location different from the host. For example, tiling may be performed by a first set of accelerators at a first location, the stitching process may be performed by a second set of accelerators at a second location, and the host may be located at a third location and configured to receive the final output from the second set of accelerators. The accelerators and the host are communicated together and are physically or wirelessly connected at one or more locations.

[0066] Figure 2A shows an exemplary inference process 200 using a conventional system. As shown in Figure 2A, the conventional inference system 200 is (as shown in Figure 1) The system can receive input data 150 and generate output data 225 by performing inference calculations for an expanded FCN 215 that processes the input data 150. Output data 225 can be substantially similar to output data 170 as shown in Figure 1. Input data 150 can be an input image as described above.

[0067] Each of the 150 input data can have a different size, which means that conventional systems need to recompile the FCN215 for the hardware accelerator to handle different input sizes. For example, if the first input data has a size of 50x50 pixels, the system can deploy the FCN215 on the hardware accelerator to be configured to handle inputs of size 50x50 pixels. However, if the second input data has a different size from the first input data, for example, 100x100 pixels, the system must recompile the FCN215 to be configured to handle inputs of size 100x100 pixels.

[0068] For input data of varying sizes, conventional systems must first determine the size of a particular input and then determine whether FCN215 needs to be recompiled to process that input. In addition, system 200 must perform extra computational checks to monitor whether memory and data structures are properly allocated. In such cases, conventional techniques for performing inference computations can incur a substantial amount of overhead, reducing the computational efficiency for generating inference outputs given inputs of varying sizes.

[0069] Figure 2B shows an exemplary inference process 250 using the exemplary inference system 100 of Figure 1.

[0070] As shown in Figure 2B, in relation to Figure 1, an inference system 100 employing the described technique can avoid recompiling the unfolded FCN for inputs of different sizes, which reduces overhead and increases computational efficiency. More specifically, system 100 first statically unfolds the FCN model 235 on a hardware accelerator. System 100 can determine alignment information based on the properties of the FCN model and determine a tiling pattern including a dummy region of the fixed-size output and a fixed size for tiling the effective region so that each effective pixel can be obtained from at least one fixed-size output. The FCN model 235 is compiled to process inputs having a fixed size. System 100 can tile the input data 150 to generate multiple fixed-size inputs 230 based on the tiling pattern and provide the fixed-size inputs 230 for performing inference calculations using the unfolded FCN 235. System 100 can obtain fixed-size outputs 240 from the unfolded FCN 235 and stitch the fixed-size outputs 240 to generate the final output data 170. Output data 170 is equivalent to output data 225 obtained by directly processing the complete input data 150 using FCN215, which has been compiled to process the complete input data 150. Details of tiling and stitching will be described later.

[0071] In some implementations, system 100 can determine a set of candidate sizes suitable for the FCN model to process random-sized inputs. System 100 can determine a set of candidate sizes from all tile sizes based on the characteristics of the FCN model (e.g., layer characteristics such as filter size and stride size). It should be noted that some tile sizes cannot be used according to their characteristics. For example, certain input sizes cannot generate output sizes based on the filter size and stride size of the FCN model. For example, system 100 can determine a set of candidate sizes based on the FCN model's characteristics. Sizes unsuitable for generating can be removed from all possible sizes.

[0072] In some implementations, system 100 can select a fixed size from multiple candidate sizes to deploy the FCN model. For example, system 100 can select a fixed size based on performance.

[0073] In some implementations, for each candidate size, system 100 can deploy each copy of the FCN model on each hardware accelerator for processing inputs having one of the candidate sizes. System 100 can measure a level of performance, such as the total execution time for performing inference computations using different copies of the FCN network processing different fixed-size inputs, or, as an alternative example, the overhead in system 100, which includes multiple hardware accelerators for performing inference computations for each deployed FCN. Based on the performance measurement, system 100 can select one of the candidate sizes as a fixed size for deploying the FCN model on a particular hardware accelerator. For example, system 100 can select a candidate size that leads to the minimum total execution time. As an alternative example, system 100 can select a candidate size that results in the least overhead. Selectively, system 100 can select a candidate size that has a satisfactory execution time and overhead for performing inference computations.

[0074] The selection of candidate sizes can be based on the characteristics of the trained FCN model. For example, assuming that the candidate size for tiling (or for the expanded FCN model) is too small, the fixed-size output generated from a fixed-size input of the candidate size may also be small and may even contain no effective region (i.e., all per-pixel values ​​in the fixed-size output are in the dummy region).

[0075] Selectively, system 100 can provide the user with a distinct range of candidate sizes to choose one candidate size within that range as a fixed size. The distinct range of candidate sizes can be discontinuous based on the characteristics of the FCN (e.g., the number and location of one or more transposed layers, and the characteristics of each layer included in the FCN). For example, the range of candidate sizes can be an even number of pixels from 10x10 pixels to 30x30 pixels. The user can then choose 16x16 pixels as a fixed size within the provided range.

[0076] Furthermore, the fixed size does not have to be a scalar. Instead, the fixed size can be a vector representing a rectangle in two-dimensional space or a block in three-dimensional space. More specifically, the fixed size can contain values ​​in each dimension. For example, if the input image is two-dimensional, system 100 can determine a fixed-size vector having a first size for a first dimension (e.g., the horizontal dimension) and a second size for a second dimension different from the first dimension (e.g., the vertical dimension). System 100 can generate multiple 30x10 pixel fixed-size inputs from an input image with a size of 300x100 pixels.

[0077] Referring to Figures 1 and 2B, after receiving input data 150, system 100 can determine multiple fixed-size inputs from the received input data. For input data with a size larger than the fixed size, system 100 can analyze the input data and tile it into multiple fixed-size inputs with or without overlap. For input data with a size smaller than the fixed size, system 100 pads zeros around the input data to reach the fixed size and expands it into an FCN model for performing inference calculations (now fixed size It can provide padded input (which is also an is).

[0078] To tile the received input data 150 into multiple fixed-size inputs 138, the host 130 may include a tiling engine 135 configured to receive the input data 150 and generate multiple fixed-size inputs 138 based on a tiling pattern. Alternatively, a suitable hardware accelerator 110 can tile the input 150 into a number of fixed-size tiles. More specifically, the host 130 can send an instruction to the hardware accelerator 110 containing binary data representing a compiled FCN model and a memory address storing the input data 150. The hardware accelerator 110 may include a suitable computing component such as a CPU and is configured to acquire one or more tiles by accessing corresponding memory addresses that store the per-pixel values ​​for one or more tiles (e.g., direct memory access). For example, the hardware accelerator can acquire a 5x5 pixel tile by accessing the corresponding memory address that stores the per-pixel values ​​of the tile without accessing the memory address that stores the pixel values ​​outside the tile. In this way, the system 100 can reduce memory traffic and increase computational efficiency as described above.

[0079] System 100 can determine a tiling pattern for tiling input data 150 into multiple fixed-size inputs 138. For example, the tiling engine 135 can tile the input data 150 into fixed-size inputs of a specific size with a specific overlap size, such that the tiled fixed-size inputs have no overlap, each have a shared overlap area of ​​a specific size, or each overlaps with each other in their respective sizes. Therefore, the total number of fixed-size inputs generated from the input data depends on the tiling pattern.

[0080] Note that if the FCN model includes one or more transposed convolutional layers with stride sizes of 2 or more, the tiling pattern is further determined based on alignment information.

[0081] The overlap size for a tiling pattern can be any suitable size smaller than the fixed size. For example, each fixed-size input may have a shared overlap size of one pixel width and the length of the edges of the fixed-size inputs. As another example, the overlap size can be 2 pixels, 3 pixels, and 5 pixels wide. The fixed size and overlap size for tiling are determined based on at least the alignment information.

[0082] System 100 can automatically determine a tiling pattern based on the characteristics of the FCN model or user instructions. For example, System 100 can tile a 100x100 pixel input image into four fixed-size inputs, each of 60x60 pixels. Each of the fixed-size inputs can have overlapping regions of 20x60 pixels, 60x20 pixels, or 20x20 pixels from one another.

[0083] Selectively, system 100 can also generate a tiling pattern such that fixed-size inputs have overlapping regions with each other. For example, a 70x30 pixel input image can be tiled into 30x30 pixel fixed-size inputs that fit the unfolded FCN model. In one scenario, four fixed-size inputs overlap each other in a 20x30 pixel region. The last fixed-size input may have a 10x30 pixel region outside the input image, which can be expanded or sewn. Note that padding may occur. In some implementations, the system may shift the overlap region of the last fixed-size input along with other inputs to reduce and even eliminate padding zeros in order to improve computational efficiency.

[0084] In some implementations, system 100 can determine a tiling pattern based on a suitable machine learning model trained on various training data. The training data can be each set of fixed-size inputs for the same copy of the input, but each is tiled based on a different tiling pattern. The machine learning model can output one or more tiling patterns for system 100 or for the user to select for system 100.

[0085] System 100 can generate output 170 by stitching fixed-size outputs using the stitching engine 140. Since System 100 obtains a tiling pattern containing the coordinates of pixels in the valid area, the system can efficiently stitch pixels from the valid area to generate complete output data. The system can employ algorithms for the stitching process, which will be described in more detail later.

[0086] System 100 can obtain, for each fixed-size output, the coordinates for a particular fixed-size output and the coordinates of the corresponding fixed-size input for generating that particular fixed-size output. The coordinates of a particular fixed-size input represent the position of the fixed-size input relative to the original input 150, and similarly, the coordinates of a particular fixed-size output represent the position of the fixed-size output relative to the corresponding final output 170. System 100 can determine each coordinate frame (e.g., a Cartesian coordinate frame, or any appropriate individual coordinate frame) and the origin of the coordinate frame for each input and corresponding output data. System 100 can determine the coordinates of the fixed-size input during the tiling process and then determine the coordinates of the corresponding fixed-size output according to the properties of the unfolded FCN model 115. Similarly, System 100 can first determine the coordinates of the fixed-size output and then determine the coordinates of the corresponding fixed-size input based on the properties of the FCN model 115. System 100 can apply one or more algorithms to generate alignment information, generate relationships between the coordinates of fixed-size inputs and fixed-size outputs, and stitch the fixed-size outputs based on those relationships. Details of the alignment information will be described later.

[0087] Once the alignment constraints for the FCN model are met, system 100 can further determine the central effective region and surrounding dummy region of each fixed-size output after associating the coordinates of the fixed-size output and the corresponding fixed-size input. The central effective region includes pixels generated using effective pixels from the corresponding fixed-size input. The dummy region includes pixels generated using one or more zero-value neighboring pixels (for example, from zero values ​​that should have been non-zero for pixels in the full input image outside the fixed-size input).

[0088] System 100 can determine one or more overlapping regions between fixed-size outputs. Selectively, the system can also determine whether at least a portion of the overlapping region belongs to the effective region of a fixed-size output. In some implementations, system 100 can determine coordinate shifts for one or more overlapping fixed-size outputs so that the effective regions of different fixed-size outputs are positioned adjacent to or touching each other without overlapping.

[0089] Figure 3A shows an exemplary fixed-size input 138 and dummy with adjacent pixel region 310. —An exemplary fixed-size output 133 having region 320 is shown.

[0090] As described above, system 100 can tile the complete input data 150 into multiple fixed-size inputs having fixed sizes that fit for the expanded FCN model. System 100 can determine a tiling pattern for the input data 150 and generate fixed-size inputs by tiling the input data 150 from top to bottom and left to right. The tiling pattern can include overlapping regions and, essentially, positions defined by the respective coordinates of each fixed-size input. For example, as shown in Figure 3, one fixed-size input 138 is positioned at a specific location in the complete input data 150. Details of the tiling pattern will be described later in relation to Figures 3B and 3C.

[0091] The position of a fixed-size input 138 can be represented using the coordinates of one or more corner pixels relative to the origin of the complete input data 150. For example, the system can determine the top-left corner pixel of the complete input data 150 as the origin (0,0). The coordinates for each fixed-size input are determined relative to the origin. For example, system 100 can use the coordinates of the top-left and bottom-right corner pixels of the fixed-size input 138 to represent its position and size.

[0092] Fixed-size inputs can be represented by any suitable coordinate frame. For example, the coordinates of each fixed-size input 138 can be represented in a Cartesian coordinate frame, a cylindrical coordinate frame, or any other suitable coordinate frame.

[0093] The tiling pattern can define the position for each fixed-size input 138 in any suitable form. For example, fixed-size inputs can be positioned in rows and columns. As another example, fixed-size inputs can be scattered and misaligned. In other words, fixed-size inputs 138 do not need to be aligned with each other in rows and columns, for example, in a zigzag pattern.

[0094] System 100 can annotate the location of fixed-size inputs using any appropriate notation. For example, System 100 can use (i,j) notation to represent a fixed-size input at the i-th location along the first dimension and the j-th location along the second dimension. For simplicity, in the following specification, System 100 annotates fixed-size inputs in a tiling grid; that is, each fixed-size input is represented by a sequential number along the row and column. Each fixed-size input can be conceived substantially as a rectangle. However, it should be acknowledged that the tiling pattern and annotations may vary based on the tiling requirements.

[0095] System 100 determines the coordinates of the top-left corner pixel

[0096]

number

[0097] For example, the bottom right corner pixel

[0098]

number

[0099] It can be displayed as follows, where i and j represent the numbering of each fixed-size input for input data 150. For example, i and j represent the respective rows and columns for the fixed-size inputs of all the numerous fixed-size inputs.

[0100] As another example, assuming the input image has 100x100 pixels, system 100 tiles the input image into a 3x3 grid (i.e., 9 fixed-size inputs) each with an overlap size. The fixed-size input in the first row of the 3x3 grid is the first fixed-size input placed in the first grid,

[0101]

number

[0102] and

[0103]

number

[0104] The coordinates can be such that a second fixed-size input placed in a second grid is

[0105]

number

[0106] and

[0107]

number

[0108] The coordinates can be such that a third fixed-size input placed in a third grid is

[0109]

number

[0110] and

[0111]

number

[0112] It may include having coordinates. The fixed-size input in the first column of a 3x3 grid may include the first fixed-size input, and the fourth fixed-size input placed in the fourth grid may include,

[0113]

number

[0114] and

[0115]

number

[0116] The coordinates can be such that the fifth fixed-size input placed in the seventh grid is

[0117]

number

[0118] and

[0119]

number

[0120] It can have coordinates. Note that the pixel values ​​of the third and fifth fixed-size inputs outside the input image may be expanded and set to zero.

[0121] To avoid redundant counting or calculation of each edge pixel of a large number of fixed-size inputs 138, in some implementations, during tiling, the system 100 can determine that for each of the fixed-size inputs 138, pixels at the top and left edges of the fixed-size input are considered to be included in the fixed-size input, while pixels at the bottom and right edges of the fixed-size input are not considered to be included in the fixed-size input.

[0122] Before tiling the input data 150 into multiple fixed-size inputs, the system 100 can determine whether the input data is smaller than a fixed size set for the system 100. In response to the determination that the input data 150 is smaller than the fixed size, the system 100 can pad zeros around the periphery of the input data 150 to reach the fixed size.

[0123] It should be noted that the term “adjacent pixel region 310” refers to a region containing adjacent pixels generated by using zero values ​​to replace the original non-zero values ​​for adjacent pixels, as described above. For example, the adjacent pixel region 310 may include a region containing one or more adjacent pixels of the fixed-size input 138 in the complete input data 150, as shown in Figure 3A. The width 315 of the adjacent pixel region 310 can represent the number of adjacent pixels contained within the adjacent pixel region 310. System 100 can determine the width 315 for the adjacent pixel region 310 based on the characteristics of the expanded FCN115 model.

[0124] System 100 can obtain the coordinates for each fixed-size output 133 with respect to the final output data 170. For example, System 100 can select the top-left corner pixel of the final output data 170 as the origin, and obtain the coordinates of the top-left corner pixel of the fixed-size output.

[0125]

number

[0126] For example, the bottom right corner pixel

[0127]

number

[0128] It can be displayed as follows, where i and j represent the numbering of each fixed-size output for output data 170. For example, i and j represent the respective rows and columns for each fixed-size output of all fixed-size outputs.

[0129] As described above, the system can further determine the effective region 330 and dummy region 320 for each fixed-size output 133 based on the characteristics of the FCN model. Generally, the effective region 330 may be located in the center of the fixed-size output 133, and the dummy region 320 may surround the effective region 330 with a width 335. The width 335 determines a specific number of pixels in each dummy region 320. The effective region includes per-pixel values ​​for pixels in the effective region calculated using the effective per-pixel values ​​in the corresponding fixed-size input 138, while the dummy region 320 includes per-pixel values ​​for pixels in the dummy region calculated using at least one or more adjacent pixels during the tiling process or through operations characterized in one or more layers in the FCN model. At least some of the per-pixel values ​​for pixels in the effective region 330 contribute to the final output 170, while dummy pixels are excluded or discarded during the stitching process.

[0130] System 100 can determine the valid region 330 and the dummy region 320 by tracing back from pixels in the fixed-size output through the FCN model to one or more pixels in the corresponding fixed-size input, according to the characteristics of the FCN model. More specifically, System 100 can execute the FirstValidPixelOffset() algorithm, as described later, to determine the width for the dummy region. The region is the remaining portion of the region in the output.

[0131] More specifically, the FirstValidPixelOffset() algorithm is used to determine the final dummy for the FCN output. The system is configured to propagate invalid information layer by layer to determine the region. In the first layer of the FCN, the layer generates a dummy region in its output by using pixels in the adjacent pixel region for the first layer. However, from the second layer onward, the dummy region of the layer output grows by using adjacent pixels and dummy pixels generated and propagated from the preceding layer.

[0132] By executing the FirstValidPixelOffset() algorithm, system 100 will Based on the characteristics of the FCN model (e.g., the respective filter size, zero-padding size, stride size, and scale factor for each layer in the FCN model), a width of 335 and the number of pixels within that width can be determined. Note that a width of 335 for the dummy region can contain all dummy pixels. However, in some implementations, a width of 335 is large enough to contain all dummy pixels and one or more valid pixels.

[0133] If the FCN model includes one or more transposition layers, the system 100 is one or more The width 335 of the dummy region 320 can be determined based on the number and position of the transposed layers. In relation to Figure 3D, which shows an exemplary process of generating an output using transposed convolutional layers 340, 345 in the FCN model, the FCN model can be equivalent to the compiled full convolutional network 115 in Figure 1. For simplicity, a well-configured system, for example, the inference system 100 in Figure 1, can perform the process in Figure 3D.

[0134] As shown in Figure 3D, the FCN model may include a transposed convolutional layer 340 configured to receive a 2x2 pixel output 341 from a preceding layer in the FCN model. The system can perform operations associated with the transposed convolutional layer 340 to produce a 4x4 pixel output 342. The transposed convolutional layer 340 includes a 3x3 pixel filter size with a stride size of 1. The transposed convolutional layer 340 does not include any zero padding. Input pixel A is associated with output pixels A1, A2, A3, C1, C2, C3, D1, D2, and D3, and input pixel B is associated with output pixels C1, C2, C3, D1, D2, D3, B1, B2, and B3. The overlapping region of output pixels associated with input pixels A and B includes pixels C1, C2, C3, D1, D2, and D3.

[0135] Note that pixels C1, C2, and C3 are also associated with the input pixel to the left of pixel A in output 341 (not shown). Similarly, pixels A1, A2, and A3 are associated with the two input pixels to the left of pixel A, and pixels D1, D2, and D3 are associated with input pixels A and B, as well as another input pixel to the right of pixel B.

[0136] Assuming that the complete input image can generate intermediate outputs through a preceding layer containing the first pixel to the left of pixel A, pixel A, and pixel B, the pixel values ​​A1, A2, A3, C1, C2, and C3 are not accurate. This is because the fixed-size input does not generate a pixel value for the first pixel, and therefore system 100 uses a zero-value adjacent pixel to represent the first pixel in order to generate partial outputs for pixels A1, A2, A3, C1, C2, and C3. However, the pixel values ​​D1, D2, D3, B1, B2, and B3 are accurate. This is because both the complete input and the fixed-size input use a zero-pixel value for the pixel to the right of pixel B.

[0137] Similarly, the transposed convolutional layer 345 has a stride size of 2 pixels in both directions and a filter size of 3x3 pixels, and is configured to receive the output 344 from the preceding layer and produce a 5x5 pixel output 345. Input pixels A are associated with pixels A1, A2, A3, C1, C2, C3, D1, D2, and D3, and input pixels B are associated with pixels D1, D2, D3, B1, B2, B3, E1, E2, and E3. The overlapping region includes the exact pixels D1, D2, and D3 because these pixels are not calculated using adjacent pixels.

[0138] Figure 3D shows only the determination of accurate and inaccurate pixel values ​​for a single transposed convolutional layer, but system 100 can determine the dummy region and effective region for a fixed-size output by analyzing the relationship between input and output for all layers of the FCN model as described above.

[0139] In addition, system 100 can determine alignment information based on the relationship between the inputs and outputs for each layer of the FCN model.

[0140] For example, as shown in Figure 3D, unlike the transposed convolutional layer 340, the transposed convolutional layer 345 has a stride of 2 pixels. Therefore, outputs including C1, C2, C3, D1, D2, D3, B1, B2, and B3 do not have corresponding pixels in output 344. System 100 can determine alignment information for the transposed convolutional layer 345 to be an integer multiple of 2 pixels. An integer multiple of 2 pixels can be, for example, 2, 4, 8, and 10 pixels, which are included in the alignment information to ensure effective mapping in the transposed convolutional layer.

[0141] If the FCN model includes two or more transposed layers, system 100 can determine overall alignment information for the entire FCN model (e.g., accumulated alignment values ​​for all layers, or overall alignment values) based on the characteristics of all transposed layers (e.g., number, location, and stride of the transposed layers). In some implementations, system 100 can determine the overall alignment information as the product of the respective stride sizes of all transposed layers.

[0142] System 100 can determine alignment information from a number of candidate alignment values ​​based on the correctness of the final output, the memory traffic during computation, and the computational efficiency. In particular, with respect to the correctness of the final output, System 100 can select an overall alignment value that ensures that each pixel of the final output can be obtained from one of the valid regions of the fixed-size output.

[0143] For FCN models that include other types of layers, such as pooling layers, system 100 can process these other types of layers as a form of convolutional layer for analyzing the tiling and stitching processes. For example, a maximum pooling 2x2 layer may be processed as a convolutional layer with a stride of 2 pixels, a filter size of 2x2 pixels, and no zero padding for analyzing the tiling and stitching processes.

[0144] For the sake of clarity, the sizes of outputs 341 and 344 are 2x2 pixels, the size of output 342 is 4x4 pixels, and the size of output 346 is 5x5 pixels, but it should be noted that the inputs and outputs can generally have any suitable size. Similarly, the filter sizes, stride, and zero padding for the transposed convolutional layers 340 and 341 can include any suitable size.

[0145] In general, when an FCN model includes one or more transposed layers, determining the dummy region in a fixed-size output can be substantially complex. However, systems performing the techniques described herein can determine the propagation of dummy regions from preceding to succeeding layers, regardless of whether the layer is a convolutional layer or a transposed convolutional layer, and theoretically, regardless of how many network layers the FCN model includes, it is possible to determine the dummy region for a fixed-size output given a fixed-size input based on the characteristics of the FCN model.

[0146] One or more layers of an FCN model may have different properties along different dimensions (e.g., height and width dimensions in the case of a two-dimensional layer). For example, the filter size, stride size, or padding size of a network layer may not be the same along the height and width dimensions (e.g., a 3x2 pixel filter size, a 2x1 pixel stride size, and a 0x1 pixel zero padding size). The techniques described herein can independently compute alignment information, dummy regions, and tiling patterns along each dimension, which can result in non-uniform fixed-size outputs along different dimensions. For example, system 100 generates non-uniform widths for dummy regions 320. It is possible that the width 335 may be uneven due to the dummy area 320. For example, the width 335 of the left and right portions of the dummy area 320 may be larger than that of the top and bottom portions.

[0147] Generally, FCN models can receive an input tensor and generate an output tensor in multiple dimensions. For example, the input tensor may have multiple channels C and multiple batches B, in addition to the height H and width W dimensions mentioned above.

[0148] An FCN model can be adapted to handle any number of dimensions of an input, as long as the dimensions are fully convoluted. For example, an FCN model can handle an image input with B × H × W × C dimensions. Assuming the batch and channel dimensions are not fully convoluted, an FCN model can handle the input only in the height and width dimensions, where the process can generally be considered a two-dimensional problem. As another example, an FCN can handle multi-dimensional audio inputs by handling only single-dimensional audio inputs if the rest of the dimensions are not fully convoluted. Alternatively, an FCN model can handle higher dimensions, e.g., dimensions higher than two, if those dimensions are fully convoluted.

[0149] System 100 can also determine the coordinates of the effective region 330 of the fixed-size output 133. Similarly, the system can determine the upper-left corner pixel of the effective region with respect to the origin of the fixed-size output 133.

[0150]

number

[0151] For example, the bottom right corner pixel

[0152]

number

[0153] It can be displayed as follows: For example, i and j represent the respective rows and columns for the corresponding fixed-size output 133 or the valid area of ​​the corresponding fixed-size output 133.

[0154] In the case of an unfolded FCN model without a transposed convolutional layer, system 100 can stitch the fixed-size output through a first algorithm described later. As another example, in the case of an unfolded FCN model with a transposed convolutional layer, system 100 can stitch the fixed-size output based on alignment information generated using a second algorithm described later.

[0155] The first algorithm can ensure that the effective regions of the fixed-size outputs do not overlap, while the second algorithm potentially causes the effective regions of the fixed-size outputs to overlap, which requires an extra step to correctly combine the fixed-size outputs. The extra step may include coordinate shifts for each effective region of the fixed-size outputs, or for each or both of the fixed-size outputs, details of which will be discussed later.

[0156] When using the first algorithm, system 100 has a dummy region 335 width as b, and fixed-size inputs, corresponding fixed-size outputs, and fixed-size outputs respectively. Mapping function for coordinates of the effective region

[0157]

number

[0158] ,

[0159]

number

[0160] and

[0161]

number

[0162] can be displayed. Each mapping function can return a specific coordinate in a specific direction (e.g., I(i,j).

[0163]

Number

[0164] represents the coordinates in the vertical or height direction. For simplicity, system 100 assumes that the fixed-size input and fixed-size output are square in the two-dimensional space, and represents the size of the fixed-size input as T I , and represents the size of the fixed-size output as T O The system represents the size of the initial input data as H I and W I Without loss of generality, it is assumed that H I >= T I and W I >= T I Note that the fixed-size input and output can be rectangular in some implementations.

[0165] System 100 can execute the following first algorithm using dynamic programming to scan from left to right and top to bottom according to each coordinate of the fixed-size output in order to generate the final output 170. The first algorithm is written as follows: Initialization: O(0,0) = (0,0,T O ,T O ) V(0,0) = (0,0,T O - b,T O - b) Left boundary tile: O(i,0) = (V(i - 1,0).hb - b,0,V(i - 1,0).hb - b + T O ,T O ) V(i,0) = (V(i - 1,0).hb,0,V(i - 1,0).hb - 2b + TO ,T O - b) Upper realm タイル: O(0,j) = (0,V(0,j - 1).wb - b,T O ,V(0,j - 1).wb - b + T O ) V(0,j) = (0,V(0,j - 1).wb,T O - b,V(0,j - 1).wb - 2b + T O ) Internal タイル: O(i,j) = (V(i - 1,j).hb - b,V(i,j - 1).wb - b,V(i - 1,j).hb - b + T O ,V(i,j - 1).wb - b + T O ) V(i,j) = (V(i - 1,j).hb,V(i,j - 1).wb,V(i - 1,j).hb - 2b + T O V(i,j - 1).wb - 2b + T O ) Right realm タイル: O(i,lastj) = (V(i - 1,lastj).hb - b,W O - T O ,V(i - 1,lastj).hb - b + T O W O ) V(i,lastj) = (V(i - 1,lastj).hb,W O - T O + b,V(i - 1,lastj).hb - 2b + T O W O ) Lower realm タイル: O(lasti,j) = (H O - T O ,V(lasti,j - 1).wb - b,H O ,V(lasti,j - 1).wb - b + T O ) V(lasti,j) = (H O- T O + b,V(lasti,j - 1).wb,H O ,V(lasti,j - 1).wb - 2b + T O ) According to the first algorithm described above, system 100 can generate effective fixed-size outputs with effective regions adjacent to each other without overlap. More specifically, system 100 can discard pixels in dummy regions and combine the effective regions in the fixed-size outputs to generate the final output. Furthermore, since the effective regions between the fixed-size outputs do not overlap, a system using the first algorithm can compute almost all pixels in the effective regions in just one step, which optimizes computational efficiency for FCN models without transposed convolutional layers. An example of this implementation will be described in more detail with reference to Figure 3B.

[0166] In the case of an FCN model that includes a transposed convolutional layer, system 100 needs to execute a second algorithm to address alignment information. The effective regions generated using the second algorithm may potentially overlap, potentially resulting in duplicate calculations for one or more pixels within the effective regions.

[0167] System 100 can obtain alignment information for a fixed-size output according to the computational requirements indicated by the transposed convolutional layer in the FCN model. For example, this requirement may be that the pixel exponents for one or more pixels in the fixed-size input traced from one or more pixels in the fixed-size output should be integers.

[0168] The second algorithm is written as follows: Initialization: O(0,0) = (0,0,T O ,T O ) V(0,0) = (0,0,T O - b,T O - b) Left border tile: U O = (V(i - 1,0).hb - b,0,V(i - 1,0).hb - b + T O ,T O ) O(i,0) = AlignOutputTile(U O ) V(i,0)=(O(i,0).ht + b,0,O(i,0).hb - b,T O - b) Upper boundary tile: U O = (0,V(0,j - 1).wb - b,T O ,V(0,j - 1).wb - b + T O ) O(0,j) = AlignOutputTile(U O ) V(0,j) = (0,O(0,j).wt + b,T O - b,O(0,j).wb - b) Interior tiles: U O = (V(i - 1,j).hb-b,V(i,j - 1).wb - b,V(i - 1,j).hb - b + T O ,V(i,j - 1).wb - b + T O ) O(i,j) = AlignOutputTile(U O ) V(i,j)=(O(i,j).ht + b,O(i,j).wt + b,O(i,j).hb - b,O(i,j).wb - b) The second algorithm is a modified version of the first algorithm. In particular, system 100 can obtain coordinates of an "unaligned" fixed-size output that does not take alignment requirements into account. The fixed-size output is U OIt is displayed as having both dummy and active regions, with the left and top dummy regions omitted. The second algorithm can determine alignment information for "unaligned" fixed-size outputs, which are based on the AlignOutputTile() function below. Based on this, it is possible to determine whether the alignment information is met. The alignment information AlignOutputTile() can be obtained using the AlignOutputTile() function based on at least one of the following: local search or analytical method. Alignment information is "unaligned". This may include coordinate shifts to shift the fixed-size output to the left and upward. In some implementations, the alignment information can represent alignment values ​​analytically determined based on the characteristics of the FCN model. Details of the alignment values ​​and the functions for obtaining them are described later.

[0169] By executing the second algorithm, system 100 can ensure that each pixel value associated with the final output can be obtained from at least one of the fixed-size outputs, and that the alignment values ​​for the fixed-size outputs ensure that each corresponding fixed-size input has integer pixel coordinates with respect to the input image. Thus, system 100 can obtain the coordinates of the effective region by subtracting the dummy region using the second algorithm.

[0170] Details of the tiling and stitching processes using the first and second algorithms are described in relation to Figures 3B and 3C, respectively.

[0171] System 100 can also use the properties of the expanded FCN model to obtain the coordinates of the fixed-size input based on the coordinates of the corresponding fixed-size output. More specifically, System 100 can obtain the coordinates of the insertion force based on the coordinates of the layer output, as well as the layer padding, stride, filter size, and scale factor. One exemplary algorithm is called "ProjectBackwards()" and is written as follows: function ProjectBackwards((ht, wt, hb, wb),layers): for layer = output to input layers: if ht == hb or wt == wb: THROW EXCEPTION; / / Layer is destroyed if layer type is “conv”: / / Convolutional layer: n = floor((m + 2p - f) / s) + 1, where n is the output size and m = input size s = stride of the layer; p = padding of the layer; f = filter size of the layer n_h = hb - ht; n_w = wb - wt / / Output tiles in h and w dimensions size / / Input tile size m_h = (n_h - 1) * s + f - 2p; m_w = (n_w - 1) * s + f - 2p / / Note: m_h x m_w is the minimum input tile size, / / Any size up to (m_h + s-1) x (m_w + s-1) works to produce n_h x n_w output. / / If the trans_conv layer precedes the conv layer in the model order and does not allow a certain size / / Choosing a larger size can be problematic. / / Sizes are not shown here for simplicity and can be investigated as backtracking. ht = ht * s; hb = ht + m_h wt = wt * s; wb = wt + m_w else if layer type is “trans_conv”: / / TransConv layer:n=(m+2p-1)*s+f s = stride of the layer; p = padding of the layer; f = filter size of the layer n_h = hb - ht; n_w = wb - wt / / Output tiles in h and w dimensions size / / Input tile size m_h = Validate((n_h - f) / s - 2 p + 1) m_w = Validate((n_w - f) / s - 2 p + 1) ht = Validate( ht / s ); hb = ht + m_h wt = Validate( wt / s ); wb = wt + m_w return (ht, wt, hb, wb) here: function Validate(value): If the value is integral: return value else: THROW EXCEPTION / / Values ​​cannot be used for coordinates of fixed-size inputs in FCN The ProjectBackward() algorithm takes a fixed-size input and moves the coordinates from the layer's output to the layer's input. The Validate() function is called to check whether it can be projected properly. This Validate() function can determine, for example, whether the output location (e.g., pixel coordinates) selected by system 100 is not suitable for alignment constraints or alignment information for one or more transposed convolutional layers (i.e., the projected coordinates contain non-integer values), and therefore, the output location that system 100 attempts to project onto a fixed-size input location is invalid and cannot be used.

[0172] In some implementations, system 100 can also obtain coordinates for a fixed-size output based on the coordinates of the corresponding fixed-size input and the properties of the expanded FCN model. One exemplary algorithm is called "ProjectForward()" and is written as follows: function ProjectForward((ht, wt, hb, wb), layers): for layer = input to output layers: if ht == hb or wt == wb: THROW EXCEPTION; / / Layer is destroyed if layer type is “conv”: / / Convolutional layer: n = floor((m+2p-f) / s)+1, where n is the output size, m = input size s = stride of the layer; p = padding of the layer; f = filter size of the layer m_h = hb - ht; m_w = wb - wt / / Input tile size in dimensions h and w / / Output tile size n_h = floor((m_h + 2 p - f) / s) + 1; n_w = floor((m_w + 2 p - f) / s) + 1 ht = Validate(ht / s); hb = ht + n_h wt = Validate(wt / s); wb = wt + n_w else if layer type is “trans_conv”: / / TransConv layer: n=(m+2p-1)*s+f s = stride of the layer; p = padding of the layer; f = filter size of the layer m_h = hb - ht; m_w = wb - wt / / Input tilizer in h and w dimensions Iz / / Output tile size n_h = (m_h + 2 p - 1) * s + f; n_w = (m_w + 2 p - 1) * s + f; ht = ht * s; hb = ht + n_h wt = wt * s; wb = wt + n_w return (ht, wt, hb, wb) Similarly, the Validate() function can be used by the ProjectForwards() algorithm to validate the projection of a fixed-size input location onto a corresponding output location, for example. A fixed-size input location for a convolutional layer having stride sizes of 2 or more is It is possible to determine whether or not it is appropriate.

[0173] Referring again to Figures 3A and 3D, the system can determine the width b for a dummy region such that the region of width b should contain at least all inaccurate pixels. In some implementations, the dummy region can contain all inaccurate pixels and one or more accurate pixels. However, the width b should not be too large to impair computational performance, because a large width b could result in the system 100 having a greater number of overlapping fixed-size outputs and fixed-size inputs during the tiling and stitching process. The system 100 can determine the minimum value for width b by calculating the first valid pixel offset for the layer output of each layer. The first valid pixel for the current layer is calculated by the system 100 without using any zero-value adjacent pixels from the output of the preceding layer. The system 100 performs the operation of the function FirstValidPixelOffset() as follows: function FirstValidPixelOffset(layers): / / The first pixel offset from which the previous layer produced a valid result. first_valid_offset = 0 for layer = input to output layers: if layer type is “conv”: s = stride of layer; p = padding of layer; f = filter size of layer first_valid_offset = ceil((first_valid_offset + p) / s) else if layer type is “trans_conv”: s = stride of layer; p = padding of layer; f = filter size of layer / / The offset of the last invalid pixel in the input operation. Can be -1 or greater. last_invalid_offset = first_valid_offset + p - 1 / / Offset of the last invalid pixel in output operation last_invalid_offset = last_invalid_offset * s + f - 1 first_valid_offset = last_invalid_offset + 1 return first_valid_offset b=FirstValidPixelOffset(layers) In general, the criteria for the first valid pixels calculated from the left and right of a fixed-size output are not perfectly symmetrical, and some pixels may remain on the right side of the fixed-size input where the filter cannot be applied. Note that this will result in one more valid pixel on the right side of the fixed-size output than on the left. (FirstValidPixelOffset()) The function's output (e.g., the first effective offset) is calculated from the left, and this value should also be correct for the right. Similarly, the analysis described above should also apply to calculations from above or below a fixed-size output.

[0174] Again, referring to the AlignOutputTile() function in the second algorithm in relation to the ProjectBackwards() function, system 100 for each of the fixed-size outputs The final output can be generated by obtaining each coordinate shift and combining the respective fixed-size outputs based on each coordinate shift.

[0175] System 100 can implement the AlignOutputTile() function using different methods. For example, system 100 can perform a local search for each coordinate shift or obtain an analytical representation for each coordinate shift. AlignOutputTile() is written as follows: function AlignOutputTile((ht, wt, hb, wb), layers): if approach == “local search”: for (hs, ws) = try all values ​​in some pattern from 0 to max_shift: try: return ProjectBackwards((ht - hs, wt-ws, hb-hs, wb-ws), layers) except: / / Projection failed, continuing to try other shift values THROW EXCEPTION / / Failed to find a valid alignment for the tile. else if approach == “analytical”: hts = int (ht / alignment) * alignment wts = int (wt / alignment) * alignment hbs = hb - (ht - hts) wbs = wb - (wt - wts) return (hts, wts, hbs, wbs) where: alignment = CalculateAnalyticalAlignment(layers) When using the local search method, system 100 can provide multiple trial shifts in each dimension. The trial shift values ​​can range from zero pixels to a predetermined maximum value for the coordinate shift (e.g., the size of the final output). System 100 needs to determine the relationship between the coordinates of the "unaligned" fixed-size output and the coordinates of the associated fixed-size input. For example, system 100 can provide the coordinates and trial shift values ​​of the "unaligned" fixed-size output 133 to the ProjectBackwards() function to search for a proven fixed-size input (i.e., the coordinates representing the fixed-size input should be integer pixels). If system 100 successfully finds the proven fixed-size input, system 100 can return the shifted fixed-size output based on a specific trial shift value.

[0176] When using an analytical method, system 100 can determine a certain alignment value by analyzing the properties of the expanded FCN model. One exemplary algorithm for the analytical representation is called "CalculateAnalyticalAlignment()" and is written as follows: function CalculateAnalyticalAlignment(layers): / / Algorithm for finding the minimum correct alignment: presence of a convolutional layer before trans_conv / / The layer facilitates the alignment required by the trans_conv layer. conv_stride_product = 1 / / Product of the strides of the back-to-back convolutional layer trans_conv_stride_product = 1 / / stride of back-to-back trans conv layer product alignment = 1 / / Required tile alignment in the FCN output layer Ment for layer = output to input layers: if layer type is “conv”: s = stride of layer conv_stride_product *= s else if layer_type is “trans_conv”: s = stride of layer trans_conv_stride_product * = s prev_layer = previous layer / / prev_layer generates input for the layer / / If the layer is the input layer for the entire FCN, then prev_layer == null if prev_layer == null OR prev_layer type ! = “trans_conv”: / / By using the greatest common divisor, the stack of the trans_conv layer is charged The alignment requirements / / To simplify things, we use the subsequent stack of the convolutional layer. gcd = GCD (conv_stride_product, trans_conv_stride_product) alignment_for_stack = trans_conv_stride_product / gcd alignment * = alignment_for_stack / / Reset the stack conv_stride_product, trans_conv_stride_product = 1, 1 return path System 100 determines a certain alignment value based on the characteristics of each layer of the FCN model. For example, this characteristic can be the layer type (e.g., convolutional, transposed convolutional, or other layers such as pooling layers), or the size for padding, filtering, and stride for the layer. As mentioned above, other types of layers in the FCN model, such as pooling layers, are treated as convolutional layers throughout the specification.

[0177] Figure 3B shows an example of a tiling and stitching process 399 performed by the exemplary inference system 100 of Figure 1. System 100 may be configured to perform the tiling and stitching process 355 using a first algorithm.

[0178] System 100 can generate a number of fixed-size inputs 350a, 350b, 350c, and 350d, each having a different size. For example, fixed-size inputs 350a to d may each have a different size. As another example, fixed-size inputs 350a to d may have the same size, as shown in Figure 3B. For ease of explanation, fixed-size inputs 350a to d are represented by squares with solid lines.

[0179] As shown in Figure 3B, each fixed-size input 350a~d can have its own adjacent pixel region 360a, 360b, 360c, or 360d. The size or width of the adjacent pixel region can be 1 pixel, 3 pixels, and 5 pixels. For ease of explanation, the adjacent pixel region is represented by a square with a dashed line. Note that the region to the left of the zero-pixel value region 360a does not contain any zero-value adjacent pixels. This is because the left edge of the fixed-size input 350a is also part of the left edge of the complete input data 150, and thus the calculation of pixels in the region to the left of the fixed-size input 350a does not result in inaccuracies in the corresponding fixed-size output.

[0180] In some implementations, fixed-size inputs 350a-d and their respective associated adjacent pixel regions 360a-d can be evenly spaced relative to the complete input data 150 and can uniformly overlap each other. As shown in Figure 3B, fixed-size inputs 350a and 350b overlap each other in overlap region 353a, fixed-size inputs 350b and 350c overlap each other in region 353b, and fixed-size inputs 350c and 350d overlap each other in region 353c. Overlapping regions 353a and 353b have the same size, but overlap region 353c can be larger than overlap regions 353a and 353b. This is due to the characteristics of the first algorithm. As shown in the first algorithm, fixed-size inputs at the right and bottom boundaries cannot extend beyond the boundaries of the input data. For example, assuming that fixed-size input 350d is arranged in the same format as the other fixed-size inputs 350a~c, fixed-size input 350d may be on the right boundary and have a portion that extends beyond the right boundary of the complete input data 150. System 100 can use the first algorithm to "move" (i.e., retile) fixed-size input 350d to the left by a few pixels, and this As a result, the pixels of the fixed-size input 350d are placed entirely within the complete input data 150. However, since the placement of the fixed-size input 350d is no longer the same as that of the other fixed-size inputs, the overlapping region 353c between the fixed-size inputs 350d and 350c can be larger than the overlapping regions 353a and 353b. If the fixed-size inputs at the right and bottom boundaries do not exceed the corresponding boundaries of the complete input data 150, the fixed-size inputs may be placed to have the same overlapping region.

[0181] After tiling the full input data 150 into a number of fixed-size inputs based on a fixed size calculated online or offline, the system 100 can process the random-size inputs based on at least a first algorithm and a stitching algorithm described in more detail below, and generate fixed-size outputs having respective valid regions that do not overlap with each other and are adjacent to each other at the edge pixels.

[0182] The system 100 can generally generate fixed-size outputs having non-overlapping valid regions in the full output data 170. However, in some situations, one or more fixed-size outputs may overlap with each other. As shown in FIG. 3B, the valid regions 370a, 370b, and 370c do not overlap with each other. However, the valid region 370d overlaps with the valid region 370c in the overlap region 373. This is because the first algorithm "moves" the right-border fixed-size input 350a to the left by a few pixels using the first algorithm, whereby the fixed-size output 370d overlaps with the adjacent fixed-size output 370c. The dummy regions 375a, 375b, 375c, and 375d associated with the corresponding valid regions can overlap. For ease of explanation, the valid regions of the fixed-size outputs are represented by squares with solid lines, and the dummy regions of the fixed-size outputs are represented by squares with dashed lines.

[0183] The region to the left of the dummy region 375a does not contain any invalid values. This is because the left edge of the fixed-size output 375a is also part of the left edge of the full output data 170. Similarly, the right edge of the dummy region 375d does not contain any invalid values.

[0184] During the stitching process, system 100 can discard per-pixel values ​​in dummy regions and stitch together per-pixel values ​​in active regions to generate complete output data 170. Each pixel value in the complete output data (or final output) is provided at least once from the per-pixel values ​​in active regions.

[0185] Figure 3C shows another example of the tiling and stitching process 355 performed by the exemplary inference system 100 in Figure 1. System 100 may be configured to perform the tiling and stitching process 399 using a second algorithm.

[0186] As described above, compared to the first algorithm, system 100 uses the second algorithm to perform several additional steps, for example, determining alignment information for the FCN model and determining the effective region by calculating coordinate shifts for the fixed-size output based on the alignment information. This is because, if the FCN model includes certain layers (e.g., transposed convolutional layers), the system needs to verify the mapping (e.g., integer coordinates) from pixels in the fixed-size output to corresponding pixels in the fixed-size input.

[0187] In addition, the second algorithm differs from the first algorithm in that it does not require the "movement" of fixed-size outputs at the right and lower boundaries of the complete input data 150.

[0188] As shown in Figure 3C, system 100 can generate a number of fixed-size inputs (e.g., fixed-size inputs 380a-d) from the complete input data 150 based on a tiling pattern. The fixed-size inputs 380a-d can overlap each other by their respective sizes or by the same size. For example, fixed-size inputs 380a and 380b can overlap each other in overlap region 385a, a second fixed-size input 380b and a third fixed-size input 380c can overlap each other in overlap region 385b, and a third fixed-size input 380c and a fourth fixed-size input 380d can overlap each other in overlap region 385c. The sizes of the overlap regions 385a-c are substantially the same, as shown in Figure 3C. In some implementations, the overlap regions 385a-c can be slightly larger than the overlap regions generated using the first algorithm. This is because the system using the second algorithm needs to tile fixed-size inputs based on alignment information.

[0189] System 100 can also determine and place zero-value adjacent pixel regions 390a~d similar to those described above. As shown in Figure 3C, for ease of explanation, the fixed-size inputs 380a, 380b, 380c and 380d are represented by solid squares, and the adjacent pixel regions 390a, 390b, 390c and 390d are represented by dashed squares.

[0190] System 100 can use the second algorithm to determine the region outside the complete input data 150, and in some cases it may not be necessary to "move" the fixed-size input 380d. As shown in Figure 3C, the fixed-size input 380d has a region 381 outside the complete input data 150. Since the fixed-size input 380d cannot be "moved", the overlapping regions 385a~c can remain the same. When dealing with certain inputs where the "move" operation is not allowed, the second algorithm is more robust than the first algorithm.

[0191] After processing all fixed-size inputs through the compiled FCN model, system 100 determines the effective regions 395a, 395b, 395c, and 395d of all fixed-size outputs, as well as the corresponding dummy regions 397a, 397b, and 397c. According to a second algorithm, it can calculate coordinate shifts for pixels in the effective regions, discard pixels in the dummy regions, and combine the pixels in the effective regions to generate the complete output data 170. The effective regions 395a-d may also overlap each other in their respective overlap regions 393a-c. Each overlap region 393a-c may be substantially the same if the overlap regions 395a-c between the fixed-size inputs are substantially the same.

[0192] Similarly, for ease of explanation, the effective regions of fixed-size outputs 395a to d are represented by solid squares, and the dummy regions of fixed-size outputs 397a to d are represented by dashed squares.

[0193] Although only four fixed-size inputs and four fixed-size outputs are shown in Figures 3B and 3C, it should be noted that system 100 can generate five or more fixed-size inputs, e.g., 5, 10, 20, 50, and more, to tile the complete input data 150. The system can generate five or more fixed-size outputs, e.g., 5, 10, 20, 50, and more, containing valid per-pixel values ​​for each pixel in the complete output data 170. Output can also be generated. Each pixel value for a pixel associated with the complete output data 170 is represented in at least a fixed-size output generated from the corresponding fixed-size input. In the case of a pixel value for a pixel in an overlapping region spanning two or more fixed-size outputs, the system 100 can select the corresponding pixel value as the pixel value for the pixel from any one of the overlapping fixed-size outputs.

[0194] After calculating all fixed-size outputs through the FCN model, the system uses the StitchOutputImage() function as if the input had been fully processed by the FCN model. To construct the complete output, we can apply O(i,j) and V(i,j) mappings: function StitchOutputImage(): for tile indices (i,j)in a top-to-bottom, left-to-right scan of the tiles: (ht_O, wt_O, hb_O, wb_O) = O(i,j) (ht_V, wt_V, hb_V, wb_V) = V(i,j) Output(ht_V:hb_V, wt_V:wb_V) = OutputTile( (ht_V-ht_O):(hb_V-ht_O), (wt_V-wt_O):(wb_V-wt_O) ) OutputTile(i,j) corresponds to the (i,j)th fixed-size input, with size T. O This represents a fixed-size output. For example, a fixed-size input in the i-th column and j-th row of a tiling grid.

[0195] Figure 4 shows an exemplary process 400 for performing inference computations of a fully convolutional network for inputs of different sizes. For simplicity, process 400 is described as being performed by one or more computer systems located in one or more locations. For example, a well-programmed neural inference system, e.g., system 100 in Figure 1, can perform process 400.

[0196] The system receives a new input which is processed by a fully convolutional neural network deployed on a hardware accelerator (410). The new input may have a first size different from the fixed size which the fully convolutional neural network is configured to process when deployed on the hardware accelerator. As described above, the new input may have a size greater than or less than the fixed size.

[0197] The system determines one or more fixed-size inputs from the new input (420). Each of the one or more fixed-size inputs has a fixed size. More specifically, the system can determine a tiling pattern for tiling the new input based on at least the characteristics of the expanded FCN model, e.g., alignment information, padding size, stride size, filter size, and scale factor.

[0198] The system provides one or more fixed-size inputs to a hardware accelerator to perform inference computations using a fully convolutional neural network (430).

[0199] The system obtains (440) from a hardware accelerator each fixed-size output generated by a fully convolutional neural network for each of one or more fixed-size inputs. Each fixed-size output may include results for one or more inaccurate pixels. As described above, the system can provide a fixed-size input for the FCN deployed on the hardware accelerator and include a host for receiving the fixed-size output from the hardware accelerator. When processing the fixed-size input, the system can use adjacent pixels surrounding the fixed-size input to determine the valid region and the dummy region for each fixed-size output. The system can provide a fixed-size input for the FCN deployed on the hardware accelerator and include a host for receiving the fixed-size output from the hardware accelerator. When processing the fixed-size input, the system can use adjacent pixels surrounding the fixed-size input to determine the valid region and the dummy region for each fixed-size output.

[0200] The system generates (450) a final output equivalent to the output generated by processing a new input using a fully convolutional neural network from each fixed-size output.

[0201] As described above, the system can combine the fixed-size outputs using different algorithms based on the characteristics of the deployed FCN. If the FCN model does not include any transposed convolutional layers, the system can combine the valid regions of each fixed-size output using a first algorithm. If the FCN model includes one or more transposed convolutional layers, the system can obtain a coordinate shift for each fixed-size output and combine the fixed-size outputs by shifting the coordinates of each fixed-size output based on the coordinate shift.

[0202] The system can determine coordinate shifts using different methods. For example, the system can determine coordinate shifts using local search. The system can generate coordinate shifts for fixed-size output by testing multiple trial shift values ​​using the ProjectBackwards() function. Alternatively, the system can generate coordinate shifts based on analyzing the properties of the expanded FCN and obtain a constant value for the coordinate shift by analytical representation using the CalculateAnalyticalAlignment() function.

[0203] Implementations of the subject matter and the operations and calculations described herein may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Implementations of the subject matter described herein may be implemented as one or more modules of computer program instructions encoded in a computer program carrier for execution by a data processing device or for controlling calculations of a data processing device. The carrier may be a tangible non-temporary computer storage medium. Alternatively or in addition, the carrier may be an artificially generated propagating signal, such as a mechanically generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable memory device, a machine-readable memory board, a random or serial-accelerated memory device, or one or more combinations thereof, or a part thereof. The computer storage medium is not a propagating signal.

[0204] The term "data processing device" includes all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A data processing device may include dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPUs (Graphics Processing Units). In addition to hardware, the device may also include code that generates an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or code constituting one or more of these.

[0205] Computer programs, which may be called or written as programs, software, software applications, apps, modules, software modules, engines, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages. Computer programs can be deployed in any form, including as standalone programs or as modules, components, engines, subroutines, or other units suitable for execution in a computing environment. Such environment may include one or more computers interconnected by a data communication network in one or more locations.

[0206] Computer programs may, but do not, correspond to files in a file system. A computer program may be stored in a single file dedicated to the program in question, or in a number of coordinated files, such as files storing one or more modules, subprograms, or parts of code, which may hold other programs or data, for example, one or more scripts stored in a markup language document.

[0207] The processes and logic flows described herein may be executed by one or more computers running one or more computer programs to perform operations by operating on input data and producing outputs. The processes and logic flows may also be executed by dedicated logic circuits, such as FPGAs, ASICs, or GPUs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0208] A computer suitable for running computer programs can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be assisted by or incorporated into dedicated logic circuits.

[0209] Generally, a computer includes one or more mass storage devices, or is operablely coupled to receive data from or transfer data to one or more mass storage devices. Mass storage devices can be, for example, magnetic, magneto-optical, or optical disks, or solid-state drives. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive.

[0210] To provide user interaction, the implementations of the subject matter described herein may be implemented on or configured to communicate with a computer having a display device for displaying information to the user, such as an LCD (liquid crystal display) monitor, and an input device, such as a keyboard and a pointing device, such as a mouse, trackball, or touchpad, thereby enabling the user to provide input to the computer. Other types of devices may also be used to provide user interaction. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, Alternatively, it may be haptic feedback. User input can be received in any form, including acoustic input, voice input, or haptic input. In addition, a computer can interact with a user by sending and receiving documents to and from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser, or by interacting with an app running on a user device, such as a smartphone or electronic tablet. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages back from the user.

[0211] This specification uses the term “configured to perform” in relation to systems, devices, and computer program components. One or more computer systems being configured to perform a particular operation or action means that the system has software, firmware, hardware, or a combination thereof installed on the system that causes the system to perform the operation or action when an operation is performed. One or more computer programs being configured to perform a particular operation or action means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform the operation or action. A dedicated logic circuit being configured to perform a particular operation or action means that the circuit has electronic logic that performs the operation or action.

[0212] Implementations of the subject matter described herein may be implemented in a computing system that includes, for example, a data server, a backend component, or a middleware component, such as an application server, or a frontend component, such as a client computer having a graphical user interface, a web browser, or an application through which a user can interact with the implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system may be interconnected by digital data communications, such as any form or medium of a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0213] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer and having a client-server relationship with each other. In some implementations, the server sends data, such as an HTML page, to a user device to display data to the user interacting with the device acting as a client and to receive user input from the user. Data generated on the user device, such as the results of user interaction, may be received from the device by the server.

[0214] This specification includes many specific implementation details, but these should not be construed as limitations on the scope of what is claimed or can be claimed, but rather as descriptions of features that may be specific to a particular implementation of a particular invention. Certain features described herein in relation to separate implementations may also be implemented in combination in one implementation. Conversely, various features described in relation to one implementation may also be implemented separately or in any appropriate subcombination in numerous implementations. Furthermore, features described above as working in a certain combination are also described above. While such claims may be made even initially, one or more features from the claimed combination may, in some cases, be removed from that combination, and the claims may be directed towards subcombinations or variations of subcombinations.

[0215] Similarly, while operations are shown in the drawings and described in the claims in a specific order, this should not be understood as requiring that such operations be performed in a specific order or sequence shown, or that all shown operations be performed, in order to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the implementation forms described above should not be understood as requiring such separation in all implementation forms, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0216] A specific implementation of the subject is described. Other implementations are included in the attached claims. For example, the operations listed in the claims can be performed in a different order and still achieve the desired results. As an example, the process shown in the attached drawings does not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, The process includes receiving a new input which is processed by a fully convolutional neural network deployed on a hardware accelerator, wherein the new input has a first size which is different from the fixed size which the fully convolutional neural network is configured to process when deployed on the hardware accelerator. The aforementioned method, The process further includes determining one or more fixed-size inputs from the new inputs, each of which has a fixed size. The aforementioned method, To perform inference calculations using the fully convolutional neural network, the hardware accelerator is provided with each of the one or more fixed-size inputs. The hardware accelerator further includes obtaining each fixed-size output generated by the fully convolutional neural network for each of the one or more fixed-size inputs, each of the fixed-size outputs comprising a central effective region and a peripheral dummy region with a width of a first number of pixels, wherein the central effective region comprises at least a portion of the final output and the peripheral dummy region comprising the results of one or more inaccurate pixels, The aforementioned method, The process further includes generating a final output from each of the fixed-size outputs that is equivalent to the output generated by processing the new input using the fully convolutional neural network, and generating the final output from each of the fixed-size outputs is The coordinates are determined for each of the fixed-size outputs such that the central effective region of the subsequent fixed-size output is positioned adjacent to the central effective region of the preceding fixed-size output, The central effective region of each of the fixed-size outputs is combined based on the determined coordinates to form the final output. Methods that include...

2. The method according to claim 1, wherein the central effective regions are combined without overlapping each other.

3. The method according to claim 1 or 2, further comprising determining the fixed size based on at least the characteristics of the fully convolutional neural network before deploying the fully convolutional neural network on the hardware accelerator.

4. The method according to claim 3, wherein determining the fixed size further comprises providing the user with a plurality of candidate sizes in order to select one of the plurality of candidate sizes as the fixed size.

5. To generate a plurality of candidate sizes for the fully convolutional neural network based on the characteristics of the fully convolutional neural network, For each of the aforementioned candidate sizes, To process the candidate size input, a copy of the fully convolutional neural network is deployed on each hardware accelerator, Measure the total execution time for performing inference calculations for the unfolded copy of the fully convolutional neural network on each of the aforementioned hardware accelerators, As the fixed size, a candidate size is selected from the plurality of candidate sizes based on the total execution time measured for at least the candidate size. The method according to claim 1, further comprising:

6. Determining the fixed size is Determining that the first size of the new input is smaller than the fixed size, A fixed-size input is generated by padding the new input with zeros up to the aforementioned fixed size. The method according to any one of claims 1 to 3, further comprising:

7. The method according to any one of claims 1 to 6, wherein the first number of pixels is determined based on the characteristics of the fully convolutional neural network.

8. The method according to any one of claims 3, 5, or 7, wherein the characteristics of the fully convolutional neural network include the respective filter size, zero-padding size, and stride size for each network layer of the fully convolutional neural network.

9. The method according to any one of claims 1 to 8, wherein determining the coordinates of a particular fixed-size output includes determining the coordinates of the particular fixed-size output based on the coordinates of the edge below the central effective area of ​​a preceding vertical fixed-size output and the coordinates of the right edge of the central effective area of ​​a preceding horizontal fixed-size output.

10. The method according to claim 9, wherein the coordinates of the specific fixed-size output are determined based on the coordinates of the central effective region of the preceding fixed-size output and the width of the surrounding dummy region.

11. A system comprising one or more computers and one or more storage devices that, when executed by the one or more computers, store instructions causing the one or more computers to perform an operation according to any one of claims 1 to 10.

12. A program that causes one or more computers to perform an operation according to any one of claims 1 to 10.

Images