Efficient implementation of inference computations for fully convolutional networks for inputs with different sizes

Abstract

Methods, systems, and apparatuses including a computer program encoded on a computer storage medium for performing an inference calculation of a fully convolutional neural network that receives inputs having different sizes, one of the methods includes receiving a new input to be processed by the fully convolutional neural network, the new input having a first size different from a fixed size that the fully convolutional neural network is configured to process, the method further includes determining one or more fixed size inputs from the new input, each fixed size input having a fixed size, the method further includes obtaining respective fixed size outputs generated by the fully convolutional neural network performing the inference calculation for each of the one or more fixed size inputs, and generating a final output from the respective fixed size outputs including one or more invalid pixel values, the final output being equivalent to an output generated by processing the new input with the fully convolutional neural network.

Description

[Background technology]

[0001] background This specification relates to neural networks, and in particular to efficient performance of inference computations for fully convolutional networks that receive inputs with different sizes.

[0002] A neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of network parameters.

[0003] A fully convolutional network is a neural network that includes only convolutional neural network layers and, optionally, other layers composed only of components that operate only in the local input domain, such as pooling layers and element-wise layers, e.g., those that apply element-wise nonlinear activation functions. Notably, unlike other types of convolutional neural networks, a fully convolutional network does not have any fully connected layers. A fully convolutional network can be configured to make pixel-wise predictions of an input (e.g., an image having multiple pixels). In other words, a fully convolutional network can be used to make a respective prediction for each pixel of the input. One example of a task that requires making pixel-wise predictions is image segmentation, where a neural network is configured to generate a respective score for each of multiple classes for each pixel of an input image. Summary of the Invention

[0004] overview This specification generally describes techniques for performing inference computations for neural networks.

[0005] According to one aspect, the described technology relates to a method executed by one or more computers. The method includes receiving a new input to be processed by a fully convolutional neural network deployed on a hardware accelerator, determining one or more fixed-size inputs from the new input, providing each of the one or more fixed-size inputs to the hardware accelerator for performing an inference calculation using the fully convolutional neural network, obtaining from the hardware accelerator a respective fixed-size output generated by the fully convolutional neural network for each of the one or more fixed-size inputs, and generating, from the respective fixed-size outputs, a final output equivalent to an output generated by processing the new input using the fully convolutional neural network. The new input has a first size different from the fixed size that the fully convolutional neural network is configured to process when deployed on the hardware accelerator. Each of the one or more fixed-size inputs has a fixed size. Each of the fixed-size outputs has one or more inaccurate pixel-by-pixel results.

[0006] The subject matter described in this specification can be implemented in particular embodiments to realize one or more of the following advantages.

[0007] The described technique enables a statically compiled fully convolutional network model deployed on a hardware accelerator to process input data having unknown or varying sizes. Generally, a fully convolutional neural network can, in principle, process inputs of any arbitrary size, but a statically compiled neural network already deployed on a hardware accelerator cannot process inputs having varying sizes. In addition, it is difficult to compile a neural network for deployment on a hardware accelerator that can dynamically process input data having unknown or varying sizes. However, the described technique can efficiently tile input data into multiple smaller, fixed-size inputs and provide the inputs for performing inference calculations on a statically compiled fully convolutional network.

[0008] The described techniques can also stitch the generated fixed-size outputs to generate a final output for a given input of random size that is equivalent to the output generated by a fully convolutional network processing a random-size input. Thus, the described techniques enable a fully convolutional network that was compiled only to receive fixed-size inputs when deployed on a hardware accelerator to generate accurate outputs for inputs of different sizes without modifying the compiled model or the operation of the hardware accelerator.

[0009] Additionally, the described techniques can automatically generate optimized parameters for tiling and stitching inputs and outputs for a network based on the characteristics of the fully convolutional network. Using these optimized parameters, the described techniques can increase computational efficiency in performing inference calculations for input data of unknown or varying sizes.

[0010] The described techniques can execute inference operations of different tiles (e.g., fixed-size inputs) in parallel, taking advantage of data sharing characteristics between adjacent accelerators, to reduce memory usage. For example, the described techniques can optimize data transfer across overlapping regions of adjacent fixed-size inputs according to input or output data having various sizes.

[0011] Furthermore, the described techniques are robust to different input sizes and hardware accelerator architectures. The described techniques can automatically identify hardware constraints or requirements, such as system memory bandwidth. Based on the identified hardware constraints or requirements, the described techniques can efficiently tile arbitrarily large inputs to fit a fully convolutional network deployed on a hardware accelerator. The system can also robustly process inputs with sizes smaller than the fixed size for the fully convolutional network by padding zeros around the input to reach the fixed size.

[0012] For example, for accelerators with modern memory addressing capabilities (e.g., accelerators including direct memory access (DMA) engines), the described techniques can reduce or eliminate overhead time associated with data manipulation for tiling inputs and stitching fixed-size outputs. As another example, for accelerators with simpler architectures or smaller memory bandwidths, the described techniques can perform operations for one model at a time. In some implementations, the described techniques determine whether an accelerator array is present in the computing system, and in response to determining that an accelerator array is present, the described techniques can execute inference operations of different tiles in parallel, taking advantage of data sharing characteristics between adjacent accelerators to reduce memory usage.

[0013] Furthermore, the techniques described herein are unique and advantageous over conventional data parallelism techniques. Generally, data parallelism techniques divide input data (e.g., an input image) into multiple disjoint parts (e.g., segments of the input image), assign the multiple parts to multiple hardware components (e.g., hardware accelerators), and process the multiple parts independently and in parallel to generate partial outputs. After all parts have been processed by the hardware components, a system configured to execute the data parallelism technique can generate a final output by collecting the partial outputs. As long as the operations are performed correctly by each hardware component for the parts they are respectively designed for, the system does not need to consider whether any part of the partial outputs is unsuitable or inaccurate for generating the final output.

[0014] However, fully convolutional networks generally do not utilize data parallelism techniques because the output generated by a fully convolutional network processing a portion of an input image (e.g., a tile of an input image as described herein) may contain one or more erroneous or inaccurate per-pixel values. This is because the system's calculations processing a tile of input may involve "neighboring pixels," which may cause some of the output pixels to be inaccurate.

[0015] Throughout this specification, the term "neighboring pixels" refers to pixels surrounding the boundary of an input to a fully convolutional network model. Neighboring pixels may include pixels added to the boundary of the input by zero padding specified by one or more layers of the fully convolutional network model. In the case of a fixed-size input to a fully convolutional network model (e.g., a tile extracted from the full input data), neighboring pixels may also include pixels that originally surrounded the fixed-size input in the full input data.

[0016] A region surrounding an input or a fixed-size input to a fully convolutional network model and including neighboring pixels is referred to throughout the specification as a "neighboring pixel region." The neighboring pixel region may include a width of one or more pixels. In some implementations, the width of the neighboring pixel region may be determined based on characteristics of the fully convolutional network model. Neighboring pixels may have or be replaced with zero pixel values ​​during calculations, causing the output from processing the neighboring pixels through the fully convolutional network model to be inaccurate.

[0017] In some implementations, the neighboring pixels are initially present in the full input data. When a fixed-size input is extracted from the full input data, the system may need one or more neighboring pixels to process the fixed-size input. However, the system may change the value of one or more non-zero neighboring pixels to zero, causing inaccuracies in the calculations at some pixel locations.

[0018] For example, the system may include one or more convolution layers with a filter size of 2 or greater. To process boundary pixels of a fixed-size input, the system may use one or more neighboring pixels outside the boundary pixels to calculate the corresponding pixel-by-pixel output. Non-zero neighboring pixels may be replaced with zero values ​​during the calculation. By using zero-value neighboring pixels to process the fixed-size input rather than the true pixel values ​​associated with the neighboring pixels, one or more pixel values ​​in the fixed-size output may be inaccurate.

[0019] As another example, a system may include one or more transposed convolution layers with a filter size greater than or equal to 2. If the calculation for one of the transposed convolution layers uses zero values ​​to replace non-zero neighboring pixels, the output pixel values ​​may be inaccurate.

[0020] In other words, zero-value adjacent pixels (e.g., non-zero pixels originally replaced with zero values) may cause one or more pixel values ​​in the output tile to be incorrect. Therefore, it is problematic for a system for performing operations in a fully convolutional network for processing a fixed-size input to generate a final output by combining fixed-size outputs without determining and discarding incorrect data. The system needs to determine both correct data (e.g., valid values) and incorrect data (e.g., dummy pixel values) based on the characteristics of the network layers in the fully convolutional network when processing a fixed-size input.

[0021] The techniques described herein can determine which per-pixel values ​​in the fixed-size outputs are inaccurate by analyzing the characteristics of network layers in a fully convolutional network and determining layer or global alignment information and appropriate fixed sizes for compiling a fully convolutional network model and tile input data. The alignment information and appropriate fixed sizes can be used for a system employing the described techniques to generate accurate values ​​for each pixel in the final output through the fully convolutional network model. The accurate value for each pixel is generated at least once in at least one fixed-size output, and the system can obtain the accurate value for the pixel from the at least one fixed-size output.

[0022] The techniques described herein can further reduce memory traffic by reducing or even avoiding calculating invalid or overlapping pixel values ​​between different fixed-size outputs. In some situations where a fixed size is determined, the techniques can optimize memory traffic between the accelerator and the host by minimizing exact pixel overlap of different fixed-size output tiles so that a valid final output can be generated based on the minimized overlap. In some situations where a fixed size has not yet been determined, the described techniques can select one of multiple candidate fixed sizes as the fixed size based on input data and hardware characteristics so that calculations to generate inaccurate or overlapping pixel values ​​are minimized or even eliminated.

[0023] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 illustrates an exemplary inference system for performing inference computations of fully convolutional networks for inputs of different sizes. [Figure 2A] FIG. 1 illustrates an exemplary inference process using a convolution system. [Figure 2B] FIG. 2 illustrates an exemplary inference process using the exemplary inference system of FIG. [Figure 3A] FIG. 10 illustrates an exemplary fixed-size input with adjacent pixel regions and an exemplary fixed-size output with dummy regions. [Figure 3B] FIG. 2 illustrates an example of the tiling and stitching process performed by the example reasoning system of FIG. 1. [Figure 3C]FIG. 2 illustrates another example of the tiling and stitching process performed by the example reasoning system of FIG. 1. [Figure 3D] FIG. 1 illustrates an exemplary process for generating outputs using a transposed convolution layer in an FCN model. [Figure 4] FIG. 1 illustrates an exemplary process for performing inference computations of a fully convolutional network for inputs having different sizes. DETAILED DESCRIPTION OF THE INVENTION

[0025] Detailed Description A fully convolutional network (FCN) can include at least one or more convolutional neural network layers and, optionally, a pooling layer, a transposed convolutional layer, and an element-wise layer (e.g., a layer that applies an element-sized activation function). An FCN can be deployed on a hardware accelerator to generate pixel-by-pixel predictions of an input (e.g., an input image having multiple pixels). In particular, the FCN is configured to generate an output having pixels related to one or more corresponding pixels of the input image and make a prediction for each pixel of the input image. In some implementations, the FCN can also relate the output pixel to the input pixel and neighboring pixels in a fixed-size neighborhood. Because an FCN model can process pixels of an input, an FCN can, in principle, process inputs of any size.

[0026] Although FCNs have the advantage over typical neural networks in that they can generate pixel-by-pixel predictions for input data provided at different sizes, several hardware limitations make it difficult or even impossible to deploy FCNs dynamically on hardware accelerators (i.e., to process inputs with various input sizes).

[0027] Dynamically deploying an FCN to handle inputs of different sizes can create problems with computational costs. First, network parameters, including data structures (e.g., matrix dimensions for computations, or padding, strides, filter sizes, and scale factors for network layers), scale with the size of the input. A change in input size may require a shuffling of current network parameters, which can lead to increased downtime (e.g., overhead) for systems with many hardware accelerators. Furthermore, to accommodate dynamic input sizes, the host must send instructions to perform inference computations using a more general execution mechanism. For example, the host may allocate larger memory for data storage (a portion of the larger memory may be used, at the expense of slower performance), perform more checks on vector or tensor sizes, or dynamically change the number of compute units used to perform computations during parallel computations. Therefore, in practice, FCNs are typically statically deployed (e.g., compiled with fixed network hyperparameters) on one or more hardware accelerators and configured to receive fixed-size inputs to avoid problems caused by dynamic deployment.

[0028] The techniques described below can solve the above problems by enabling statically compiled FCNs that have already been deployed (or will be deployed) on a hardware accelerator to efficiently process inputs of different sizes.

[0029] The described techniques can tile input data having a particular size into multiple smaller inputs, each having a fixed size. A statically compiled FCN on the hardware accelerator can process each of the multiple fixed-size inputs and generate a corresponding fixed-size output. The described techniques can then stitch the fixed-size outputs to generate a final output as if the input had been fully processed by an FCN compiled for that input size.

[0030] In general, the described techniques can provide a method for determining specific “tiling and stitching” parameters for tiling a specific-sized input into multiple fixed-sized inputs and stitching multiple fixed-sized outputs generated from multiple fixed-sized inputs to generate a final output equivalent to the output generated by processing the input entirely through an FCN compiled for that specific size. More specifically, according to the characteristics of the FCN model, the described techniques can generate fixed-sized outputs having different regions by processing the fixed-sized input tiles through the FCN model. The different regions can include dummy regions and valid regions. By analyzing the characteristics of the FCN model (e.g., padding, stride, filter size, scale factor, and layer type for all layers in the FCN model), the described techniques can determine valid regions in the fixed-sized output where per-pixel values ​​are accurate, i.e., no zero-valued neighboring pixels are used to generate the output pixel values, and dummy regions in the fixed-sized output where per-pixel values ​​are at least “not perfectly” accurate, i.e., where per-pixel values ​​are generated by the FCN by utilizing at least one zero-valued neighboring pixel. The described techniques can combine (e.g., "stitch") accurate per-pixel values ​​from all fixed-size outputs to generate a final output, and can ensure that each accurate per-pixel value corresponding to a pixel in the final output is generated and obtained from at least one fixed-size output. This is in contrast to traditional parallelization techniques, where input data is easily split to generate output data independently, without having to consider inaccuracies introduced by neighboring pixels.

[0031] The described techniques can also determine the fixed size by various means before the FCN is compiled and deployed on a hardware accelerator. First, the described techniques can suggest multiple candidate sizes based on the characteristics of the FCN model and the hardware accelerator on which the FCN will be deployed. The multiple candidate sizes are valid and suitable for the tiling and stitching process performed by the described techniques. For example, if the FCN includes one or more transposition layers, the multiple candidate sizes can be determined based on alignment information of the output tiles. Throughout the specification, the term "alignment information" refers to data representing constraints or requirements for the alignment of fixed-size outputs. The alignment information is obtained by the system so that the fixed-size outputs can be properly projected to fixed-size inputs and vice versa.

[0032] The system may also determine coordinate shifts between the fixed-size outputs and corresponding fixed-size inputs based on the alignment information to obtain an appropriate tiling pattern, which may include one or more fixed sizes (e.g., one or more candidate sizes selected automatically or by a user), an overlap size for the fixed-size inputs at a particular fixed size, and, optionally, coordinates of the fixed-size inputs and outputs, particularly coordinates for the dummy and valid areas of the fixed-size outputs.

[0033] The determined tiling pattern must satisfy at least two criteria: (i) the alignment information should be correct, i.e., the tiling pattern should have fixed-size outputs accurately positioned so that each fixed-size output can be accurately projected onto the fixed-size input or vice versa, and (ii) each pixel value for the complete output data should be generated and extracted from at least the valid area of ​​one fixed-size output. Optionally, the system can determine a tiling pattern that minimizes the overlap area for the fixed-size inputs to enhance computational performance and optimize computational resource usage. Details about determining a tiling pattern based on alignment information are provided below.

[0034] In some implementations, the described techniques can selectively select one of the candidate sizes included in the appropriate tiling pattern as a fixed size based on performance metrics (e.g., total execution time or overhead). The described techniques can also generate a range of candidate sizes for deploying an FCN model on a hardware accelerator and provide the range of candidate sizes for user selection. A user can select one size from the range of candidate sizes as a fixed size, for example, according to the characteristics of the FCN model, the hardware accelerator, or the specific computational requirements of the task.

[0035] The tiling pattern described above may include a fixed size for tiling the complete input data onto one or more fixed-size inputs and a size of the overlap region for generating the fixed-size inputs. Generally, the system can tile the fixed-size inputs so that they often overlap each other, thereby ensuring accurate or correct pixel-by-pixel values ​​associated with all pixels in the final output, i.e., each accurate value is generated and obtained from at least one fixed-size output. The tiling pattern described above may further include data representing valid and dummy regions for the fixed-size outputs. The fixed-size outputs generally include a dummy region of substantial size due to one or more zero-valued neighboring pixels used to generate the output pixel values. The system may employ one or more algorithms to determine alignment information based on the characteristics of the FCN model, and may apply additional algorithms to determine a relationship (e.g., a mapping) between the coordinates of the fixed-size outputs and the coordinates of the corresponding fixed-size inputs, determine the valid region for the fixed-size outputs, and determine a coordinate shift for the fixed-size outputs based on the mapping and valid region described above. Details of these algorithms are described below.

[0036] After determining the tiling pattern, the system can stitch the fixed-size outputs by combining the effective areas of each fixed-size output. Note that the tiling pattern is generated based on alignment information for the FCN model. Because the system can generate an appropriate tiling pattern for the fixed-size input, the stitching process is highly efficient because the system has coordinate information for all pixels in the effective area of ​​the fixed-size output. In some implementations, the system can take pixel values ​​in the effective area of ​​the fixed-size output at least once for each pixel located in the complete output data to generate the complete output data. Details of stitching are described below in conjunction with specific algorithms and Figures 3A-3D.

[0037] Furthermore, the described techniques can perform “tiling and stitching” analysis both online and offline. To deploy a compiled FCN on a hardware accelerator in a format similar to one previously deployed using the described techniques, the host processor can perform the analysis offline by reusing previously saved parameters for “tiling” the complete input data and “stitching” the output tiles to generate complete output data of unknown or varying sizes. The previously saved parameters can include at least a tiling pattern, such as alignment information for the FCN, a fixed size for the tiling, overlap regions for the fixed-size input and / or fixed-size output, and dummy and valid regions for the fixed-size output. The system can reuse these parameters to process new complete input data and generate complete output data as if the new complete input had been processed directly by the FCN model compiled for the size of the new complete input data. The host processor can generate a new set of “tiling and stitching” parameters for processing the input data in situations where a new FCN is deployed.

[0038] 1 illustrates an exemplary inference system 100 for performing inference computations on fully convolutional networks for inputs having different sizes. Inference system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations that can implement the systems, components, and techniques described below.

[0039] 1, system 100 includes a host 130 and an accelerator 110 in communication with each other. Generally, system 100 receives input data 150 and generates output data 170 using a trained FCN 115 deployed on hardware accelerator 110. Generally, output data 170 can have a size that is smaller than, larger than, or equal to the size of input data 150.

[0040] More specifically, the system can use the deployed FCN 115 for tasks such as object detection and classification (e.g., face detection), image segmentation, image generation, image super-resolution, image completion, and image colorization, to name a few. For example, if the task is image segmentation, the input data 150 can be an image input, and the system 100 can generate output data 170 having pixel-by-pixel predictions, i.e., a respective prediction for each pixel of the input image or for each pixel of the output image generated by the FCN 115. The output data 170 can also include a respective score distribution for each pixel, assigning a respective score to each of a number of categories. In that case, the system 100 can detect the presence, shape, and location of objects from the input image. As another example, if the task is image super-resolution, the system 100 can use the deployed FCN 115 to enhance the image resolution for the input image by predicting pixels to be added around each input pixel. In that case, output image 170 may have a higher resolution than input image 150, and one or more output cells may be associated with each pixel in the input image.

[0041] System 100 can compile an FCN for processing inputs having a fixed size in compilation engine 160 included in host 130 and deploy the compiled FCN on hardware accelerator 110. To compile the FCN, host 130 can receive data 155 representing a trained FCN model in compilation engine 160, compile the trained FCN model, and generate instructions (e.g., binary data) to deploy the trained FCN model on hardware accelerator 110. In some implementations, compilation engine 160 can recompile the trained FCN model to process different input data having various input sizes. Recompiling the trained FCN model and deploying the recompiled FCN model on hardware accelerator 110 is described in more detail below.

[0042] In general, the compiled FCN 115 can process any suitable input having the same size (e.g., fixed-size input 138). Thus, the hardware accelerator 110 can use the compiled FCN model 115 to perform inference calculations for the provided fixed-size input 138.

[0043] The compilation engine 160 can apply conventional compilation techniques to compile the FCN on the hardware accelerator 110. In general, the compilation engine 160 can decode program code written in any suitable high-level language and encoded into machine-readable binary code on the hardware accelerator with data representing the characteristics of the FCN. The data representing the characteristics of the FCN can include hyperparameters that define the structure of the FCN (e.g., input size, number of layers, number of nodes in each layer, layer type and position, padding, stride, filter size, and scale factor for one or more layers), and layer weights obtained from the training process. During compilation, the system 100 needs to allocate respective computational resources based on the characteristics of the FCN. For example, the system 100 needs to allocate respective data structures to provide respective computations for performing inference calculations. As another example, the system needs to allocate respective memories for storing respective data structures and associated computation results during inference operations for the deployed FCN.

[0044] Conventionally, system 100 must allocate respective data structures and memory according to input size. For example, data structures allocated for layer weight matrices, activation inputs, and outputs are based at least on the input size. Respective memory allocated for storing layer structures and associated computational results is also based on the input size. Thus, deployed FCNs are configured to receive fixed-size inputs when deployed. Systems (e.g., system 100) therefore often statically compile FCNs to receive fixed-size inputs, allowing the system to allocate computational resources efficiently and all at once during compilation.

[0045] In some implementations, host 130 can perform tiling pattern analysis to determine tiling parameters, e.g., an appropriate fixed size, for compiling FCN 115 based on the characteristics of the FCN model and associated hardware accelerator. It should be noted that one or more hosts different from host 130 (e.g., offline analysis / compilation hosts) can perform the tiling pattern analysis offline or in advance of host 130. The one or more hosts can then compile and deploy FCN 115 on host 130 (e.g., one or more communicatively coupled computers) or deploy FCN 115 as an “application” on one or more edge devices (e.g., mobile phones or tablets) for processing inputs of random or unknown size. Determining the fixed size is described in more detail below.

[0046] The system 100 may include any suitable type of hardware accelerator 110 for performing inference calculations for the FCN model. For example, the hardware accelerator 110 may be a CPU, a GPU, or a TPU. The hardware accelerator 110 may include components such as a memory for storing parameters of the FCN model. Furthermore, the hardware accelerator 110 may include one or more computation units for parallel computation.

[0047] Input data 150 may have one or more sizes that differ from the fixed size that compiled FCN 115 is configured to process its input in. For example, input data 150 may include multiple image frames, each having a respective size that differs from the fixed size.

[0048] The generated output data 170 are outputs generated by the system 100 performing inference operations for the trained and statically deployed FCN model 115 for the input data 150. The generated output data 170 can each have a respective size related to the size of the corresponding input data.

[0049] As a simple example, if the input data (e.g., an input image) has a size of 500x500 pixels, the generated output 170 may have a size of 50x50 pixels, 500x500 pixels, or 1000x1000 pixels, with each pixel of the output data being associated with pixels within an 8x8, 10x10, or 20x20 pixel neighborhood of the input image based on the characteristics of the FCN model (e.g., the filter size, stride size, padding size, and scale factor for each layer of the FCN model). In general, the size of the output generated from an input by a trained FCN is a function of the characteristics of the FCN model.

[0050] For example, for ease of explanation, a naive FCN model may include two network layers, each with a filter size of 2x2 pixels, a stride size of 1 pixel, and 1x1 zero padding, such that each of the two layers can generate a 4x4 pixel output by processing a 3x3 pixel input. When a 3x3 input passes through both layers of the network, a 5x5 output is generated; similarly, a 5x5 input generates a 7x7 output through the naive FCN model.

[0051] Assume that a naive FCN model has been compiled to receive 5x5 pixel tiles tiled from input data 150. The system can generate a fixed-size output of 7x7 pixels by analyzing the characteristics of the FCN model. The system can then generate final output 170 by determining valid and dummy regions for the fixed-size output and associating per-pixel values ​​in the valid regions with corresponding pixels in the final output for all fixed-size outputs. Referring to the above example, the valid region of the fixed-size output generated by the naive FCN model can have a size of 3x3 pixels, with per-pixel values ​​calculated using values ​​in the 5x5 pixel input tiles (i.e., the per-pixel values ​​in the valid region are not generated based on any padded zeros).

[0052] As another example, an FCN model may include two convolutional layers, each with a filter size of 3x3 pixels, a stride size of 1, and no zero padding. For a fixed-size input of 50x50 pixels, the fixed-size output generated by the FCN model processing the fixed-size input may have 46x46 pixels. The system 100 may determine that there are no dummy regions in the fixed-size output, and that the valid region of the fixed-size output is 46x46 pixels.

[0053] As another example, an FCN model may include two convolutional layers, each with a filter size of 3x3 pixels, a stride size of 1, and zero padding of a single pixel. A fixed-size output from an FCN model processing a fixed-size input of 50x50 pixels may have a size of 50x50 pixels. The system 100 may determine dummy regions with a width of 2 pixels on all sides of the output data (e.g., the output image), and the valid region of the fixed-size output is 46x46 pixels. The process of determining the dummy and valid regions of the fixed-size output is described in more detail below in connection with the FirstValidPixelOffset() algorithm.

[0054] Generally, when an FCN model includes one or more convolutional layers with a stride size of 2 or more, the input and output sizes can be many-to-one mappings, and are no longer one-to-one mappings. For example, an FCN model can include a first convolutional layer with a filter size of 3x3 pixels, a stride size of 1, and a single pixel of zero padding, and a second convolutional layer with a filter size of 5x5 pixels, a stride size of 2, and a single pixel of zero padding. The FCN model can generate outputs of the same size (e.g., 24x24 pixels) by processing inputs with different sizes (e.g., a 50x50 pixel input and a 49x49 pixel input). This is because a stride size of 2 in the second convolutional layer can trigger a rounding process when processing the input through each network layer.

[0055] In addition, the FCN model may include one or more transposed convolutional layers. For example, the FCN model may include a transposed convolutional layer with a filter size of 5 pixels, no zero padding, and a stride of 2 pixels. The transposed convolutional layer may be added to a second layer with a stride size of 2 pixels. The transposed convolutional layer is generally configured to increase (e.g., blow up) the output size from the input provided by the preceding layer by a factor based on the stride size of the transposed convolutional layer. For the above example, the transposed convolutional layer may generate a 51×51 pixel output by processing a 24×24 pixel output from the second convolutional layer. That is, the FCN model may process a 50×50 pixel or a 49×49 pixel input to generate a 51×51 pixel output.

[0056] A transposed convolutional layer can generate or expand a dummy region even if the padding size is zero. The size of the dummy region can be based on the characteristics of the transposed convolutional layer, such as the relationship between the filter size and the stride size. For example, if the transposed convolutional layer has a stride size smaller than the filter size, the output can include a dummy region because the calculation involves neighboring pixel regions when a fixed-size input is extracted from the full input. As described above, by extracting a fixed-size input from the full input data, the FCN can involve one or more zero-valued neighboring pixels instead of true pixel values ​​in one or more calculations, which causes inaccurate calculations of boundary pixels in the fixed-size input and generates an output having a dummy region containing inaccurate pixel values ​​and a valid region of true pixel values ​​surrounded by the dummy region.

[0057] Referring again to the above example, when a fixed-size input is extracted from full input data (e.g., input data 150), dummy regions are generated. Otherwise, if an FCN is compiled to process full input data directly, dummy regions are not generated. For example, an FCN model including one or more transposed convolutional layers can generate an output with dummy regions. This is because when a fixed-size input is extracted from the full input data, the pixel values ​​of one or more non-zero neighboring pixels that contribute to the output are replaced with zero values.

[0058] To determine the valid and dummy regions of the fixed-size output, the system 100 can trace one or more pixels in the input image for the output pixel, or vice versa, by analyzing the characteristics of the FCN model and the coordinates of the output pixel. For example, the system 100 can arrange 50×50 pixels in an input tile to generate a fixed-size output of 51×51 pixels. As mentioned above, tiles generated from the input data 150 by the host 130 or hardware accelerator 110 with adequate computing power can overlap each other at one or more pixels. Therefore, the system can employ a specific algorithm (e.g., the ProjectBackwards() algorithm, as described in more detail below) to trace back the corresponding input pixel used to generate the output pixel in the fixed-size output through the deployed FCN model.

[0059] 1 , host 130 can communicate with hardware accelerator 110 by transmitting data, instructions, or both. Host 130 and hardware accelerator 110 can communicate through a wired or wireless connection and, in some cases, can be located remotely from one another. For example, host 130 can be a server in a different physical location from where accelerator 110 is located.

[0060] The host 130 may receive input data 150 of a size larger than the fixed size and generate a plurality of fixed-size inputs 138, each having a size smaller than the input data 150. The host 130 may provide the fixed-size inputs 138 to the hardware accelerator 110 and receive a plurality of corresponding fixed-size outputs 133 from the hardware accelerator 110. The received fixed-size outputs 133 are generated by the hardware accelerator 110 performing inference operations on the unfolded FCN 115 for the provided plurality of fixed-size inputs 138.

[0061] In some implementations, as described above, a hardware accelerator including a hardware component such as a CPU can perform a tiling process to tile the input data 150 into multiple fixed-size inputs 138.

[0062] In situations where the input data 150 has a size smaller than the fixed size, the system 100 can pad the input data 150 with zeros to reach the fixed size and provide it to the hardware accelerator 110 for performing the inference calculations.

[0063] To generate output data 170, host 130 may further include a stitching engine 140 configured to combine the received fixed-size outputs 133. Stitching engine 140 may perform the stitching process by determining alignment information for each of the fixed-size outputs 133 based on the fixed size and characteristics of the unfolded FCN model, and generate final output 170 that is an equivalent output as if it had been obtained by directly processing input data 150 with the same FCN model, but unfolded to process input having the size of input data 150, without tiling.

[0064] In some implementations, a hardware accelerator including hardware components capable of performing the stitching process can generate a final output 170 on the hardware accelerator based on the fixed size output 133 and provide the final output 170 to the host 130 or on a display of a user interface.

[0065] In some implementations, the tiling and stitching processes described throughout this specification may not be performed remotely from the host. For example, any suitable accelerator including appropriate hardware components, such as a CPU, may perform the tiling and stitching processes on the accelerator. Furthermore, the tiling and stitching processes may be performed in a different physical location than the host. For example, tiling may be performed by a first set of accelerators in a first location, the stitching process may be performed by a second set of accelerators in a second location, and the host may be located in a third location and configured to receive the final output from the second set of accelerators. The accelerators and the host are communicatively connected and may be physically or wirelessly connected in one or more locations.

[0066] FIG. 2A illustrates an exemplary inference process 200 using a conventional system. As shown in Figure 2A, a conventional inference system 200 can receive input data 150 (as shown in Figure 1) and generate output data 225 by performing inference computations for an unfolded FCN 215 that processes the input data 150. The output data 225 can be substantially similar to the output data 170 as shown in Figure 1. The input data 150 can be an input image, as described above.

[0067] Each of the input data 150 can have a different size, which requires conventional systems to recompile the FCN 215 for the hardware accelerator to process the different input sizes. For example, if the first input data has a size of 50x50 pixels, the system can deploy the FCN 215 on the hardware accelerator to be configured to process inputs of size 50x50 pixels. However, if the second input data has a different size than the first input data, e.g., 100x100 pixels, the system must recompile the FCN 215 to be configured to process inputs of size 100x100.

[0068] For input data having different sizes, conventional systems must first determine the size of the particular input and then determine whether FCN 215 needs to be recompiled to process the particular input. In addition, system 200 must perform extra computational checks to monitor whether memory and data structures are properly allocated. In that case, conventional techniques for performing inference computations can incur a substantial amount of overhead, reducing the computational efficiency for generating inference outputs given inputs of various sizes.

[0069] FIG. 2B shows an example inference process 250 using the example inference system 100 of FIG.

[0070] As shown in FIG. 2B, an inference system 100 employing the techniques described in connection with FIG. 1 can avoid recompiling an unfolded FCN for inputs with different sizes, which reduces overhead and improves computational efficiency. More specifically, the system 100 first statically unfolds an FCN model 235 on a hardware accelerator. The system 100 can determine alignment information based on characteristics of the FCN model and determine a tiling pattern including a fixed size for tiling dummy and valid regions of the fixed-size output so that each valid pixel can be obtained from at least one fixed-size output. The FCN model 235 is compiled to process an input having a fixed size. The system 100 can tile input data 150 to generate multiple fixed-size inputs 230 based on the tiling pattern and provide the fixed-size inputs 230 for performing inference calculations using the unfolded FCN 235. The system 100 can obtain fixed-size outputs 240 from the unfolded FCN 235 and stitch the fixed-size outputs 240 to generate final output data 170. The output data 170 is equivalent to the output data 225 that would be obtained by directly processing the full input data 150 with the FCN 215 that was compiled to process the full input data 150. Tiling and stitching are described in more detail below.

[0071] In some implementations, the system 100 can determine a set of candidate sizes suitable for the FCN model to process random-sized inputs. The system 100 can determine the set of candidate sizes from all tile sizes based on the characteristics of the FCN model (e.g., layer characteristics such as filter size and stride size). It should be noted that some tile sizes cannot be used according to the characteristics. For example, a specific input size cannot generate an output size based on the filter size and stride size of the FCN model. For example, the system 100 can remove sizes from all possible sizes that are not suitable for the FCN model to generate candidate sizes.

[0072] In some implementations, the system 100 can select a fixed size from multiple candidate sizes for deploying the FCN model. For example, the system 100 can select the fixed size based on performance.

[0073] In some implementations, for each of the candidate sizes, the system 100 can deploy a respective copy of the FCN model on a respective hardware accelerator for processing inputs having one of the candidate sizes. The system 100 can measure a level of performance, such as the total execution time for performing an inference calculation using different copies of the FCN network processing different fixed-size inputs, or, as another example, the overhead in a system 100 including multiple hardware accelerators for performing an inference calculation for each deployed FCN. Based on the performance measurement, the system 100 can select one of the candidate sizes as the fixed size for deploying the FCN model on a particular hardware accelerator. For example, the system 100 can select the candidate size that results in the smallest total execution time. As another example, the system 100 can select the candidate size that results in the least overhead. Alternatively, the system 100 can select a candidate size that has a satisfactory execution time and overhead for performing the inference calculation.

[0074] The selection of the candidate size can be based on the characteristics of the trained FCN model. For example, if the candidate size for tiling (or for the unfolded FCN model) is too small, the fixed-size output generated from the fixed-size input of the candidate size may also be small and may not even contain any valid regions (i.e., all per-pixel values ​​in the fixed-size output are in dummy regions).

[0075] Alternatively, the system 100 may provide a user with a discrete range of candidate sizes to select one candidate size within the range as a fixed size. The discrete range of candidate sizes may be non-contiguous based on the characteristics of the FCN (e.g., the number and position of one or more transposed layers and the characteristics of each layer included in the FCN). For example, the candidate size range may be an even number of pixels, from 10x10 pixels to 30x30 pixels. The user may select 16x16 pixels as a fixed size within the provided range.

[0076] Furthermore, the fixed size need not be a scalar. Instead, the fixed size can be a vector representing a rectangle in two-dimensional space or a block in three-dimensional space. More specifically, the fixed size can include a respective value in each dimension. For example, if the input image is two-dimensional, the system 100 can determine a fixed-size vector having a first size for a first dimension (e.g., the horizontal dimension) and a second size for a second dimension (e.g., the vertical dimension) that is different from the first dimension. The system 100 can generate multiple fixed-size inputs of 30×10 pixels from an input image having a size of 300×100 pixels.

[0077] 1 and 2B, after receiving input data 150, system 100 can determine multiple fixed-size inputs from the received input data. For input data having a size larger than the fixed size, system 100 can analyze the input data and tile the input data into multiple fixed-size inputs, with or without any overlap. For input data having a size smaller than the fixed size, system 100 can pad zeros around the input data to reach the fixed size and provide the padded inputs (which are now also fixed-size) to the unfolded FCN model for performing inference calculations.

[0078] To tile the received input data 150 into multiple fixed-size inputs 138, the host 130 may include a tiling engine 135 configured to receive the input data 150 and generate the multiple fixed-size inputs 138 based on a tiling pattern. Alternatively, a suitable hardware accelerator 110 may tile the input 150 into multiple tiles of fixed size. More specifically, the host 130 may send instructions to the hardware accelerator 110, including binary data representing the compiled FCN model and memory addresses for storing the input data 150. The hardware accelerator 110 may include a suitable computational component, such as a CPU, and may be configured to retrieve one or more tiles by accessing (e.g., direct memory access) corresponding memory addresses that store per-pixel values ​​for one or more tiles. For example, the hardware accelerator may retrieve a 5x5 pixel tile by accessing corresponding memory addresses that store per-pixel values ​​for the tile without accessing memory addresses that store pixel values ​​outside the tile. In this manner, the system 100 may reduce memory traffic and increase computational efficiency, as described above.

[0079] The system 100 can determine a tiling pattern for tiling the input data 150 into multiple fixed-size inputs 138. For example, the tiling engine 135 can tile the input data 150 into fixed-size inputs at specific sizes with specific overlap sizes, e.g., the tiled fixed-size inputs do not have any overlap, or each have a shared overlap region of a specific size, or each overlap each other at their respective sizes. Thus, the total number of fixed-size inputs generated from the input data depends on the tiling pattern.

[0080] Note that if the FCN model includes one or more transposed convolutional layers with a stride size greater than or equal to two, the tiling pattern is further determined based on the alignment information.

[0081] The overlap size for the tiling pattern can be any suitable size smaller than the fixed size. For example, each fixed size input can have a shared overlap size that is one pixel wide and the length of the edge of the fixed size input. As another example, the overlap sizes can be two pixels, three pixels, and five pixels wide. The fixed size and overlap size for the tiling are determined based at least on the alignment information.

[0082] The system 100 can automatically determine the tiling pattern based on the characteristics of the FCN model or user instructions. For example, the system 100 can tile a 100x100 pixel input image into four fixed-size inputs, each 60x60 pixels. Each of the fixed-size inputs can have an overlapping area of ​​20x60 pixels, 60x20 pixels, or 20x20 pixels.

[0083] Optionally, the system 100 can generate a tiling pattern such that the fixed-size inputs have respective overlap regions with each other. For example, a 70×30 pixel input image can be tiled into a 30×30 pixel fixed-size input that is compatible with the unfolded FCN model. In one situation, four fixed-size inputs overlap each other in a 20×30 pixel region. Note that the last fixed-size input can have a 10×30 pixel region outside the input image, which can be expanded or padded with zeros. In some implementations, the system can shift the overlap region of the last fixed-size input with the other inputs to reduce and even eliminate padded zeros for increased computational efficiency.

[0084] In some implementations, the system 100 can determine the tiling pattern based on an appropriate machine learning model trained on various training data. The training data can be sets of fixed-size inputs for the same copy of the input, but each tiled based on a different tiling pattern. The machine learning model can output one or more tiling patterns for the system 100 or for a user to select from for the system 100.

[0085] System 100 can generate output 170 by stitching fixed-size outputs using stitching engine 140. Because system 100 has obtained a tiling pattern that includes coordinates of pixels in the active area, the system can efficiently stitch pixels from the active area to generate the complete output data. The system can employ an algorithm for the stitching process, which is described in more detail below.

[0086] For each fixed-size output, the system 100 can obtain coordinates for the particular fixed-size output and coordinates of the corresponding fixed-size input for generating the particular fixed-size output. The coordinates of the particular fixed-size input represent the position of the fixed-size input relative to the original input 150; similarly, the coordinates of the particular fixed-size output represent the position of the fixed-size output relative to the corresponding final output 170. The system 100 can determine a respective coordinate frame (e.g., a Cartesian coordinate frame, or any suitable individual coordinate frame) and the origin of the coordinate frame for each input and corresponding output data. The system 100 can determine the coordinates of the fixed-size input during the tiling process and then determine the coordinates of the corresponding fixed-size output according to the characteristics of the unfolded FCN model 115. Similarly, the system 100 can first determine the coordinates of the fixed-size output, and then determine the coordinates of the corresponding fixed-size input based on the characteristics of the FCN model 115. The system 100 can apply one or more algorithms to generate alignment information, generate relationships between the coordinates of the fixed-size inputs and the fixed-size outputs, and stitch the fixed-size outputs based on the relationships. The alignment information is described in more detail below.

[0087] Once the alignment constraints for the FCN model are satisfied, the system 100 can further determine a central valid region and a peripheral dummy region for each fixed-size output after associating the coordinates of the fixed-size output and the corresponding fixed-size input. The central valid region includes pixels generated using valid pixels from the corresponding fixed-size input. The dummy region includes pixels generated using one or more zero-valued neighboring pixels (e.g., from zero values ​​that would have been non-zero for pixels in the full input image outside the fixed-size input).

[0088] System 100 can determine one or more overlapping regions between the fixed-size outputs. Optionally, the system can also determine whether at least a portion of the overlapping region belongs to the effective region of the fixed-size output. In some implementations, system 100 can determine a coordinate shift for one or more overlapping fixed-size outputs, such that the effective regions of different fixed-size outputs are positioned adjacent to or abutting each other without overlapping.

[0089] FIG. 3A shows an exemplary fixed size input 138 with a contiguous pixel region 310 and an exemplary fixed size output 133 with a dummy region 320 .

[0090] As described above, the system 100 can tile the complete input data 150 into multiple fixed-size inputs having fixed sizes that fit the unfolded FCN model. The system 100 can determine a tiling pattern for the input data 150 and generate the fixed-size inputs by tiling the input data 150 from top to bottom and left to right. The tiling pattern can include overlapping regions and, essentially, positions defined by the respective coordinates of each fixed-size input. For example, as shown in FIG. 3, one fixed-size input 138 is located at a specific position in the complete input data 150. Details of the tiling pattern will be described below in conjunction with FIGS. 3B and 3C.

[0091] The position of the fixed size input 138 may be represented using the coordinates of one or more corner pixels relative to the origin of the complete input data 150. For example, the system may determine the upper left corner pixel of the complete input data 150 as the origin (0,0). The coordinates for each fixed size input are determined with respect to the origin. For example, the system 100 may use the coordinates of the upper left corner pixel and the lower right corner pixel of the fixed size input 138 to represent the position and size of the input 138.

[0092] The fixed-size inputs may be represented by any suitable coordinate frame. For example, the coordinates of each fixed-size input 138 may be represented in a Cartesian coordinate frame, a cylindrical coordinate frame, or any other suitable coordinate frame.

[0093] The tiling pattern may define a position for each fixed-size input 138 in any suitable manner. For example, the fixed-size inputs may be positioned in rows and columns. As another example, the fixed-size inputs may be scattered and mismatched. In other words, the fixed-size inputs 138 need not be aligned with each other in rows and columns, for example, in a zigzag pattern.

[0094] System 100 can annotate the locations of fixed-size inputs with any suitable notation. For example, system 100 can use (i,j) notation to represent a fixed-size input at the ith position along the first dimension and the jth location along the second dimension. For simplicity, in the following specification, system 100 annotates the fixed-size inputs in a tiling grid. That is, each fixed-size input is represented by a sequential number along the row and column. Each fixed-size input can be thought of as substantially rectangular. However, it should be appreciated that the tiling pattern and annotations can vary based on tiling requirements.

[0095] The system 100 calculates the coordinates of the top left corner pixel as

[0096]

number

[0097] and the bottom right corner pixel is

[0098]

number

[0099] where i and j represent the numbering of each fixed-size input with respect to the input data 150. For example, i and j represent the respective row and column for the fixed-size input of all the multiple fixed-size inputs.

[0100] As another example, suppose the input image has 100x100 pixels, the system 100 tiles the input image into a 3x3 grid (i.e., 9 fixed-size inputs) with respective overlap sizes. The fixed-size inputs in the first row of the 3x3 grid are arranged such that the first fixed-size input placed in the first grid is

[0101]

number

[0102] and

[0103]

number

[0104] a second fixed size input arranged in a second grid, the second fixed size input having coordinates of

[0105]

number

[0106] and

[0107]

number

[0108] a third fixed size input arranged in a third grid, the third fixed size input having coordinates of

[0109]

number

[0110] and

[0111]

number

[0112] The fixed-size inputs in the first column of the 3x3 grid can include a first fixed-size input, and a fourth fixed-size input located in the fourth grid can have coordinates of:

[0113]

number

[0114] and

[0115]

number

[0116] and a fifth fixed size input arranged in a seventh grid may have coordinates of

[0117]

number

[0118] and

[0119]

number

[0120] Note that pixel values ​​of the third and fifth fixed size inputs outside the input image may be extended and set as zero.

[0121] To avoid redundantly counting or calculating edge pixels for each of multiple fixed-size inputs 138, in some implementations, during tiling, system 100 can determine that for each of fixed-size inputs 138, pixels at the top and left edges of the fixed-size input are considered to be included in the fixed-size input, while pixels at the bottom and right edges of the fixed-size input are not considered to be included in the fixed-size input.

[0122] Before tiling the input data 150 into multiple fixed-size inputs, the system 100 may determine whether the input data is smaller than the fixed size established for the system 100. In response to determining that the input data 150 is smaller than the fixed size, the system 100 may pad zeros around the perimeter of the input data 150 to reach the fixed size.

[0123] Note that the term "neighboring pixel region 310" refers to a region containing neighboring pixels generated by using zero values ​​to replace original non-zero values ​​for the neighboring pixels, as described above. For example, the neighboring pixel region 310 may include a region containing one or more neighboring pixels of the fixed-size input 138 in the complete input data 150, as shown in FIG. 3A. The width 315 of the neighboring pixel region 310 may represent the number of neighboring pixels included in the neighboring pixel region 310. The system 100 may determine the width 315 for the neighboring pixel region 310 based on the characteristics of the deployed FCN 115 model.

[0124] The system 100 can obtain coordinates for each fixed size output 133 with respect to the final output data 170. For example, the system 100 can select the top left corner pixel of the final output data 170 as the origin and define the coordinates of the top left corner pixel of the fixed size output as

[0125]

number

[0126] and the bottom right corner pixel is

[0127]

number

[0128] where i and j represent the numbering of each fixed-size output with respect to output data 170. For example, i and j represent the respective row and column for each fixed-size output of all fixed-size outputs.

[0129] The system may further determine a valid region 330 and a dummy region 320 for each fixed-size output 133 based on characteristics of the FCN model, as described above. Generally, the valid region 330 may be located at the center of the fixed-size output 133, and the dummy region 320 may surround the valid region 330 with a width 335. The width 335 determines the specific number of pixels in each dummy region 320. The valid region includes per-pixel values ​​for pixels in the valid region that are calculated using valid per-pixel values ​​in the corresponding fixed-size input 138, and the dummy region 320 includes per-pixel values ​​for pixels in the dummy region that are calculated using at least one or more neighboring pixels during a tiling process or through operations characterized in one or more layers in the FCN model. The per-pixel values ​​for pixels in the valid region 330 contribute, at least in part, to the final output 170, while the dummy pixels are eliminated or discarded during the stitching process.

[0130] According to the characteristics of the FCN model, system 100 can determine valid region 330 and dummy region 320 by tracing back from a pixel in the fixed-size output through the FCN model to one or more corresponding pixels in the fixed-size input. More specifically, system 100 can execute the FirstValidPixelOffset() algorithm, as described below, to determine the width for the dummy region, and the valid region is the remaining portion of the region in the output.

[0131] More specifically, the FirstValidPixelOffset() algorithm is configured to propagate invalid information layer by layer to determine the final dummy region of the FCN output. In the first layer of the FCN, the layer generates dummy regions in its output by using pixels in the neighboring pixel region for the first layer. However, from the second layer onwards, the dummy regions of the layer output grow by using neighboring pixels and dummy pixels generated and propagated from the previous layer.

[0132] By executing the FirstValidPixelOffset() algorithm, the system 100 can determine the width 335, and essentially the number of pixels within the width 335, based on the characteristics of the FCN model (e.g., the respective filter sizes, zero padding sizes, stride sizes, and scale factors for all layers in the FCN model). Note that the width 335 of the dummy region can include all dummy pixels. However, in some implementations, the width 335 is large enough to include all dummy pixels and one or more valid pixels.

[0133] If the FCN model includes one or more transposed layers, system 100 can determine width 335 of dummy region 320 based on the number and position of the one or more transposed layers. With reference to FIG. 3D , which illustrates an exemplary process for generating an output using transposed convolutional layers 340, 345 in an FCN model, the FCN model can be equivalent to compiled fully convolutional network 115 of FIG. 1. For simplicity, an appropriately configured system, such as inference system 100 of FIG. 1, can perform the process of FIG. 3D.

[0134] As shown in FIG. 3D , the FCN model may include a transposed convolution layer 340 configured to receive a 2×2 pixel output 341 from a previous layer in the FCN model. The system may perform operations associated with the transposed convolution layer 340 to generate a 4×4 pixel output 342. The transposed convolution layer 340 includes a filter size of 3×3 pixels with a stride size of 1. The transposed convolution layer 340 does not include any zero padding. Input pixel A is associated with output pixels A1, A2, A3, C1, C2, C3, D1, D2, and D3, and input pixel B is associated with output pixels C1, C2, C3, D1, D2, D3, B1, B2, and B3. The overlap region of the output pixels associated with input pixels A and B includes pixels C1, C2, C3, D1, D2, and D3.

[0135] Note (not shown) that pixels C1, C2, and C3 are also associated with input pixels to the left of pixel A in output 341. Similarly, pixels A1, A2, and A3 are associated with two input pixels to the left of pixel A, and pixels D1, D2, and D3 are associated with input pixels A and B and another input pixel to the right of pixel B.

[0136] Assuming a full input image can generate intermediate outputs through previous layers including the first pixel to the left of pixel A, pixel A, and pixel B, the pixel values ​​for A1, A2, A3, C1, C2, and C3 are not accurate because the fixed-size input does not generate a pixel value for the first pixel, and therefore system 100 uses zero-value neighboring pixels to represent the first pixel to generate partial outputs for pixels A1, A2, A3, C1, C2, and C3. However, the pixel values ​​for D1, D2, D3, B1, B2, and B3 are accurate because both the full input and the fixed-size input use zero pixel values ​​for the pixels to the right of pixel B.

[0137] Similarly, transposed convolution layer 345 includes a stride size of 2 pixels in both directions and a filter size of 3x3 pixels, and is configured to receive output 344 from the previous layer and generate a 5x5 pixel output 345. Input pixel A is associated with pixels A1, A2, A3, C1, C2, C3, D1, D2, and D3, and input pixel B is associated with pixels D1, D2, D3, B1, B2, B3, E1, E2, and E3. The overlap region includes the exact pixels D1, D2, and D3 because these pixels are not calculated using neighboring pixels.

[0138] Although FIG. 3D only depicts determining the correct and incorrect pixel values ​​for only one transposed convolutional layer, system 100 can determine the dummy and valid regions for a fixed-size output by analyzing the relationship between the inputs and outputs for all layers of the FCN model as described above.

[0139] In addition, the system 100 can determine alignment information based on the relationship between the inputs and outputs for each layer of the FCN model.

[0140] 3D , unlike transposed convolutional layer 340, transposed convolutional layer 345 has a stride of 2 pixels. Thus, outputs including C1, C2, C3, D1, D2, D3, B1, B2, and B3 do not have corresponding pixels in output 344. System 100 can determine alignment information for transposed convolutional layer 345 to be an integer multiple of 2 pixels. Integer multiples of 2 pixels can be, for example, 2, 4, 8, and 10 pixels, included in the alignment information to ensure valid mapping in the transposed convolutional layer.

[0141] If the FCN model includes two or more transposed layers, the system 100 can determine global alignment information (e.g., accumulated alignment values ​​for all layers, or global alignment values) for the entire FCN model based on the characteristics of all the transposed layers (e.g., the number, positions, and strides of the transposed layers). In some implementations, the system 100 can determine the global alignment information as the product of the respective stride sizes of all the transposed layers.

[0142] The system 100 can determine the alignment information from multiple candidate alignment values ​​based on the correctness of the final output, memory traffic during calculation, and computational efficiency. In particular, with respect to the correctness of the final output, the system 100 can select an overall alignment value that ensures that each pixel of the final output can be obtained from the valid area of ​​one of the fixed-size outputs.

[0143] For FCN models that include other types of layers, such as pooling layers, system 100 can treat the other types of layers as a form of convolutional layer for analyzing the tiling and stitching process. For example, a max-pooling 2x2 layer can be treated as a convolutional layer with a stride of 2 pixels, a filter size of 2x2 pixels, and no zero padding for analyzing the tiling and stitching process.

[0144] For ease of explanation, outputs 341 and 344 are sized 2x2 pixels, output 342 is sized 4x4 pixels, and output 346 is sized 5x5 pixels, but it should be noted that the inputs and outputs can generally have any suitable sizes. Similarly, the filter sizes, strides, and zero padding for transposed convolutional layers 340 and 341 can include any suitable sizes.

[0145] In general, when an FCN model includes one or more transposed layers, determining the dummy region in the fixed-size output may be substantially complicated. However, a system implementing the techniques described herein can determine the propagation of the dummy region from a preceding layer to a succeeding layer regardless of whether the layer is a convolutional layer or a transposed convolutional layer, and theoretically can determine the dummy region for a fixed-size output given a fixed-size input based on the characteristics of the FCN model regardless of how many network layers the FCN model includes.

[0146] One or more layers of the FCN model may have different characteristics along different dimensions (e.g., the height and width dimensions in the case of a two-dimensional layer). For example, the filter size, stride size, or padding size of a network layer may not be the same along the height and width dimensions (e.g., a filter size of 3x2 pixels, a stride size of 2x1 pixels, and a zero padding size of 0x1 pixels). The techniques described herein may calculate alignment information, dummy regions, and tiling patterns independently along each dimension, which may result in non-uniform fixed-size outputs along different dimensions. For example, the system 100 may generate non-uniform widths for the dummy regions 320, i.e., the widths 335 may be non-uniform for the dummy regions 320. For example, the widths 335 of the left and right portions of the dummy regions 320 may be larger than the top and bottom portions.

[0147] In general, an FCN model can receive an input tensor and generate an output tensor in multiple dimensions. For example, the input tensor can have multiple channels C and multiple batches B, in addition to the height H and width W dimensions as described above.

[0148] An FCN model can be adapted to process each of the multiple dimensions of an input, as long as the dimensions are fully convolutional. For example, an FCN model can process an image input having dimensions B x H x W x C. Assuming that the batch and channel dimensions are not fully convolutional, the FCN model can process the input only in the height and width dimensions, where the process can generally be thought of as a two-dimensional problem. As another example, an FCN can process an audio input with multiple dimensions by processing only a single-dimensional audio input if the rest of the dimensions are not fully convolutional. Alternatively, an FCN model can process higher dimensions, e.g., dimensions greater than two, if these dimensions are fully convolutional.

[0149] The system 100 can also determine the coordinates of the effective area 330 of the fixed size output 133. Similarly, the system can determine the top left corner pixel of the effective area relative to the origin of the fixed size output 133.

[0150]

number

[0151] and the bottom right corner pixel is

[0152]

number

[0153] For example, i and j represent the row and column, respectively, for the corresponding fixed-size output 133 or the effective area of ​​the corresponding fixed-size output 133.

[0154] For an unfolded FCN model without transposed convolutional layers, system 100 can stitch fixed-size outputs through a first algorithm described below. As another example, for an unfolded FCN model with transposed convolutional layers, system 100 can stitch fixed-size outputs based on alignment information generated using a second algorithm described below.

[0155] The first algorithm can ensure that the effective areas of the fixed-size outputs do not overlap, while the second algorithm potentially allows the effective areas of the fixed-size outputs to overlap, which requires extra steps to properly combine the fixed-size outputs. The extra steps can include coordinate shifts for the effective area of ​​each of the fixed-size outputs, or for each of the fixed-size outputs, or both, with coordinate shifts being described in more detail below.

[0156] When using the first algorithm, the system 100 calculates the width of the dummy region 335 as b and the mapping functions for the fixed-size input, the corresponding fixed-size output, and the coordinates of the valid area of ​​the fixed-size output, respectively.

[0157]

number

[0158] ,

[0159]

number

[0160] and

[0161]

number

[0162] Each mapping function can return a particular coordinate to a particular direction (e.g., I(i,j)).

[0163]

number

[0164] represents a coordinate in the vertical or height direction. For simplicity, the system 100 assumes that the fixed-size input and the fixed-size output are square in two-dimensional space, and the size of the fixed-size input is T I , the size of the fixed-size output is T O The system uses the size of H I and W I and without loss of generality, H I >=T I and W I >=T I It is assumed that: Note also that fixed size inputs and outputs can be rectangular in some implementations.

[0165] The system 100 can execute the following first algorithm using dynamic programming to scan from left to right and top to bottom according to the respective coordinates of the fixed size output to generate the final output 170. The first algorithm is written as follows: Initialization: O(0,0) = (0,0,T O ,T O ) V(0,0) = (0,0,T O - b,T O - b) Left border tile: O(i,0) = (V(i - 1,0).hb - b,0,V(i - 1,0).hb - b + T O ,T O ) V(i,0) = (V(i - 1,0).hb,0,V(i - 1,0).hb - 2b + TO ,T O - b) Upper realm タイル: O(0,j) = (0,V(0,j - 1).wb - b,T O ,V(0,j - 1).wb - b + T O ) V(0,j) = (0,V(0,j - 1).wb,T O - b,V(0,j - 1).wb - 2b + T O ) Internal タイル: O(i,j) = (V(i - 1,j).hb - b,V(i,j - 1).wb - b,V(i - 1,j).hb - b + T O ,V(i,j - 1).wb - b + T O ) V(i,j) = (V(i - 1,j).hb,V(i,j - 1).wb,V(i - 1,j).hb - 2b + T O ,V(i,j - 1).wb - 2b + T O ) Right realm タイル: O(i,lastj) = (V(i - 1,lastj).hb - b,W O - T O ,V(i - 1,lastj).hb - b + T O W O ) V(i,lastj) = (V(i - 1,lastj).hb,W O - T O + b,V(i - 1,lastj).hb - 2b + T O W O ) Lower realm タイル: O(lasti,j) = (H O - T O ,V(lasti,j - 1).wb - b,H O ,V(lasti,j - 1).wb - b + T O ) V(lasti,j) = (H O - TO + b,V(lasti,j - 1).wb,H O ,V(lasti,j - 1).wb - 2b + T O ) According to the first algorithm described above, the system 100 can generate valid fixed-size outputs with valid regions adjacent to each other without overlap. More specifically, the system 100 can discard pixels in the dummy regions and combine the valid regions in the fixed-size outputs to generate the final output. Furthermore, because the valid regions between the fixed-size outputs do not overlap, the system using the first algorithm can calculate almost all pixels in the valid region in just one go, which optimizes computational efficiency for FCN models without transposed convolutional layers. An example of this implementation is described in more detail with reference to FIG. 3B.

[0166] For FCN models that include transposed convolutional layers, system 100 must execute a second algorithm to address alignment information. The valid regions generated using the second algorithm may potentially overlap, resulting in duplicate calculations for one or more pixels in the valid regions.

[0167] The system 100 can obtain alignment information for the fixed-size output according to the computational requirements imposed by the transposed convolutional layer in the FCN model. For example, the requirements can be that the pixel index for one or more pixels in the fixed-size input that are traced from one or more pixels in the fixed-size output should be integers.

[0168] The second algorithm is written as follows: Initialization: O(0,0) = (0,0,T O ,T O ) V(0,0) = (0,0,T O - b,T O - b) Left border tile: U O = (V(i - 1,0).hb - b,0,V(i - 1,0).hb - b + T O ,T O ) O(i,0) = AlignOutputTile(U O ) V(i,0)=(O(i,0).ht + b,0,O(i,0).hb - b,T O - b) Top border tiles: U O = (0,V(0,j - 1).wb - b,T O ,V(0,j - 1).wb - b + T O ) O(0,j)=AlignOutputTile(U O ) V(0,j)= (0,O(0,j).wt + b,T O - b,O(0,j).wb - b) Interior tiles: U O = (V(i - 1,j).hb-b,V(i,j - 1).wb - b,V(i - 1,j).hb - b + T O ,V(i,j - 1).wb - b + T O ) O(i,j) = AlignOutputTile(U O ) V(i,j)=(O(i,j).ht + b,O(i,j).wt + b,O(i,j).hb - b,O(i,j).wb - b) The second algorithm is a modified version of the first algorithm. In particular, the system 100 can obtain coordinates of an "unaligned" fixed-size output that does not consider alignment requirements. The fixed-size output is U OThe second algorithm can determine alignment information for the "unaligned" fixed-size output and determine whether the "unaligned" fixed-size output satisfies the alignment information based on the following AlignOutputTile() function. The alignment information can be obtained using the AlignOutputTile() function based on at least one of a local search or an analytical method. The alignment information can include coordinate shifts for shifting the "unaligned" fixed-size output left and up. In some implementations, the alignment information can represent analytically determined alignment values ​​based on the characteristics of the FCN model. Details of the alignment values ​​and functions for obtaining the alignment values ​​are described below.

[0169] By executing the second algorithm, the system 100 can ensure that each pixel value associated with the final output can be obtained from at least one of the fixed-size outputs, and the alignment values ​​for the fixed-size outputs can ensure that each corresponding fixed-size input has integer pixel coordinates with respect to the input image. Thus, the system 100 can obtain the coordinates of the valid area by subtracting the dummy area using the second algorithm.

[0170] Details of the tiling and stitching process using the first and second algorithms are described in connection with Figures 3B and 3C, respectively.

[0171] System 100 can also use the properties of the deployed FCN model to obtain the coordinates of the fixed-size inputs based on the coordinates of the corresponding fixed-size outputs. More specifically, system 100 can obtain the coordinates of the insertion forces based on the coordinates of the layer outputs, as well as the layer padding, stride, filter size, and scale factor. One exemplary algorithm is called "ProjectBackwards()" and is written as follows: function ProjectBackwards((ht, wt, hb, wb),layers): for layer = output to input layers: if ht == hb or wt == wb: THROW EXCEPTION; / / The layer is destroyed if layer type is “conv”: / / Conv layer: n=floor((m+2p-f) / s)+1, n is the output size, m=input size s = stride of the layer; p = padding of the layer; f = filter size of the layer n_h = hb - ht; n_w = wb - wt / / Output tile size in h and w dimensions / / Input tile size m_h = (n_h - 1) * s + f - 2p; m_w = (n_w - 1) * s + f - 2p / / Note: m_h x m_w is the minimum input tile size, / / Any size up to (m_h + s-1) x (m_w + s-1) works to produce n_h x n_w output / / If the trans_conv layer precedes the conv layer in the model order and does not allow for a certain size / / Selecting a larger size can be a problem / / The size can be investigated as a backtracking, not shown here for simplicity ht = ht * s; hb = ht + m_h wt = wt * s; wb = wt + m_w else if layer type is “trans_conv”: / / TransConv layer:n=(m+2p-1)*s+f s = stride of the layer; p = padding of the layer; f = filter size of the layer n_h = hb - ht; n_w = wb - wt / / Output tile size in h and w dimensions / / Input tile size m_h = Validate((n_h - f) / s - 2 p + 1) m_w = Validate((n_w - f) / s - 2 p + 1) ht = Validate( ht / s ); hb = ht + m_h wt = Validate( wt / s ); wb = wt + m_w return (ht, wt, hb, wb) where: function Validate(value): if value is integral: return value else: THROW EXCEPTION / / Values ​​cannot be used for coordinates of fixed size inputs in FCN The ProjectBackward() algorithm calls the Validate() function to check whether the coordinates of a fixed-size input can be properly projected from the layer's output to the layer's input. This Validate() function can determine, for example, whether an output location (e.g., pixel coordinate) selected by system 100 is not suitable (i.e., the projected coordinate contains a non-integer value) due to alignment constraints or alignment information for one or more transposed convolutional layers, and therefore the output location that system 100 attempts to project to the fixed-size input location is invalid and cannot be used.

[0172] In some implementations, the system 100 can also obtain coordinates for the fixed-size output based on the coordinates of the corresponding fixed-size input and the properties of the deployed FCN model. One exemplary algorithm is called "ProjectForward()" and is written as follows: function ProjectForward((ht, wt, hb, wb), layers): for layer = input to output layers: if ht == hb or wt == wb: THROW EXCEPTION; / / The layer is destroyed if layer type is “conv”: / / Conv layer: n=floor((m+2p-f) / s)+1, n is the output size, m=input size s = stride of the layer; p = padding of the layer; f = filter size of the layer m_h = hb - ht; m_w = wb - wt / / input tile size in h and w dimensions / / Output tile size n_h = floor((m_h + 2 p - f) / s) + 1; n_w = floor((m_w + 2 p - f) / s) + 1 ht = Validate(ht / s); hb = ht + n_h wt = Validate(wt / s); wb = wt + n_w else if layer type is “trans_conv”: / / TransConv layer: n=(m+2p-1)*s+f s = stride of the layer; p = padding of the layer; f = filter size of the layer m_h = hb - ht; m_w = wb - wt / / input tile size in h and w dimensions / / Output tile size n_h = (m_h + 2 p - 1) * s + f; n_w = (m_w + 2 p - 1) * s + f; ht = ht * s; hb = ht + n_h wt = wt * s; wb = wt + n_w return (ht, wt, hb, wb) Similarly, the Validate() function can be used by the ProjectForwards() algorithm to validate the projection of fixed-size input locations onto corresponding output locations, e.g., to determine whether fixed-size input locations for a convolutional layer with a stride size of 2 or greater are not suitable.

[0173] 3A and 3D , the system can determine the width b for the dummy region such that the region of width b should include at least all incorrect pixels. In some implementations, the dummy region can include all incorrect pixels and one or more correct pixels. However, the width b should not be so large as to impair computational performance, because a large width b could result in the system 100 generating a larger number of overlapping fixed-size outputs and fixed-size inputs during the tiling and stitching process. The system 100 can determine the minimum value for width b by calculating the first valid pixel offset for the layer output of each layer. The first valid pixel for the current layer is calculated by the system 100 without using any zero-valued neighboring pixels from the output of the previous layer. The system 100 performs the operation of the function FirstValidPixelOffset() as follows: function FirstValidPixelOffset(layers): / / The first pixel offset where the previous layer produced a valid result first_valid_offset = 0 for layer = input to output layers: if layer type is “conv”: s = stride of layer; p = padding of layer; f = filter size of layer first_valid_offset = ceil((first_valid_offset + p) / s) else if layer type is “trans_conv”: s = stride of layer; p = padding of layer; f = filter size of layer / / The offset of the last invalid pixel in the input operation. Can be -1 or greater. last_invalid_offset = first_valid_offset + p - 1 / / Offset of the last invalid pixel in the output operation last_invalid_offset = last_invalid_offset * s + f - 1 first_valid_offset = last_invalid_offset + 1 return first_valid_offset b=FirstValidPixelOffset(layers) Note that in general, the criteria for the first valid pixel calculated from the left and right of the fixed-size output are not perfectly symmetrical; some pixels may remain on the right side of the fixed-size input where no filter may be applied, resulting in one more valid pixel on the right side of the fixed-size output than on the left. The output of the FirstValidPixelOffset() function (e.g., the first valid offset) is calculated from the left, and this value should also be correct for the right. Similarly, the above analysis should also apply for calculations from the top or bottom of the fixed-size output.

[0174] Again, referring to the AlignOutputTile() function in the second algorithm in conjunction with the ProjectBackwards() function, the system 100 can obtain a respective coordinate shift for each of the respective fixed-size outputs and generate the final output by combining the respective fixed-size outputs based on the respective coordinate shifts.

[0175] The system 100 can implement the AlignOutputTile() function using different methods. To name a few, the system 100 can perform a local search for each coordinate shift or obtain an analytical expression for each coordinate shift. AlignOutputTile() is written as follows: function AlignOutputTile((ht, wt, hb, wb), layers): if approach == “local search”: for (hs, ws) = try all values ​​in some pattern from 0 to max_shift: try: return ProjectBackwards((ht - hs, wt-ws, hb-hs, wb-ws), layers) except: / / Projection failed, keep trying other shift values THROW EXCEPTION / / Failed to find a valid alignment for the tile else if approach == “analytical”: hts = int (ht / alignment) * alignment wts = int (wt / alignment) * alignment hbs = hb - (ht - hts) wbs = wb - (wt - wts) return (hts, wts, hbs, wbs) where: alignment = CalculateAnalyticalAlignment(layers) When using a local search method, the system 100 can provide multiple trial shifts in each dimension. The trial shift values ​​can range from zero pixels to a predetermined maximum value for the coordinate shift (e.g., the size of the final output). The system 100 needs to determine the relationship between the coordinates of the "unaligned" fixed-size output and the coordinates of the associated fixed-size input. As an example, the system 100 can provide the coordinates of the "unaligned" fixed-size output 133 and the trial shift values ​​to the ProjectBackwards() function to search for the verified fixed-size input (i.e., the coordinates representing the fixed-size input should be in integer pixels). If the system 100 successfully finds the verified fixed-size input, the system 100 can return a shifted fixed-size output based on the particular trial shift value.

[0176] When using analytical methods, the system 100 can determine a constant alignment value by analyzing the properties of the deployed FCN model. One exemplary algorithm for analytical expression is called "CalculateAnalyticalAlignment()" and is written as follows: function CalculateAnalyticalAlignment(layers): / / Algorithm for finding minimum correct alignment: presence of conv layer before trans_conv / / The layer facilitates the alignment required by the trans_conv layer conv_stride_product = 1 / / product of strides of back-to-back conv layers trans_conv_stride_product = 1 / / product of strides of back-to-back trans conv layers alignment = 1 / / Desired tile alignment in the FCN output layer for layer = output to input layers: if layer type is “conv”: s = stride of layer conv_stride_product *= s else if layer_type is “trans_conv”: s = stride of layer trans_conv_stride_product * = s prev_layer = previous layer / / prev_layer yields the input for the layer / / prev_layer == null if the layer is the input layer for the entire FCN if prev_layer == null OR prev_layer type ! = “trans_conv”: / / Alignment requirements imposed by a stack of trans_conv layers by using the greatest common denominator / / Use subsequent stacks of conv layers to make it easier gcd = GCD (conv_stride_product, trans_conv_stride_product) alignment_for_stack = trans_conv_stride_product / gcd alignment * = alignment_for_stack / / Reset the stack conv_stride_product, trans_conv_stride_product = 1, 1 return alignment The system 100 determines a constant alignment value based on the characteristics of each layer of the FCN model. For example, the characteristics can be the layer type (e.g., convolution, transposed convolution, or other layer such as a pooling layer), or the size for the padding, filter, and stride for the layer. As mentioned above, other types of layers in the FCN model, such as pooling layers, are treated as convolution layers throughout this specification.

[0177] Figure 3B shows an example of a tiling and stitching process 399 performed by example reasoning system 100 of Figure 1. System 100 may be configured to perform tiling and stitching process 355 using a first algorithm.

[0178] System 100 may generate multiple fixed-size inputs 350a, 350b, 350c, and 350d, each having a respective size. For example, fixed-size inputs 350a-d may each have a different size. As another example, fixed-size inputs 350a-d may have the same size, as shown in FIG. 3B. For ease of illustration, fixed-size inputs 350a-d are represented by squares with solid lines.

[0179] As shown in FIG. 3B , each fixed-size input 350a-d can have a respective neighboring pixel region 360a, 360b, 360c, or 360d. The size or width of the neighboring pixel region can be 1 pixel, 3 pixels, and 5 pixels. For ease of illustration, the neighboring pixel regions are represented by dashed squares. Note that the region to the left of zero pixel value region 360a does not contain any zero-value neighboring pixels. Because the left edge of fixed-size input 350a is also part of the left edge of full input data 150, computations on pixels in the region to the left of fixed-size input 350a do not introduce inaccuracies into the corresponding fixed-size output.

[0180] In some implementations, fixed-size inputs 350a-d and their associated adjacent pixel regions 360a-d may be evenly spaced relative to the complete input data 150 and may overlap each other uniformly. As shown in FIG. 3B , fixed-size inputs 350a and 350b overlap each other in overlap region 353a, fixed-size inputs 350b and 350c overlap each other in region 353b, and fixed-size inputs 350c and 350d overlap each other in region 353c. While overlap regions 353a and 353b have the same size, overlap region 353c can be larger than overlap regions 353a and 353b. This is due to the characteristics of the first algorithm. As shown in the first algorithm, the fixed-size inputs at the right and bottom boundaries cannot exceed the boundaries of the input data. For example, assuming fixed-size input 350d is arranged in the same format as other fixed-size inputs 350a-c, fixed-size input 350d may be at the right boundary and have a portion that extends beyond the right boundary of full input data 150. System 100 may use a first algorithm to "shift" (i.e., re-tile) fixed-size input 350d to the left by a number of pixels, so that the pixels of fixed-size input 350d are fully located within full input data 150. However, because fixed-size input 350d's placement is no longer the same as the other fixed-size inputs, overlap area 353c between fixed-size inputs 350d and 350c may be larger than overlap areas 353a and 353b. If the fixed-size inputs at the right and bottom boundaries do not extend beyond the corresponding boundaries of full input data 150, the fixed-size inputs may be arranged to have the same overlap area.

[0181] After tiling the complete input data 150 into multiple fixed-size inputs based on fixed sizes calculated online or offline, the system 100 can process the random-size inputs based on at least a first algorithm and a stitching algorithm described in more detail below to generate fixed-size outputs having respective valid areas that do not overlap each other and are adjacent to each other at edge pixels.

[0182] System 100 can generate fixed-size outputs with valid areas that generally do not overlap in complete output data 170. However, in some circumstances, one or more fixed-size outputs may overlap one another. As shown in FIG. 3B , valid areas 370a, 370b, and 370c do not overlap one another. However, valid area 370d overlaps valid area 370c in overlap area 373. This is because the first algorithm “shifts” right-bound fixed-size input 350a to the left by several pixels, causing fixed-size output 370d to overlap adjacent fixed-size output 370c. Dummy areas 375a, 375b, 375c, and 375d associated with corresponding valid areas may overlap. For ease of illustration, the valid areas of the fixed-size outputs are represented by solid-line squares, and the dummy areas of the fixed-size outputs are represented by dashed-line squares.

[0183] The area to the left of dummy area 375a does not contain any invalid values ​​because the left edge of fixed size output 375a is also part of the left edge of full output data 170. Similarly, the right edge of dummy area 375d does not contain any invalid values.

[0184] During the stitching process, the system 100 can discard the per-pixel values ​​in the dummy regions and connect the per-pixel values ​​in the valid regions to generate the complete output data 170. Each pixel value in the complete output data (or final output) is provided at least once from the per-pixel values ​​in the valid regions.

[0185] Figure 3C shows another example of a tiling and stitching process 355 performed by example reasoning system 100 of Figure 1. System 100 may be configured to perform tiling and stitching process 399 using a second algorithm.

[0186] As described above, compared to the first algorithm, the system 100 performs some additional steps using the second algorithm, such as determining alignment information for the FCN model and determining the valid region by calculating coordinate shifts for the fixed-size output based on the alignment information. This is because, when the FCN model includes a certain layer (e.g., a transposed convolutional layer), the system needs to verify the mapping (e.g., integer coordinates) from pixels in the fixed-size output to corresponding pixels in the fixed-size input.

[0187] Additionally, the second algorithm differs from the first algorithm by not needing to perform fixed-size output "movements" at the right and bottom boundaries of the complete input data 150.

[0188] As shown in FIG. 3C , the system 100 can generate multiple fixed-size inputs (e.g., fixed-size inputs 380a-d) from the complete input data 150 based on a tiling pattern. The fixed-size inputs 380a-d can overlap each other at their respective sizes or the same size. For example, the fixed-size input 380a and the fixed-size input 380b can overlap each other at an overlap region 385a, the second fixed-size input 380b and the third fixed-size input 380c can overlap each other at an overlap region 385b, and the third fixed-size input 380c and the fourth fixed-size input 380d can overlap each other at an overlap region 385c. The sizes of the overlap regions 385a-c are substantially the same, as shown in FIG. 3C . In some implementations, the overlap regions 385a-c can be slightly larger than the overlap regions generated using the first algorithm. This is because the system using the second algorithm needs to tile a fixed size input based on alignment information.

[0189] System 100 can also determine and locate zero-value neighboring pixel regions 390a-d similar to those described above. As shown in Figure 3C, for ease of illustration, fixed size inputs 380a, 380b, 380c, and 380d are represented by solid squares, and neighboring pixel regions 390a, 390b, 390c, and 390d are represented by dashed squares.

[0190] System 100 can use the second algorithm to determine the region outside of full input data 150, and may not need to "move" fixed-size input 380d. As shown in FIG. 3C, fixed-size input 380d has region 381 outside of full input data 150. Because fixed-size input 380d is not "moved," overlap regions 385a-c can remain the same. The second algorithm is more robust than the first algorithm when processing certain inputs where "moving" operations are not allowed.

[0191] After processing all fixed-size inputs through the compiled FCN model, system 100 can determine valid regions 395a, 395b, 395c, and 395d and corresponding dummy regions 397a, 397b, and 397c of all fixed-size outputs, calculate coordinate shifts for pixels in the valid regions according to a second algorithm, discard pixels in the dummy regions, and combine pixels in the valid regions to generate complete output data 170. The valid regions 395a-d can also overlap each other in respective overlap regions 393a-c. The respective overlap regions 393a-c can be substantially the same if the overlap regions 385a-c between the fixed-size inputs are substantially the same.

[0192] Similarly, for ease of illustration, the valid areas of fixed size outputs 395a-d are represented by solid squares, and the dummy areas of fixed size outputs 397a-d are represented by dashed squares.

[0193] 3B and 3C, it should be appreciated that system 100 can generate more than four fixed-size inputs, e.g., 5, 10, 20, 50, and more, for tiling complete input data 150. The system can also generate more than five fixed-size outputs, e.g., 5, 10, 20, 50, and more, that contain valid per-pixel values ​​for each pixel in complete output data 170. Each per-pixel value for a pixel associated with complete output data 170 is represented in at least the fixed-size output generated from the corresponding fixed-size input. In the case of per-pixel values ​​for a pixel in an overlap region across two or more fixed-size outputs, system 100 can select the corresponding pixel value from any one of the overlapping fixed-size outputs as the per-pixel value for the pixel.

[0194] After computing all fixed-size outputs through the FCN model, the system can apply the O(i,j) and V(i,j) mappings to construct the complete output as if the input had been fully processed by the FCN model using the StitchOutputImage() function as follows: function StitchOutputImage(): for tile indices (i,j)in a top-to-bottom, left-to-right scan of the tiles: (ht_O, wt_O, hb_O, wb_O) = O(i,j) (ht_V, wt_V, hb_V, wb_V) = V(i,j) Output(ht_V:hb_V, wt_V:wb_V) = OutputTile( (ht_V-ht_O):(hb_V-ht_O), (wt_V-wt_O):(wb_V-wt_O) ) OutputTile(i,j) is a fixed-size input of size T O represents a fixed-size output of, for example, a fixed-size input at the ith column and jth row of a tiling grid.

[0195] 4 illustrates an exemplary process 400 for performing inference computations of a fully convolutional network for inputs having different sizes. For convenience, process 400 is described as being performed by one or more computer systems located at one or more locations. For example, a suitably programmed neural inference system, such as system 100 of FIG. 1, can perform process 400.

[0196] The system receives a new input to be processed by a fully convolutional neural network deployed on a hardware accelerator (410). The new input can have a first size that is different from the fixed size that the fully convolutional neural network is configured to process when deployed on the hardware accelerator. As described above, the new input can have a size that is larger than the fixed size or smaller than the fixed size.

[0197] The system determines one or more fixed-size inputs from the new input (420). Each fixed-size input of the one or more fixed-size inputs has a fixed size. More specifically, the system can determine a tiling pattern for tiling the new input based on at least characteristics of the deployed FCN model, such as alignment information, padding size, stride size, filter size, and scale factor.

[0198] The system provides each of the one or more fixed-size inputs to a hardware accelerator to perform inference computations using a fully convolutional neural network (430).

[0199] The system obtains from the hardware accelerator a respective fixed-size output generated by the fully convolutional neural network for each of one or more fixed-size inputs (440). Each fixed-size output may include one or more inaccurate pixel-by-pixel results. As described above, the system may include a host for providing the fixed-size input for the FCN deployed on the hardware accelerator and receiving the fixed-size output from the hardware accelerator. When processing the fixed-size input, the system may use neighboring pixels surrounding the fixed-size input to determine valid and dummy regions for each fixed-size output.

[0200] From each fixed-size output, the system generates a final output (450) that is equivalent to the output generated by processing a new input with a fully convolutional neural network.

[0201] As described above, the system can combine fixed-size outputs using different algorithms based on the characteristics of the deployed FCN. If the FCN model does not include any transposed convolutional layers, the system can combine the valid regions of each fixed-size output using a first algorithm. If the FCN model includes one or more transposed convolutional layers, the system can combine the fixed-size outputs by obtaining a coordinate shift for each fixed-size output and shifting the coordinates of each fixed-size output based on the coordinate shift.

[0202] The system can determine the coordinate shift using different methods. For example, the system can determine the coordinate shift using a local search. The system can generate a coordinate shift for a fixed-size output by testing multiple trial shift values ​​using the ProjectBackwards() function. Alternatively, the system can generate a coordinate shift based on analyzing the characteristics of the deployed FCN and obtain a constant value for the coordinate shift through an analytical expression using the CalculateAnalyticalAlignment() function.

[0203] Implementations of the subject matter and the acts and operations described herein may be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or a combination of one or more of them. Implementations of the subject matter described herein may be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a computer program carrier for execution by or to control the operation of a data processing apparatus. The carrier may be a tangible, non-transitory computer storage medium. Alternatively, or in addition, the carrier may be an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus. The computer storage medium may be, or be a portion of, a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not a propagated signal.

[0204] The term "data processing apparatus" includes all kinds of apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. A data processing apparatus can include special-purpose logic circuitry, such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), or a GPU (graphics processing unit). In addition to hardware, an apparatus can also include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0205] A computer program, which may be called or written as a program, software, software application, app, module, software module, engine, script, or code, may be written in any type of programming language, including compiled or interpreted languages, or declarative or procedural languages. A computer program may be deployed in any form, including as a stand-alone program or as a module, component, engine, subroutine, or other unit suitable for execution in a computing environment. The environment may include one or more computers interconnected by a data communications network in one or more locations.

[0206] A computer program may, but need not, correspond to a file in a file system. A computer program can be stored in part of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, subprograms, or portions of code.

[0207] The processes and logic flows described herein may be performed by one or more computers executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry, such as an FPGA, ASIC, or GPU, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0208] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by, or incorporated in, special-purpose logic circuitry.

[0209] Generally, a computer also includes one or more mass storage devices, or is operatively coupled to receive data from or transfer data to one or more mass storage devices. The mass storage devices can be, for example, magnetic, magneto-optical, or optical disks, or solid-state drives. However, a computer need not have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0210] To provide for user interaction, implementations of the subject matter described herein may be implemented on or configured to communicate with a computer having a display device, e.g., an LCD (liquid crystal display) monitor, for displaying information to a user, and an input device, e.g., a keyboard and pointing device, e.g., a mouse, trackball, or touchpad, by which the user can provide input to the computer. Other types of devices may also be used to provide for user interaction. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback. Input from the user may be received in any form, including acoustic input, voice input, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's device in response to a request received from the web browser, or by interacting with an app running on the user device, e.g., a smartphone or electronic tablet. The computer can also interact with the user by sending text messages or other types of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages back from the user.

[0211] This specification uses the term "configured to" in connection with systems, devices, and computer program components. To say that one or more computer systems are configured to perform a particular operation or action means that the system has software, firmware, hardware, or a combination thereof installed on the system that, when run, causes the system to perform the operation or action. To say that one or more computer programs are configured to perform a particular operation or action means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action. To say that special purpose logic circuitry is configured to perform a particular operation or action means that the circuitry has electronic logic that performs the operation or action.

[0212] Implementations of the subject matter described herein may be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0213] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server sends data, e.g., HTML pages, to a user device, e.g., to display data to and receive user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of user interaction, may be received at the server from the device.

[0214] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what is or may be claimed, but rather as descriptions of features that may be particular to particular implementations of a particular invention. Certain features described herein in the context of separate implementations may also be implemented in combination in one implementation. Conversely, various features described in the context of one implementation may also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, while features may be described above as operative in a combination and even initially claimed as such, one or more features from a claimed combination may in some cases be deleted from that combination, and the claims may be directed to subcombinations or variations of the subcombinations.

[0215] Similarly, although operations are shown in the figures and recited in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence shown, or that all of the shown operations be performed, to achieve desired results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated into one software product or packaged in multiple software products.

[0216] Specific implementations of the subject matter have been described. Other implementations are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequence, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method implemented by one or more computers, comprising: receiving a new input to be processed by a fully convolutional neural network deployed on a hardware accelerator, the new input having a first size different from a fixed size that the fully convolutional neural network is configured to process when deployed on the hardware accelerator; The method comprises: determining one or more fixed-size inputs from the new input, each fixed-size input having a fixed size; The method comprises: providing each of the one or more fixed-size inputs to the hardware accelerator for performing an inference calculation using the fully convolutional neural network; obtaining from the hardware accelerator a respective fixed-size output generated by the fully convolutional neural network for each of the one or more fixed-size inputs, each of the respective fixed-size outputs including a central valid region and a peripheral dummy region a first number of pixels wide, the central valid region including at least a portion of a final output, and the peripheral dummy region including one or more incorrect pixel-by-pixel results; The method comprises: generating, from each of the fixed-size outputs, a final output equivalent to an output generated by processing the new input with the fully convolutional neural network, wherein generating the final output from each of the fixed-size outputs comprises: determining data representing a respective coordinate shift for each respective said fixed size output; combining the central effective areas of each of the fixed-size outputs based on the determined data and a relationship between coordinates of each of the fixed-size outputs and coordinates of each corresponding fixed-size input used to generate each of the fixed-size outputs.

2. The fully convolutional neural network includes one or more transposed convolutional layers; The method of claim 1 , wherein determining data representing a respective coordinate shift for each respective said fixed size output comprises determining alignment information.

3. 3. The method of claim 1, further comprising determining the fixed size based on at least a characteristic of the fully convolutional neural network before deploying the fully convolutional neural network on the hardware accelerator.

4. The method of claim 3 , wherein determining the fixed size further comprises providing a user with a plurality of candidate sizes to select one of the candidate sizes as the fixed size.

5. generating a plurality of candidate sizes for the fully convolutional neural network based on characteristics of the fully convolutional neural network; For each of the candidate sizes: deploying a copy of the fully convolutional neural network on each hardware accelerator to process inputs of the candidate size; measuring the total execution time of performing inference computations for the unrolled copies of the fully convolutional neural network on the respective hardware accelerators; selecting a candidate size from the plurality of candidate sizes as the fixed size based at least on the total execution time measured for the candidate size; The method of claim 1 further comprising:

6. Determining the fixed size includes: determining that the first size of the new input is smaller than the fixed size; generating a fixed size input by padding zeros around the new input up to the fixed size; The method of any one of claims 1 to 3, further comprising:

7. The method of any one of claims 1 to 6, wherein the first number of pixels is determined based on characteristics of the fully convolutional neural network.

8. Determining data representative of each of said coordinate shifts comprises:

8. The method of claim 1, further comprising determining each of the coordinate shifts using a local search, the local search comprising determining a relationship between coordinates of fixed-size outputs and coordinates of corresponding fixed-size inputs used to generate the fixed-size outputs.

9. Determining data representative of each of said coordinate shifts comprises: determining global alignment information based on characteristics of the fully convolutional neural network; and determining a respective coordinate shift for each of the respective fixed size outputs based on the determined global alignment information; The method according to any one of claims 1 to 7, comprising:

10. 10. The method of claim 3, wherein the characteristics of the fully convolutional neural network include a respective filter size, a zero-padding size, a stride size, and a scale factor for each network layer of the fully convolutional neural network.

11. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 10.

12. A program which, when executed by one or more computers, causes said one or more computers to perform the operations of the method of any one of claims 1 to 10.

Images