Convolutional neural network with soft kernel selection
By introducing conditional convolutional layers into convolutional neural networks and utilizing soft kernel selection techniques to generate input-related weights, the computational resource and performance bottlenecks of convolutional neural networks are solved, resulting in significant performance improvements and a simplified training process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GOOGLE LLC
- Filing Date
- 2020-01-23
- Publication Date
- 2026-04-14
AI Technical Summary
Existing convolutional neural networks have bottlenecks in terms of computational resources and performance, making it difficult to achieve significant performance improvements across various tasks, while the training process is also complex.
Conditional convolutional layers are used to enhance the computational capacity of neural networks by generating weights that are relevant to the input, while minimizing the increase in computational overhead. Soft kernel selection techniques are also used to improve performance.
While minimizing computational resource consumption, it significantly improves the performance of convolutional neural networks, especially in tasks such as image classification and object detection, and the training process is also simpler.
Smart Images
Figure CN113348472B_ABST
Abstract
Description
[0001] Cross-reference to related applications
[0002] This application claims priority to Provisional U.S. Patent Application No. 62 / 796,045, filed January 23, 2019, the entire contents of which are incorporated herein by reference. Technical Field
[0003] This manual relates to the use of convolutional neural networks to process input. Background Technology
[0004] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output in response to a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer serves as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates its output from the received input based on the current values of its corresponding set of parameters.
[0005] Convolutional neural networks have achieved state-of-the-art performance on many tasks in computer vision. Summary of the Invention
[0006] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which uses a convolutional neural network to process network inputs to generate network outputs for the network inputs. Specifically, the convolutional neural network includes one or more conditional convolutional layers, each of which has increased computational capacity relative to a regular convolutional layer, while the increase in computational overhead is minimal, i.e., the increase in processing power for performing neural network inference is minimal compared to a neural network with only regular convolutional neural network layers.
[0007] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.
[0008] The described technique allows for an increased size of neural networks with only a slight increase in inference cost. Specifically, by augmenting the neural network using the described input-dependent conditional computation (i.e., using one or more conditional convolutional layers), the performance of the neural network on various tasks (e.g., image classification or object detection) can be improved relative to the network's performance if regular convolutional layers are replaced. More specifically, these additional performance gains are achieved at least in part because the weights for the various kernels used in the conditional convolutional layers are generated in an input-dependent manner. Furthermore, this improvement can be achieved with minimal computational overhead, i.e., minimal increase in computational complexity and resource consumption (e.g., processing power and memory consumption). In particular, the conditional convolutional layer still performs only a single convolution and therefore incurs only the additional computational cost of generating weights and determining the weighted sum of the kernels, which is minimal relative to the cost of performing the convolution. Thus, significant performance improvements can be achieved without a corresponding significant increase in resource usage.
[0009] Some conventional methods for increasing model capacity utilize conditional computation. In conditional computation models, this increase is achieved by starting only a portion of the entire network for each example. However, conditional computation models are often difficult to train because they need to learn discrete routing decisions from individual examples to different parts of the network. Unlike these methods, the technique described does not require discrete routes to examples and can therefore be easily optimized using gradient descent.
[0010] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the following description. Further features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description
[0011] Figure 1 An example neural network system is shown.
[0012] Figure 2 The diagram illustrates the operation of an example conditional convolutional layer.
[0013] Figure 3 This is a flowchart of an example process for processing network input to generate image processing output.
[0014] Figure 4 This is a flowchart of an example process for generating the output of a conditional convolutional layer during the training of a neural network.
[0015] Figure 5 This is a table showing the performance of a neural network with conditional convolutional layers relative to a baseline neural network.
[0016] The same reference numerals and names in various figures indicate the same elements.
[0017] Specific implementation method
[0018] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which uses a convolutional neural network to process network inputs to generate network outputs for the network inputs, i.e., to perform a machine learning task on the received network inputs. Specifically, the convolutional neural network includes one or more conditional convolutional layers, each of which has increased computational capacity relative to a regular convolutional layer, while the increase in computational overhead is minimal, i.e., the increase in processing power for performing neural network inference is minimal compared to a neural network with only regular convolutional neural network layers.
[0019] In some cases, the network input includes visual data, and the task is a computer vision task.
[0020] In some cases, the network input includes one or more images, and the task is an image processing task.
[0021] For example, an image processing task could be image classification, where the image processing output is a set of scores, each score corresponding to a different object category, and representing the probability that one or more images depict objects belonging to that object category.
[0022] As another example, an image processing task could be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the likelihood that the region depicts an object of interest.
[0023] As another example, an image processing task could be image segmentation, where the image processing output defines a corresponding probability for each pixel in one or more images for each category in a predetermined set of categories. For example, the set of categories could be foreground and background. As yet another example, the set of categories could be object categories.
[0024] As another example, an image processing task could be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images.
[0025] As another example, an image processing task could be motion estimation, where the network input comprises multiple images, and the image processing output, for each pixel of one of the input images, defines the motion of the scene depicted at pixels between the images in the network input.
[0026] As another example, the task could include real-time server-side video processing for autonomous vehicles, or real-time onboard perception.
[0027] In some other cases, the network input can be a sequence, while the output can be a sequence of different types. For example, the network input can be a sequence of source language text, while the output can be a text sequence of target language text, which is a translation of the input sequence into the target language. As another example, the network input can be an audio sequence representing utterance, while the output sequence can be a text sequence as a transcription of utterance. As yet another example, the input sequence can be a text sequence or a sequence of linguistic features of text, while the output sequence can be an audio sequence, such as a sequence of audio samples, or amplitude values of lexicalized input text.
[0028] Figure 1 An example neural network system 100 is shown. The neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the following systems, components and techniques are implemented.
[0029] System 100 receives network input 102 and processes network input 102 to generate network output 122 for network input 102.
[0030] Specifically, system 100 generates network output 122 by processing network input 102 through convolutional neural network 110.
[0031] The convolutional neural network 110 includes one or more conditional convolutional layers 120 (hereinafter referred to as "first convolutional layers"). Like regular convolutional layers, the conditional convolutional layer 120 receives an input tensor and generates an output tensor from the input tensor. However, unlike regular layers, the conditional convolutional layer 120 implements "soft kernel selection," a type of input-conditional computation that increases the computational capacity of the layer with minimal increase in computational overhead.
[0032] Although Figure 1 The diagram shows a single conditional convolutional layer 120, but in reality, the convolutional neural network 110 can have any number of conditional convolutional layers 120. For example, some or all of the regular convolutional layers in an existing neural network architecture can be replaced with conditional convolutional layers 120 to generate an architecture for the convolutional neural network 110.
[0033] Therefore, the convolutional neural network 110 includes one or more conditional convolutional layers 120, and in some cases, includes one or more other types of neural network layers, such as regular convolutional layers, fully connected layers, pooling layers, softmax layers, etc. Generally, the convolutional neural network 110 can have any architecture, which includes at least one conditional convolutional layer 120 and is adapted to receive the input type required for the machine learning task and generate the output type.
[0034] In particular, each conditional convolutional layer 120 has a set of multiple kernels.
[0035] When a given input tensor is received from convolutional layer 120, system 100 generates corresponding input-related weights for each of the multiple kernels from the input tensor. These input-related weights are called "input-related" because system 100 generates weights based on the input tensor to convolutional layer 120, where different input tensors result in different weights for the various kernels of conditional convolutional layer 120. (See below for reference.) Figure 3 A more detailed description of the generation of input-related weights.
[0036] Then, system 100 generates input-related kernels by calculating a weighted sum of the kernel set based on the corresponding input-related weights (i.e., a sum of kernels with each assigned weight, which is used to generate the kernels). This can be viewed as performing soft kernel selection among multiple kernels in the conditional convolutional layer 120.
[0037] Then, system 100 generates an output tensor for conditional convolutional layer 120 from the input tensor of the conditional convolutional layer. As part of generating the output tensor, the system performs convolution between the input tensor and the input kernel. Because a weighted sum of kernels is performed before applying a single input-dependent kernel, the size and computational complexity of the convolution are the same as those applied to a regular convolutional layer with a single kernel.
[0038] System 100 can also perform other operations on the output of the convolution, such as applying batch normalization, activation functions, or both to the convolution output to generate the output tensor of the conditional convolutional layer 120.
[0039] Because the processing required to generate weights and compute weighted sums is much cheaper than performing convolutions, and because the size of a convolution is the same as that of a layer with only a single kernel, the described technique increases the size and performance of convolutional neural networks with minimal increase in computational overhead.
[0040] Figure 2 The operation of an example conditional convolutional layer is shown. Figure 2 In the example, the conditional convolutional layer has three kernels: W1, W2, and W3.
[0041] like Figure 2 As shown, the conditional convolutional layer receives an input tensor (“PREV LAYER OUTPUT”) generated by the previous layer in the convolutional neural network. Typically, the input tensor is a multidimensional array. For example, the input tensor could be an X×Y×Z feature map generated from an input image. As another example, the input tensor could be an X×D matrix of feature vectors generated from an input sequence.
[0042] The system uses a routing function (“ROUTE FN”) to generate corresponding input-related weights for each of the three kernels W1, W2 and W3, and then combines the three kernels (“COMBINE”) according to the corresponding input-related weights to generate a combined kernel.
[0043] The system then computes the convolution (CONV) between the input tensor (“PREV LAYER OUTPUT”) and the combined kernel to generate the convolutional output.
[0044] For a given input tensor x, these operations can be represented as:
[0045] Output(x)=(α1·W1+α2·W2+α3·W3)*x,
[0046] Where α1, α2, and α3 are the weights related to the input, and * denotes convolution.
[0047] The type of convolution applied by a conditional convolutional layer depends on the architecture of the convolutional neural network. For example, the convolution can be a standard convolution, a pointwise convolution, or a depthwise convolution, depending on the architecture of the convolutional neural network and the placement of the conditional convolutional layer within that architecture—different conditional convolutional layers within the same neural network can perform different types of convolutions.
[0048] Advantageously, the kernels are combined before performing a single convolution, i.e., instead of computing three separate convolutions with three separate kernels, and therefore only a single convolution needs to be performed to generate the convolution output.
[0049] exist Figure 2 In the example, the conditional convolutional layer then applies batch normalization (“BN”) to the convolutional output to generate a batch-normalized output, and then applies the rectified linear unit activation function (ReLU) to generate the output tensor for the conditional convolutional layer. However, in general, any given conditional convolutional layer can have regular neural network operations applied to the convolutional output to generate the output tensor for that layer. In some cases, the system directly uses the convolutional output as the output of the conditional convolutional layer.
[0050] The system can then provide the output tensor as input to the next layer in the convolutional neural network.
[0051] Figure 3 This is a flowchart of an example process 300 for processing network input to generate network output. For convenience, process 300 will be described as being executed by a system of one or more computers located at one or more locations. For example, a properly programmed neural network system (e.g., Figure 1The neural network system 100 can execute process 300.
[0052] The system receives the input tensor of the first convolutional layer in a multi-layered image processing neural network (step 302). Typically, the input tensor is derived from the network input, i.e., it has been processed by one or more other layers in the convolutional neural network to generate the network input. The first convolutional layer is a conditional convolutional layer, which, unlike a regular convolutional layer, has multiple kernels.
[0053] The system determines the corresponding input-related weights for each of the multiple kernels from the input tensor (step 304).
[0054] Specifically, the system processes the input tensor through a non-linear routing function with multiple parameters to generate corresponding input-related weights for multiple kernels.
[0055] Typically, a routing function comprises one or more neural network layers that work together to transform the input tensor into a vector with corresponding values for each kernel in the kernel.
[0056] In some implementations, the neural network layer includes a global average pooling layer that performs global average pooling across the spatial dimensions of the input tensor. That is, the non-linear routing function first applies global average pooling to the input tensor to generate a pooled tensor, and then applies additional parameter-dependent operations to the pooled tensor to generate input-dependent weights. Convolutional operations operate only above the local receptive field; therefore, by employing global average pooling, the routing function allows these local operations to be adapted using the global context throughout the entire input tensor.
[0057] As an example, additional, parameter-related operations could include processing the average pooling tensor through one or more fully connected layers, and then processing the output of the fully connected layers with a non-linearity (e.g., a sigmoid activation function) to generate weights.
[0058] In some implementations, when a neural network includes multiple conditional convolutional layers, the system can independently perform routing for each conditional convolutional layer, i.e., generate routing weights.
[0059] In some other implementations, when the neural network includes multiple conditional convolutional layers, the system can share routing weights among some of the conditional convolutional layers to reduce computational overhead. For example, the system can share routing weights among each conditional convolutional layer in a given block of multiple convolutional layers; that is, it can compute new weights only for the first layer in each block and then maintain the same weights for the remaining layers in the block. As another example, the system can share routing weights between every other conditional convolutional layer or every other three conditional convolutional layers in the neural network. Nevertheless, in each of these examples, the routing weights of any given conditional convolutional layer still depend on the current network input and will typically differ for different network inputs (and therefore for different inputs to the conditional convolutional layer).
[0060] The system generates an input-related kernel by calculating a weighted sum of multiple kernels based on the corresponding input-related weights (step 306).
[0061] The system generates the output tensor of the first convolutional layer from the input tensor of the first convolutional layer (step 308). As part of generating the output tensor, the system performs convolution between the input-associated kernel and the input tensor. Therefore, even though the first convolutional layer has multiple kernels, the system uses only a single combined kernel to perform a single convolution when generating the output of the first convolutional layer.
[0062] After the neural network has been trained as part of generating new network outputs for new network inputs, the system can execute process 300. The system can also execute process 300 during training of the neural network on a set of training inputs to determine trained values for the parameters of the convolutional neural network, such as the kernels of conditional convolutional layers and the parameters of the routing functions of conditional convolutional layers. When process 300 is executed during training, the system can compute gradients of the network parameters with respect to a target function that measures the quality of the network output relative to the network inputs, and use these gradients to update the parameters using conventional machine learning techniques such as Adam, rmsProp, or stochastic gradient descent.
[0063] However, in some other cases, the system computes the output tensor of the conditional convolutional layer in a different manner during the training of the neural network. That is, after training, the system can use procedure 300 to compute the output, while during training, the system can use the following reference... Figure 4 The different processes described are used to calculate the output.
[0064] Figure 4This is a flowchart of an example process 400 for generating output tensors for conditional convolutional layers during training. For convenience, process 400 will be described as being performed by a system of one or more computers located at one or more locations. For example, a properly programmed neural network system (e.g., Figure 1 The neural network system 100 can execute process 400.
[0065] The system receives the layer input of the first convolutional layer (step 402).
[0066] Based on the current value of the kernel, the system performs a corresponding convolution between the layer input and each of the multiple kernels to generate a corresponding convolution output for each kernel (step 404). That is, unlike process 300, the system performs a different convolution for each of the multiple kernels in the first convolutional layer.
[0067] The system processes the input tensor through the routing function based on the current value of the parameters of the nonlinear routing function to generate corresponding input-related weights for each of the multiple kernels (step 406).
[0068] The system generates the overall convolutional output for the first convolutional layer by calculating a weighted sum of the convolutional outputs of the kernel, based on the weights associated with the input (step 408). Therefore, unlike in process 300, the system combines the convolutional outputs to generate the overall convolutional output after individual convolutions have been performed.
[0069] Once the network output has been generated for the training input, the system can determine the gradient of the network parameters with respect to the objective function, which includes the parameters of multiple kernels and non-linear routing functions, and use the gradient to update the current values of the parameters and the kernels.
[0070] In some implementations, the system determines whether to use process 300 or process 400 during training for a given conditional convolutional layer based on the number of kernels the conditional convolutional layer has. In particular, many hardware accelerators used for neural network training are optimized for training large batches of convolutions and struggle to fully utilize them at small batch sizes. Therefore, when the number of kernels is small (e.g., less than or equal to 4), it may be more efficient to train the conditional convolutional layer using process 400 and large batches of convolutions, and then use process 300 for inference after training. When the number of kernels is large (e.g., greater than 4), it may be more efficient to directly use process 300 to train the conditional convolutional layer.
[0071] Figure 5 Table 500 shows the performance of a neural network with conditional convolutional layers relative to a baseline neural network.
[0072] Specifically, for each of the five baseline neural networks (MobileNetV1 (1.0x), MobileNetV2 (1.0x), MnasNet-A1, ResNet-50, and EfficientNet-B0), Table 500 shows the computational cost of the number of multiply-accumulate (MADD) operations required to compute the output of the baseline network, as well as the top-1 percentage of accuracy achieved by the baseline neural network on the ImageNet 2012 dataset image classification task.
[0073] For each of the five baseline neural networks, Table 500 also shows the performance of a modified version of the neural network (“CondConv”), which was generated by replacing the convolutional layers in the baseline neural networks with conditional convolutional layers having eight kernels. As can be seen from Table 500, CondConv improves the accuracy of all baseline neural networks while the relative increase in inference cost is small (<10%).
[0074] This specification uses the term "configured" in relation to system and computer program components. Configuring one or more computer systems to perform a specific operation or action means that software, firmware, hardware, or a combination thereof are installed on the system that, in operation, causes the system to perform said operation or action. Configuring one or more computer programs to perform a specific operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.
[0075] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, or in computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by or control of the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing device.
[0076] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an implementation environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.
[0077] A computer program, also referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, such as in a markup language document, a single file dedicated to the program in question, or one or more scripts in multiple coordination files, such as files that store one or more modules, subroutines, or portions of code. A computer program can be deployed to run on one computer or multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.
[0078] In this specification, the term "database" is used broadly to refer to any data collection: data that does not need to be structured in any particular way, or does not need to be structured at all, and data can be stored in one or more locations on a storage device. Therefore, for example, an index database may include multiple data collections, each of which can be organized and accessed in different ways.
[0079] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components, installed at one or more locations on one or more computers. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.
[0080] The processes and logic flows described in this specification can be executed by one or more programmable computers that implement one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC) or by a combination of special-purpose logic circuitry and one or more programmable computers.
[0081] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The components of a computer are the central processing unit for executing or carrying out instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Typically, a computer will also include or be operatively coupled to receive data from or transfer data to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device (e.g., a Universal Serial Bus (USB) flash drive), to name just a few.
[0082] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, such as semiconductor memory devices, such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.
[0083] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, verbal, or tactile input. Additionally, a computer can interact with a user by sending and receiving documents to and from the device used by the user; for example, sending a webpage to the web browser on the user's device in response to a request received from a web browser. Furthermore, a computer can interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and, in turn, receiving response messages from the user.
[0084] The data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive portions of machine learning training or production (i.e., inference, workload).
[0085] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.
[0086] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer, web browser, or app with a graphical user interface through which a user can interact with the implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0087] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.
[0088] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while features may be described above as operating in certain combinations, and even initially claimed in this way, in some cases, one or more features from a claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.
[0089] Similarly, although operations are depicted in a specific order in the accompanying drawings and described in the claims, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or requiring all illustrated operations to achieve the desired effect. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, but rather it should be understood that the described program components and systems can generally be integrally formed into a single software product or packaged into multiple software products.
[0090] Specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.
Claims
1. A computer-implemented method for processing network inputs using an image processing neural network to generate network outputs from the network inputs, wherein, The image processing neural network is a convolutional neural network and includes multiple convolutional layers, wherein the method includes: The input tensor is received from the first convolutional layer of the plurality of convolutional layers in the image processing neural network, the first convolutional layer having a plurality of kernels, and the input tensor is derived from the network input; Determining the corresponding input-related weights for each of the plurality of kernels from the input tensor includes processing the input tensor through a non-linear routing function with multiple parameters to generate the corresponding input-related weights for the plurality of kernels; An input-related kernel is generated by calculating a weighted sum of the plurality of kernels according to the corresponding input-related weights; and Generating the output tensor of the first convolutional layer from the input tensor of the first convolutional layer includes performing convolution between the input-associated kernel and the input tensor.
2. A computer-implemented method for performing computer vision tasks, comprising the method according to claim 1, wherein, The network input includes visual data, and generating the network output includes performing the computer vision task.
3. The computer-implemented method according to claim 2, wherein, The network input includes one or more images, and the computer vision task includes one of the following: Image classification, wherein the network output includes multiple scores, each score corresponding to a different object category representing the probability that the one or more images depict an object belonging to that object category; Object detection, wherein the network output identifies one or more regions in the one or more images, and for each region, identifies the probability that the region depicts an object of interest; Image segmentation, wherein the network output defines a corresponding probability for each category in a predetermined set of categories for each pixel in the one or more images; Depth estimation, wherein the network output defines a corresponding depth value for each pixel in the one or more images, or Motion estimation, wherein the network output, for each pixel of one of the one or more images, defines the motion of the scene depicted at that pixel between the images in the network input.
4. The method according to claim 2, wherein, The nonlinear routing function includes one or more neural network layers.
5. The method according to claim 4, wherein, The one or more neural network layers include a global average pooling layer that performs global average pooling across the spatial dimensions of the input tensor.
6. The method according to claim 5, wherein, The global average pooling layer is followed by one or more fully connected layers and nonlinearity.
7. The method according to claim 2, wherein, The generation of output tensors also includes: Batch normalization and activation functions are applied to the output of the convolution.
8. The method according to any one of claims 2 to 7, wherein, The convolution is a depthwise convolution.
9. The method according to any one of claims 2 to 7, wherein, The convolution is a point convolution.
10. A method for training a convolutional neural network according to any one of claims 1 to 9, to determine trained values of the plurality of kernels of the first convolutional layer of the convolutional neural network, and to determine trained values of the parameters of the routing function according to any one of claims 2 to 6, the method comprising: Receives multiple training inputs; Each training input is processed by the convolutional neural network to generate a training output for each training input, including for each training input: Receive the layer input of the first convolutional layer; Based on the current value of the kernel, perform a corresponding convolution between the layer input and each of the plurality of kernels to generate a corresponding convolution output for each kernel; The input tensor is processed by the nonlinear routing function according to the current value of the parameter of the nonlinear routing function to generate the corresponding input-related weights for each of the plurality of kernels; as well as The overall convolutional output of the first convolutional layer is generated by calculating a weighted sum of the convolutional outputs of the kernel based on weights associated with the input. For each training input, the gradient of the parameters of the nonlinear routing function relative to the plurality of kernels and the objective function, the objective function measuring the quality of the training output of the training input, is determined; as well as The gradient is used to update the current value of the parameter and the kernel.
11. A computer-readable storage medium storing one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform a corresponding operation of the method according to any one of claims 1 to 10.
12. A system comprising one or more computers and one or more storage instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform a corresponding operation of the method according to any one of claims 1 to 10.