Simultaneous training of functional subnetworks of neural networks

Abstract

An artificial neural network [200] is trained, the artificial neural network [200] comprising a first subnetwork [311-315] implementing a known function and a second subnetwork [310] implementing an unknown function. The first subnetwork is trained separately and in parallel based on a corresponding known training data set to determine a first parameter value defining the first subnetwork. The first subnetwork is executed on a plurality of processing elements in a processing system. Input values ​​[320, 325] from a network training data set are provided to the artificial neural network comprising the trained first subnetwork. An error value is generated by comparing an output value [335] generated by the artificial neural network with a labeled output value of the network training data set. The second subnetwork is trained by back-propagating the error value to modify a second parameter value defining the second subnetwork without modifying the first parameter value.

Description

Background Art

[0001] A deep neural network (DNN) is a class of artificial neural networks (ANNs) that can learn how to perform tasks such as image recognition, natural language processing, and game playing. The DNN architecture includes a stack of layers that implement the function of converting an input volume (such as a digital image) into an output volume (such as a labeled feature detected in a digital image). For example, the layers in a DNN can be divided into convolutional layers, pooling layers, and fully connected layers representing a convolutional neural network (CNN). Multiple groups of convolutional layers, pooling layers, and fully connected layers can be overlapped to form a complete DNN. For example, a DNN may include a group of convolutional layers that receive inputs and provide outputs to a group of pooling layers, which provide outputs to another group of convolutional layers. A second group of convolutional layers provides outputs to another group of pooling layers, which provide outputs to one or more groups of fully connected layers, which generate output volumes. The functions implemented by the layers in a DNN are explicit (i.e., known or predetermined) or hidden (i.e., unknown). A CNN is a deep neural network (DNN) that performs deep learning on tasks containing multiple hidden layers. For example, a DNN for implementing computer vision includes an explicit function (such as a direction map) and multiple hidden functions in a hierarchical structure of the visual stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] By referencing the accompanying drawings, the present disclosure may be better understood, and its numerous features and advantages become apparent to those skilled in the art.The same reference numbers used in different drawings indicate similar or identical items.

[0003] Figure 1 is a block diagram of a processing system according to some embodiments.

[0004] Figure 2 is a block diagram illustrating a deep neural network (DNN) that includes convolutional layers and is trained to perform tasks such as image recognition according to some embodiments.

[0005] Figure 3 is a block diagram illustrating training a subnetwork that implements an unknown function within an instance of an artificial neural network that also includes a subnetwork that implements a known function, according to some embodiments.

[0006] Figure 4 is a diagram showing the steps that are performed to perform quality assurance steps according to some embodiments. Figure 3 Block diagram of an instance of the CNN shown.

[0007] Figure 5 is a block diagram illustrating training a sub-network that implements an unknown function within a cutout portion of an artificial neural network that also includes a sub-network that implements a known function, according to some embodiments.

[0008] Figure 6 is a flowchart of a method for training an artificial neural network (such as a CNN or DNN) including a first subnetwork according to some embodiments.

[0009] Figure 7 A flowchart of a method for training a subnetwork that implements an unknown function within an artificial neural network (such as a CNN or DNN), according to some embodiments, wherein the artificial neural network includes subnetworks that implement known and unknown functions.

[0010] Figure 8 is a flow chart of a method for generating and training a neural network based on previously trained sub-networks, according to some embodiments. DETAILED DESCRIPTION

[0011] The functionality of a deep neural network (DNN) is represented by different sets of parameters for different layers. The parameters of the convolutional layer define a set of learnable filters (or kernels) that convolve the incoming data over the width and height of the input volume to produce a two-dimensional (2-D) activation map of the filter. The parameters of the pooling layer define how to divide the input volume into sub-regions. For example, the pooling layer can be configured to divide the input image into a set of non-overlapping rectangles and generate the maximum value of each sub-region. The parameters of the fully connected layer define the high-level reasoning performed by the DNN based on the connection with the activations in the previous layer (such as the previous pooling layer). The parameters of the DNN are determined by training the DNN with a training data set including a set of input volumes and a set of corresponding (known or labeled) output values. For example, a facial recognition DNN can be trained using images that are known to include individuals to be recognized by the facial recognition DNN in other images. The training images are called labeled data, which are defined as a set of samples that have been labeled with one or more tags. During training, the input data from the training data set are sequentially provided to the DNN, and the errors between the output values ​​generated by the DNN and the known output values ​​are accumulated. The accumulated error is back-propagated to modify the parameters of the DNN. The above process is repeated until the convergence criterion is met. However, training large DNNs is a computationally intensive task that may take hours, days, or even months depending on the size of the network.

[0012] Training an artificial neural network is typically a sequential process. Back-propagated errors are used to modify the parameters that define the artificial neural network. Examples of training parameters include connection weights for connections between nodes in the network. The accuracy of the function (e.g., pattern recognition) implemented by the artificial neural network improves with each iteration of the sequential process because the parameters are modified based on the back-propagated errors generated for each consecutive sample in the training set. Therefore, it is difficult to parallelize the training process by subdividing the training data set and training multiple instances of the network in parallel because each instance of the network will be trained based on only a portion of the training set. Therefore, merging instances of networks trained in parallel using subsets of the training data set into a single trained network may not result in a network that performs its function with the accuracy of a network trained sequentially using the complete training data set.

[0013] These deficiencies in conventional sequential training processes are addressed by parallelizing the training of an artificial neural network (such as a CNN or DNN) comprising a first subnetwork that implements a known function (and therefore can be trained using a corresponding known training data set) and a second subnetwork that implements an unknown function that does not have a corresponding training data set. The first subnetwork is trained separately and in parallel based on the known training data set. Then, an artificial neural network comprising the first subnetwork and the second subnetwork is trained based on the network training data set by providing input values ​​from the network training data set to the artificial neural network and accumulating errors between output values ​​representing the artificial neural network and labeled output values ​​from the network training data set. The accumulated errors are back-propagated to modify the parameters of the second subnetwork. During the training of the artificial neural network, the previously trained parameters of the first subnetwork are not modified. This process is iterated until a convergence criterion for the parameters of the second subnetwork is met. The first subnetwork and the second subnetwork are significantly smaller than the artificial neural network, and therefore, training the first subnetwork and the second subnetwork separately is significantly faster than training the artificial neural network.

[0014] A quality assurance step is performed to train the parameters of the artificial neural network given the parameter values ​​determined for the trained first subnetwork and the second subnetwork. During the quality assurance step, input values ​​of the training data set are provided to an instance of the artificial neural network, the instance being defined by the modified parameters of the first subnetwork and the second subnetwork. Error values ​​generated by the artificial neural network are back-propagated to modify the parameters of the first subnetwork and the second subnetwork defined in the artificial neural network, and the above process is iterated until a convergence criterion is met. It is desirable to start the quality assurance step using the parameter values ​​determined for the first subnetwork and the second subnetwork trained separately to speed up the convergence of the quality assurance step (relative to techniques that use back-propagation to train the entire artificial neural network) because the parameter values ​​of the trained first subnetwork and the second subnetwork are expected to be significantly closer to the converged values ​​of the complete neural network than other possible initial values ​​of the parameters (such as, arbitrarily or randomly selected parameter values).

[0015] Figure 1 is a block diagram of a processing system 100 according to some embodiments. The processing system 100 includes or has access to a memory 105 or other storage component implemented using a non-transitory computer-readable medium such as a dynamic random access memory (DRAM). However, the memory 105 may also be implemented using other types of memory including static random access memory (SRAM), non-volatile RAM, etc. The processing system 100 also includes a bus 110 to support communications between entities implemented in the processing system 100 (such as the memory 105). Some embodiments of the processing system 100 include a plurality of busses 110 that are not shown in the figure for clarity. Figure 1 Other buses, bridges, switches, routers, etc. shown in FIG.

[0016] The processing system 100 includes a graphics processing unit (GPU) 115 configured to render an image for presentation on a display 120. For example, the GPU 115 may render an object to generate pixel values, which are provided to the display 120, which uses the pixel values ​​to display an image representing the rendered object. Some embodiments of the GPU 115 may also be used for general computing. In the illustrated embodiment, the GPU 115 implements a plurality of processing elements 116, 117, 118 (collectively referred to herein as "processing elements 116-118"), which are configured to execute instructions simultaneously or in parallel. In the illustrated embodiment, the GPU 115 communicates with the memory 105 via the bus 110. However, some embodiments of the GPU 115 communicate with the memory 105 by direct connection or via other buses, bridges, switches, routers, etc. The GPU 115 may execute instructions stored in the memory 105, and the GPU 115 may store information such as the results of the executed instructions in the memory 105. For example, memory 105 may store a copy 125 of instructions from program code to be executed by GPU 115 .

[0017] The processing system 100 also includes a central processing unit (CPU) 130 that implements a plurality of processing elements 131, 132, 133 (collectively referred to herein as "processing elements 131-133"). The processing elements 131-133 are configured to execute instructions simultaneously or in parallel. The CPU 130 is connected to the bus 110 and can therefore communicate with the GPU 115 and the memory 105 via the bus 110. The CPU 130 can execute instructions such as program code 135 stored in the memory 105, and the CPU 130 can store information such as the results of the executed instructions in the memory 105. The CPU 130 is also capable of initiating graphics processing by issuing a draw call to the GPU 115.

[0018] An input / output (I / O) engine 140 processes input or output operations associated with display 120 and other elements of processing system 100 (e.g., keyboard, mouse, printer, external disk, etc.). I / O engine 140 is coupled to bus 110, enabling I / O engine 140 to communicate with memory 105, GPU 115, or CPU 130. In the illustrated embodiment, I / O engine 140 is configured to read information stored on an external storage component 145, which is implemented using a non-transitory computer-readable medium such as a compact disk (CD), a digital video disk (DVD), etc. I / O engine 140 may also write information such as results processed by GPU 115 or CPU 130 to external storage component 145.

[0019] An artificial neural network (such as a CNN or a DNN) is represented as a program code configured using a corresponding parameter set. Therefore, the artificial neural network can be executed on a GPU 115 or a CPU 130, or other processing units including a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a process in memory (PIM), etc. If the artificial neural network implements a known function that can be trained using a corresponding known data set, the artificial neural network is trained (i.e., the values ​​of the parameters defining the artificial neural network are established) by providing the input values ​​of the known training data set to the artificial neural network executed on the GPU 115 or the CPU 130, and then comparing the output values ​​of the artificial neural network with the labeled output values ​​in the known training data set. An error value is determined based on the comparison, and it is back-propagated to modify the values ​​of the parameters defining the artificial neural network. This process is iterated until the values ​​of the parameters meet the convergence criteria.

[0020] However, as discussed herein, artificial neural networks are generally composed of subnetworks that perform known (or explicit) functions and subnetworks that perform unknown (or implicit) functions. It is a time-consuming and resource-consuming process to sequentially train an artificial neural network including subnetworks that implement known and unknown functions based on a network training data set. In order to reduce the time and resources consumed by training an artificial network, the artificial neural network is subdivided into a first subnetwork that performs a known function (which has a corresponding known training data set) and a second subnetwork that performs an unknown function and therefore does not have a known training data set. The first subnetwork is trained separately and in parallel based on the corresponding known training data set. For example, a separate instance of the first subnetwork is executed simultaneously on a processing element 116-118 in a GPU 115 or a processing element 131-133 in a CPU 130 or a combination extracted from all processing elements 116-118, 131-133. In some embodiments, multiple instances of a single first subnetwork can be trained simultaneously on a processing element, and then the best trained instance is selected to be integrated into the artificial neural network. A separate instance of the first subnetwork can also be trained for different variations of a known function.

[0021] Once the first subnetwork has been trained, the artificial neural network is trained based on the network training data set. In this training phase, the parameters of the first subnetwork remain constant because it is expected that the parameters are accurately defined by training the first subnetwork based on a known data set. The input values ​​from the network training data set are provided to the artificial neural network, which is executed on one, some or all of the processing elements 116-118, 131-133. An error value is generated by comparing the output value of the artificial neural network with the labeled output value in the network training data set. The error value is back-propagated and used to modify the parameters of the second subnetwork. This process is iterated until the value of the parameter defining the second subnetwork meets the convergence criterion. For example, when the measure of the value change of the parameter between two iterations is below a threshold, the value of the parameter defining the second subnetwork converges. Then, the converged values ​​of the parameters defining the artificial neural network (e.g., the parameters defining the first subnetwork and the second subnetwork) are stored in a non-transitory computer-readable medium such as the memory 105 or the external storage component 145. In some embodiments, the stored values ​​for the parameters of the first and second subnetworks are subsequently read from the non-transitory computer-readable medium and used to construct other neural networks, possibly in combination with other subnetworks that may or may not be trained.

[0022] Once the values ​​of the parameters defining the first subnetwork and the second subnetwork are determined, a quality assurance step is performed on the complete artificial neural network. An instance of the artificial neural network is executed in the GPU 115 or the CPU 130. An instance is defined by integrating the first subnetwork and the second subnetwork into the complete artificial neural network. An input value is provided to an instance of the artificial neural network, which generates a corresponding output value based on the current value of the parameter defining the artificial neural network. An error value is determined by comparing the output value with the labeled output value in the training data set. The error value is back-propagated and used to modify the values ​​of the parameters defining the first subnetwork and the second subnetwork in the artificial neural network. This process is iterated until the values ​​of the parameters defining the artificial neural network meet the convergence criteria. Then, the artificial neural network is considered to be trained to perform its assigned tasks. In some embodiments, the simultaneous training of the first subnetwork, the training of the unknown function in the second subnetwork, and the subsequent quality assurance training of the artificial neural network are iterated once or multiple times to train the artificial neural network.

[0023] Figure 2 is a block diagram illustrating a deep neural network (DNN) 200 trained to perform tasks such as image recognition according to some embodiments. Figure 1DNN 200 is executed on processing elements 116-118 in GPU 115 or processing elements 131-133 in CPU 130 shown in FIG. DNN 200 is configured to receive an input value, such as a portion 205 of image 210, and to generate an output value 215 based on a function implemented in DNN 200 and the values ​​of parameters defining the function.

[0024] DNN 200 includes a convolution layer 220 that implements a convolution function defined by a set of parameters, which are trained based on one or more training data sets. The parameters include a set of learnable filters (or kernels) that have a small receptive field and extend through the entire depth of the input volume of the convolution layer 220. The parameters may also include a depth parameter, a step parameter, and a zero padding parameter that control the size of the output volume of the convolution layer 220. The convolution layer 220 applies a convolution operation to the input value and provides the result of the convolution operation to a subsequent layer in the DNN 200. For example, a portion 205 of an image 210 is provided as an input 225 to the convolution layer 220, which applies a convolution operation to the input 225 based on a set of parameters to generate a corresponding output value 230. In some embodiments, the convolution layer 220 is identified as a subnetwork of the DNN 200. The subnetwork then represents a convolutional neural network (CNN). However, convolutional layer 220 may be part of a larger subnetwork of DNN 200, or convolutional layer 220 may be further subdivided into multiple subnetworks of DNN 200.

[0025] The results generated by the convolution layer 220 are provided to the pooling layer 235 in the DNN 200. The pooling layer 235 combines the outputs of the neuron clusters of the convolution layer 220 into a smaller number of neuron clusters output from the pooling layer 235. The pooling layer 235 generally implements a known (or explicit) function. For example, the pooling layer 235 that implements maximum pooling can assign the maximum value of the values ​​of the neurons in the cluster output from the convolution layer 220 to the single neuron output from the pooling layer 235. For another example, the pooling layer 235 that implements average pooling can assign the average value of the values ​​of the neurons in the cluster output from the convolution layer 220 to the single neuron output from the pooling layer 235. Therefore, a predetermined training data set can be used to train the known (or explicit) functionality of the pooling layer 235. In some embodiments, the pooling layer 235 is identified as a subnetwork of the DNN 200. However, the pooling layer 235 may be part of a larger sub-network of the DNN 200, or the pooling layer 235 may be further subdivided into multiple sub-networks of the DNN 200.

[0026] In the illustrated embodiment, DNN 200 also includes an additional convolutional layer 240 that receives input from convolutional layer 235 and an additional pooling layer 245 that receives input from additional convolutional layer 240. However, additional convolutional layer 240 and additional pooling layer 245 are optional and are not present in some embodiments of DNN 200. In addition, some embodiments of DNN 200 may include a greater number of convolutional layers and pooling layers. Additional convolutional layer 240 and additional pooling layer 245 may be identified as a subnetwork of DNN 200, a part of a subnetwork of DNN 200, or may be subdivided into multiple subnetworks of DNN 200.

[0027] The output from the additional pooling layer 245 is provided to the fully connected layers 250, 255. The neurons in the fully connected layers 250, 255 are connected to each neuron in another layer (such as, the additional pooling layer 245 or other fully connected layers). The fully connected layers 250, 255 generally implement functionality representing high-level reasoning for producing output values ​​215. For example, if the DNN 200 is trained to perform image recognition, the fully connected layers 250, 255 implement functionality for labeling portions of an image that have been "recognized" by the DNN 200. Examples of labels include the names of people whose faces are detected in the image 210, the types of objects detected in the image, and the like. As discussed herein, the functions implemented in the fully connected layers 250, 255 are represented by the values ​​of parameters determined using the training data set. The fully connected layers 250, 255 are identified as a subnetwork of the DNN 200, a portion of a subnetwork of the DNN 200, or are subdivided into multiple subnetworks of the DNN 200.

[0028] Figure 3 is a block diagram illustrating training a subnetwork to implement an unknown function within an instance 300 of an artificial neural network that also includes a subnetwork to implement a known function, according to some embodiments. Figure 1 Instance 300 is executed on a processing element of the processing elements 116-118, 131-133 shown. The DNN is implemented using interconnected subnetworks 310, 311, 312, 313, 314, 315 (collectively referred to herein as "subnetworks 310-315"). Subnetworks 310-315 implement different functions, which are defined by the values ​​of parameters that characterize subnetworks 310-315. In the illustrated embodiment, subnetwork 310 implements an unknown function and therefore has no known training data set. Although in Figure 3 , a single subnetwork 310 implementing an unknown function is shown, but some embodiments of the artificial neural network include multiple subnetworks implementing one or more unknown functions. Subnetworks 311-315 implement known functions with corresponding known training data sets. Therefore, subnetworks 311-315 have been trained individually and in parallel based on the corresponding known training data sets.

[0029] An instance 300 of a DNN is trained using a network training dataset including input values ​​320 and 325 and labeled output values ​​330. The instance 300 of the DNN may receive the input values ​​320, 325 and generate an output value 335. Then, an error value of the instance 300 of the DNN is determined by comparing the output value 335 with the labeled output value 330. As indicated by the solid line, the subnetwork 310 is identified as a training subnetwork, which means that the parameters defining the subnetwork 310 are modified based on the back-propagated error values. As indicated by the dashed lines, the subnetworks 311-315 are identified as non-training subnetworks, which means that the parameters defining the subnetworks 311-315 are not modified based on the back-propagated error values ​​because the parameters are previously trained based on a known training dataset associated with the functions of the subnetworks 311-315.

[0030] The training subnetwork 310 is then trained by assuming that the error values ​​generated by the instance 300 of the DNN are generated by inaccurate values ​​of the parameters defining the training subnetwork 310. Therefore, the values ​​of the parameters are modified based on the error values ​​generated during the current iteration to reduce the error values ​​generated during subsequent iterations. During the training process, the values ​​of the parameters defining other (non-training) subnetworks 311-315 remain constant. For example, the values ​​of the parameters defining the subnetwork 310 in the instance 300 of the DNN are iteratively modified to reduce the error values ​​generated by the instance 300 of the DNN, while the values ​​of the parameters defining the subnetworks 311-315 remain constant.

[0031] Figure 4 is a diagram showing the steps that are performed to perform quality assurance steps according to some embodiments. Figure 3 A block diagram of an example 400 of a DNN is shown. Figure 1 Instance 400 is executed on a processing element of one of the processing elements 116-118, 131-133 shown. During the quality assurance step, as indicated by the solid lines, all sub-networks 310-315 are considered training sub-networks. The complete DNN (e.g., as shown in FIG. 4 ) is trained based on the network training data set 320, 325, 330 using the values ​​of the parameters defining the previously trained sub-networks 310-315 as initial values ​​for the iterative training process. Figure 3 ). An instance 400 of a DNN receives input values ​​320, 325 and generates an output value 405 based on the current values ​​of the parameters defining the DNN (e.g., the parameters defining the subnetworks 310-315). The output value 405 is compared to the labeled output value 330 to determine an error value, which is back-propagated and used to modify the parameter values ​​defining the DNN. This process is iterated until a convergence criterion is met, such as a measure of the rate of change of the error value or the value of the error value is below a threshold.

[0032] Independently and in parallel training of subnetworks 311-315 implementing known functions allows subnetwork training to be performed simultaneously or in parallel on different processing elements, thereby reducing the time required to train subnetworks 311-315. The values ​​of the parameters defining the training subnetworks 311-315 may be closer to those obtained by using a training data set (e.g., using Figure 4 The values ​​of the parameters determined by training the complete DNN based on the input values ​​320, 325, 330 of the training dataset) are obtained by training the complete DNN. Therefore, it is expected that the convergence of the training process of the complete DNN (including training the sub-networks 310-315 and performing quality assurance steps) will consume less time, energy, and fewer resources than the conventional practice of training the complete CNN based on the input values ​​320, 325, 330 of the training dataset using arbitrary or random initial values ​​of the parameters.

[0033] Figure 5 is a block diagram illustrating training a subnetwork to implement an unknown function within a cutout portion 500 of an artificial neural network that also includes a subnetwork to implement a known function, according to some embodiments. Figure 1 The cutting portion 500 is performed on the processing elements of the processing elements 116-118, 131-133 shown. The cutting portion 500 of the artificial neural network is implemented using interconnected sub-networks 510, 511, 512, 513, 514, 515, 516 (collectively referred to herein as "sub-networks 510-516"). The sub-networks 510-516 implement different functions, which are defined by the values ​​of the parameters that characterize the sub-networks 510-516. In the illustrated embodiment, the sub-network 510 implements an unknown function and therefore has no known training data set. Although in Figure 5 , a single subnetwork 510 implementing an unknown function is shown, but some embodiments of the artificial neural network include multiple subnetworks implementing one or more unknown functions. Subnetworks 511-516 implement known functions with corresponding known training data sets. Therefore, subnetworks 511-516 have been trained separately and in parallel based on the corresponding known training data sets.

[0034] Subnetwork 510 is substantially surrounded by subnetworks 511-516. As used herein, the phrase "substantially surrounded" indicates that the input to subnetwork 510 is provided only (or primarily) by one or more of subnetworks 511-516, and no (or a small amount) output from subnetwork 510 is exposed to any subnetwork other than subnetworks 511-516. In that case, subnetwork 510 is trained based on a combined training data set consisting of known training data sets of subnetworks 511-516. For example, as discussed herein, subnetworks 511-516 may be trained individually and in parallel based on corresponding known training data sets. Subnetwork 510 is then trained by training cutout 500 based on the combined training data set, while the parameters of previously trained subnetworks 511-516 remain constant. Once trained, cutout 500 is integrated into a complete artificial neural network, which is trained based on a network training data set as discussed herein.

[0035] Figure 6 is a flow chart of a method 600 for training an artificial neural network (such as a CNN or a DNN) including a first subnetwork according to some embodiments. The method 600 Figure 1 The processing system 100 shown is implemented in some embodiments.

[0036] At block 605, the artificial neural network is divided into subnetworks that perform known functions in the artificial neural network and subnetworks that perform unknown functions. The definition of functions is not strict. The DNN architecture may define functions based on the network topology or the internal prototype operations performed by the subnetworks or any other reasonable criteria. For example, Figure 2 The artificial neural network of the deep neural network 200 shown, which includes a convolutional layer 220, a pooling layer 235, a convolutional layer 240, a pooling layer 245, and fully connected layers 250, 255, is divided into a first sub-network including the convolutional layer 220, a second sub-network including the pooling layer 235, a third sub-network including the convolutional layer 240, a fourth sub-network including the pooling layer 245, a fifth sub-network including the fully connected layer 250, and a sixth sub-network including the fully connected layer 255. Other divisions of the neural network at higher or lower granularity layers are also possible. For example, the division of the layer may occur within a single layer, such as within the convolutional layer 220.

[0037] In different processing elements (such as Figure 1A first subnetwork with a known function is trained separately and in parallel on the processing elements 116-118, 131-133 shown in the figure. The first subnetwork is trained based on a known data set corresponding to the function implemented in the first subnetwork. In the illustrated embodiment, the values ​​of the parameters defining the training subnetwork are modified based on a training data set specific to each subnetwork. This training data set is known because the function of the subnetwork (the output value generated in response to the input) is also known. For example, at box 610, the values ​​of the parameters defining the first known subnetwork are modified based on an error value, and the error value is determined by comparing the output value of this subnetwork with the labeled output value of the training data set of this subnetwork. At box 615, the values ​​of the parameters defining the second known subnetwork are modified based on the error value, and the error value is determined by comparing the output value of the second subnetwork with the labeled output value of the training data set of the second known subnetwork. At box 620, the values ​​of the parameters defining the Nth known subnetwork are modified based on the error value, and the error value is determined by comparing the output value of the Nth subnetwork with the labeled output value of the known training data set of the Nth subnetwork. The processes represented by blocks 610, 615, 620 are performed simultaneously or in parallel.

[0038] At box 625, the parameters of the second subnetwork implementing the unknown function are modified based on the network training data set for the artificial neural network. The values ​​of the parameters defining the first subnetwork in the artificial neural network are set to be equal to the values ​​of the parameters determined in boxes 610, 615, and 620, so that the first subnetwork is integrated back into the entire artificial neural network. The values ​​of the parameters defining the second subnetwork are set to random values ​​or arbitrary values, or any other criteria for setting initial values ​​are used. Input values ​​from the network training data set are provided to the artificial neural network, which generates output values ​​based on the current values ​​of the parameters. The output values ​​are compared with the labeled output values ​​of the network training data set to determine an error value, which is back-propagated to modify the values ​​of the parameters defining the second subnetwork. During this process, the parameters defining the first subnetwork remain constant, and the process is iterated until the convergence criteria of the second subnetwork are met.

[0039] At block 630, a quality assurance step is performed. In the quality assurance step, input values ​​from the network training data set are provided to an artificial neural network, which generates output values ​​based on the current values ​​of the parameters. The output values ​​are compared to the labeled output values ​​of the network training data set to determine an error value. The quality assurance step performed at block 630 differs from the step performed at block 625 in that the error values ​​are back-propagated to modify the values ​​of the parameters defining the first subnetwork and the second subnetwork. The quality assurance step is iterated until a convergence criterion for the artificial neural network is met. Once the process converges, the values ​​of the parameters defining the first subnetwork and the second subnetwork are stored in a database using a processor such as a processor. Figure 1The memory 105 or the external storage component 145 shown may be implemented in a storage component of a non-transitory computer-readable medium.

[0040] Figure 7 7 is a flow chart of a method 700 for training a subnetwork implementing an unknown function within an artificial neural network (such as a CNN or a DNN) that includes subnetworks implementing known and unknown functions, according to some embodiments. Figure 1 The method 700 is implemented in some embodiments of the processing system 100 shown. Figure 6 Some implementations of method 600 are shown.

[0041] At block 705, the artificial neural network generates output values ​​based on input training values ​​from the training data set and current values ​​of parameters of subnetworks that make up the neural network. As discussed herein, subnetworks that implement unknown functions in an artificial neural network are identified as training subnetworks.

[0042] At block 710, the output values ​​of the artificial neural network are compared to the labeled output values ​​of the training data set. An error value is determined based on the comparison. For example, the error value may indicate a percentage of people or objects in one or more training images that were correctly identified by the artificial neural network, as indicated by the comparison of the output values ​​to the labeled output values.

[0043] At block 715, the values ​​of the parameters defining the training subnetwork (i.e., the subnetwork implementing the unknown function) are modified based on the error value. The values ​​of the parameters defining the other subnetworks implementing the known functions in the artificial neural network are kept constant, i.e., the values ​​of the parameters defining the training subnetwork are modified under the assumption that the error is caused only by the wrong values ​​of the parameters defining the training subnetwork.

[0044] At decision block 720, convergence criteria are evaluated for the value of the parameter defining the training subnetwork. For example, the value of the parameter determined in the current iteration of method 700 can be compared with the value of the parameter determined in the previous iteration of method 700. The value of the parameter meeting the convergence criteria is that the measure of the change of the value between the current iteration and the previous iteration is lower than a threshold value. Other convergence criteria can be applied in other embodiments, such as comparing the output of the network with the labeled data. In this case, a convergence test (block 720) can be placed between block 710 and block 715. If the convergence criteria are not met, then method 700 flows back to block 705 to start another iteration of method 700 with the modified value of the parameter. Modification is completed by back propagation or any other method used in training the neural network. If the convergence criteria are met, then method 700 enters block 725.

[0045] At block 725, the converged values ​​of the parameters defining the training subnetwork are stored in a storage component implemented using a non-transitory computer-readable medium. For example, the converged values ​​are stored in Figure 1 The stored values ​​may then be accessed for subsequent quality assurance steps performed on the neural network. As discussed herein, the stored values ​​may also be accessed and combined with other parameter values ​​of other sub-networks to form a new neural network.

[0046] Figure 8 is a flow chart of a method 800 for generating and training a neural network based on previously trained sub-networks, according to some embodiments. In the illustrated embodiment, a first neural network has been subdivided into a plurality of sub-networks, which are then trained using a network training data set. For example, Figure 6 The method 600 and Figure 7 The embodiment of the method 700 shown is used to train a first neural network. The values ​​of the parameters defining the subnetworks in the first neural network are stored in a memory such as Figure 1 The training of the first neural network is performed by Figure 1 The processing system 100 shown may be executed by other processing systems.

[0047] At block 805, values ​​of parameters defining a subset of sub-networks of a first neural network are read from a non-transitory computer-readable storage medium. In some embodiments, values ​​of parameters defining sub-networks of other neural networks are also read from a non-transitory computer-readable storage medium.

[0048] At block 810, the subnetworks are combined to form a second neural network. As discussed herein, combining the subnetworks to form the second neural network includes configuring program code and parameter values ​​to represent the second neural network, e.g., connecting the outputs of one set of subnetworks to the inputs of another set of subnetworks. Various criteria for integrating subnetworks from multiple networks into a new network may be selected, e.g., functional integration, information data flow, real-time aspects, and many other aspects. Thus, the second neural network may be executed on one or more processing elements. For example, the values ​​of the parameters defining the subset of subnetworks of the first neural network and the values ​​of the parameters defining the subnetworks of the other neural networks, if any, are interconnected to construct the second neural network so that it can be used in a manner such as Figure 1 A second neural network is shown executing on one or more of the processing elements 116 - 118 , 131 - 133 .

[0049] At block 815, a second neural network is trained based on the network training data set. The values ​​of the parameters defining the second neural network are initially set equal to the corresponding values ​​of the parameters defining the subnetworks that make up the second neural network. Figure 6 The method 600 and Figure 7 Some embodiments of the method 700 shown are used to train a second neural network. Starting the training process using previously trained values ​​of the parameters defining the sub-network can speed up the training process so that the training takes less time and fewer resources than starting the training process of the second neural network using arbitrary or random values ​​of the parameters defining the second neural network.

[0050] In a first aspect, a method includes training an artificial neural network, the artificial neural network including a first subnetwork that implements a known function and a second subnetwork that implements an unknown function, the method including separately and in parallel training the first subnetwork based on a corresponding known training data set to determine a first parameter value that defines the first subnetwork, providing an input value from the network training data set to the artificial neural network including the trained first subnetwork, generating an error value by comparing an output value produced by the artificial neural network with a labeled output value of the network training data set, using the error value to modify a second parameter value that defines the second subnetwork without modifying the first parameter value, and storing the first parameter value and the second parameter value.

[0051] In one embodiment of the first aspect, the method includes training a first subnetwork, including simultaneously providing input values ​​of a corresponding known training data set to the first subnetwork, simultaneously generating an error value of the first subnetwork, and simultaneously modifying a first parameter value. In one embodiment, the method includes iteratively performing simultaneously providing input values ​​of a corresponding known training data set to the first subnetwork, simultaneously generating an error value of the first subnetwork, and simultaneously modifying a first parameter value defining the first subnetwork until a convergence criterion for the first parameter value is met. In a specific embodiment, the method includes training a second subnetwork based on a cut training set formed by a subset of the known training data set corresponding to a first subset of the first subnetwork, wherein the first subset contains the second subnetwork.

[0052] In another embodiment of the first aspect, the method includes providing input values ​​of a network training data set to an instance of an artificial neural network defined by modified parameter values ​​of a first subnetwork and a second subnetwork, generating an error value by comparing output values ​​from the artificial neural network with labeled values ​​in the network training data set, using the error value to modify first parameter values ​​and second parameter values ​​defining the first subnetwork and the second subnetwork, and storing the modified parameter values ​​defining the first subnetwork and the second subnetwork. In a specific embodiment, the method includes iteratively performing providing input values ​​of the network training data set to the artificial neural network, generating error values, and modifying first parameter values ​​defining the first subnetwork and the second subnetwork until a convergence criterion for the first parameter value is met.

[0053] In another embodiment of the first aspect, the method includes reading stored parameter values ​​of a subset of a first subnetwork of an artificial neural network, and using the stored parameter values ​​of the subset of the first subnetwork of the artificial neural network to define parameter values ​​of a different artificial neural network.

[0054] In a second aspect, a processing system includes a plurality of processing elements configured to train an artificial neural network including a first subnetwork implementing a known function and a second subnetwork implementing an unknown function by: training the first subnetwork separately and in parallel based on a corresponding known training data set to determine a first parameter value defining the first subnetwork, providing input values ​​from the network training data set to the artificial neural network including the trained first subnetwork, generating an error value by comparing output values ​​produced by the artificial neural network with labeled output values ​​of the network training data set, training the second subnetwork by modifying a second parameter value defining the second subnetwork using the error value without modifying the first parameter value, and an input / output engine configured to store the first parameter value and the second parameter value in a storage component.

[0055] In another embodiment of the second aspect, a plurality of processing elements are configured to simultaneously provide input values ​​of a corresponding known training data set to a first subnetwork, simultaneously generate an error value of the first subnetwork, and simultaneously modify a first parameter value. In a specific embodiment, a plurality of processing elements are configured to iteratively simultaneously provide input values ​​of a corresponding known training data set to a first subnetwork, simultaneously generate an error value of the first subnetwork, and simultaneously modify a first parameter value defining the first subnetwork until a convergence criterion for the first parameter value is met. In a more specific embodiment, a plurality of processing elements are configured to train a second subnetwork based on a cut training set formed by a subset of a known training data set corresponding to a first subset of the first subnetwork, wherein the first subset contains the second subnetwork.

[0056] In another embodiment of the second aspect, a plurality of processing elements are configured to provide input values ​​of a network training data set to an instance of an artificial neural network defined by modified first and second parameter values ​​of a first subnetwork and a second subnetwork, generate an error value by comparing an output value from the artificial neural network with a labeled value in the network training data set, and use the error value to modify the first and second parameter values ​​defining the first and second subnetworks. In addition, the input / output engine is configured to store the modified parameter values ​​defining the first and second subnetworks in a storage component. In a specific embodiment, a plurality of processing elements are configured to iteratively provide input values ​​of a network training data set to an artificial neural network, generate an error value, and modify the first and second parameter values ​​defining the first and second subnetworks until a convergence criterion for the first parameter value is met.

[0057] In another embodiment of the second aspect, the input / output engine is configured to read parameter values ​​of the subset of the first subnetwork of the artificial neural network from the storage component, and at least one of the plurality of processing elements is configured to use the parameter values ​​of the subset of the first subnetwork of the artificial neural network to define parameter values ​​of a different artificial neural network.

[0058] In a third aspect, a method includes reading, using an input / output engine of a processing system, a first parameter value defining a first subnetwork implementing a known function and a second parameter value defining a second subnetwork implementing an unknown function, wherein the first subnetwork has been trained based on a corresponding known training data set. The method also includes generating an artificial neural network at the processing system by combining the first subnetwork and the second subnetwork, providing input values ​​of the network training data set to the artificial neural network, generating an error value of the artificial neural network at the processing system by comparing output values ​​from the artificial neural network with labeled output values ​​in the network training data set, and modifying the second parameter value defining the second subnetwork at the processing system based on the error value.

[0059] In one embodiment of the third aspect, iteratively performing the input values ​​of the network training data set to the artificial neural network, generating an error value of the artificial neural network, and modifying the second parameter value defining the second subnetwork until the convergence criteria of the second parameter value defining the second subnetwork are met. In another embodiment, the method includes storing the first parameter value and the second parameter value in a storage component. In a specific embodiment, the method includes providing the input values ​​of the network training data set to an instance of the artificial neural network defined by the modified parameter value of the second subnetwork. The method also includes generating an error value by comparing the output value from the artificial neural network with the labeled value in the network training data set, and using the error value to modify the first parameter value and the second parameter value defining the first subnetwork and the second subnetwork. In another specific embodiment, the method includes iteratively performing the input values ​​of the network training data set to the artificial neural network, generating an error value, and modifying the first parameter value and the second parameter value defining the first subnetwork and the second subnetwork until the convergence criteria of the first parameter value and the second parameter value are met. In an even more specific embodiment, the method includes storing the modified parameter values ​​defining the first subnetwork and the second subnetwork in a storage component, and using a subset of the stored parameter values ​​to define the parameter values ​​of different artificial neural networks.

[0060] In some embodiments, the apparatus and techniques described above are implemented in a system that includes one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as those described above with reference to Figures 1 to 7Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used to design and manufacture these IC devices. These IC devices can implement the methods described herein directly in transistor circuits or as programmable code executed on such circuits.

[0061] Computer-readable storage media may include any non-transitory storage media or a combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or storage media based on microelectromechanical systems (MEMS). Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disc or a flash memory based on a universal serial bus (USB)), or connected to a computer system via a wired or wireless network (e.g., a network accessible storage device (NAS)). The subnetworks and networks described herein may be stored in such storage media.

[0062] In some embodiments, certain aspects of the above-mentioned techniques may be implemented by one or more processors of a processing system that executes software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate the one or more processors to perform one or more aspects of the above-mentioned techniques. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, a cache, a random access memory (RAM), or one or more other non-volatile memory devices, etc. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executable by one or more processors.

[0063] It should be noted that not all activities or elements described above in the general description are required, a portion of a particular activity or device may not be required, and one or more other activities may be performed or one or more other elements may be included in addition to those described. Further, the order in which the activities are listed is not necessarily the order in which they are performed. In addition, these concepts have been described with reference to specific embodiments. However, it will be appreciated by those of ordinary skill in the art that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the following claims. Therefore, the present specification and drawings should be regarded as illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the present disclosure.

[0064] Benefits, other advantages, and solutions to problems are described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any features that may cause any benefit, advantage, or solution to appear or become more apparent should not be construed as being key, essential, or basic features of any or all claims. In addition, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter may be modified and practiced in different but equivalent ways as would be apparent to one skilled in the art having the benefit of the teachings herein. Unless otherwise described in the claims below, it is not intended to limit the details of construction or design shown herein. Therefore, it is apparent that the specific embodiments disclosed above may be altered or modified, and all such changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is as set forth in the claims below.

Claims

1. A method for training an artificial neural network, the artificial neural network comprising a first sub-network for implementing a known function and a second sub-network for implementing an unknown function, the artificial neural network being used for image data, the method comprising: training the first sub-network separately and in parallel based on a corresponding known training data set to determine first parameter values ​​defining the first sub-network; providing input values ​​from a network training data set to the artificial neural network including the trained first subnetwork; generating an error value by comparing an output value generated by the artificial neural network with labeled output values ​​of the network training data set; using the error value to modify a second parameter value defining the second subnetwork without modifying the first parameter value; as well as storing the first parameter value and the second parameter value, Wherein, after the training of the artificial neural network is completed, the second sub-network is substantially surrounded by the first sub-network, so that the input to the second sub-network is provided by the first sub-network, and the output from the second sub-network is provided to the first sub-network.

2. The method of claim 1, wherein training the first subnetwork comprises simultaneously providing input values ​​of the corresponding known training data set to the first subnetwork, simultaneously generating error values ​​of the first subnetwork, and simultaneously modifying the first parameter value.

3. The method of claim 2, wherein providing the input value of the corresponding known training data set to the first subnetwork, generating the error value of the first subnetwork, and modifying the first parameter value defining the first subnetwork are iteratively performed simultaneously until a convergence criterion for the first parameter value is met.

4. The method of claim 3, further comprising: The second sub-network is trained based on a cut training set formed by a subset of a known training data set corresponding to a first subset of the first sub-network, wherein the first subset contains the second sub-network.

5. The method of claim 1, further comprising: providing the input values ​​of the network training data set to an instance of the artificial neural network defined by the first parameter values ​​and the second parameter values ​​of the first sub-network and the second sub-network; generating an error value by comparing an output value from the artificial neural network with labeled values ​​in the network training data set; modifying the first parameter value and the second parameter value defining the first subnetwork and the second subnetwork using the error value; as well as Modified parameter values ​​defining the first sub-network and the second sub-network are stored.

6. The method of claim 5, wherein providing the input values ​​of the network training data set to the artificial neural network, generating the error value, and modifying the first parameter values ​​defining the first subnetwork and the second subnetwork are performed iteratively until a convergence criterion for the first parameter value is met.

7. The method of claim 1, further comprising: reading stored parameter values ​​of a subset of the first subnetwork of the artificial neural network; as well as The stored parameter values ​​of the subset of the first sub-network of the artificial neural network are used to define parameter values ​​of a different artificial neural network.

8. A processing system, comprising: a plurality of processing elements configured to train an artificial neural network, the artificial neural network comprising a first subnetwork implementing a known function and a second subnetwork implementing an unknown function, the artificial neural network for image data, wherein, after training of the artificial neural network, the second subnetwork is substantially surrounded by the first subnetwork such that an input to the second subnetwork is provided by the first subnetwork and an output from the second subnetwork is provided to the first subnetwork by: training the first sub-network separately and in parallel based on a corresponding known training data set to determine first parameter values ​​defining the first sub-network; providing input values ​​from a network training data set to the artificial neural network including the trained first subnetwork; generating an error value by comparing an output value generated by the artificial neural network with labeled output values ​​of the network training data set; training the second subnetwork by modifying second parameter values ​​defining the second subnetwork using the error value without modifying the first parameter values; and An input / output circuit is configured to store the first parameter value and the second parameter value in a storage component.

9. The processing system of claim 8, wherein the plurality of processing elements are configured to simultaneously provide input values ​​of the corresponding known training data set to the first sub-network, simultaneously generate error values ​​of the first sub-network, and simultaneously modify the first parameter value.

10. The processing system of claim 9, wherein the plurality of processing elements are configured to iteratively and simultaneously provide the input values ​​of the corresponding known training data set to the first subnetwork, simultaneously generate the error value of the first subnetwork, and simultaneously modify the first parameter value defining the first subnetwork until a convergence criterion for the first parameter value is met.

11. The processing system of claim 10, wherein the plurality of processing elements are configured to: The second sub-network is trained based on a cut training set formed by a subset of a known training data set corresponding to a first subset of the first sub-network, wherein the first subset contains the second sub-network.

12. The processing system of any one of claims 8 to 11, wherein the plurality of processing elements are configured to: providing the input values ​​of the network training data set to an instance of the artificial neural network defined by the first parameter values ​​and the second parameter values ​​of the first sub-network and the second sub-network; generating an error value by comparing an output value from the artificial neural network with labeled values ​​in the network training data set; as well as The first parameter value and the second parameter value defining the first subnetwork and the second subnetwork are modified using the error value, and wherein The input / output circuit is configured to store modified parameter values ​​defining the first sub-network and the second sub-network in the storage component.

13. The processing system of claim 12, wherein the plurality of processing elements are configured to iteratively provide the input values ​​of the network training data set to the artificial neural network, generate the error value, and modify the first parameter value and the second parameter value defining the first subnetwork and the second subnetwork until a convergence criterion for the first parameter value is met.

14. The processing system of any one of claims 8 to 11, wherein: The input / output circuit is configured to read parameter values ​​of a subset of the first sub-network of the artificial neural network from the storage component; and At least one of the plurality of processing elements is configured to use the parameter values ​​of the subset of the first sub-network of the artificial neural network to define parameter values ​​of a different artificial neural network.

15. A method comprising: Reading, using input / output circuitry of the processing system, a first parameter value defining a first subnetwork implementing a known function and a second parameter value defining a second subnetwork implementing an unknown function, wherein the first subnetwork has been trained based on a corresponding known training data set; generating, at the processing system, an artificial neural network by combining the first sub-network and the second sub-network, the artificial neural network for image data; Providing input values ​​of a network training data set to the artificial neural network; generating, at the processing system, an error value for the artificial neural network by comparing output values ​​from the artificial neural network with labeled output values ​​of the network training data set; as well as modifying, at the processing system, the second parameter value defining the second subnetwork based on the error value; wherein, after training of the artificial neural network, the second subnetwork is substantially surrounded by the first subnetwork such that input to the second subnetwork is provided by the first subnetwork and output from the second subnetwork is provided to the first subnetwork.

Images