A multi-node neural network constructed from pre-trained small networks

Abstract

A method for training a large neural network using a number of pre-trained smaller neural networks. The large neural network is created by stacking multiple pre-existing pre-trained neural networks in multiple stages. Each of the pre-trained neural networks is scaled up to provide a larger sparse neural network, each pre-trained neural network having a first number of multi-dimensional nodes. The values ​​of the larger sparse neural networks are then superimposed onto the larger neural network. The pre-trained neural network can be created from publicly available pre-trained neural networks. By replacing and / or retraining one of the sub-networks used to create the large neural network, the larger neural network can be adapted for use in different tasks.

Description

Technical Field

[0001] This invention generally relates to the field of artificial intelligence, and more particularly to training neural networks. Background Technology

[0002] Artificial neural networks are increasingly used in artificial intelligence and machine learning applications. In an artificial neural network, a set of inputs is propagated through one or more intermediate or hidden layers to generate an output. Layers that connect the inputs to the outputs are connected by multiple sets of weights generated during the training or learning phase by determining a set of mathematical operations that transform the inputs into the outputs, moving the computation of the probability of each output between layers. Once the weights are determined, they can be used during the inference phase to determine the output from a set of inputs.

[0003] The development of neural networks has focused on improving their capabilities. The ability of a neural network to absorb information is limited by the number of its parameters. The success of neural networks largely stems from building increasingly larger networks. While such networks can perform better on a wide range of tasks, their scale makes them more expensive to use. Larger networks consume more storage space, making them more difficult to allocate, and they run longer, thus requiring more expensive hardware. This is especially important when building models for real-world applications. Summary of the Invention

[0004] One general aspect includes a computer-implemented method for training a neural network, the neural network including a number of nodes. The computer-implemented method includes: instantiating a first plurality of pre-trained neural subnetworks, each pre-trained neural subnetwork having a first number of multidimensional nodes, at least some of the multidimensional nodes having non-zero weights. The computer-implemented method further includes: scaling up a plurality of the first plurality of pre-trained neural subnetworks to have a second larger number of multidimensional nodes, such that a plurality of the first plurality of pre-trained neural subnetworks have a sparse number of non-zero weights associated with the second larger number of multidimensional nodes. The computer-implemented method further includes: creating the neural network by superimposing the non-zero weights of the first plurality of pre-trained neural subnetworks (represented in the multidimensional nodes of the neural network). The computer-implemented method further includes: receiving data for a first task to be computed by the neural network. The computer-implemented method further includes: performing the first task to generate a solution to the first task from the neural network.

[0005] The implementation can include any one or more of the above methods, and the method further includes: creating the neural network, which may further include: creating a second plurality of neural subnetworks with a second larger number of multidimensional nodes by superimposing the non-zero weights of the first plurality of neural subnetworks; and creating the neural network with multidimensional nodes by superimposing the non-zero weights of the second plurality of neural subnetworks onto the nodes of the neural network. The implementation can include any one or more of the above methods, and the method further includes: connecting each of the first plurality of neural subnetworks such that each of the first plurality of pre-trained neural subnetworks is connected to a selective node of another of the first plurality of networks, the selective node being smaller than all nodes in the plurality of nodes of the other of the first plurality of networks arranged in a first-level neural subnetwork that may include a subset of the first plurality of subnetworks. The implementation can include any one or more of the above methods, and the method further includes: connecting each of the subset of the first plurality of neural subnetworks in the first level to a selective node in the nodes of the second plurality of neural subnetworks in a second-level neural subnetwork that may include a subset of the first level. The implementation can include any one or more of the above methods, and the method further includes: retraining the neural network for a new task by replacing at least a subset of the first plurality of neural subnetworks for the new task. Implementations may include any one or more of the methods described above, wherein the retraining further includes retraining the neural network for the new task by: calculating correlation parameters among the trained first plurality of neural subnetworks; predicting an empirical distribution of labels in the training data for the new task based on the first task; training each of the first plurality of networks using the training data for the new task; and replacing a plurality of the first plurality of neural subnetworks with the retrained neural subnetworks. Implementations may include any one or more of the methods described above, wherein replacing a neural subnetwork may include: replacing a plurality of the first plurality of neural subnetworks when there exists a greater than a maximum number of pre-trained neural subnetworks. Implementations may include any one or more of the methods described above, wherein replacing a neural subnetwork may include: replacing a neural subnetwork with general performance determined relative to the training data for the new task. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0006] Another general aspect includes a processing device. The processing device includes a non-transient memory, which may include instructions. The processing device also includes one or more processors communicating with the memory, wherein the one or more processors create a neural network by executing the instructions to: instantiate a first plurality of pre-trained neural subnetworks, each pre-trained neural subnetwork having a first number of multidimensional nodes, at least some of which have non-zero weights; amplify a plurality of the first plurality of pre-trained neural subnetworks to have a second larger number of multidimensional nodes, such that the plurality of the first plurality of pre-trained neural subnetworks have a sparse number of non-zero weights associated with the second larger number of multidimensional nodes; and create the neural network by superimposing the non-zero weights of the first plurality of neural subnetworks (represented in the multidimensional nodes of the neural network). Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, respectively, for performing the above-described actions of the instructions.

[0007] Implementations may include a processing device comprising any one or more of the features described above, wherein the processor executes instructions to: retrain the neural network for a new task by replacing at least a subset of the first plurality of neural subnetworks for the new task. Implementations may include a processing device comprising any one or more of the features described above, wherein the retraining further includes: retraining the neural network for the new task by executing instructions to: compute correlation parameters among the trained first plurality of neural subnetworks; predict an empirical distribution of labels in the training data for the new task; train each of the first plurality of networks using the training data for the new task; and replace a plurality of the first plurality of neural subnetworks with the retrained neural subnetworks. Implementations may include a processing device comprising any one or more of the features described above, wherein the replacement may include: replacing a plurality of the first plurality of neural subnetworks when there exists a greater than a maximum number of pre-trained neural subnetworks. Implementations may include a processing device comprising any one or more of the features described above, wherein replacing at least a subset of the first plurality of neural subnetworks for the new task may include: replacing neural subnetworks with general performance determined relative to the training data for the new task. The implementation may include a processing device comprising any one or more of the features described above, wherein the processor executes instructions to: create a second plurality of neural subnetworks having a second larger number of multidimensional nodes by superimposing non-zero weights of the first plurality of neural subnetworks; connect each of the first plurality of neural subnetworks such that each of the first plurality of neural subnetworks and the second plurality of neural subnetworks is connected to a selective node of another of the first plurality of neural subnetworks, the selective node being smaller than all nodes of the other of the plurality of neural subnetworks, such that the plurality of neural subnetworks are arranged in a certain level of neural subnetwork, the connected selective node creating at least two levels of recursive connections of the first plurality of neural subnetworks.

[0008] One general aspect includes a non-transitory computer-readable medium storing computer instructions for training a neural network in such a way as to train a plurality of neural subnetworks, each having a first number of multidimensional nodes. The instructions cause one or more processors to perform the training in such a way as to: instantiate a first plurality of pre-trained neural subnetworks, each having a first number of multidimensional nodes, at least some of which have non-zero weights; scale up a plurality of the first plurality of pre-trained neural subnetworks to have a second larger number of multidimensional nodes, such that each of the first plurality of pre-trained neural subnetworks has a sparse number of non-zero weights associated with the second larger number of multidimensional nodes; create a second plurality of neural subnetworks having the second larger number of multidimensional nodes by superimposing the non-zero weights of the first plurality of neural subnetworks into the second plurality of neural subnetworks; scale up a plurality of the second plurality of neural subnetworks to have a third number of multidimensional nodes, such that a plurality of the second plurality of neural subnetworks has a sparse number of non-zero weights associated with the third number of multidimensional nodes; and create the neural network by superimposing non-zero weights into the multidimensional nodes of a plurality of neural networks in the third plurality of networks. The instructions also cause the one or more processors to receive data for a first task to be computed by the neural network, and cause the one or more processors to compute the task data to generate a solution for the first task from the neural network.

[0009] The non-transient computer-readable medium may include any of the features described above, and further includes the processor execution instructions to: retrain the neural network for a new task by replacing at least a subset of the first plurality of neural subnetworks for the new task. The non-transient computer-readable medium may include any of the features described above, and further includes the processor execution instructions to retrain the neural network for the new task by executing the instructions to: compute correlation parameters among the trained first plurality of neural subnetworks; predict an empirical distribution of labels in the training data for the new task based on the first task; train each of the first plurality of networks using the training data for the new task; and replace a plurality of the first plurality of neural subnetworks with the retrained neural subnetworks. The non-transient computer-readable medium may include any of the features described above, and further includes the processor execution instructions to: replace a plurality of the first plurality of neural subnetworks when there is a greater than maximum number of pre-trained neural subnetworks. The non-transient computer-readable medium may include any of the features described above, and further includes the processor execution instructions to: replace neural subnetworks with general performance determined relative to the training data for the new task. The non-transitory computer-readable medium may include any of the above features, and also includes the processor executing instructions to: connect each of the first plurality of neural subnetworks such that each of the first plurality of neural subnetworks and the second plurality of neural subnetworks is connected to a selective node of the other of the first plurality of neural subnetworks and the second plurality of neural subnetworks, the selective node being less than all nodes of the first plurality of networks and the second plurality of networks, such that a plurality of the first plurality of neural subnetworks and the second plurality of neural subnetworks are arranged in a certain level of neural subnetwork, the connection creating at least two levels of recursive connections between the first plurality of neural subnetworks and the second plurality of neural subnetworks.

[0010] This invention provides a brief overview of some concepts, which will be further described in the specific embodiments. This invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter. The subject matter claimed in the patent application is not limited to solutions to any or all of the problems mentioned in the background art. Attached Figure Description

[0011] Various aspects of the invention are shown by way of example and are not limited to the accompanying drawings, in which the same reference numerals denote the same or similar elements.

[0012] Figure 1 illustrates a method for a prior art process used to train a large neural network;

[0013] Figure 2 A flowchart outlining the methods for performing the described tasks is shown;

[0014] Figure 3 A schematic diagram showing multi-level nested and superimposed subnetworks to create large neural networks is shown;

[0015] Figure 4 The connection between a single neural network node and a supernode is illustrated graphically.

[0016] Figure 5 It shows in Figure 2 The flowchart of the corresponding steps performed in step 225;

[0017] Figure 6 A flowchart illustrating the updating of one or more subnetworks is shown;

[0018] Figure 7 A block diagram of a processing apparatus that can be used to implement various embodiments is shown. Detailed Implementation

[0019] This invention and its embodiments relate to a novel method for training a large neural network using a number of pre-trained smaller neural networks. The pre-trained smaller neural networks can be considered as subnetworks of a larger neural network. This invention provides a large-scale neural network defined by a network designer, where multiple pre-existing pre-trained smaller neural networks are reused to create a large neural network through multi-level stacking. Each of the pre-trained neural networks is scaled up to produce a larger sparse neural network, whose values ​​are then superimposed onto a larger neural network for a defined task. The pre-trained neural networks can be created from existing available neural networks that have been trained using labeled training data associated with a specific task. Once scaled up, nodes with sparse values ​​in a first pre-trained network are determined for each of the pre-trained neural networks. This can be achieved by stacking the non-zero weights of the scaled-up pre-trained neural networks onto a larger network, creating a neural network with a large number of multi-dimensional nodes. By replacing and / or retraining one of the subnetworks used to create the large neural network, the larger neural network can be adapted for use in different tasks.

[0020] Neural networks can take many different forms depending on the types of operations performed within them. A neural network consists of an input layer and an output layer, with multiple intermediate hidden layers. Most neural networks perform mathematical operations on the input data through a series of computational (hidden) layers with multiple computational nodes, each node being trained using training data.

[0021] Each node in a neural network computes an output value by applying a specific function to the input values ​​from the previous layer. The function applied to the input values ​​is determined by a weight vector and biases. In a neural network, learning occurs by iteratively adjusting these biases and weights. The weight vector and biases are called filters and represent specific features of the input (e.g., a specific shape).

[0022] A layer in an artificial neural network can be represented as a set of interconnected nodes or artificial neurons, indicated by circles, and a set of connections from the output of one artificial neuron to the input of another. The nodes or artificial neurons / synapses of an artificial neural network are implemented by the processing system as mathematical functions that receive one or more inputs and sum them to produce an output. Typically, each input is individually weighted, and this summation is implemented through the node's mathematical function to provide the node's output. Nodes and their connections typically have weights that adjust as the learning process progresses. Nodes are often aggregated into layers. Different layers can perform different types of transformations on their inputs. Signals may be transmitted from the first layer (input layer) to the last layer (output layer) after multiple traversals of the layers.

[0023] Artificial neural networks are "trained" by providing input and then examining and correcting the output. For example, a neural network trained to identify dog ​​breeds processes a set of images and calculates the probability that a dog in the images is of a certain breed. Users can view the results, select the probability the neural network should display (above a certain threshold, etc.), and receive suggested labels. Each of these mathematical operations is considered a layer in itself, and complex neural networks have multiple layers. Due to the depth provided by numerous intermediate or hidden layers, neural networks can model complex nonlinear relationships during training.

[0024] There are many publicly available pre-trained neural networks that can be downloaded and used for free. Each of these pre-trained neural networks can operate on a processing device and has been trained to perform a specific task. For example, several pre-trained networks, such as GoogLeNet and SqueezeNet, have been trained on the ImageNet dataset. These are just two examples of pre-trained networks, and it should be understood that there are networks available for tasks other than image recognition that can be trained on datasets other than ImageNet.

[0025] According to the present invention, a pre-trained network with a finite number of nodes is used as a building block for creating a large trained neural network.

[0026] Figure 1 shows a flowchart illustrating one embodiment of a process for training a conventional neural network to generate a set of weights. The training process can be performed by one or more processing devices, including cloud-based processing devices, thereby allowing access to additional or more powerful processing. In step 100, training input is received, such as a set of images in the example above (e.g., the image input in Figure 1). The training input can be applied to a first network task, such as the example of identifying dog breeds described above. In step 120, the input is propagated by connecting it to subsequent layers, current filters, or layers of a set of weights. For example, the output of each layer can then be received in a subsequent layer so that the value received as the output from one layer is used as the input to the next layer. The input from the first layer is propagated in this way through all intermediate or hidden layers until the input reaches the network output. Once trained, in step 130, the neural network can acquire test data and provide an output. In the dog breed example in the previous paragraph, the input is image data of multiple dogs, and the intermediate layers use the current weight values ​​to calculate the probability that the dog in the image is a certain breed, returning the suggested dog breed label in step 130. The user can then check the accuracy of the results so that the training system can select the probabilities the neural network should return and determine whether the current set of weights provides sufficiently accurate labels; if so, training is complete. If the results are not accurate enough, the network can be retrained by repeating steps 100 and 120. However, if a different network task is required in step 140, a new set of training data must be provided in step 150, and the training process must be repeated for the new training data in step 120. Then, in step 130, new training data can be fed into the network for output to a new task. When no new task is available, the training process ends in step 160.

[0027] Neural networks are typically feedforward networks, where data flows from the input layer through intermediate layers to the output layer without looping back. First, in the supervised learning training phase, as shown in Figure 1, the neural network creates mappings of virtual neurons and assigns random numerical values, or "weights," to the connections between them. The weights are multiplied by the input and the result is returned as the output. If the network cannot accurately identify a particular pattern, the algorithm adjusts the weights. In this way, the algorithm can make certain parameters have a greater influence (by increasing the corresponding weight) or a smaller influence (by decreasing the weight), adjusting the weights accordingly until a set of weights is determined that provides sufficiently correct mathematical operations to adequately process the data.

[0028] Figure 2A flowchart illustrating one embodiment of the process for training a neural network provided by the present invention is shown. In step 210, a pre-trained neural network is accessed and utilized, instead of training a single large neural network using training data. Typically, such pre-trained neural networks are publicly available and have training data input for a specific task. Compared to large-scale trainable networks, such pre-trained networks are smaller and generally more task-oriented. The pre-trained neural network used herein has a certain number of nodes (N), which are only a fraction of the number of nodes (M) that a user of the present invention can create in a large neural network.

[0029] In large neural networks, each pre-trained neural network with N nodes can be viewed as one of multiple (e.g., "first" multiple) sub-networks nested at multiple levels within the larger network. In embodiments of the present invention, "N" can be on the order of hundreds or thousands of nodes. In another aspect, in step 220, nodes at different levels of each pre-trained network (and the sub-networks created from the pre-trained network) can selectively connect to other nodes at different levels to reduce the number of direct connections between nodes at different levels. In one embodiment, step 220 is optional and does not need to be performed. The following is in conjunction with... Figure 3 and Figure 4 This multi-level nesting will be described further.

[0030] A sparse neural network can be viewed as a matrix with a large percentage of zero values ​​in the weighted average of the network nodes; conversely, a dense network has multiple non-zero weights. In step 225, for a given size of the desired large neural network with M nodes, each of the pre-trained neural networks can be scaled up to the size of the large neural network, thereby creating a second plurality of neural networks. In embodiments of the present invention, “M” can be on the order of millions or billions of nodes. Typically, this second plurality of neural networks includes sparse networks (even if the scaled-up pre-trained networks are dense networks). That is, for each pre-trained network with N nodes, these nodes can be conceptually identified as a two-dimensional or three-dimensional matrix of computation nodes; for a given desired neural network with M nodes (also used as a two-dimensional or three-dimensional matrix of computation nodes), each pre-trained network can be “scaled up” to a certain number of nodes (M) and the matrix size of the large network. When scaling up smaller networks, each scaled-up pre-trained neural network now includes a sparse neural network. Since the scaled-up pre-trained networks are sparse networks, they can be combined into the desired large neural network by stacking.

[0031] Using an image recognition example, the multiple pre-trained neural networks collected in step 210 can be scaled up and then superimposed onto a large neural network with M nodes, the large network having training weights that can be used to solve a given image recognition problem (e.g., dog breed recognition).

[0032] Once trained, in step 230, the neural network can acquire task data and provide output. In the previous dog breed example, the input was image data of multiple dogs, and the intermediate layer used weight values ​​to calculate the probability that a dog in an image is of a certain breed, returning the suggested dog breed label in step 230. The user can then check the accuracy of the results so that the training system can select the probabilities the neural network should return and determine whether the current set of weights provides a sufficiently accurate label; if so, training is complete.

[0033] When a new task is presented in step 240, the neural network training can be updated by updating one or more of the smaller-scale (N-node) network, as follows: Figure 6 As stated above.

[0034] Figure 3 A schematic block diagram of multi-level nested and stacked subnetworks to create a large trained neural network is illustrated graphically. As previously mentioned, neural networks typically consist of multiple layers of nodes, including an input layer, an output layer, and one or more hidden layers. Nodes in the layers are connected to form a network of interconnected nodes. These connections between nodes are used to transmit signals from one node to another. As the number of layers in a neural network increases, interconnecting each node in one layer to each node in other layers can be problematic and may affect network performance and increase complexity. As discussed above in step 220, selectively connecting different layers of the network provides multi-level nesting of the network, which improves the efficiency of the present invention. Figure 3 Describe the process of step 220.

[0035] Figure 3 A three-layer node (layer 1, layer 2, and layer 3) is shown, with multiple neural networks "nested" in subsequent layers. Figure 3Multiple ("X") pre-trained networks 300a-300x are shown, each with N nodes and conceptually providing a first level ("Layer 1") of multi-level nesting in subnetworks. The pre-trained networks 300a-300x can be viewed as matrices of two dimensions (A x B) or three dimensions (A x B x C). In one embodiment, each node in each pre-trained matrix 300a-300x can be coupled to every other node in each matrix. For simplicity, each pre-trained matrix 300a-300x is shown as a two-dimensional 3x4 matrix. The first multi-level nesting produces a "Y" subnetwork (320a, ..., 302y) with 9x16 nodes in this example, and a third-level neural network 325m with 27x64 nodes (i.e., "M" nodes in this example). It should be understood that the array shown at 325m is merely illustrative.

[0036] In some neural networks, each node in the network can connect to every other node in the network, regardless of which level the node is operating at. According to the present invention, multi-level nesting involves selectively connecting nodes of each smaller subnetwork (including the pre-trained network of level 1) to nodes in subnetworks of different levels. Thus, for example, network 300a has a connection 350 to a representative node in network 320a of layer 2, and network 300n has a connection 352 to a representative node in network 320y of layer 2. Similarly, network 320a has a connection 354 to a representative node in network 325m of layer 3.

[0037] This is Figure 4 It is shown graphically in the middle. Figure 4 A 2x2 pre-trained network 400a is shown, where each node is connected to every other node in network 400a, and one node in this pre-trained network is coupled to a supernode 450a. Each supernode may have one or more pre-trained networks 400 connected to it. It should be understood that each of the supernodes 450a-450h may have one or more pre-trained networks selectively connected to it.

[0038] return Figure 3 The connections of each pre-trained network can be controlled using virtual crossover switches 302a-302x. Therefore, each subnet is connected by a hierarchical crossover switch (or other interconnect topology) to form connections hierarchically within a larger network. Thus, the switch functionality controls weights, neurons, filters, channels, amplitudes, gradients, and activations.

[0039] Typically, the internal connections of a virtual crossover switch can be configured to be selectively turned on or off to represent a pruning network (a small network that performs as well as a large network for one type of task), where the same connections can be turned off or on for another pruning network. Figure 3 In this approach, the weights of the best-effort pruning network are superimposed based on the similarity of their weight distributions. In the basic example where each weight is represented by a 4-bit binary value, the probability of overlapping weight distributions among small subnets with 175 billion parameters is high. Considering that 175 billion in a 1000x1000 matrix divided by 24 values ​​yields approximately 2.7K matches.

[0040] It should be recognized that the multi-level nesting technique described above does not need to be used in every embodiment of the technology described herein. In alternative embodiments, all nodes at each level are connected to each other; in other embodiments, nodes at all levels are connected.

[0041] In another aspect of the invention, a large-scale densely trained neural network is created by superimposing amplified pre-trained networks. Figure 5 It shows Figure 2 An embodiment of step 220. Combined with... Figure 3 To describe the lower half Figure 5 .like Figure 5 As shown, step 225 can occur directly after step 210 or after step 220. Initially, in step 420, each of the first plurality of pre-trained subnetworks is scaled up to the required number of nodes in a larger network (i.e., M nodes) – a large neural network. Scaling each pre-trained neural network may include scaling on the same dimension as the required large network of M nodes or any other suitable dimension. Once scaled to the larger scale, each of the plurality of smaller pre-trained networks includes a sparsely padded subnetwork. In step 430, for each of the scaled-up networks, the method determines the nodes with values ​​and the nodes without values ​​in the scaled-up network. In step 440, the method creates a second plurality of networks with M multidimensional nodes by superimposing a plurality of the first plurality of padded nodes onto the nodes of the larger network. Finally, in step 460, a neural network with M multidimensional nodes is created by superimposing a plurality of the second plurality of networks determined to have weight values ​​(positioning the weight values ​​within the nodes of the larger network).

[0042] Figure 3 The process is illustrated graphically by connections 502, 504, and 506, which show how individual scaling nodes are positioned within larger scaling networks 362, 364, and 366, resulting in an M-node network 390. It should be understood that... Figure 3The number of nodes shown is only for a 4x4 network, but the scaling factor for each of the pre-trained subnetworks can be much larger, resulting in an M-node network that is even larger. Figure 3 Only a small portion of networks 362, 364, 366, and 390 is shown; it should be understood that, in this example, network 390 can have the same number of M nodes as network 325m.

[0043] like Figure 2 As shown, once a large neural network is created, new tasks can be proposed that require modification or retraining of the large neural network. Figure 6 This illustrates the method used to update the neural network. Figure 2 One embodiment of step 250. Initially, in step 610, a method collects pre-trained sub-networks for a new task and pre-existing training data. This training data includes labeled data labeled with one or more tags that identify certain attributes or features, classifications, or included objects. In step 620, correlation parameters between each of the pre-trained sub-networks and the pre-existing training data are determined. This can determine whether the pre-trained network performs well, poorly, or generally well for the new task. In one embodiment, a maximum correlation algorithm can be used to determine the correlation parameters between the existing pre-trained networks and the new task training data. In step 630, the method predicts an empirical distribution of the training data class labels for the new task based on the existing training task. If the number of pre-trained networks exceeds a specified maximum value, the correlation prediction is used to select a pre-trained network. In step 640, if necessary, one or more new sub-networks are trained using the new task training data; in step 645, the newly trained one or more sub-networks are pruned. In step 640, if one or more of the pre-trained networks exhibit general performance characteristics, further training may be necessary. In this context, general performance is defined as the network's performance on the task being neither good nor bad. In step 645, pruning is a compression method involving removing unnecessary weights or nodes from the trained network. In step 650, it is determined whether a newly trained subnetwork can be added to the pre-trained network, which can be used to build a newly trained M-node network for the new task. This determination is based on the network designer's specifications, who has determined a maximum number of pre-trained networks based on any number of given factors, including network performance, processing power, and other constraints. If the maximum allowed number of pre-trained networks is not reached in step 650, then in step 670, multiple pre-trained networks can be updated using newly trained networks. If the maximum allowed number of pre-trained networks has been reached, then in step 660, the method removes one or more networks with general performance. In this case, networks with general performance are those whose performance on their pre-training task is neither good nor bad.

[0044] Figure 7 A block diagram of a network device 700 that can be used to implement various embodiments is shown. A particular network device may utilize all or only a portion of the components shown, and the degree of integration between devices may vary. Furthermore, network device 700 may include multiple instances of components, such as multiple processing units, processors, memory, transmitters, receivers, etc. Network device 700 may include a central processing unit (CPU) 710, memory 720, mass storage device 730, I / O interface 760, and network interface 750 connected to a bus 770. Bus 770 may be one or more of any type of bus architecture, including a storage bus or storage controller, peripheral bus, etc.

[0045] CPU 710 may include any type of electronic data processor. Memory 720 may include any type of system memory, such as static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or combinations thereof. In an embodiment, memory 720 may include ROM used at power-on and DRAM used to store programs and data during program execution.

[0046] In one embodiment, memory 720 is non-transient. In one embodiment of a network device used to create the neural network described herein, memory 720 may include one or more of training engine 720A, pruning engine 720B, overlay engine 720C, training data 720D, subnetwork 720E, and task execution engine 720F.

[0047] Training engine 720A includes code executable by CPU 710 to perform the training of the neural network described herein. Pruning engine 720B includes code executable by CPU to perform network pruning as described herein. Overlay engine 720C includes code executable by CPU to perform the overlay of network nodes with weights as described herein. Training data 720D may include training data for an existing or new task that can be utilized by the CPU and training engine 720A to perform the training of the neural network described herein. Subnetwork 720E may include code executable by CPU to run and instantiate each of the pre-trained networks or other subnetworks described herein. Task execution engine 720F may include code executable by a processor to present tasks to a large neural network to obtain results, as described herein.

[0048] Mass storage device 730 may include any type of storage device for storing data, programs, and other information, and enabling access to this data, programs, and other information via bus 770. For example, mass storage device 730 may include one or more of solid-state drives, hard disk drives, disk drives, optical disk drives, etc. Mass storage device 730 may include training data and executable code that can be transferred to memory 720 to implement any particular engine or data described herein.

[0049] Mass storage devices can also store any components described as being in or shown in memory 720 for read by a CPU and executed in memory 720. Mass storage devices can include executable code in a non-volatile form for each component shown in memory 720. Mass storage device 730 can be a non-transitory computer-readable medium, which includes all types of computer-readable media, including magnetic storage media, optical storage media, and solid-state storage media, but specifically excluding signals. It should be understood that software can be installed in and sold with network devices. Alternatively, software can be obtained and loaded into network devices, including via optical disc media or by any means from a network or distribution system, including, for example, obtaining software from a server owned by the software creator or from a server not owned by the software creator but used by the software creator. For example, software can be stored on a server for distribution over the Internet.

[0050] Network device 700 also includes one or more network interfaces 750, which may include wired links such as Ethernet cables and / or wireless links for accessing nodes or one or more networks 780. Network interface 750 allows network device 700 to communicate with remote units via network 780. For example, network interface 750 may provide wireless communication via one or more transmitter / transmit antennas and one or more receiver / receive antennas. In embodiments, network device 700 is coupled to a local area network (LAN) or wide area network (WAN) 780 for data processing and communication with other processing units, the Internet, remote storage facilities, and other remote devices.

[0051] This invention provides a large-scale neural network defined by the network designer, which reuses multiple pre-existing, pre-trained smaller neural networks to create a large neural network through multi-level stacking. Therefore, this network can provide the same performance as a custom-trained large neural network, with lower energy consumption and higher flexibility. The large neural network is continuously updated through learning by pruning pre-trained subnetworks and adding new subnetworks, using new tasks to train these new subnetworks. Given a defined number of subnetworks, networks with general performance can be removed.

[0052] For the purposes of this document, it should be noted that the dimensions of the various features depicted in the accompanying drawings are not necessarily drawn to scale.

[0053] For the purposes of this document, the description may use the terms "embodiment," "one embodiment," "some embodiments," or "another embodiment" to describe different or the same embodiments.

[0054] For the purposes of this document, a connection can be a direct connection or an indirect connection (e.g., through one or more other components). In some cases, when one element is said to be connected to or coupled to another element, that element can be directly connected to the other element or indirectly connected to the other element through an intermediate element. When one element is said to be directly connected to another element, there is no intermediate element between that element and the other element. If two devices are directly or indirectly connected so that they can transmit electronic signals between them, then they are in “communication”.

[0055] While the invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations can be made to the invention without departing from its scope. Therefore, the specification and drawings are to be regarded only as a description of the invention as defined by the appended claims, and are intended to cover any and all modifications, alterations, combinations, or equivalents falling within the scope of the invention.

[0056] For purposes of illustration and description, a detailed description has been presented above, but this description is not intended to be exhaustive or to limit the subject matter claimed herein to one or more precise forms disclosed. Many modifications and variations can be made based on the above guidance. The various embodiments described were chosen to better explain the principles of the disclosed technology and its practical application, thereby enabling those skilled in the art to better utilize the techniques of the various embodiments and various modifications suitable for the intended particular use. The scope of the invention is intended to be defined by the appended claims.

Claims

1. A computer-implemented method for training a neural network, characterized in that, The method includes: Instantiate a first plurality of pre-trained neural subnetworks, each pre-trained neural subnetwork having a first number of N multi-dimensional nodes, at least some of the multi-dimensional nodes having non-zero weights; Each of the first plurality of pre-trained neural subnetworks is amplified to have a second larger number of M multi-dimensional nodes, such that each amplified pre-trained neural network in the first plurality of pre-trained neural subnetworks comprises a sparse neural network in which the weights of the M multi-dimensional nodes in the sparse neural network have a large proportion of zero values, and each amplified pre-trained neural network in the first plurality of pre-trained neural subnetworks has a sparse number of non-zero weights associated with the second larger number of M multi-dimensional nodes, wherein the N multi-dimensional nodes are a part of the M multi-dimensional nodes; By superimposing the non-zero weights of each amplified pre-trained neural network of the first plurality of pre-trained neural sub-networks, a second plurality of neural sub-networks with M multidimensional nodes are created. The neural network with multidimensional nodes is created by superimposing the non-zero weights of the second plurality of neural subnetworks onto the nodes of the neural network; Receive data for a first task to be computed by the neural network, wherein the data includes image data, and the first task includes an image recognition task; Perform the first task to generate a solution for the first task from the neural network.

2. The method according to claim 1, characterized in that, include: The neural network is retrained for the new task by replacing at least a subset of the first plurality of pre-trained neural subnetworks.

3. The method according to claim 2, characterized in that, The retraining also includes retraining the neural network for the new task in the following manner: Calculate the correlation parameters between the first and second neural subnetworks trained and the training data for the new task; Based on the first task, predict the empirical distribution of labels in the training data of the new task; Each of the first plurality of neural subnetworks is trained using the training data from the new task; Replace multiple of the first plurality of pre-trained neural subnetworks with retrained neural subnetworks.

4. The method according to claim 2 or 3, characterized in that, The replacement includes replacing multiple of the first plurality of pre-trained neural subnetworks when there is more than a maximum number of pre-trained neural subnetworks.

5. The method according to claim 3, characterized in that, The replacement includes replacing the neural subnetwork with general performance determined relative to the training data for the new task.

6. The method according to any one of claims 1 to 3, characterized in that, The method includes: connecting each of the first plurality of pre-trained neural subnetworks such that each of the first plurality of pre-trained neural subnetworks is connected to a selective node of another of the first plurality of pre-trained neural subnetworks, the selective node being smaller than all nodes in a plurality of nodes of the other of the first plurality of pre-trained neural subnetworks arranged in a first-level neural subnetwork comprising a subset of the first plurality of pre-trained neural subnetworks.

7. The method according to claim 6, characterized in that, The method further includes connecting each of the subsets of the first plurality of pre-trained neural subnetworks in the first level to a selective node in the nodes of the second plurality of neural subnetworks in the second level neural subnetwork that includes the subset of the first level.

8. A processing device, characterized in that, include: Includes non-transitory memory for instructions; One or more processors communicating with the memory, wherein the one or more processors create a neural network by executing the instructions, such that: Instantiate at least a first plurality of pre-trained neural subnetworks, each pre-trained neural subnetwork having a first number of N multi-dimensional nodes, at least some of the multi-dimensional nodes having non-zero weights; Each of the first plurality of pre-trained neural subnetworks is amplified to have a second larger number of M multi-dimensional nodes, such that each amplified pre-trained neural network in the first plurality of pre-trained neural subnetworks comprises a sparse neural network in which the weights of the M multi-dimensional nodes in the sparse neural network have a large proportion of zero values, and each amplified pre-trained neural network in the first plurality of pre-trained neural subnetworks has a sparse number of non-zero weights associated with the second larger number of M multi-dimensional nodes, wherein the N multi-dimensional nodes are a part of the M multi-dimensional nodes; By superimposing the non-zero weights of each amplified pre-trained neural network of the first plurality of pre-trained neural sub-networks, a second plurality of neural sub-networks with M multidimensional nodes are created. The neural network with multidimensional nodes is created by superimposing the non-zero weights of the second plurality of neural subnetworks onto the nodes of the neural network; The processing device is configured to: perform a first task including an image recognition task; receive data for the first task to be computed by the neural network, the data including image data; and perform the first task to generate a solution for the first task from the neural network.

9. The processing apparatus according to claim 8, characterized in that, The processor executes instructions to: retrain the neural network for the new task by replacing at least a subset of the first plurality of pre-trained neural subnetworks for the new task.

10. The processing apparatus according to claim 9, characterized in that, The retraining also includes: retraining the neural network for the new task by executing instructions so that: Calculate the correlation parameters between the first and second neural subnetworks trained and the training data for the new task; Based on the new task, predict the empirical distribution of labels in the training data for the new task; Each of the first plurality of neural subnetworks is trained using the training data from the new task; Replace multiple of the first plurality of pre-trained neural subnetworks with retrained neural subnetworks.

11. The processing apparatus according to claim 9 or 10, characterized in that, The replacement includes replacing multiple of the first plurality of pre-trained neural subnetworks when there is more than a maximum number of pre-trained neural subnetworks.

12. The processing apparatus according to claim 10, characterized in that, Replacing at least a subset of the first plurality of neural subnetworks for the new task includes replacing neural subnetworks with general performance determined relative to the training data for the new task.

13. The processing apparatus according to any one of claims 8 to 10, characterized in that, The processor executes instructions so as to: Each of the first plurality of pre-trained neural subnetworks is connected such that each of the first plurality of pre-trained neural subnetworks is connected to a selective node of another of the first plurality of pre-trained neural subnetworks, the selective node being smaller than all nodes in a plurality of nodes of the other of the first plurality of pre-trained neural subnetworks arranged in a first-level neural subnetwork comprising a subset of the first plurality of pre-trained neural subnetworks.

14. The processing apparatus according to claim 13, characterized in that, The processor executes instructions so as to: Each of the subsets of the first plurality of pre-trained neural subnetworks in the first level is connected to a selective node in the second plurality of neural subnetworks in the second level neural subnetwork that includes the subsets of the first level.

15. A non-transitory computer-readable medium, characterized in that, The computer instructions for training a neural network are stored, which, when executed by one or more processors, cause the one or more processors to perform the method as described in any one of claims 1 to 7.

Images