Systems and methods for neural architecture search
Through the neural architecture search method, the preferred model is identified and the weight update strategy is optimized, the structure selection and training problems in neural network design are solved, training efficiency and performance are improved, and more efficient resource utilization and task execution capabilities are achieved.
Patent Information
- Application Number
- CN201910926760.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-01-31
- Filing Date
- 2019-09-27
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-09-27
AI Technical Summary
In the design process of neural networks, it is difficult to choose appropriate neuronal structures and weight update strategies, resulting in long training time and large resource consumption, affecting the efficiency and performance of the network.
Using neural architecture search method, by identifying preferred models, using the analysis and training of multiple candidate models, controlling the importance of weight updates, based on Fisher information and loss functions, the architecture and weight update strategies of neural networks are optimized, and provided for execution. The neural network for the selected task.
It improves the training efficiency and performance of neural networks, reduces time-dependent bias, provides a more reliable preferred model, reduces resource consumption, and enhances the network's ability when performing tasks.
Smart Images

Figure CN110956260B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of neural architecture search. In particular, the present disclosure relates to systems and methods for designing neural networks. Background Art
[0002] By providing increased computing power and access to more and more suitable training data, machine learning techniques have found increasing application, providing improved solutions to many technical problems. In particular, the use of neural networks has become increasingly common as a method that provides elegant and effective solutions to many such technical problems.
[0003] Neural networks typically have a structure that includes multiple interconnected neurons. Each neuron in the network is arranged to receive input data, and the neuron scales the received data and performs a function on it before providing certain output data. Subsequently, the output data can be fed into subsequent neurons as input data for subsequent neurons. Then, these subsequent neurons can also scale the data before performing further functions on it. This process can be repeated for multiple different neurons until the final output is determined. These networks can be trained by using input data with corresponding known outputs, wherein training involves repeatedly operating the network on the data to provide an output and performing a comparison with the output. That is, the output of one input data is compared with the known output of the input data. Then, the weights of the network (which affects the scaling of the input data) can be updated based on the results of the above comparison, so that the network "learns" to provide the correct output when given input data is provided. By repeating this process many times using suitable training data, studies have shown that neural networks have a strong universal approach ability (for example, neural networks can solve the problems they are trained for).
[0004] There are a large number of different structures for neural networks. For example, a feedforward neural network includes multiple layers of neurons. In a feedforward network, the output of a neuron in a layer is fed into each neuron in the next layer, so that each neuron receives input from the neurons in the layer above it. Different structures of neural networks include recurrent neural networks, in which the input of the previous layer can also be fed into the next layer as input data. Another type of network is a residual network, which has a similar arrangement to a simple feedforward network, except that the network may include branches that connect the output of a neuron or multiple layers of neurons to neurons in layers other than the next layer. Different types of networks have been found to be particularly effective for solving certain types of technical problems. Therefore, when designing a neural network, it may be helpful to choose a type of network based on the problem that the network is intended to solve.
[0005] While the choice of neural network type can have an important impact on the overall ability of the network to perform a selected task, the choice of constituent neuron types can also have a significant impact on the ability of a network to perform a selected task. For example, a well-trained network composed of a first choice of neurons may not perform as well as a poorly trained network with a different, better choice of neurons. Since the training of a network can be very time-consuming, the initial selection of appropriate neurons can significantly reduce the time and resources spent on providing a neural network that satisfactorily performs the selected task. In addition to increasing the efficiency of training neural networks, the ability to select the appropriate neuron structure can make the difference between providing a neural network that can solve a technical problem and one that cannot.
[0006] Embodiments of the present disclosure may address these and other technical problems by providing a neural architecture search system and method to provide a neural network for solving or contributing to a selected technical problem. Summary of the invention
[0007] Various aspects of the disclosure are set out in the independent claims, with optional features set out in the dependent claims. Various aspects of the invention may be provided in combination with each other, and features of one aspect may be applied to other aspects.
[0008] In one aspect, a computer-implemented method of neural architecture search is provided to provide a neural network for performing a selected task. The method comprises the following steps:
[0009] i. Obtain a computational graph, the computational graph comprising: a plurality of nodes connected by a plurality of edges, wherein each node and / or edge is associated with a corresponding one of a plurality of weights, and is used to scale input data provided to (e.g., along the edge) each of the nodes according to the corresponding weight. Each node is used to: receive at least one input data from a previous node via an edge connecting each node to the previous node; perform an operation on the input data to provide output data, wherein each input data is scaled according to a corresponding weight associated with the node and / or edge; and provide the output data to a subsequent node in the graph via the edge. The computational graph defines a first candidate model and a second candidate model, each candidate model being a subgraph in the computational graph, the subgraph having nodes, edges, and weights selected from the plurality of nodes, the plurality of edges, and the associated plurality of weights, wherein some of the selected nodes, edges, and weights are shared between the first model and the second model. For example, it should be understood that the first candidate model and / or the second candidate model may have some selections of nodes, edges, and weights that are not in the other candidate model, e.g., not all nodes, edges, and weights are shared between the two models.
[0010] ii. Based on training the first model to perform the selected task, the weight of the first model is updated.
[0011] iii. Based on training the second model to perform the same selected task as the first model, updating the weights of the second model. Updating the weights of the second model includes updating some weights shared between the first model and the second model updated in step (ii). Updating the shared weights is controlled based on an indication of the importance of the nodes and / or edges associated with the weights to the trained first candidate model. For example, for weights associated with nodes and / or edges that are not important to the first model, the update to the weights may be greater than when the associated nodes and edges are more important to the first model.
[0012] iv. Identify a preferred model of the neural network, the preferred model comprising nodes, edges, and associated weights selected from the computational graph. The preferred model is determined based on an analysis of the first training candidate model and the second training candidate model.
[0013] v. Providing a neural network for performing the above selected tasks based on the above preferred model.
[0014] Embodiments of the present disclosure may improve the identification of a preferred model for a neural network for performing a selected task. Controlling the update of shared weights based on an indication of importance may reduce the magnitude of updates to weights associated with important features of a first model. For example, when training a second model, weights that are less important to a first model may be changed more than weights that are more important to the first model. Then, when identifying a preferred model for a neural network, an analysis of the first and second training candidate models may be more representative of the model's ability to perform a selected task. For example, a change in the weights of a first model may be less than the change that would otherwise result from the sequential training of the first and second candidate models. This, in turn, may more reliably identify a preferred model as part of a neural architecture search. Thus, an improved neural network for performing a selected task may be provided.
[0015] The neural network used to perform the selected task may include a neural network for at least one of: (i) natural language processing, (ii) image recognition, (iii) classification and / or modeling of physical systems, (iv) data processing, and (v) search result generation. It should be understood that the use of neural networks can be applied to any number of technical problems, for example, it can contribute to technical solutions to any number of technical problems. For example, the network provided by the neural architecture search method described herein can be used to process input data and provide output based on its controllable technical operation.
[0016] The indication of importance may be determined based on an indication of Fisher information for the weights. The indication of importance may be determined for each shared weight; the indication of importance may be determined for all weights. Controlling updates to shared weights based on the indication of importance may include scaling the magnitude of the updates based on the value of the Fisher information for the weight. For example, the maximum possible change for a given weight may be limited based on its Fisher information. The indication of importance may be implemented in a loss function used when training a candidate model.
[0017] An indication of importance can be determined based on identifying a loss in performance associated with a node / edge in a candidate model. For example, an indication of importance for a given corresponding weight can provide an indication of how sensitive the performance of the model is to a notional change in the given weight. A small change in a weight that results in a large drop in performance may indicate that the weight is very important. A suitable loss function can be used to represent the performance of the model. The indication of importance can be defined as the norm of the gradient of the loss function for the given weight. The method may include determining Fisher information for each weight. The Fisher information for each weight can be determined based on a diagonal approximation of the Fisher matrix for the weight. Using the Fisher information can be a good indication of the indication of importance associated with the weight.
[0018] Training the second candidate model includes training using a training loss function that includes a component indicating Fisher information of weights in the second model. The component may indicate Fisher information of weights in the second model, for example, the component indicates the importance of (some) weights in the second model to the first model. If more than two candidate models are used, the indication of Fisher information may be used for all other candidate models that have been trained (in a batch of sampled candidate models). The loss function used may include the following terms:
[0019]
[0020] Among them, θ i Represents each weight, Denotes the weight θ i Fisher information, m j represents the jth candidate model, represents the optimal value of the weight.
[0021] The loss function may include a scaling factor for suppressing exploding gradients associated with any weight in the second model. The scaling factor may indicate the magnitude of a weight, for example, which may suppress any changes to the weight that would cause its magnitude to increase substantially. The scaling factor may include the norm of the weight. For example, the scaling factor term may include:
[0022]
[0023] The loss function may include a term indicating the cross-entropy of the model.
[0024] Before training the second model, the weights of the second model can be initialized based on the trained weights of the first model. This can reduce the time to train each individual model because the trained weights that worked well in the first model are likely to work well in the second model as well. For example, these weights can provide a starting point that is closer to the optimal weights for the second model than using randomly assigned weights.
[0025] Identifying a preferred model of a neural network may include analyzing a first candidate model and a second candidate model using a controller having a strategy based on at least one of (i) reinforcement learning and (ii) evolutionary learning. The controller may be arranged to perform a reinforcement learning analysis and / or an evolutionary learning analysis on a computational graph. For example, the controller itself may include a neural network, such as a recurrent neural network. The controller may include a long short term memory (LSTM) network. The controller may include a plurality of controller parameters. The controller may operate a strategy (e.g., it may perform a selected operation based on a function set forth in the strategy). The controller parameters may affect the strategy so that modifying the parameters may modify the results of the actions performed by the strategy. The controller may be trained by updating the controller parameters. For example, a reward that the controller seeks to maximize may be established, for example, this may be a reduction in variance associated with a baseline mean of the controller.
[0026] The controller can be trained to recommend candidate models, for example, from a selected subset of nodes of the model and possible subgraphs. The controller can be updated (or trained) based on an indication of a preferred model from a subset of candidate models presented to the controller. The performance of the preferred model can be tested using a validation set. Subsequently, the controller can be trained to recommend better candidate models by learning which model worked well previously. In order to train the controller, the controller's strategy can be updated by updating the values of the controller parameters. The controller can provide a set of recommended candidate models, which are then trained sequentially and then analyzed to help identify the preferred model. This process can be performed iteratively.
[0027] Identifying a preferred model can be an iterative process in which multiple candidate models are obtained, then sequentially trained, and then evaluated. Based on the evaluation (e.g., analysis) of the trained models, a controller can be updated. The updated controller can then recommend a subsequent set of candidate models to be subsequently trained and evaluated. The process can be repeated until a preferred model is identified. For example, identifying a preferred model can include:
[0028] i. Determine the best performing model among the two models based on testing the first candidate model and the second candidate model using the validation dataset.
[0029] ii. Keep the value of the trained weights constant and update the controller’s strategy based on the analysis of the best candidate model above.
[0030] iii. Operate the controller with the updated policy to obtain recommendations for the third candidate model and the fourth candidate model. Some nodes, edges, and weights are shared between the third model and the fourth model.
[0031] iv. Perform the selected task based on the training of the third model and update the weights of the third model.
[0032] v. Based on training the fourth model to perform the same selected task as the third model, updating the weights of the fourth model. Updating the weights of the fourth model includes updating some weights shared between the third model and the fourth model. Based on an indication of the importance of the nodes and / or edges associated with the weights to the trained third candidate model, controlling the update of the shared weights.
[0033] vi. Identify a preferred model based on the analysis of the third training candidate model and the fourth training candidate model.
[0034] These steps for identifying a preferred model can be repeated with different recommendation candidate models until at least one of the following is met: (i) a time threshold is reached, and (ii) the difference between the best candidate model and the next recommended candidate model is below a threshold difference. For example, a threshold number of epochs can be used, after which a preferred model can be established. The candidate model recommended by the controller can be compared with the last model evaluated by the controller. The iterative process can stop when the similarity between the two models is above a certain threshold. The best performing recommendation candidate model from the last iteration of the above steps can then be used as a model architecture for a neural network for performing a selected task. Providing a neural network can include providing a neural network having an architecture corresponding to a subgraph of the best performing recommendation candidate. The weights of the nodes in the subgraph can be retrained. For example, the retraining can start from the weights found during the search process, or can start from a set of random weights. The trained neural network can be provided as a neural network for performing a selected task.
[0035] The values of the trained weights may not remain constant. For example, the policy and weights may be updated simultaneously. In such an example, the controller may be configured such that the update to the policy and the update to the weights are independent of each other. For example, the update to the weights may be based on training using a training data set, while the update to the policy may be based on training the policy using a validation data set.
[0036] When minimizing the loss function, stopping criteria can be introduced. For example, these can inhibit overtraining of the model, e.g., training too specifically on its training dataset so that the model has difficulty generalizing to unseen data. Stopping criteria can include using a validation dataset to identify the performance of the model on a separate dataset above a threshold level. For example, stopping criteria can be based on an indication of the accuracy of the model, e.g., once above a certain threshold level, the model can be considered to have been sufficiently trained.
[0037] Each candidate model may be trained multiple times. A first candidate model and a second candidate model may be trained and then analyzed. Thereafter, the two models are trained again and then analyzed again. Subsequent training of the two models may include updating some weights shared between the first and second models, and wherein updating the shared weights is controlled based on an indication of the importance of the nodes and / or edges associated with the weights to the other trained candidate model. Other candidate models may be recommended after a threshold number of iterations, or a preferred candidate model may be identified. For example, each model may be trained over multiple generations.
[0038] Candidate models can be obtained based on selecting nodes, and the selection is made based on the above-mentioned selected task and an indication of the performance of the node on tasks similar to the selected task. The node includes at least one of the following: (i) a neuron, and (ii) a neuron block. For example, if the node includes a neuron block, the internal workings of each neuron block can remain constant during the training of the candidate model. When recommending and training candidate models, the arrangement of blocks (e.g., the number / order / weight between blocks) can be changed. A combination of neurons and neuron blocks can be used. For example, nodes can be selected so that the model includes nodes that have previously performed well on similar tasks. This can make convergence on a preferred model more reliable.
[0039] Operating the controller includes using a reward based on: (i) a component indicating the fitness of the candidate model, and (ii) a component indicating the complexity of the candidate model. The two components of the reward are separated and / or scaled to prevent the reward from being negative.
[0040] The weights may include a plurality of matrices that provide numerical weights for scaling the value of information as it flows through the model. The method may include identifying weights that have a low rate of change in response to training. For example, a second-order derivative may indicate that the weight value does not change much in response to training (e.g., a change below a threshold level), which may indicate that it is close to an optimal value. The gradient associated with the cost function may be monitored for each given weight. This may indicate the sensitivity of the model to that weight. The weights are used to scale the information passed between nodes. The weights may be applied to at least one of: each node and each edge. Applying weights on each edge may make the dimensionality easier to manage. The weights may be stored in the edges.
[0041] Training and analyzing the candidate models can be an iterative process in which the first and second candidate models are trained and analyzed multiple times. Initially, the training of the second candidate model can be performed without each update of the weights being based on an indication of the importance of the nodes and / or edges associated with the weights to the trained first candidate model. The iterative process can then transition to an iteration in which the update of the weights is controlled based on an indication of the importance of the nodes and / or edges associated with the weights to the trained first candidate model. This can reduce any bias towards the earlier trained models.
[0042] Aspects of the present disclosure include a computer program product comprising program instructions for programming a processor to perform the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Some embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which:
[0044] Figure 1 A schematic diagram of an example feed-forward network is shown;
[0045] Figure 2 A schematic diagram showing an example graph including a feed-forward network with two possible node options at the second and third layers;
[0046] Figure 3a Shows from Figure 2 A schematic diagram of an example feed-forward network with available options shown;
[0047] Figure 3b Shows from Figure 2 A schematic diagram of another example feed-forward network with available options shown;
[0048] Figure 4 A flowchart illustrating an example method of neural architecture search;
[0049] Figure 5a and Figure 5b A comparison between the results obtained using a traditional cross entropy loss system and the results obtained using the weight plasticity loss system of the present disclosure is shown;
[0050] Figures 6a to 6f A comparison between the results obtained using a traditional cross entropy loss system and the results obtained using the weight plasticity loss system of the present disclosure is shown.
[0051] In the drawings, like reference numbers are used to indicate like elements. DETAILED DESCRIPTION
[0052] Embodiments of the present disclosure relate to systems and methods for neural architecture search. When faced with the task of having to provide a neural network for performing a selected task, neural architecture search is used to identify a preferred model for such a neural network. The preferred model can be identified based on an analysis of multiple candidate models. Identifying the preferred model involves providing a computational graph consisting of multiple different nodes connected by multiple edges. The computational graph can be considered to provide multiple subgraphs (the subgraphs can be considered "candidate models"). Each subgraph is a selection of some nodes and edges in the computational graph.
[0053] The computation graph includes a plurality of weights that scale information as it flows through the network (e.g., between different nodes). Weights may be applied between nodes so that each piece of input data received at a node may be scaled based on the previous node from which such data was received. A plurality of different subgraphs may be identified, and any two of these different subgraphs may include some of the same nodes and weights. The identified subgraphs represent a subset of all possible structures of nodes and edges in the computation graph.
[0054] The identified subgraphs are sequentially trained using the training data associated with the selected task. The training of the subgraph includes operating the subgraph using each item of training data having a known output associated therewith. For each item of training data, the output of operating the subgraph using the input data is compared with the known output associated with the training data. By comparing the two outputs, the weights of the subgraph can be modified based on the difference between the two outputs (e.g., so that if the network is operated on the data again, the network will provide an output corresponding to the known output). This process is repeated multiple times, and once the ability of the subgraph to perform the selected task converges, for example, the subgraph is determined to be "trained."
[0055] Once the first subgraph is trained, the second subgraph is subsequently trained in much the same manner. However, the disclosed embodiment places restrictions on how the weights of the second subgraph are updated. The first subgraph and the second subgraph may have overlaps in their selected nodes and edges (and therefore associated weights). The weights that also function in the first subgraph (hereinafter referred to as "shared weights") will have their values updated and improved based on training the first subgraph to perform the selected task. When the weights associated with the nodes in the second subgraph are updated in response to the training, the update of the shared weights is controlled based on an indication of the importance of each shared weight to the first subgraph.
[0056] A plurality of different subgraphs may be trained sequentially, and based on analysis of the trained subgraphs, a preferred model of a neural network may be identified. In some examples, a controller (e.g., a reinforcement learning controller) may be used to evaluate the trained subgraphs and to recommend other subgraphs for sampling based on the results of the evaluation of the trained subgraphs. This process may be repeated until a preferred model is identified. The preferred model may be any subgraph, for example, any combination of nodes and edges (and associated weights) of a computational graph. Based on the preferred model, a neural network may be provided for performing the above-described tasks.
[0057] By controlling the updating of weights that are important to at least one identified subgraph trained earlier based on an indication of the importance of the weights to the subgraph trained earlier, embodiments can provide less time-dependent bias to the neural architecture search. For example, weights associated with nodes and edges that are important to the subgraph trained earlier may not change their associated weights much in response to subsequent training of different subgraphs. Models using these nodes and edges may be well suited to provide a neural network for performing a selected task. By controlling the updating of weights associated with these nodes and edges indicated as important, the earlier trained model may not lose the ability to perform the task well despite the subsequent model being trained (thus changing some nodes in the earlier trained model). Therefore, subsequent analysis of the candidate models after training may not be biased towards the later trained models. On the contrary, this can make the earlier trained models still favored (if relevant), so that they (e.g., the weights used by these models and the weights shared with the candidate model that will be the last model) can play a role in the final architecture of the neural network, while they may have been ignored due to the change of important weights before. Therefore, the results of the neural architecture search can provide an improved neural network to perform the selected task.
[0058] Embodiments of the present disclosure will now be described with reference to the accompanying drawings.
[0059] Figure 1 A schematic diagram of a simple feed-forward neural network 100 is shown. It should be understood that the present disclosure applies to all forms of neural networks, Figure 1 Network 100 is merely used as a simple example to illustrate the teachings of the present disclosure.
[0060] Figure 1 The feedforward neural network 100 has four layers: a first layer 10, a second layer 20, a third layer 30, and a fourth layer 40. The first layer 10 includes a node 11. The second layer 20 includes two nodes 21 and 22, and the third layer 30 also has two nodes 31 and 32. The fourth layer 40 has a single node 41. The edges connecting different nodes are Figure 1 For example, it can be seen that one edge connects node 11 to node 21, and another edge connects node 11 to node 22. The network 100 includes multiple weights θi,j , where each θ i,j represents the scaling factor applied to the input to node i received from the output of node j. At each node, the scaling factor is applied to each edge reaching that node. It should be understood that weights can be applied to each edge or each node. In the following, weights are referred to as being associated with a node, but this should not be considered limiting. Instead, the association of a weight with a node means that the node will process input data that has been scaled according to the weight.
[0061] The nodes 11 in the first layer 10 are used to receive input. The input may be some form of data to be processed by the neural network 100. Processing the data may cause the neural network 100 to provide an output. The output may be based on an analysis of the data as performed by the neural network 100. The output may provide an indication of the content of the input data. For example, the input data may include a photograph and the output may provide an indication of what is shown in the photograph.
[0062] The node 11 in the first layer 10 is used to process the received input data based on a specific function specified at the node 11. The node 11 in the first layer 10 is arranged to then provide the processed data to each of the two nodes 21, 22 in the second layer 20. Each of the two nodes 21, 22 in the second layer 20 is used to then process the data received from the first layer 10 based on the specific function specified to the node 21, 22 and the scaling factor applied to the processed data output from the node 11 in the first layer 10.
[0063] Each node 21, 22 in the second layer 20 is used to subsequently output its processed data and provide the data to each node 31, 32 in the third layer 30. Likewise, each of the two nodes 31, 32 in the third layer 30 is used to subsequently process the data received from the second layer 20 based on the specific function assigned to the node and the scaling factor applied to the processed data. Figure 1 It can be seen that each node 31, 32 in the third layer 30 receives input data from two previous nodes 21, 22 (nodes in the second layer 20) instead of receiving input data from one node. Each node 31, 32 in the third layer 30 is used to process its input data and provide its processed output data to the node 41 in the fourth layer 40. The node 41 in the fourth layer 40 is used to subsequently provide an output value based on processing the data received from the third layer 30, and the processing of the data is based on the specific function specified for the node 41 and the scaling factor applied to the processed data.
[0064] It should be understood that a skilled artisan will be aware of examples of the types of functionality provided by each node in the network 100. There are many factors that affect how well a neural network performs its chosen task. For example, the size and / or depth of the network (e.g., the number of nodes and / or the number of layers of nodes) can affect the performance of the neural network. Figure 1 It will be appreciated that the performance of the network 100 will depend on the weights θ applied to each node in the network 100. i,j Embodiments of the present disclosure may enable improved selection of nodes of a network, and may therefore enable provision of an improved neural network to perform a selected task.
[0065] When determining which nodes to select for network 100, a computational graph can be constructed. The computational graph includes a plurality of nodes connected by a plurality of edges. Each node is used to receive at least one input data from a previous node connected to the node via an edge. Each node is arranged to process the input data by applying a weight to the input data and performing an operation on the weighted input data to provide output data. Different nodes can perform different operations, for example, they can have different functions. The graph includes a weight associated with each node for scaling the input data of the node. Each node is used to provide its output data to a subsequent node in the graph via an edge. The computational graph can be considered to define a plurality of candidate models, each of which is a subgraph in the computational graph. Each subgraph is composed of nodes and edges selected from a computational graph of a plurality of nodes and a plurality of edges, and weights associated with the selected nodes. Some nodes and their associated weights are shared in two different subgraphs.
[0066] Each candidate model may include a fixed, selected arrangement of nodes. The selected architecture (e.g., the node arrangement) may vary for different candidate models. The functionality provided at each node may vary for different candidate models. At each of these nodes, multiple choices for the functionality of the node may be provided. Each choice of node may be considered to represent a series of candidate nodes clustered at a selected location in the graph (e.g., in the region where the selected node will be located in the final neural network). The computational graph may include weights for the connections between each candidate node in one cluster and each candidate node in the next cluster.
[0067] Figure 2 A schematic diagram representing a functional diagram of a computation graph 200 is shown. Figure 2 Figure 200 and Figure 1 The neural network 100 shown corresponds to the graph 200 having four layers, the first and fourth layers each having one node, and the second and third layers each having two nodes.
[0068] exist Figure 2In the diagram 200, two candidate nodes 21a and 21b are provided as candidates for the first node 21 of the second layer 20. Similarly, two candidate nodes 22a and 22b are provided as candidates for the second node 22 of the second layer 20. The same is true for the third layer 30. Two candidate nodes 31a and 31b are provided as candidates for the first node 31 of the third layer 30, and two candidate nodes 32a and 32b are provided as candidates for the second node 32 of the third layer 30. The candidate nodes are shown as squares, with Figure 1 The corresponding nodes are shown as circles with dashed lines.
[0069] Figure 2 The dashed lines in show possible routes along which information can flow through the network, for example, these dashed lines show all possible edges. It should be understood that some of these dashed lines will exist in each model, and the choice of those edges that exist will depend on the candidate nodes that exist in the selected model. Note that in Figure 2 In the example of , each candidate node in the third layer 30 has 4 different input sources from the second layer 20, and the same is true for the node 41 in the fourth layer 40. Once the computational graph 200 including the candidate models (subgraphs) is created, a preferred candidate model can be identified. The preferred candidate model can be a specific subgraph (for example, a candidate node may be selected for each node in the network). The specific subgraph can be a combination identified in one of the candidate models, or it can be a new combination of candidate nodes.
[0070] To identify a preferred candidate model, each candidate model in a subset of the subgraphs included in the computation graph is trained. This process is performed in an iterative manner. For example, a first candidate model can be trained, and then, once the first candidate model is trained to a certain threshold level, a second candidate model can be subsequently trained. This process is repeated until a selected number of candidate models have been trained (e.g., all candidate models in the subset). As discussed in more detail below, once the candidate models are trained, a controller having a training strategy can analyze the computation graph and the trained models to help identify a candidate model selected for the neural network.
[0071] It should be understood that the neural network diagram here is for illustration only. Figure 2 It is computationally feasible to train and then test all possible subgraph combinations and then select the best performing model as the model on which the neural network is based, given the computational graph 200 shown in FIG. 2 . However, neural networks can be much larger than this. Therefore, analyzing a large neural network with many different choices for each node in the network can have an almost infinite number of possible structures. To identify a preferred network from this number of structures, only a subset of the total number of structures would be used, otherwise the computational requirements would be impractical.
[0072] Figure 3a and3b Two schematic diagrams showing candidate models of neural networks. Figure 3a Candidate model 301 is shown in which nodes 21a, 22b, 31a, and 32b are selected. Figure 3b Candidate model 302 is shown, in which 21a, 22a, 31a, and 32a are selected. Figure 3a and Figure 3b Describes the training of candidate models.
[0073] The candidate model has a set of selected candidate nodes and a set of selected weights θ i,j The weight θ i,j A scaling factor is applied to each point-to-point connection in the model. The scaling factor for each weight in the model can be "learned" by training the model.
[0074] For example, the weights can be learned by using gradient-based learning. This involves defining a cost function and minimizing it. Minimization of the cost function can be achieved by following the gradient of the cost function to minimize it. Stochastic gradient descent can be used as an optimizer. A gradient descent optimization algorithm can be used, which uses backpropagation. For the sake of brevity, this will not be discussed in detail.
[0075] Each weight θ i,j You can train according to the following formula:
[0076]
[0077] Among them, t represents time, α represents the learning rate, and J m (θ i,j (t) ) is a cost function calculated on a subset of training data of size m. For example, it should be understood that represents the gradient function, such as the derivative. The training is based on the data set Among them, x i is the i-th input data, y i For the input data x i The associated correct output. At each time step, the stochastic gradient descent algorithm updates the weights until the transformation is reached.
[0078] Given a family of density models (p model;θ ), a conventional neural network can be trained using maximum likelihood (e.g., maximum likelihood function). For example, it should be understood that these density models represent the probability density function of the model for a given set of weights. That is, a neural network is trained to identify the so-called "optimal weights" of the network based on, for example,
[0079]
[0080] By assuming And through Bayes' theorem, we can get:
[0081] P model;θ (y i , x i )=n -1 p model;θ (y i , x i )
[0082] The maximum likelihood estimator can identify the "optimal weights" by minimizing the following equation:
[0083]
[0084] The disclosed embodiments further constrain the training process. A modified loss function used in training can be defined. When training each model, updates to the weights are calculated based on minimizing the modified loss function. For selected weights, the modified loss function includes a component that takes into account the importance of the selected weight in any previously trained models. Since different subgraphs can include the same nodes (and weights associated with the nodes), weights that have been trained for a previous model can be trained as part of a later model. This can be done from Figure 3a and Figure 3b These two figures show Figure 2 Two possible subgraphs (models) 301 and 302 of the computation graph 200. It can be seen that both subgraphs 301 and 302 include nodes 11, 21a, 31a, and 41. Therefore, the two models (subgraphs 301 and 302) share the weight θ 21a,11 ,θ 31a,21a , and θ 41,31a .
[0085] When updating a weight in a candidate model, the modified loss function may take into account how important the weight (and its associated node) was in an earlier trained model. For example, this may be based on an indication of Fisher information for the weight (and node). Fisher information may provide an indication of the amount of information a node has about the operation of the model. For a selected weight, the Fisher information provides an indication of the curvature of the log-likelihood for the weight. A Fisher information matrix may be calculated.
[0086] For example, for a selected subgraph (model) to be trained, the subgraph m can be obtained. iThe cross-entropy loss function The cross entropy loss function may be defined such that minimizing the cross entropy likelihood provides the maximum likelihood for the classification problem. The Fisher information for each weight may be sequentially updated based on the cross entropy loss function for the subgraph. The updates may be based on the derivative of the cross entropy loss function. For example, the squared gradient of the cross entropy loss function involving the selected subgraph may be calculated for each weight (e.g., for all calculate ). This squared gradient can then be used to update the Fisher information of each weight. For example, for all f∈{1,...,|θ|}, the Fisher information of each weight can be updated in the order j∈{1,...,n}. Updated to:
[0087]
[0088] After each model training, a validation batch may be used to compute updates to the Fisher matrix, for example, determined based on previously trained candidate models.
[0089] For example, a loss function can be defined for each architecture (subgraph) The loss function is hereinafter referred to as a weight plasticity loss function. The loss function may include a term indicating Fisher information, e.g., a term indicating Fisher information for each weight. For example, for each weight, the loss function may include a component based on: (i) the difference between the value of the weight and the optimal value for the weight, and (ii) an indication of the Fisher information for the weight, e.g.,
[0090]
[0091] Therefore, minimizing the weight plasticity loss function can be based at least in part on minimizing this component. After each model training, the optimal weights can be rewritten
[0092] The weight plasticity loss function for each weight may also include an indication of the cross entropy loss function for the subgraph. Thus, the weight plasticity loss function may include two terms:
[0093]
[0094] One and / or both of these two terms can be scaled, for example, by a corresponding hyperparameter. Such a hyperparameter can be learned. Such a hyperparameter can be controlled to provide a selected learning rate for the equation.
[0095] As each candidate model is trained, a weight plasticity loss function can be minimized. This can reduce the magnitude of updates to weights in a candidate model that are important to other candidate models. The weight plasticity loss function balances minimizing changes in important weights with minimizing the performance of each model when performing the chosen task, so that each model can still achieve high performance on the task without sacrificing the performance of earlier trained models on the same task.
[0096] The weight plasticity loss function may include a term indicating the modulus of each weight. For example, this may be based on the value of the modulus squared, e.g., the weight plasticity loss function may include the following terms:
[0097]
[0098] This can force lower weights on the subgraph. This has been shown to provide improved performance for neural networks designed to perform selected tasks. Each subgraph m j The weight plasticity loss function can be defined as:
[0099]
[0100] The training of each subgraph can be based on minimizing this function when updating the weights of that subgraph. Each of these three components can be adjusted according to hyperparameters to control its relative contribution. The values of these hyperparameters can be learned.
[0101] The updates to the Fisher information matrix can be clipped, for example, they can be clipped based on a selected threshold of the updates. This helps prevent gradient explosions associated with the loss function. For example, each update can be scaled based on the value of the corresponding weight. The updates to the Fisher information matrix can be scaled based on the norm of each weight matrix. For example, the updates to the Fisher information matrix can be clipped by the norm of each weight matrix according to the following formula:
[0102]
[0103] The parameters α and β can be adjusted, for example, they can be learned. These parameters can provide a degree of constraint on the learning of the control model.
[0104] Once the weights of each candidate model have been updated based on the training, an analysis of the candidate models can be performed. The analysis can include sequentially evaluating the performance of different candidate models using a validation set, for example, where the validation set is a data set different from the training data set, from which the performance of the model (on unseen data) in performing a selected task can be determined. The best performing model can be selected. In some examples, the best performing model can be considered a preferred model, and a neural network based on the preferred model can be provided. Providing a neural network can include creating a neural network using the preferred model and a set of random weights, and then training the network.
[0105] Figure 4 Another example of using an iterative approach is shown, which enables the controller to evaluate well-performing trained candidate models and learn these candidate models, and recommend candidate models for training and evaluation based thereon.
[0106] Figure 4 A method 400 of providing a neural network for performing a selected task is shown. At step 410, a computational graph is obtained. This may be the same as described above.
[0107] At step 420, a set of recommended subgraphs is obtained. Initially, the set of subgraphs may be randomly selected. However, as the loop iterates, the set of subgraphs will be selected by the controller. The set of subgraphs may also be initially selected by the controller.
[0108] At step 430, each of these subgraphs is trained in an iterative manner as described above. That is, the training uses the training data and minimizes the weight plasticity loss function so that the weights of each candidate model are updated based on the importance of these weights to other previously trained candidate models.
[0109] At step 440, each candidate model completes training. These candidate models may have been given limited training time, for example, only a small number of batches. Therefore, each candidate may be better than it was in the past, but not fully trained. A time threshold or batch threshold can be used to control the amount of training each model receives. These models are then tested on a validation data set to identify the best candidate model. The best candidate model is then evaluated by a controller. The controller can use a reinforcement learning algorithm or an evolutionary learning algorithm to update the parameters of its strategy based on the results of the validation data of the best candidate model. For example, the controller can use a reinforcement learning algorithm.
[0110] Reinforcement learning algorithms use a trial and error process guided by an agent. The agent can interact with its environment in discrete or continuous time. At each time, it receives a state S from the environment. t =s, based on this state, the agent generates action At = a. The action is selected by the controller's strategy Complete, this is the probability distribution of the action state. Then, the agent receives the new state S t+1 and reward R t+1 . The goal is to maximize some form of future reward. Reinforcement learning involves changing policy in response to received rewards and striving to maximize future rewards. To this end, reinforcement learning algorithms can address the exploration-exploitation trade-off. For example, a bandit algorithm can be used. In the context of neural architecture search, this can include identifying new, untested combinations / selections of nodes with unknown performance, and sticking to known combinations / selections of nodes while making small changes to that combination / selection. The parameters of the controller's policy can be learned using policy gradient methods (e.g., methods based on the gradient of a loss function).
[0111] Policy gradient methods can be used with recurrent policies suitable for neural architecture search. T A sequence of actions defines an architecture (e.g., a subgraph). Each action can define an operation such as a ReLU or softmax function (e.g., an operation on a node). Each action can define instructions on how to compose the operation (e.g., define the inputs to the operation). A policy (e.g., a recurrent neural network policy) can be used to sequentially generate actions a i , until reaching action a indicating that the architecture is complete T . Status S T can be considered as the hidden state of the neural network. The policy can be defined as:
[0112]
[0113] This is a stochastic policy where actions can be sampled according to a probability distribution over the space of all possible actions. The goal is to maximize the expected reward of the policy, where:
[0114]
[0115] Among them, D defines the set of all possible architectures (subgraphs), is the probability of sampling architecture a1, ..., T, and R(a1, ..., T) is the inverse of the validation perplexity of the sampled architecture a1, ..., T. The parameters θ of this strategy are p The goal is to find the policy parameters that maximize L(θ) by following the gradient p ):
[0116]
[0117] In practice, this can include calculating for:
[0118]
[0119] The gradient can be calculated using, for example, an unbiased Monte Carlo estimate. Stochastic gradient descent can be used, for example, to update the parameter θ as follows p :
[0120]
[0121] For example, the algorithm REINFORCE can be used. A Boltzmann softmax equation can be used that divides the exponential in the original softmax by a temperature T that decays over time. This temperature (and its decay) can be used to initially favor exploration (e.g., as opposed to exploitation).
[0122] When computing the policy's reward, a validation set can be used (different from the training dataset used to update the weights). This can enable the policy to select generally applicable models (rather than models that are over-specialized / overfit to the training dataset). For example, perplexity can be computed on the validation dataset (or a minibatch of it).
[0123] At step 450, the controller's policy may be updated. This may be based on the method described above. The updated policy may then select / identify a different subgraph that it deems better. The controller may be configured to recommend other subgraphs that the controller determines based on its updated policy to be suitable for performing the selected task.
[0124] At step 460, the controller is operable to recommend a subgraph for the selected task using the updated strategy. For example, the controller may select a set of actions corresponding to a series of operations of the subgraph. The controller may select a plurality of such subgraphs. These subgraphs then constitute candidate models for the next round of testing subgraphs.
[0125] In some examples, the controller may not recommend a subgraph. Steps 420 to 470 may loop without recommending a new subgraph, so that the candidate model obtained in step 420 is trained for several iterations. During each iteration of training these candidate models, the strategy may still be updated based on the evaluation of the training model. In some examples, the controller may not recommend a subgraph, but may still provide an indication of which selection of the training subgraph is the preferred subgraph. For example, step 460 may be skipped.
[0126] At step 470, it may be determined whether to iterate (e.g., repeat steps 420 to 460). For example, the subgraphs recommended by the controller may be checked to determine whether to iterate. Different criteria may be used in this step to determine whether to keep iterating. The recommended subgraphs may be used as the obtained subgraph set in step 420, and steps 420 to 460 may be repeated using these newly recommended candidate subgraphs. By iterating this process, the method may get closer to obtaining a preferred model for performing the selected task.
[0127] Criteria can be applied to determine that a recommended subgraph or the best subgraph from a set of previous candidate models should be used as the preferred model that the neural network will use to perform the selected task. For example, a recommended candidate model and a previous best candidate model can be compared. If the two are very similar in function, for example, having an architectural similarity above a certain threshold level / containing a threshold number of the same neurons (for example, in the same position in the network), it can be determined that the recommended model / previous best model should be the preferred model. If the controller recommends multiple subgraphs, a comparison can be performed to see if any of the subgraphs have been previously recommended by the controller. If a threshold number of recommended subgraphs are the same as the previous candidate models, it can be determined that no meaningful changes have occurred and it can be determined that the threshold has been reached. Additionally and / or alternatively, a threshold time limit or a threshold number of iterations can be applied, and once the threshold is reached, the method stops iterating.
[0128] At step 470, it is determined that iteration should be stopped and the method proceeds to step 480. At step 480, a neural network is created based on the subgraph in the preferred model.
[0129] In step 490, the neural network is trained so that the weights are updated to convergence, for example, starting from random initialization. The network is then provided as a selected neural network to perform a selected task.
[0130] Reference is now made to Figures 5 and 6, which illustrate the results obtained using the methods described herein. Appendix I provides a statistical demonstration of the use of the weight plasticity loss function described herein.
[0131] Figure 5a and Figure 5b The results of test accuracy are shown when two models are trained sequentially. The two models are tested on the handwritten digit recognition dataset (MNIST). The two models are simple feed-forward networks, model 1 has 4 layers and model 2 has 6 layers. In both cases, the first model is trained until convergence, so as to obtain the optimal weights for model 1 The optimal weights for Model 1 include some weights shared with Model 2. Model 2 is then trained on the same dataset as Model 1. Training Model 2 includes updating the shared weights. Figure 5aand Figure 5b It shows how the test accuracy of the two models varies depending on the loss function used when training the models.
[0132] Figure 5a The results are shown when two models are trained sequentially using a cross entropy function, for example, by training with maximum likelihood to obtain optimal weights. The line from top left to top right shows the performance of Model 1 after training. This is a flat line, indicating that the performance achieved by Model 1 remains at a test accuracy of approximately 92%. The line shows the performance of Model 1 as a baseline. The line from bottom left to top right shows the test accuracy of Model 2 when it is trained. As expected, the test accuracy of Model 2 increases with increased training. It can be seen that Model 2 did not reach convergence at the number of iterations shown, but a significant increase in the ability of Model 2 to perform the task can be observed.
[0133] Figure 5a The line from top left to middle right in shows how the training of Model 2 (and therefore the update of the shared weights between the two models) affects the test accuracy of Model 1. It can be seen that as Model 2 is trained (and the shared weights are changed), the performance of Model 1 decreases. Figure 5a It is clear that the performance of Model 1 has dropped significantly, so any subsequent analysis of Model 1 will show that Model 1 is not performing as well as it could. As can be seen, over 500 epochs of training, the test accuracy of Model 1 has dropped by nearly 30%.
[0134] Figure 5b The results are shown when two models are trained sequentially using the following weight plasticity loss function:
[0135]
[0136] Among them, θ s are the weights shared between Model 1 and Model 2, are the diagonal elements of the corresponding weights shared by the Fisher approximation obtained from the first fit of the first subgraph (Model 1). Appendix I provides a statistical proof of the weight plasticity loss function of this form.
[0137] These lines are used with Figure 5a As can be seen, the performance of the trained model 1 and the performance of the trained model 2 are similar. Figure 5aThe main difference is the test accuracy of Model 1 while Model 2 is being trained. It can be seen that sequentially training Model 2 only causes very minor changes in the test accuracy of Model 1. The second optimization (e.g., training of Model 2) mainly modifies weights with low Fisher information. These weights are considered less important for the first model (e.g., because the Fisher information indicates the curvature of the log-likelihood for each specific weight).
[0138] Using a loss function to train model 2 based on the importance of nodes and / or edges associated with weights to model 1 can maintain the performance of model 1 during the training of the subsequent model. Figure 5a and Figure 5b , it can be seen that analysis of Model 1 and Model 2 at iteration 500 of training Model 2 will provide an indication of which model has a higher test accuracy. By training using a weight plasticity loss function, the test accuracy of Model 1 is maintained while training Model 2.
[0139] Figures 6a to 6f Additional comparative data between the use of weight plasticity loss function and cross entropy loss function is shown. The weight plasticity loss function used is:
[0140]
[0141] The Fisher information of each weight is updated as follows:
[0142]
[0143] Fisher's update was whittled down to:
[0144]
[0145] for Figure 6a and Figure 6b , the values of α and β are set to 1 and 100 respectively, 50 models are sampled per epoch, and 10 batches are sampled per epoch. For the use of weight plasticity loss function, the weight plasticity loss function is introduced in the third epoch, and the Fisher information is set to 0 in each epoch. For the use of cross entropy loss function, the cross entropy loss function is used throughout the process. The increase in the value of β imposes a greater constraint (e.g., penalty) on the previously learned weights.
[0146] Figure 6a Shown for each trained model is the average difference in perplexity between the performance of: (i) the model after training all 50 models sequentially, and (ii) the model when trained independently. The average difference over 50 generations is shown. Perplexity provides an indication of how well the model's predictions are sampled - low perplexity indicates that the model makes good predictions. Figure 6a Error bars of ±2 standard deviations are also shown. Figure 6a It can be seen that the use of the weight plasticity loss function (as shown by the lower of the two lines on the right and the narrower of the two error bars) reduces the average difference in perplexity between models that have been trained sequentially and models that have been trained independently. This is clearly beneficial. This means that the model used to update the controller's policy is more likely to be the correct choice (e.g., the choice of the best candidate model is less likely to be biased by its position in the order in which the models were trained).
[0147] Figure 6b The maximum difference in perplexity between the performance of (i) the models after training all 50 models sequentially and (ii) the models when trained independently is shown. Again, the weight plasticity loss function significantly reduces the maximum difference in perplexity. Additionally, the fluctuation in this value is smaller for the weight plasticity loss function. This shows another clear benefit of using the weight plasticity loss function.
[0148] Figure 6c to Figure 6f The values of α and β set to 70 and 10 respectively are shown, with 70 models per generation and 15 batches per model. For the weight plasticity model, the weight update is based on the weight plasticity loss function at the eighth generation, and the Fisher information matrix is set to 0 every three generations. For each loss function, the policy is trained using 2000 steps per generation and the policy is not kept fixed. The reinforcement learning algorithm REINFORCE is used as part of the controller with the policy to be determined.
[0149] Figure 6c The average reward of the model is plotted against the number of policy steps taken. The policy aims to maximize the reward based on the candidate model, so higher reward values are beneficial. Figure 6c As can be seen in Figure 1, the average reward is much higher when using the weight plasticity loss function than when using the cross entropy function. The error bars shown represent 0.5 standard deviations. It can be seen that the average reward is higher with the weight plasticity loss function, and the standard deviation of this reward is lower.
[0150] Figure 6d The entropy of the policy is shown. The entropy of a policy provides a measure of the uncertainty in the probability distribution of predictions made by the policy - the higher the entropy, the higher the uncertainty. Figure 6d It can be seen that the entropy of the policy obtained using the weight plasticity loss function is much smaller than the entropy of the policy obtained using the cross entropy loss function. Therefore, using the weight plasticity loss function can provide a more reliable neural network to perform the selected task.
[0151] Figure 6e and Figure 6f Shown with Figure 6a and6b Similar trends are observed when comparing models trained as part of a sequence to their independently trained counterparts, the difference has much lower mean and standard deviation across multiple candidate models than when using the cross entropy loss model. Additionally, it is evident from Figures 6e and 6f that the use of the weight plasticity loss function is responsible for this performance improvement. As mentioned above, the weight plasticity loss function was introduced only until the eighth generation. Prior to this point, the two loss functions were equal. Figure 6e and 6f It can be seen that the improvement in the performance of the weight plasticity loss function occurs after the eighth epoch. Before this, the two loss functions perform the same.
[0152] In addition to this empirical data showing improved performance due to the use of a weight plasticity loss function, Appendix I provides a statistical justification for the choice of a weight plasticity loss function.
[0153] The disclosure herein refers to nodes of a neural network, and the functions (e.g., activation functions) used in neural networks are well known. Such functions may include: ReLU, Tanh, Sigmoid, Softmax, etc. In particular, commonly used nonlinear functions. These functions may be used for individual neurons in the network. Embodiments of the present disclosure encompass each individual node as an individual neuron or a block of neurons (as described herein). The computational graph may include a combination of the two.
[0154] Embodiments of the present disclosure may use blocks of neurons at nodes. Each block of neurons may have its own topology and content. For example, a block may include a sequence of selected neurons connected by a selected set of edges, with fixed weights associated with them. Because these blocks can be selected based on prior knowledge of the performance of the blocks on similar tasks, the use of blocks of neurons can improve neural architecture search. This can also reduce the number of weights that need to be trained as part of the method because the weights within the block can remain fixed. The neural architecture search method may then only need to identify edges and weights between different blocks, for example, fewer edges and weights between features than when the blocks are expanded and each neuron requires edges / weights.
[0155] Such examples may be referred to as neural architecture search with blocks. A controller may be used (e.g., with an LTSM strategy) that samples blocks at nodes and determines edges connecting those blocks. In some examples, the weights inside a block may not be fixed, e.g., only component neurons are selected, but the weights between those neurons may be edited during training. In these examples, it should be understood that if two candidate models share the same block, the two candidate models will share a large number of weights (e.g., all weights in the block shared by the two candidate models). Therefore, using a weight plasticity loss function may have specific applications in such situations.
[0156] The reward to be maximized by the controller can be modified to include a component that takes into account the complexity of the candidate model. For example, the reward can be based on the number of components within the candidate model. This can be in addition to a reward based on the fitness of the candidate model, such as the validation perplexity of the model. It should be understood that the introduction of a component based on the complexity of the candidate model into the reward function can be applicable to all methods described herein.
[0157] For example, a density-based reward can be introduced. This reward can be used to favor models with fewer weights. The density of a model can be defined as:
[0158]
[0159] Among them, θ maximum is the maximum number of weights that may be included in the model. The reward can be based on this density, for example, as:
[0160]
[0161] Among them, a and b are the hyperparameters to be learned, valid ppl is the validation perplexity, i.e., how well the model performs on the validation dataset. The two components of the reward can be separated (as shown above) to prevent the reward from becoming negative. Thus, using this reward can identify models that perform well and have less weight.
[0162] Embodiments described herein include the use of an iterative process in which candidate models are trained multiple times before a preferred model is selected. In some examples, the same set of candidate models may be trained multiple times, for example, the set of candidate models may be trained in multiple different batches. In the case where a controller is used to evaluate candidate models and update its strategy for identifying preferred candidate models, before retraining a set of models and then re-evaluating the set of models with the controller, the iterative process may include multiple repetitions of training the set of candidate models and then evaluating the set of models using the controller to update the strategy. In some examples, in response to updating the strategy based on a set of trained candidate models, the controller may recommend new candidate models for testing. However, it should be understood that in some examples, the above-described iterative process may not be required. For example, a single operation of sequential training of candidate models may be sufficient to identify a preferred model. Since the embodiments described herein enable sequentially trained models to be trained so that any updated weights reduce damage to previously trained models, a single sequential training of these models is still sufficient to identify a preferred model.
[0163] It should be understood that the terms "subgraph" and "model" can be used interchangeably. Obtaining a computational graph can be considered as obtaining data indicative of the computational graph, such as an array of matrices and operations that control and scale the flow of information between different functions. A particular subgraph (e.g., a collection of nodes connected by edges containing weights) can be considered a candidate model. If the candidate model is determined to be a preferred candidate model, the particular architecture of the subgraph can be used as the architecture of the neural network (e.g., without the other nodes and edges in the computational graph).
[0164] It should be understood that the selected task can be any of a variety of selected tasks. Neural networks can be applied to many technical problems. For example, the selected tasks include: (i) natural language processing, (ii) image recognition, (iii) classification and / or modeling of physical systems, (iv) data processing, and (v) search result generation. This list should not be considered restrictive, because any instance in which input data can be processed to provide output data and based on which technical operations can be performed can benefit from the use of improved neural networks to process the above data. It should be understood that the present disclosure is not limited to any type of neural network provided. The present disclosure is applied to any network in which weights are used as part of a neural architecture search for this type of network, where weights can be shared between different nodes in a computational graph. For example, suitable neural networks can include convolutional neural networks, deep residual networks, capsule networks, recurrent neural networks, multi-layer perceptrons, etc.
[0165] The neural networks described herein (e.g., neural networks for performing selected tasks) may include recurrent neural networks, which are used as encoders in important sequence-to-sequence models. These networks can be applied to areas such as translation, customer churn detection, and email classification. For example, an improved translation neural network can identify key phrases to be translated in a sentence. This can enable a neural network capable of receiving input data in the form of a text string in a first language or encoding to process the input data to provide output data in the form of a text string in a different language or encoding. In other examples, neural networks can be applied to image recognition systems. For example, at a security checkpoint such as an airport, a camera can capture an image of an individual, and a neural network can be arranged to classify / recognize the individual. Improved neural network architectures can improve the ability of these neural networks to recognize individuals in photos.
[0166] It should be understood that any suitable type of controller / strategy can be used when analyzing candidate models for training. Although the use of reinforcement learning methods has been described, other examples can be used. For example, evolutionary networks (and associated neuroevolutionary methods) can be used. In such examples, the topology of the network and its weights can be updated during training (e.g., the network architecture can evolve). Weights can also be inherited by child nodes in a controlled manner so that parent weights are applied to child nodes where appropriate.
[0167] It should be understood from the above discussion that the embodiments shown in the drawings are only schematic and include features that can be generated, removed, or replaced as described herein and as claimed. With reference to the drawings, it will be understood that the schematic functional block diagram is used to indicate the functions of the systems and devices described herein. In addition, the processing functions can also be provided by devices supported by electronic devices. However, it will be appreciated that the functions do not have to be divided in this way and should not be understood to imply any specific hardware structure other than the hardware described and claimed below. The functions of one or more elements shown in the drawings can be further subdivided and / or distributed throughout the device of the present disclosure. In some embodiments, the functions of one or more elements shown in the drawings can be integrated into a single functional unit. For example, weights have been shown as applied to nodes or edges. Both examples should not be considered restrictive, and in each figure, weights can be applied at any position.
[0168] As will be appreciated by those skilled in the art, in the context of the present disclosure, each example described herein may be implemented in a variety of different ways. Any feature of any aspect of the present disclosure may be combined with any other aspect of the present disclosure. For example, method aspects may be combined with device aspects, and features described with reference to the operation of specific elements of these devices may be provided in a method that does not use a specific type of device. In addition, unless it is explicitly stated that some other features are essential to its operation, each feature of each embodiment is intended to be separated from the features described in conjunction with it. Each of these separable features may be combined with any other feature of the embodiment describing the feature, or may be combined with any other feature or feature combination of any other embodiment described herein. In addition, without departing from the present invention, equivalents and modifications not described above may also be adopted.
[0169] Some features of the method described herein can be implemented in hardware, and one or more functions of the device can be implemented in method steps. It should also be understood in the context of the present disclosure that the method described herein need not be performed in the order in which they are described, nor in the order in which they are depicted in the accompanying drawings. Therefore, the various aspects of the present disclosure described with reference to a product or device are also intended to be implemented as a method, and vice versa. The method described herein can be implemented in a computer program or in hardware or in any combination thereof.
[0170] For example, each node may include a signal processing unit for receiving input signals, processing input signals, and providing output signals. The node may include circuitry for: receiving at least one input signal indicating an input data from a previous node connected to the node via an edge, performing an operation on the input data to provide output data, optionally scaling the input data according to a weight associated with the node; and providing an output signal indicating the output data to a subsequent node in the graph via the edge. The edge may include a communication path for transmitting signals between nodes. For example, the communication path may be used to receive an output signal from a previous node, optionally scale the output signal according to a weight, and send the output signal to a subsequent node. Local data storage may be used to determine the final destination to which the output signal is sent via the communication path.
[0171] Computer programs include software, middleware, firmware, and any combination thereof. Such programs may be provided as signals or network messages and may be recorded on a computer-readable medium, such as a tangible computer-readable medium that may store computer programs in a non-transitory form. The methods described herein and / or the functions of components such as nodes and edges may be provided in hardware, which may include computers, handheld devices, programmable processors, general-purpose processors, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and logic gate arrays.
[0172] In some examples, one or more storage elements may store data and / or program instructions for implementing the operations described herein. Embodiments of the present disclosure provide a tangible, non-transitory storage medium including program instructions that can be used to program a processor to perform any one or more of the methods described and / or claimed herein and / or provide a data processing device described and / or claimed herein.
[0173] Other examples and variations of the present disclosure will be apparent to those skilled in the art in light of the context of the present disclosure.
[0174] Appendix I
[0175] A statistical justification for the weight plasticity loss function used above is provided here.
[0176] Consider the following problem: (θ1, θ2, θ s ) are disjoint partitions of parameter θ. and Two architectures, where D is the dataset. Assume that the parameter θ s Shared by two\models. Hope to find and Independent working parameters (θ1, θ2, θ s ). Assume and θ2 for a given θ s and data set D. Under this assumption, we seek to maximize the posterior probability of the parameter θ given the parameter set.
[0177] Lemma 1. Given the setting described above,
[0178]
[0179] Proof: Using Bayes' theorem, we have
[0180]
[0181] Where the conditional independence assumption is used in the third equation:
[0182] Now derive the denominator of equation (4.2)
[0183] Lemma 2. Assumption:
[0184] Prior distribution follows normal distribution
[0185] ·Card(θ1)+Card(θ s )=p1+p s =p,
[0186] In the maximum likelihood estimation The log-posterior probability distribution evaluated under (Right now, ) is divided into negative Hessians that conform to (θ1, θ s ) of 4 pieces:
[0187]
[0188] have
[0189]
[0190] in
[0191] Proof: Yes
[0192]
[0193] in Make H p (θ1,θ s )=H(θ1,θ s )+σ -2 lp (θ1,θ s ). Using l p (θ1,θ s ) in its maximum likelihood estimate The Taylor expansion around
[0194]
[0195] Because it is evaluated based on maximum likelihood estimation, the first-order derivative is zero. Now the negative Hessian is partitioned into:
[0196]
[0197] Available
[0198]
[0199] definition Then get
[0200]
[0201] in,
[0202] therefore,
[0203]
[0204] Given equation (4.4), we can prove Lemma 2, because
[0205]
[0206] where the scaling is arranged so that the mean z and the covariance matrix Integrating over the normal distribution of
[0207] Following Lemma 1 and Lemma 2, we get
[0208]
[0209] in,
[0210] Equation (4.5) depends only on the second model Therefore, from the first model The information learned in the conditional posterior probability The last term on the right side of equation (4.5) Model must be maintained and The interaction between them.
[0211] Posterior probability is difficult to solve. We approximate it by using the maximum likelihood estimate The Gaussian approximation of the posterior distribution comes from the second-order Taylor expansion in equation (4.3) and
[0212]
[0213] in, is the negative Hessian of the log-likelihood. Therefore, by defining the observed Fisher information I,
[0214] We assume normality of the prior distribution. The third term on the right side of equation (4.5), i.e., logp(θ2,θ s ) represents l2 regularization.
[0215] Using the diagonal approximation of the Fisher information, Directly get H 1s is a zero matrix, so that Ω=I ss +σ -2 I p
[0216] Given the above discussion, we define the weight plasticity loss (to be minimized) as
[0217]
[0218] in, are the diagonal elements of the corresponding parameters shared from the Fisher approximation, which is derived from The first fit of is the cross entropy loss function. The intuition behind (4.6) is that because the Fisher information represents the curvature of the log-likelihood with respect to a particular parameter, we allow the second optimization to modify parameters that have low Fisher information and are therefore less important to the first model.
Claims
1. A computer-implemented method of providing a computer program product, the computer program product comprising an image recognition neural network for processing input image data to provide an output indicative of content of the input image data, the method comprising: Performing a neural architecture search, the neural architecture search comprising: (i) obtaining a computational graph, the computational graph comprising: a plurality of nodes connected by a plurality of edges, and a plurality of weights for scaling input data provided to the nodes along the edges, wherein each node is used to: receiving at least one item of input data from a previous node connected to the node via an edge; performing operations on the input data to provide output data, wherein each item of input data is scaled according to a weight associated with the node and / or edge; and providing the output data to a subsequent node in the graph via an edge; wherein the computation graph defines a first candidate model and a second candidate model, each candidate model is a subgraph in the computation graph, the subgraph having nodes, edges, and weights selected from the plurality of nodes, the plurality of edges, and the associated plurality of weights, wherein some of the selected nodes, edges, and weights are shared between the first candidate model and the second candidate model; (ii) updating the weight of the first candidate model based on training the first candidate model to process the input image data; (iii) processing the input image data based on training the second candidate model, updating the weight of the second candidate model, wherein: Before training the second candidate model, initializing the weight of the second candidate model based on the trained weight of the first candidate model; Updating the weights of the second candidate model comprises updating some weights shared between the first candidate model and the second candidate model updated in step (ii), wherein updating the shared weights is controlled based on an indication of the importance of the nodes and / or edges associated with the weights to the trained first candidate model, the indication of the importance being determined using a measurement based on a gradient or a higher-order derivative of a loss function selected with respect to the weights; controlling an update of the weight of a share based on the indication of importance comprises scaling a magnitude of the update based on a value of the measurement; and Training the second candidate model comprises training using a training loss function, the training loss function comprising a component indicative of the measurement of weights in the second candidate model, wherein the training loss function comprises a scaling factor for suppressing an exploding gradient associated with any of the weights in the second candidate model; (iv) identifying a preferred model of the neural network, the preferred model comprising nodes, edges, and associated weights selected from the computational graph, wherein the preferred model is determined based on an analysis of the first training candidate model and the second training candidate model; A computer program product is provided, the computer program product comprising an image recognition neural network for processing input image data to provide an output indicative of content of the input image data.
2. The method according to claim 1, wherein: For each shared weight, the indication of importance is determined using an indication of Fisher information for the weight.
3. The method according to claim 2, wherein: The magnitude of the update is scaled based on the value of the Fisher information of the weight.
4. The method according to claim 3, wherein: The components are indicative of the Fisher information of weights in the second candidate model.
5. The method according to claim 4, wherein: The scaling factor comprises a norm of the weight.
6. The method according to claim 1, wherein: Identifying the preferred model of the neural network includes analyzing the first candidate model and the second candidate model using a controller having a strategy based on at least one of (i) reinforcement learning and (ii) evolutionary learning.
7. The method according to claim 6, wherein: Identifying the preferred model includes: Determine the best performing model among the first candidate model and the second candidate model based on testing the first candidate model and the second candidate model using a validation data set; maintaining the values of the trained weights constant, and updating the strategy of the controller based on the analysis of the best candidate model; operating the controller with the updated policy to obtain recommendations for a third candidate model and a fourth candidate model, wherein some nodes, edges, and weights are shared between the third candidate model and the fourth candidate model; Processing the input image data based on training the third candidate model, updating the weight of the third candidate model; updating weights of the fourth candidate model based on training the fourth candidate model to process the input image data, wherein updating the weights of the fourth candidate model includes updating some weights shared between the third candidate model and the fourth candidate model, and wherein updating the shared weights is controlled based on an indication of the importance of nodes and / or edges associated with the weights to the trained third candidate model; and The preferred model is identified based on an analysis of the third training candidate model and the fourth training candidate model.
8. The method according to claim 7, wherein: The steps of claim 7 are repeated with different suggested candidate models until at least one of: (i) a time threshold is reached, and (ii) a difference between the best candidate model and a subsequent suggested candidate model is below a threshold difference.
9. The method according to claim 8, wherein: The best performing recommendation candidate model from the last iteration of the steps of claim 7 is used as the model architecture of the neural network for processing the input image data.
10. The method according to claim 1, wherein: A candidate model is obtained based on selecting the node, the selection being based on image recognition and an indication of the performance of the node on a task similar to the image recognition.
11. The method according to claim 10, wherein: The node includes at least one of: (i) a neuron, and (ii) a neuron block.
12. The method according to claim 6, wherein: Operating the controller includes using a reward based on: (i) a component indicating a goodness of fit of a candidate model, and (ii) a component indicating a complexity of the candidate model.
13. The method according to claim 12, wherein: The two components of the reward are separated and / or scaled to prevent the reward from being negative.
14. The method according to claim 1, wherein: Steps (ii), (iii), and (iv) are repeated a selected number of times, wherein, initially, steps (ii), (iii), and (iv) are repeated without controlling the updating of the weights based on an indication of the importance of nodes and / or edges associated with the weights to the trained first candidate model; and Steps (ii), (iii), and (iv) are then repeated, and in step (iii), the updating of the weights is controlled based on an indication of the importance of the nodes and / or edges associated with the weights to the trained first candidate model.
15. A computer program product comprising program instructions for programming a processor to perform the method according to any one of claims 1 to 14.
