Method and apparatus for weight sharing neural network with random architecture

Abstract

The present disclosure provides a method for training a weight-shared neural network with random architecture. The method comprises: selecting one mini-batch from a plurality of mini-batches, a training dataset for a task is grouped into the plurality of mini-batches, and each of the plurality of mini-batches comprises a plurality of instances; randomly selecting a plurality of network architectures of the neural network for the selected mini-batch; obtaining a loss of each instance by applying each instance of the selected mini-batch to one network architecture of the plurality of network architectures; and updating shared weights of the neural network based on the loss of each instance of the selected mini-batch.

Description

Technical Field

[0001] All aspects of this disclosure relate to artificial intelligence in general, and more specifically to training and inference methods for weight-shared neural networks with stochastic architectures. Background Technology

[0002] Deep neural networks (DNNs) are widely used to process complex data in a wide range of real-world scenarios. Traditionally, designing a DNN to perform a specific machine learning task is a labor-intensive process that requires a lot of trial and error by experts, who need to manually optimize the architecture of the DNN during the iterations of the training and testing process.

[0003] Neural Architecture Search (NAS) techniques have been proposed to automatically search for neural network architectures to alleviate manual labor. However, early NAS techniques typically required training thousands of models from scratch and were computationally intensive, making them difficult to implement in practice.

[0004] Several methods have been proposed for automatically searching network architectures while reducing computational requirements. One promising approach is to share weights across network architectures instead of training thousands of individual network architecture models from scratch. Specifically, a weight-sharing neural network can be trained that can mimic any of its network architectures. Different network architectures are different subsets of the neural network and share the weights contained within it.

[0005] Enhancements are needed to improve the performance of weight-sharing technology. Summary of the Invention

[0006] According to one implementation, a method for training a weight-sharing neural network with a randomized architecture is provided. The method includes: selecting a mini-batch from a plurality of mini-batches, wherein a training dataset for a task is grouped into the plurality of mini-batches, and each mini-batch includes a plurality of instances; randomly selecting a plurality of network architectures of the neural network for the selected mini-batch; obtaining a loss for each instance of the selected mini-batch by applying it to one of the network architectures; and updating the shared weights of the neural network based on the loss for each instance of the selected mini-batch.

[0007] According to one embodiment, a method for inference using a weight-shared neural network is provided. The method includes: receiving input data; randomly selecting one or more network architectures of the neural network; inferring one or more output data based on the input data using the selected one or more network architectures; and obtaining final inference data based on the one or more output data.

[0008] By using the training method of this disclosure, the diversity of weight-sharing neural networks with stochastic architectures can be explored to enhance the performance of weight-sharing network models. Conversely, by using the inference method of this disclosure, the diversity of weight-sharing neural networks with stochastic architectures can be explored during inference to enhance protection against attacks on the network architecture. Other advantages of this disclosure will be explained in the description below. Attached Figure Description

[0009] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided for illustrative purposes and not for limiting the scope of the disclosure.

[0010] Figure 1 An exemplary weight-sharing neural network with randomized architecture (NSA) according to one implementation is shown.

[0011] Figure 2 An exemplary method for training a weight-sharing NSA is shown according to one implementation.

[0012] Figure 3 An exemplary method for inference using weight-sharing NSA according to one implementation is shown.

[0013] Figure 4 A graph showing the accuracy and loss between training and testing according to one implementation scheme is presented.

[0014] Figure 5 An exemplary method for training a weight-sharing NSA is shown according to one implementation.

[0015] Figure 6 A graph showing the accuracy and loss between training and testing according to one implementation scheme is presented.

[0016] Figure 7 An exemplary method for training a weight-sharing NSA is shown according to one implementation.

[0017] Figure 8 An exemplary computing system according to one implementation is shown. Detailed Implementation

[0018] This disclosure will now be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and thus implement embodiments of this disclosure, and not to impose any limitation on the scope of this disclosure.

[0019] This disclosure describes methods and systems for implementing computer programs that execute on one or more computers, which are equipped with task neural networks configured to perform specific machine learning tasks. The task neural network is implemented as a weight-sharing neural network with a stochastic architecture. As an example, the specific machine learning task could be a machine learning image processing task. As another example, the specific machine learning task could be classifying resources or documents. As another example, the specific machine learning task could be scoring the probability that a particular advertisement will be clicked. As another example, the specific machine learning task could be scoring the probability that a user will respond favorably to a recommendation. As another example, the specific machine learning task could be language translation. As another example, the specific machine learning task could be an audio processing task. As another example, the specific task could be a health prediction task. As another example, the specific task could be a proxy control task performed in a control system for autonomous driving, a control system for industrial facilities, or the like.

[0020] Figure 1 An exemplary weight-sharing NSA (Network with Random Architecture) according to one implementation is shown.

[0021] Unlike conventional DNNs, weight-shared NSA can be defined as a neural network with a fixed set of weights and a random sampling architecture during training and / or inference.

[0022] like Figure 1 As shown, label 10 represents an input node, where the input data is received by the neural network. The input data can be processed as needed, for example, it can be projected onto a feature map in the delayed semantic space. Each of labels 20-50 represents a computation node that includes at least one operation. For example, in the case of a CNN network, each of nodes 20-50 may include a convolution operation. Label 60 represents an output node, where the output data is provided by the neural network, for example. The data to be output can be processed as needed, for example, the feature map in the delayed semantic space can be projected onto user-recognizable output data. Nodes 10-60 are also referred to as layers or modules.

[0023] Figure 1 The diagram illustrates three exemplary network architectures A to C, and a weight-sharing NSA D comprising these three network architectures A to C. Each of network architectures A to C includes a set of computational nodes 10-60 and an edge pattern, wherein each edge connects two nodes. Specifically, network architecture A includes nodes 10-60, an edge connecting nodes 10-60 in series, and an edge E connecting two nodes 20 and 50. A Network architecture B includes nodes 10-60, edges connecting nodes 10-60 in series, and two edges E connecting a pair of nodes 10 and 30 and a pair of nodes 40 and 60, respectively.B The network architecture C includes nodes 10-60, edges connecting nodes 10-60 in series, and three edges E connecting pairs of nodes 10 and 40, pairs of nodes 30 and 60, and pairs of nodes 10 and 60, respectively. C Edge E connecting non-adjacent nodes A E B E C This can be called a bypass side. All nodes connected in series can be called the main chain. Each of the network architectures A through C is a subset of NSA D, which includes, for example... Figure 1 The different architectures A through C are shown. Therefore, architectures A through C can share the weights of NSA D, such as the weights related to the operations included in the nodes and the weights of the edges.

[0024] In the connection view, different network architectures, such as architectures A to C, activate different skip connection patterns in a fixed number of computational operations, and the different skip connection patterns are represented by edges. Each network architecture, such as A to C, is represented as a directed graph of nodes connected by edges. It should be understood that the directed graph representing the computational nodes of the network architecture can be parameterized as a discrete adjacency matrix.

[0025] It should be understood that Figure 1 The structure of NSA D in the text is illustrative; NSA D can include far more architectures, such as hundreds, thousands, tens of thousands, or even more. Furthermore, the structure of the NSA, and correspondingly, the structure of its network architecture, can be much more complex; for example, the NSA can include more than... Figure 1 The diagram shows more nodes, and the operations represented by nodes 20-50 may include any appropriate operations for performing the task, and this disclosure is not limited to NSA-specific operations.

[0026] Figure 2 An exemplary method for training a weight-sharing NSA is shown according to one implementation.

[0027] In execution Figure 2 Before implementing training methods, a network architecture space can be established. As an example, a space similar to... Figure 1 The architecture space of NSA D in this example consists of six nodes connected in series, and bypass edges connecting any two non-adjacent nodes. Assuming the possible network architectures in the architecture space are represented as directed graphs of computational nodes, there are 1023 possible network architectures in this architecture space; in other words, the size of the architecture space is 1023. It should be understood that if a main chain including all nodes connected in series is not required in the architecture space, the size of the architecture space will be larger.

[0028] In one implementation, the neural network can be designed to represent an architecture space. The neural network can be divided into three levels, each level comprising eight convolutional modules. In this implementation, each level can have a structure similar to the NSA D discussed above. Sub-architectures can be sampled from each level, and thus the architecture can be sampled by concatenating the three sub-architectures. As an example, wide convolutions with a widening factor (e.g., a widening factor of 10, etc.) can be used in the nodes for feature extraction. Uniform summation can be performed before each convolutional module to aggregate the incoming feature maps, and batch normalization (BN) can be performed after the convolutional modules; that is, ReLU-convolution-BN triples can be included in at least some of the nodes in each node.

[0029] It should be understood that the size of the entire architecture space discussed above would be enormous. In one implementation, architecture space refinement can be performed, where a subset of architectures can be sampled from the entire architecture space. For example, to avoid meaningless architecture samples, a knowledge-guided sampler, such as the Erdos-Renyi (ER) model (ref. Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. In Proceedings of the IEEE International Conference on Computer Vision, 460 pp. 1284-1293, 2019), can be used to sample a subset of architectures from the entire architecture space. For example, an architecture sampler with a probability of 0.3 for activating any of the possible skip connection patterns can be used to sample the architectures. As an example, multiple 500 architectures can be sampled using a sampler to form a refined architecture space. It should be understood that the size of the refined architecture space is not limited to a specific number; for example, the size can be 500, 5000, 50000, or other numbers. A weight-sharing NSA can be a refined architecture space that includes the sampled network architecture. In some implementations, a weight-sharing NSA can also be the entire architecture space. It should be understood that a weight-sharing NSA can be represented by a set of weights contained within it.

[0030] The training dataset can be divided into multiple mini-batches, each containing multiple instances. For example, the training dataset can be divided into M mini-batches, each containing N training data instances. The size of the NSA is S, meaning that the NSA representing the refined architecture space includes S network architectures. In one implementation, S is not less than 500. In another implementation, S is not less than 500 and not greater than 5000. The weights of the NSA can be initialized in any suitable manner; for example, all weights can be initialized to 1, or any weight can be initialized to a random number.

[0031] In step 210, a mini-batch is selected from multiple mini-batches, for example, a mini-batch is randomly selected from M mini-batches, and the selected mini-batch includes N instances of the training data.

[0032] In step 220, a network architecture is selected from the NSA for the selected mini-batch. For example, a network architecture A is randomly sampled from the S architectures of the NSA with distribution P(A). In the exemplary NSA discussed above, the randomly selected network architecture A comprises 24 cascaded computation nodes and a set of bypass edges, each node comprising a ReLU-convolution-BN triplet.

[0033] In step 230-1, the first instance from the selected minibatch of N instances is applied to the selected network architecture A to obtain a loss value. Similarly, in steps 230-2 to 230-N, each of the second to Nth instances from the selected minibatch is applied to the selected network architecture A to obtain a loss value. The loss value for one instance can be expressed by the formula:

[0034] L = -log p(y i |x i ;W,A),A~p(A) (1)

[0035] Where W represents the weights included in the network architecture, (x i y i () represents an instance of training data, for example, x i This represents the data input to the network architecture, y i This represents the data expected to be output by the network architecture. p(y) i |x i W, A) is the prediction distribution, from which the loss value L = -log p(y) is obtained. i |x i (W, A). It should be understood that although steps 230-1 to 230-N are described as parallel processes, these steps may also be executed in one or more loops, in which the steps are executed one by one.

[0036] In step 240, the N loss values ​​of the selected mini-batch of N instances are averaged to obtain the mini-batch average loss value. The mini-batch average loss value can then be expressed by the formula:

[0037]

[0038] Where B represents the selected mini-batch of training data, and |B| represents the number of instances included in the mini-batch. In this example, |B| equals N.

[0039] In step 250, the weights of the selected network architecture are updated based on the average loss. Therefore, the shared weights of the NSA are updated. For example, the gradient of the weights can be computed by backpropagating the average loss of mini-batches along the selected network architecture, and this gradient can be used to update the weights of the selected network architecture accordingly. The shared weights of the selected network architecture include the weights of the operations contained in the network architecture and the weights of the edges contained in the network architecture. In the example of using ReLU-convolution-BN triples in a node, the shared weights include the weights of the convolution operation and the weights of the BN operation. The edge weights can be referred to as summation weights.

[0040] Then, steps 210-250 can be repeated for another mini-batch selected from the M mini-batches. Steps 210-250 can be repeated M times for the M mini-batches of the training dataset. The process of traversing the training dataset by repeating steps 210-250 M times can be called an epoch. Multiple epochs can be performed on the training dataset until a convergence condition is met. For example, a convergence condition could be that the loss value stabilizes below a threshold.

[0041] Figure 3 An exemplary method for inference using weight-sharing NSA according to one implementation is shown.

[0042] Inference can be performed using a trained weight-sharing NSA. For example, a weight-sharing NSA could be trained to perform inference using the above reference. Figure 2 The process shown or through the following reference Figure 5 The process shown is a task neural network used to perform a specific task.

[0043] In step 310, input data is received. In some examples, the task-oriented neural network may be configured to receive and process an input image to generate a network output for that input image, i.e., to perform some kind of machine learning task for image processing. For example, a particular machine learning task could be image classification, and the output generated by the neural network for a given image could be a score for each of a set of object categories, where each score represents an estimated probability that the image contains an object belonging to that category. As yet another example, a particular machine learning task could be object detection, and the output generated by the neural network could be the location of an identifier depicting a particular type of object in the input image.

[0044] As another example, the input to the task neural network is an internet resource (e.g., a webpage), a document, or a portion of a document, or features extracted from an internet resource, document, or portion of a document. The specific machine learning task can be classifying the resource or document (“resource classification”). That is, the output generated by the task neural network for a given internet resource, document, or portion of a document can be a score for each topic in a set of topics, where each score represents an estimated probability that the internet resource, document, or portion of a document is related to that topic.

[0045] As another example, a specific task could be a health prediction task, where the input is a patient’s electronic health record data and the output is a prediction related to the patient’s future health, such as a predicted treatment that should be prescribed to the patient along with a score, the likelihood of the patient experiencing an adverse health event, or a predicted diagnosis of the patient along with a score.

[0046] As another example, a specific task can be an agent control task, where the input is an observation characterizing the state of the environment, and the output defines the action to be performed by the agent in response to that observation. The agent can be, for example, a real-world or simulated robot, a control system for autonomous driving, a control system for industrial facilities, or a control system that controls different types of agents.

[0047] As another example, the input to a task neural network is features of the impression context of a particular ad, and the output generated by the task neural network can be a score representing the estimated probability that a particular ad will be clicked.

[0048] As another example, the input to the task neural network is features for personalized recommendations to a user, such as features characterizing the context of the recommendation, for example, features characterizing the user's previous actions. The output generated by the task neural network can be a score for each of a set of content items, where each score represents an estimated probability that the user will make a favorable response to the recommended content item.

[0049] In step 320, one or more network architectures of the NSA are randomly selected. For example, each of one or more network architectures from the NSA is randomly selected with a uniform distribution or probability.

[0050] In step 330, one or more output data are inferred from the input data by one or more selected network architectures. For example, the selected one or more network architectures process the input data to obtain one or more inference outputs. An example of inference from the output of a network architecture used to perform a specific task was shown in the discussion above in step 310, and will not be explained again here.

[0051] In step 340, final inference data is obtained based on one or more output data. For example, in the case where multiple inference outputs are obtained from multiple network architectures, the multiple inference outputs are aggregated to obtain the final inference data. For example, the multiple inference outputs are averaged to obtain the final inference data. For example, voting can be performed based on the multiple inference outputs to obtain the final inference data.

[0052] It should be understood that in existing NSA methods, the random selection of the network architecture for weight-sharing NSAs is performed only during the training process. In the embodiments of this disclosure, by randomly selecting one or more network architectures of a trained weight-sharing NSA during inference, the diversity of NSA architectures can be leveraged to enhance protection against attacks on the network architecture. Furthermore, by aggregating the inference results from multiple randomly selected network architectures from the trained NSA, the accuracy and robustness of the trained neural network model can be enhanced.

[0053] Figure 4 A graph showing the accuracy and loss between training and testing according to one implementation scheme is presented.

[0054] In this implementation plan, according to Figure 2 The training is performed using the method described above, and based on... Figure 3 The method executes the inference for the test.

[0055] The loss value is calculated based on the loss function shown in equation (2) discussed above. The accuracy value can be expressed by the formula:

[0056]

[0057] in This refers to the validation dataset used during the testing process. This represents an architecture randomly selected from the trained NSA. It should be understood that both training and validation data include data pairs (x...). i y i Furthermore, this data is only used at different stages.

[0058] Figure 4 The graphs are based on a typical CIFAR-10 image classification task. The NSA size S is set to 500 and 5000 respectively. The accuracy and loss for each batch of data are calculated according to equations (2) and (3), and the average of the accuracy and loss is taken as the accuracy and loss for the entire dataset, as shown in the graphs.

[0059] As shown in the figure, there are differences between training accuracy and test accuracy, as well as between training loss and test loss. This can be referred to as the training / test discrepancy problem.

[0060] To mitigate the training / test discrepancy problem, a solution could be to explore the diversity of randomly selected network architectures during the NSA training process.

[0061] Figure 5 An exemplary method for training a weight-sharing NSA is shown according to one implementation.

[0062] The establishment of the network architecture space is similar to the above reference. Figure 2 As explained, weight-sharing NSA can be a refined architecture space that includes the sampled network architecture, or in some implementations, it can be the entire architecture space.

[0063] The training dataset is grouped in mini-batches, similar to the examples above. Figure 2 The grouping is explained. For example, the training data is divided into M mini-batches, each containing N training data instances. The size of the NSA is S, meaning that the NSA representing the refined architecture space includes S network architectures. In one implementation, S is not less than 500. In another implementation, S is not less than 500 and not greater than 5000. The weights of the NSA can be initialized in any suitable manner.

[0064] In step 510, a mini-batch is selected from multiple mini-batches, for example, a mini-batch is randomly selected from M mini-batches, and the selected mini-batch includes N instances of the training data.

[0065] In step 520, multiple network architectures are selected from the NSA for the chosen mini-batch. For example, N network architectures A1 to A2 can be randomly sampled from the S network architectures of the NSA with a uniform distribution or probability P(A). N Each of the N network architectures corresponds to one instance from the selected mini-batch. In the exemplary NSA discussed above, network architectures A1 to A2 are randomly selected. N Each of them consists of 24 serially connected operation nodes and a set of bypass edges, and each node includes a ReLU-convolution-BN operation triplet.

[0066] In step 530-1, the first instance from the selected minibatch of N instances is applied to the first network architecture in the selected network architecture to obtain a loss value. Similarly, in steps 530-2 to 530-N, each of the second to Nth instances from the selected minibatch is applied to one of the second to Nth network architectures to obtain a loss value. The loss value for an instance based on a network architecture can be expressed by the formula:

[0067] L = -log p(y i |x i W, A i A i ~p(A) (4)

[0068] Where W represents the weights included in the network architecture, (x i y i () represents an instance of training data, for example, x i This represents the data input to the network architecture, y i A represents the data expected to be output by the network architecture. i This represents one of the selected architectures. p(y) i |x i W, A i ) is the prediction distribution, from which the loss value L = -log p(y) is obtained. i |x i W, A i It should be understood that although steps 530-1 to 530-N are described as parallel processes, these steps may also be executed in one or more loops, in which the steps are executed one by one.

[0069] In step 540, the N loss values ​​of the selected mini-batch of N instances are averaged to obtain the average loss value of the mini-batch. The average loss value of a mini-batch can then be expressed by the formula:

[0070]

[0071] Where B represents the selected mini-batch of training data, and |B| represents the number of instances included in the mini-batch. In this example, |B| equals N.

[0072] In step 550, the weights of the selected network architecture are updated based on the average loss. Therefore, the shared weights of the NSA are updated. For example, the first set of gradients for the weights included in architecture A1 can be calculated by backpropagating the average loss of mini-batches along the selected network architecture A1, the second set of gradients for the weights included in architecture A2 can be calculated by backpropagating the average loss along the selected network architecture A2, and so on. N The Nth gradient of the weights in the network can be obtained by traversing the selected network architecture A. N The average loss is calculated using backpropagation. Then, N sets of gradients can be used to update the weights contained in the selected N network architectures. For example, the accumulated gradients of the weights can be used to update each weight contained in the selected N network architectures. As another example, the average of the gradients of the weights can be used to update each weight contained in the selected N network architectures. The shared weights of the selected network architectures include the weights of the operations contained in the network architecture and the weights of the edges contained in the network architecture. In the example of using ReLU-convolution-BN triples in a node, the shared weights include the weights of the convolution operation and the weights of the BN operation. The edge weights can be referred to as summation weights.

[0073] Then, steps 510-550 can be repeated for another mini-batch selected from the M mini-batches. Steps 510-550 can be repeated M times for the M mini-batches of the training dataset. The process of traversing the training dataset by repeating steps 510-550 M times can be called an epoch. Multiple epochs can be performed on the training dataset until a convergence condition is met. For example, a convergence condition could be that the loss value stabilizes below a threshold.

[0074] Although step 520 shows that N network architectures are selected for a selected mini-batch including N instances, it is also applicable to select fewer network architectures for a mini-batch, for example, selecting N / 2 or N / 3 network architectures for a mini-batch while requesting more than one network architecture.

[0075] Although the average loss is updated by backpropagating along multiple selected network architectures based on gradients to update the NSA weights, it is also applicable to have multiple loss values ​​updated by backpropagating along multiple selected network architectures based on gradients to update the NSA weights.

[0076] Figure 6 A graph showing the accuracy and loss between training and testing according to one implementation scheme is presented.

[0077] In this implementation plan, according to Figure 5 The training is performed using the method described above, and based on... Figure 3 The method performs the inference for the test. The loss value is calculated based on the loss function shown in equation (5) discussed above, and the accuracy value is calculated based on the accuracy function shown in equation (3) discussed above.

[0078] Figure 6 The chart is based on and Figure 4 The charts were obtained under similar conditions, but the training process differed. Specifically, a typical CIFAR-10 image classification task was performed. The NSA size S was set to 500 and 5000, respectively. The accuracy and loss for each batch of data were calculated according to equations (5) and (3), and the average of the accuracy and loss was used as the accuracy and loss for the entire dataset, as shown in the chart.

[0079] As shown in the table, with Figure 4 Compared to the example shown, Figure 6 The differences between training accuracy and test accuracy, as well as between training loss and test loss, were significantly reduced.

[0080] According to one implementation, to further explore the diversity of network architectures randomly selected during the training and inference processes of the NSA, additional weights for each network architecture of the NSA can be configured in addition to the shared weights of the NSA. Considering computational and storage resource requirements, architecture-specific weights can be low-dimensional. In one implementation, weights used for summation aggregation (i.e., edge weights) and weights used for affine transformations in BN operations (i.e., weights used for BN operations) can be configured as architecture-specific weights. Therefore, weights other than architecture-specific weights can be configured as shared weights of the NSA; for example, in the example of ReLU-convolution-BN operation triples used in nodes, the weights of the convolution operation can be configured as shared weights of the NSA. Thus, in addition to the shared weight set for all network architectures of the NSA, S architecture-specific weight sets (e.g., affine weights and biases for BN, and summation coefficients for aggregation) are configured for each of the S network architectures of the NSA. Only when in such cases... Figure 2 and Figure 5 The set of architecture-specific weights for a network architecture is updated only when a network architecture is selected at the training step illustrated in the example.

[0081] Back Figure 5 In an implementation scheme with architecture-specific weights configured, the shared weight set and the S architecture-specific weight sets of the NSA are initialized before step 510.

[0082] In step 510, a mini-batch is selected from multiple mini-batches, for example, a mini-batch is randomly selected from M mini-batches, and the selected mini-batch includes N instances of the training data.

[0083] In step 520, multiple network architectures are selected from the NSA for the chosen mini-batch. For example, N network architectures A1 to A2 can be randomly sampled from the S network architectures of the NSA with a uniform distribution or probability P(A). N Each of the N network architectures corresponds to one instance in the selected mini-batch. For each of the N selected network architectures, the shared weights contained in the network architecture come from the shared weight set of the NSA, and the architecture-specific weights of the network architecture are one of the S architecture-specific weight sets corresponding to the network architecture. Therefore, the shared weights contained in the network architecture and the architecture-specific weights of the network architecture constitute the weights of the network architecture, or in other words, constitute the network architecture.

[0084] In step 530-1, the first instance of the selected minibatch of N instances is applied to the first network architecture of the selected network architecture to obtain a loss value, where the first network architecture has shared weights and architecture-specific weights. Similarly, in steps 530-2 to 530-N, each of the second to Nth instances of the selected minibatch is applied to one of the second to Nth network architectures to obtain a loss value, where each of the second to Nth network architectures has shared weights and architecture-specific weights. The loss value for an instance based on a network architecture can be calculated according to the above equation (4).

[0085] In step 540, the average of the N loss values ​​of the selected mini-batch N instances is calculated to obtain the average loss value of the mini-batch.

[0086] In step 550, the weights of the selected network architecture are updated based on the average loss. For example, the first set of gradients for the weights included in architecture A1 (including shared weights and architecture-specific weights) can be calculated by backpropagating the average loss of mini-batches along the selected network architecture A1, the second set of gradients for the weights included in architecture A2 can be calculated by backpropagating the average loss along the selected network architecture A2, and so on. N The Nth gradient of the weights in the network can be obtained by traversing the selected network architecture A. N The average loss is calculated using backpropagation. Then, the weights contained in the selected N network architectures can be updated using N sets of gradients. For example, each shared weight in the selected N network architectures can be updated using the accumulated gradient of the shared weights. Similarly, each architecture-specific weight in the selected N network architectures can be updated using the accumulated gradient of the architecture-specific weights, and thus, as... Figure 5The exemplary training step shown updates N architecture-specific weight sets for N network architectures. As another example, the average of the gradients of the weights can be used to update each weight included in the selected N network architectures. In one implementation, the shared weights of the selected network architectures include weights containing at least a portion of the operations within the network architecture, and the architecture-specific weights of the selected network architectures include weights containing both the partial operations and edge weights within the network architecture. In the example using ReLU-convolution-BN triples in a node, the shared weights include the weights of the convolution operation, and the architecture-specific weights include the weights of the BN operation and the edge weights.

[0087] It should be understood that solutions with specific weights can also be applied in a similar manner. Figure 2 In the method shown.

[0088] Figure 7 An exemplary method for training a weight-sharing network with a random architecture, according to one implementation, is shown.

[0089] In step 710, a mini-batch is selected from multiple mini-batches, wherein the training dataset for the task is grouped into multiple mini-batches, and each mini-batch in the multiple mini-batches includes multiple instances.

[0090] In step 720, multiple network architectures of the neural network are randomly selected for the selected mini-batch.

[0091] In step 730, the loss for that instance is obtained by applying each instance of the selected mini-batch to one of the selected network architectures.

[0092] In step 740, the shared weights of the neural network are updated based on the loss of each instance in the selected mini-batch.

[0093] In one implementation, the neural network includes a set of nodes and a set of edges, where each node represents at least one operation and each edge connects two nodes. Each network architecture of the neural network is represented as a directed graph of nodes connected by edges.

[0094] In one implementation, the shared weights of the neural network include at least some of the operations of the nodes. In one implementation, at least some of the operations include convolution operations.

[0095] In one implementation, the gradient of the shared weights of the neural network is computed by backpropagating the average loss of each instance of the selected mini-batch along the selected multiple network architectures, or by backpropagating the loss of each instance of the selected mini-batch along the corresponding network architecture of the selected multiple network architectures, and the shared weights of the neural network are updated by accumulating the gradients of each shared weight in the shared weights.

[0096] In one implementation, the neural network also includes architecture-specific weights for each network architecture. The architecture-specific weights of each of the selected plurality of network architectures are updated based on the loss for each instance of the selected mini-batch.

[0097] In one implementation, the architecture-specific weights of each network architecture of the neural network include at least one of the following: the weights of the edges of the network architecture, and the weights of a portion of the computations of the network architecture. In one implementation, the portion of the computations includes batch normalization (BN) operations.

[0098] In one implementation, the gradient of the architecture-specific weights contained in the selected multiple network architectures is computed by backpropagating the average loss of each instance of the selected mini-batch along the selected multiple network architectures, or by backpropagating the loss of each instance of the selected mini-batch along the corresponding network architecture in the selected multiple network architectures, and the architecture-specific weights of each of the selected multiple network architectures are updated by accumulating the gradients of each architecture-specific weight contained in the architecture-specific weights of the network architectures.

[0099] In one implementation, the neural network includes a main chain comprising a set of nodes connected in series by edges, and each network architecture of the neural network includes a main chain.

[0100] Figure 8 An exemplary computing system 800 according to one embodiment is illustrated. The computing system 800 may include at least one processor 810. The computing system 800 may also include at least one storage device 820. The storage device 820 may store computer-executable instructions that, when executed, cause the processor 810 to: select a mini-batch from a plurality of mini-batches, wherein a training dataset for a task is grouped into the plurality of mini-batches, and each mini-batch includes a plurality of instances; randomly select a plurality of network architectures of a neural network for the selected mini-batch; obtain a loss for each instance of the selected mini-batch by applying it to one of the plurality of network architectures; and update the shared weights of the neural network based on the loss of each instance of the selected mini-batch.

[0101] Storage device 820 may store computer-executable instructions that, when executed, cause processor 810 to: receive input data; randomly select one or more network architectures of a neural network; infer one or more output data based on the input data using the selected one or more network architectures; and obtain final inference data based on the one or more output data.

[0102] It should be understood that the storage device 820 may store computer-executable instructions, which, when executed, cause the processor 810 to perform actions according to the following: Figures 1 to 7 Any operation of the embodiments described in this disclosure.

[0103] Embodiments of this disclosure can be embodied in a computer-readable medium, such as a non-transitory computer-readable medium. A non-transitory computer-readable medium may include instructions that, when executed, cause one or more processors to perform actions according to the following combination... Figures 1 to 8 Any operation of the embodiments described in this disclosure.

[0104] It should be understood that all operations in the above methods are merely exemplary, and this disclosure is not limited to any operation in the method or the order of such operations, and should cover all other equivalents under the same or similar concepts.

[0105] It should also be understood that all modules in the aforementioned apparatus can be implemented in various ways. These modules can be implemented as hardware, software, or a combination thereof. Furthermore, any one of these modules can be further functionally divided into sub-modules or combined together.

[0106] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to limit them to the aspects shown herein. All structural and functional equivalents of elements throughout the various aspects described herein, known or to be known hereafter by those skilled in the art, are expressly incorporated herein by reference and are intended to be covered by the claims.

Claims

1. A method for training a weight-sharing neural network with a stochastic architecture, the neural network including a neural network for performing image processing, the method comprising: Mini-batches are selected from multiple mini-batches, the training dataset for the task is grouped into the multiple mini-batches, and each of the multiple mini-batches includes multiple instances, each of the multiple instances including an image instance; Randomly select multiple network architectures of the neural network for the selected mini-batch; The loss of the instances is obtained by applying each instance of the selected mini-batch to one of the multiple network architectures; and The shared weights of the neural network are updated based on the loss for each instance of the selected mini-batch. The neural network further includes architecture-specific weights for each network architecture of the neural network, and the method further includes: The architecture-specific weights of each of the selected multiple network architectures are updated based on the loss of each instance in the selected mini-batch.

2. The method of claim 1, wherein the neural network comprises a set of nodes and a set of edges, each node representing at least one operation, each edge connecting two nodes, and each network architecture of the neural network being represented as a directed graph of nodes connected by edges.

3. The method of claim 2, wherein the shared weights of the neural network include at least a portion of the computations of the nodes.

4. The method of claim 3, wherein the at least part of the operation includes a convolution operation.

5. The method of claim 3, wherein updating the shared weights of the neural network based on the loss for each instance of the selected mini-batch further comprises: The gradient of the shared weights of the neural network is calculated by backpropagating the average loss of each instance of a selected mini-batch along the selected multiple network architectures, or by backpropagating the loss of each instance of a selected mini-batch along the corresponding network architecture of the selected multiple network architectures; and the shared weights of the neural network are updated by using the accumulated or average value of the gradients of each shared weight.

6. The method of claim 2, wherein the architecture-specific weights of each network architecture of the neural network include at least one of the following: the weights of the edges of the network architecture, and the weights of a portion of the computations of the network architecture.

7. The method of claim 6, wherein the portion of the operation includes batch normalization (BN) operation.

8. The method of claim 6, wherein updating the architecture-specific weights of each of the selected plurality of network architectures based on the loss for each instance of the selected mini-batch further comprises: The gradient of the architecture-specific weights contained in the selected network architectures is calculated by either backpropagating the loss of each instance of a selected mini-batch along the selected network architectures, or by backpropagating the loss of each instance of a selected mini-batch along the corresponding network architecture in the selected network architectures; and The architecture-specific weights of each of the selected plurality of network architectures are updated by using the sum or average of the gradients of each architecture-specific weight in the architecture-specific weights contained in the network architecture.

9. The method of claim 2, wherein the neural network includes a main chain, the main chain including the set of nodes connected in series by edges, and each network architecture of the neural network includes the main chain.

10. The method according to claim 1, further comprising: Repeat the steps of claim 1 until all of the plurality of mini-batches have been selected once.

11. The method according to claim 10, further comprising: Repeat the repetition steps as described in claim 10 until the convergence condition is met.

12. A method for inference using a weight-sharing neural network, the neural network including a neural network for performing an image processing task, the method comprising: Receive input data, including images; Randomly select one or more network architectures of the neural network; One or more output data are inferred from the input data by one or more selected network architectures respectively; as well as The final inference data is obtained based on the one or more output data. The neural network includes: shared weights of the neural network, and architecture-specific weights for each network architecture of the neural network.

13. The method of claim 12, wherein the neural network comprises a set of nodes and a set of edges, each node representing at least one operation, each edge connecting two nodes, and each network architecture of the neural network being represented as a directed graph of nodes connected by edges.

14. The method of claim 13, wherein the shared weights of the neural network include at least a portion of the operations of the nodes, and the architecture-specific weights of each network architecture of the neural network include at least one of: the weights of the edges of the network architecture, and the weights of a portion of the operations of the network architecture.

15. The method of claim 12, wherein the neural network is trained by using the method of any one of claims 1 to 11.

16. A computer system, the computer system comprising: One or more processors; and One or more storage devices storing computer-executable instructions that, when executed, cause the one or more processors to perform operations according to any one of claims 1 to 15.

17. One or more computer-readable storage media storing computer-executable instructions, which, when executed, cause one or more processors to perform operations according to any one of claims 1 to 15.

Images