Network structure search method and device

Abstract

The present disclosure proposes a network structure search method and device, which relates to the field of computers. The method includes: constructing the optimization of the network structure as the maximum a posteriori of the weight of the network structure; performing the optimization calculation of the network structure based on the maximum a posteriori of the weight of the network structure to obtain the distribution of the weight of the better network structure; sampling according to the distribution of the weight of the better network structure to obtain multiple weights; selecting a target network structure that meets the performance requirements according to the candidate network structure corresponding to each weight obtained by sampling. The distribution of the weight of the better network structure can be regarded as a surface estimate of the candidate network structure, and multiple candidate network structures can be obtained based on the distribution, giving multiple solutions.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a network structure search method and device. Background Art

[0002] When experts model specific decision-making tasks, they need to combine domain knowledge and real-world application scenarios to complete the design of the network structure. However, manually designing the network structure requires extremely high professional knowledge and design experience from the designer. For example, the network structures such as ResNet, which shines in image classification tasks, and Transformer, which dominates machine translation tasks, all come from the careful design of experts. This undoubtedly brings great challenges to the design of network structures.

[0003] In order to deal with the design problem of network structure, network structure search, which is a machine-automatic design of network structure, has attracted much attention in the industry. Network structure search methods generally start with a set of predefined operation sets, use specific search strategies to obtain a large number of candidate network structures based on the search space, train and evaluate the candidate network structures, adjust the search strategy based on the evaluation ranking information of the candidate network structures, and further obtain new candidate network structures. When the search is terminated, the candidate network structure with the highest ranking is used as the final optimal network structure for the final performance evaluation.

[0004] The related technology mainly encodes the network structure into a set of vectors in a continuous space, and then optimizes the network structure through a differentiable network structure search algorithm based on maximum likelihood estimation. Each search can only obtain a single-point estimate of the best candidate network structure and provide a solution. Summary of the invention

[0005] The disclosed embodiment constructs the optimization of the network structure as the maximum a posteriori of the weights of the network structure. The optimization calculation of the network structure is performed based on the maximum a posteriori of the weights of the network structure to obtain the distribution of weights of a better network structure, which can be regarded as a surface estimate of the candidate network structure. Based on the distribution, multiple candidate network structures can be obtained and multiple solutions can be given.

[0006] Some embodiments of the present disclosure provide a network structure search method, including:

[0007] The optimization of the network structure is constructed as the maximum a posteriori of the weights of the network structure;

[0008] The network structure is optimized by calculating the maximum a posteriori of the weight of the network structure to obtain the distribution of the weight of the better network structure;

[0009] Sampling is performed according to the distribution of weights of a better network structure to obtain multiple weights;

[0010] According to the candidate network structures corresponding to each weight obtained by sampling, the target network structure that meets the performance requirements is selected.

[0011] In some embodiments, constructing the optimization of the network structure as the maximum a posteriori of the weights of the network structure includes: constructing the weights of the network structure as equal to the maximum a posteriori of the weights of the network structure under conditions of a given data set and a hyper-network parameter of the network structure, wherein the given data set includes multiple sets of data pairs of input data and their target labels.

[0012] In some embodiments, the maximum a posteriori of the weights of the network structure is constructed based on the prior distribution of the weights of the network structure and the weights of the network structure under the optimal discriminant probability model, wherein the discriminant probability model is the discriminant probability of the target label under the conditions of input data, super-network parameters of the network structure and weights of the network structure.

[0013] In some embodiments, performing an optimization calculation of the network structure based on the maximum a posteriori of the network structure's weights to obtain a more optimal distribution of the network structure's weights includes: performing variational inference learning on the posteriori of the network structure's weights to obtain an approximation of the posteriori of the network structure's weights; and performing a maximum a posteriori calculation of the network structure's weights based on the approximation of the posteriori of the network structure's weights to obtain a more optimal distribution of the network structure's weights.

[0014] In some embodiments, performing variational inference learning on the posterior of the weights of the network structure to obtain an approximation of the posterior of the weights of the network structure includes:

[0015] Constructing a proxy Gaussian distribution, wherein the proxy Gaussian distribution has a mathematical expectation and a standard deviation as Gaussian distribution parameters and a weight of a network structure as a variable;

[0016] Calculating target Gaussian distribution parameters that minimize the distance between the proxy Gaussian distribution and the posterior of the weight of the network structure;

[0017] The proxy Gaussian distribution under the target Gaussian distribution parameters is used as an approximation of the posterior of the weights of the network structure.

[0018] In some embodiments, calculating the target Gaussian distribution parameters that minimize the distance between the proxy Gaussian distribution and the posterior of the weights of the network structure includes:

[0019] Constructing the posterior of the weights of the network structure according to the prior distribution of the weights of the network structure and the discriminant probability model, wherein the discriminant probability model is the discriminant probability of the target label under the conditions of the input data, the hypernetwork parameters of the network structure and the weights of the network structure;

[0020] Calculate target Gaussian distribution parameters that minimize the difference between the distance between the proxy Gaussian distribution and the prior distribution of the weight of the network structure and the mathematical expectation, wherein the mathematical expectation is the mean of the discriminant probability model based on the proxy Gaussian distribution.

[0021] In some embodiments, calculating the target Gaussian distribution parameter that minimizes the difference between the distance between the proxy Gaussian distribution and the prior distribution of the weights of the network structure and the mathematical expectation includes:

[0022] Extracting each Monte Carlo sample from the proxy Gaussian distribution as each posterior sample of the weight of the network structure;

[0023] Based on each posterior sample of the weight of the network structure, the target Gaussian distribution parameters that minimize the target difference are calculated, and the target difference is determined according to the total difference between the first information and the second information and the third information under each posterior sample. The first information is determined according to the proxy Gaussian distribution under each posterior sample, the second information is determined according to the prior distribution of the weight of the network structure under each posterior sample, and the third information is determined according to the discriminant probability model under each posterior sample.

[0024] In some embodiments, extracting each Monte Carlo sample from the proxy Gaussian distribution as each posterior sample of the weight of the network structure includes: sampling based on the standard Gaussian distribution, shifting using the mathematical expectation of the proxy Gaussian distribution, scaling using the standard deviation of the proxy Gaussian distribution, and introducing random variable noise to obtain the posterior sample of the weight of the network structure.

[0025] In some embodiments, the prior distribution of the weights of the network structure is determined according to a proportional mixture distribution of two preset Gaussian distributions.

[0026] In some embodiments, based on the candidate network structures corresponding to each weight obtained by sampling, selecting a target network structure that meets the performance requirements includes:

[0027] Perform performance evaluation on the candidate network structure corresponding to each weight obtained by sampling;

[0028] According to the performance evaluation results, a target network structure that meets the performance requirements is selected from multiple candidate network structures.

[0029] In some embodiments, a network structure performance evaluator is used to perform performance evaluation on the candidate network structure corresponding to each weight obtained by sampling, and the network structure performance evaluator includes at least one of a Jacob covariance evaluator and a network synaptic flow evaluator.

[0030] In some embodiments, the selected multiple target network structures are deployed on different types of hardware devices according to business needs.

[0031] Some embodiments of the present disclosure provide a network structure search device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the network structure search method of each embodiment based on instructions stored in the memory.

[0032] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the network structure search method of each embodiment.

[0033] Some embodiments of the present disclosure provide a network structure search device, including:

[0034] The distribution determination module is configured to construct the optimization of the network structure as the maximum a posteriori of the weights of the network structure; and to calculate the optimization of the network structure based on the maximum a posteriori of the weights of the network structure to obtain a distribution of the weights of a better network structure;

[0035] A sampling module is configured to obtain a plurality of weights by sampling according to the distribution of weights of a better network structure;

[0036] The selection module is configured to select a target network structure that meets the performance requirements based on the candidate network structure corresponding to each weight obtained by sampling. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.

[0038] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.

[0039] Figure 1 A schematic diagram showing a directed acyclic graph in a network structure search and the network structure of one of its subgraphs.

[0040] Figure 2 A flowchart of a network structure search method according to some embodiments of the present disclosure is shown.

[0041] Figure 3 A schematic diagram of the structure of a network structure search device according to some embodiments of the present disclosure is shown.

[0042] Figure 4 A schematic diagram of the structure of a network structure search device according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure.

[0044] Unless otherwise specified, descriptions such as “first” and “second” in the present disclosure are used to distinguish different objects and are not used to indicate meanings such as size or time sequence.

[0045] Network structure search is mainly based on a given search space to search for a better network structure. The search space can be represented as a directed acyclic graph, in which any subgraph represents a network structure, each node represents a local calculation, such as matrix multiplication, and the directed connections between nodes represent the flow of information. Figure 1 As shown in the figure, the left side is a directed acyclic graph. Assuming that the subgraph in which node 1 points to node 2 and node 4, and node 2 points to node 3 is selected, the subgraph can be converted into the network structure on the right, which contains 4 computing nodes tanh(1), ReLU(2), ReLU(3), and tanh(4). The input x[t], h[t-1], and output h[t] are fixed nodes. avg means averaging, ReLU means linear rectification, and tanh means hyperbolic tangent.

[0046] Given a dataset Where (x i ,y i ) is the i-th group of input data x i and its target label y i There are N data pairs of this type. x represents the input data x i A collection of y represents the target label y i A collection of .

[0047] According to the relevant technology, the optimization of the network structure (the network structure is represented by the weight of the network structure) is constructed by optimizing the discriminant probability model P(y|x,ω,α) through maximum likelihood estimation (Maximum Likelihood Estimation, MLE), and combining gradient descent to obtain a single point estimate of the best candidate network structure. The weight of the candidate network structure obtained by the MLE search method is marked as α MLE , see formula (1) for details. It should be noted that the weight of the network structure can characterize the corresponding network structure. The weight of the network structure has a one-to-one correspondence with the network structure. By searching for the optimal weight of the network structure, the optimal network structure is also obtained.

[0048] α MLE = arg max α logP(y|x,ω,α)=arg max α ∑ i log P(yi |x i ,ω,α)(1)

[0049] Among them, P(y|x, ω, α) represents the probability of y occurring under the conditions of x, ω, α, and is differentiable with respect to α. i |x i ,ω,α) means that at x i , y under the conditions of ω, α i The probability of occurrence, α represents the weight of the network structure, which can characterize the corresponding network structure, ω represents the super network parameter of the network structure (such as the super network weight), arg max x The f(x) function represents the calculation of the variable point x corresponding to the maximum value of f(x), for example, arg max α logP(y|x, ω, α) represents α corresponding to the maximum value of logP(y|x, ω, α).

[0050] As mentioned above, the MLE search method can only obtain a single-point estimate of the best candidate network structure each time and give a solution.

[0051] The present disclosure can obtain a distribution of weights of a better network structure based on the maximum a posteriori of the weights of the network structure, which can be regarded as a surface estimate of the candidate network structure. Based on the distribution, multiple candidate network structures can be obtained and multiple solutions can be given. The specific solution is described below.

[0052] Figure 2 A flowchart of a network structure search method according to some embodiments of the present disclosure is shown.

[0053] like Figure 2 As shown, the network structure search method of this embodiment includes the following steps.

[0054] In step 210, the optimization of the network structure is constructed as the maximum a posteriori (MAP) of the weights of the network structure.

[0055] The optimization of the network structure is constructed as the maximum a posteriori of the weight of the network structure, that is, the optimization task of the network structure is constructed as the maximum a posteriori of the weight of the network structure, specifically including: constructing the weight of the network structure as the maximum a posteriori of the weight of the network structure under the conditions of the given data set and the super network parameters of the network structure, and the weight distribution of the candidate network structure obtained by the MAP search method is marked as α MAP , see formula (2) for details.

[0056] As mentioned above, the given data set includes multiple sets of data pairs of input data and their target labels. As an example, each set of data pairs of input data and their target labels in the given data set includes, for example, but is not limited to, input images and their category labels, thereby searching for various network structures for completing image classification tasks. Obviously, if data pairs of other tasks are given, various network structures of other tasks can also be searched, which will not be listed here one by one.

[0057] A prior is imposed on the weight α of the network structure, thereby introducing a prior regularization term in the optimization problem. Thus, the maximum a posteriori of the weight of the network structure is constructed according to the prior distribution P(α) of the weight of the network structure and the weight of the network structure under the optimal discrimination probability model P(y|x, ω, α), see formula (2) for details. As mentioned above, the discrimination probability model is the discrimination probability of the target label under the conditions of the input data, the hypernetwork parameters of the network structure and the weight of the network structure.

[0058] α MAP = arg max α P(α|x,y,ω)=arg max α logP(y|x,ω,α)+logP(α) (2)

[0059] Among them, P(α|x, y, ω) represents the probability of α occurring under the conditions of x, y, ω, and is the posterior weight of the network structure. The meanings of other symbols are as mentioned above and will not be repeated here. The subsequent formulas are also handled in the same way.

[0060] The prior distribution of the weight of the network structure is determined according to the proportional mixture distribution of two preset Gaussian distributions, see formula (3) for details. Thus, a heavier tail is provided in the prior distribution of the weight of the network structure, rather than an ordinary Gaussian prior.

[0061]

[0062] Among them, α j is the jth weight of the network structure, and are two preset Gaussian distributions, β and (1-β) are the proportional coefficients of the mixture of the two preset Gaussian distributions, is in α j The probability density function of this Gaussian distribution evaluated at has a mean of μ1 and a variance of is in α j The probability density function of this Gaussian distribution evaluated at has a mean of μ2 and a variance of In some embodiments, σ1>σ2, but the present disclosure does not limit the size relationship between σ1 and σ2.

[0063] In step 220, the network structure is optimized by calculating the maximum a posteriori of the weights of the network structure to obtain a better distribution of the weights of the network structure.

[0064] P(α|x,y,ω)∝P(α)Π i P(y i |x i ,ω,α), however, P(y i |x i ,ω,α) is a complex function of α represented by a neural network, and direct reasoning is difficult to calculate. In order to calculate the posterior distribution of the weights of the network structure of a given data set, the present disclosure adopts a variational inference learning method to solve this problem.

[0065] The optimization calculation of the network structure based on the maximum a posteriori of the network structure weights to obtain a better distribution of the network structure weights includes: performing variational inference learning on the posteriori of the network structure weights to obtain an approximation of the posteriori of the network structure weights; and performing the maximum a posteriori calculation of the network structure weights based on the approximation of the posteriori of the network structure weights to obtain a better distribution of the network structure weights.

[0066] The variational inference learning of the posterior of the weight of the network structure is performed to obtain an approximation of the posterior of the weight of the network structure, including: constructing a proxy Gaussian distribution, wherein the proxy Gaussian distribution Q(α|θ) is a Gaussian distribution with a mathematical expectation μ and a standard deviation ρ as Gaussian distribution parameters θ=(μ, ρ) and a weight α of the network structure as a variable; calculating a target Gaussian distribution parameter θ that minimizes the distance between the proxy Gaussian distribution Q(α|θ) and the posterior of the weight of the network structure P(α|x, y, ω) * , where the distance is, for example, the Kullback-Leibler (KL) distance; the target Gaussian distribution parameter θ * The proxy Gaussian distribution Q(α|θ * ) is used as an approximation of the posterior of the weights of the network structure. For details, see formula (4).

[0067] Calculating the target Gaussian distribution parameters that minimize the distance between the proxy Gaussian distribution and the posterior of the weight of the network structure includes: constructing the posterior of the weight of the network structure according to the prior distribution of the weight of the network structure and the discriminant probability model, wherein the discriminant probability model is the discriminant probability of the target label under the conditions of input data, the super network parameters of the network structure and the weight of the network structure; calculating the target Gaussian distribution parameters that minimize the difference between the distance between the proxy Gaussian distribution Q(α|θ) and the prior distribution P(α) of the weight of the network structure and the mathematical expectation, wherein the mathematical expectation E Q(α|θ)(log P(y|x, ω, α)) is the mean of the discriminant probability model log P(y|x, ω, α) based on the proxy Gaussian distribution Q(α|θ), see formula (4) for details.

[0068]

[0069] Among them, arg min x The f(x) function represents the calculation of the variable point x corresponding to the minimum value of f(x), for example, Let θ be the one that minimizes the KL distance between Q(α|θ) and P(α|x, y, ω) * .

[0070] Directly optimizing the objective function in formula (4) is computationally very difficult, and the present disclosure uses unbiased Monte Carlo gradient estimation and Gaussian variational posteriors to describe it. The expected lower limit of the marginal possibility of the network structure is minimized by unbiased Monte Carlo gradient estimation. Assuming that the variational posterior of the weight of the network structure is a diagonal Gaussian distribution, the posterior sample of the weight α of the network structure can be obtained by sampling the standard Gaussian distribution, shifting it with the mathematical expectation μ, and scaling it with the standard deviation σ. σ=log(1+exp(ρ)), so σ is always non-negative. Therefore, the posterior sample of the weight α of the network structure can be obtained according to formula (5).

[0071] α=μ+log(1+exp(ρ))·z (5)

[0072] where z is a random variable noise from a standard Gaussian distribution p(z).

[0073] That is, extracting each Monte Carlo sample from the proxy Gaussian distribution as each posterior sample of the weight of the network structure includes: sampling based on the standard Gaussian distribution, shifting using the mathematical expectation of the proxy Gaussian distribution, scaling using the standard deviation of the proxy Gaussian distribution, and introducing random variable noise to obtain the posterior samples of the weight of the network structure.

[0074] Then, using the reparameterization strategy, the objective function in formula (4) is transformed into formula (6), and the θ that minimizes the loss L(y, x, ω, θ) is calculated as θ * .

[0075]

[0076] where α i represents the i-th Monte Carlo sample extracted from Q(α|θ) in each optimization process (i.e., each posterior sample of the weight of the network structure), α i The random noise samples in z are sampled from a standard Gaussian distribution p(z)i , M is the number of samples in the optimization process.

[0077] That is, calculating the target Gaussian distribution parameter that minimizes the difference between the distance and the mathematical expectation between the proxy Gaussian distribution and the prior distribution of the weight of the network structure includes: extracting each Monte Carlo sample from the proxy Gaussian distribution as each posterior sample of the weight of the network structure; based on each posterior sample of the weight of the network structure, calculating the target Gaussian distribution parameter that minimizes the target difference, wherein the target difference is determined according to the total difference of the first information minus the second information minus the third information under each posterior sample, and the first information is determined according to each posterior sample α i The proxy Gaussian distribution Q(α i |θ) is determined, and the second information is determined according to each posterior sample α i The prior distribution P(α) of the weight of the network structure under i ) is determined, the third information is determined according to each posterior sample α i The discriminant probability model P(y|x, ω, α i )Sure.

[0078] In step 230, a plurality of weights are obtained by sampling according to the distribution of weights of a preferred network structure.

[0079] The distribution of weights of better network structures can be regarded as a surface estimate of candidate network structures, and multiple candidate network structures can be obtained through sampling based on the distribution.

[0080] In step 240, based on the candidate network structures corresponding to each weight obtained by sampling, a target network structure that meets the performance requirements is selected.

[0081] According to the candidate network structures corresponding to each weight obtained by sampling, selecting a target network structure that meets the performance requirements includes: performing performance evaluation on the candidate network structures corresponding to each weight obtained by sampling; and selecting multiple or one target network structure that meets the performance requirements from multiple candidate network structures according to the performance evaluation results.

[0082] The performance of the candidate network structure corresponding to each weight obtained by sampling is evaluated by using a network structure performance evaluator, and the network structure performance evaluator includes at least one of a Jacobian convariance evaluator and a network synaptic flow evaluator. These network structure performance evaluators do not need to be retrained. The optimal network structure is further quickly evaluated by a training-free network structure performance estimator, which is designed to perform performance evaluation on the network structure at initialization without training, which greatly reduces the computational cost of the network structure performance estimator.

[0083] The disclosed embodiment of the present invention is based on a search method for the distribution of network structure weights based on variational reasoning and a network structure selection scheme for sampling and evaluating from the distribution, and obtains a distribution of weights of a better network structure, which can be regarded as a surface estimate of a candidate network structure, and is of great significance for efficiently obtaining a large number of candidate network structures. Specifically, by introducing a priori constraints on network structure weights, the network structure optimization model is transformed from the previous single point estimation to a maximum a posteriori surface estimate, and the strategy of variational reasoning is used to further transform the network structure search into the optimization of the network structure distribution. For the optimized network structure distribution, a high-performance network structure is further selected from the distribution through evaluation to complete the network structure search, so that multiple groups of high-performance network structures can be obtained through a single search.

[0084] Based on a given data set, multiple network structures that can complete corresponding tasks (or have corresponding functions) can be obtained, so as to provide multiple solutions according to business needs. For example, when a network structure with a certain function needs to be deployed on different types of hardware devices (such as mobile phones, personal computers, servers, etc.), the present disclosure can provide multiple network structures for business parties to choose from.

[0085] In addition, in variational inference, noise is introduced during the optimization of the network structure distribution, which is beneficial to enhance the robustness of the network structure search.

[0086] In addition, in variational inference, a proxy Gaussian distribution is used to approximate the distribution of weights of the network structure. The proxy Gaussian distribution can quantitatively express the mean and variance of the distribution, and the variance can be used to quantitatively evaluate the uncertainty of the searched network structure.

[0087] Figure 3 A schematic diagram showing the structure of a network structure search device according to some embodiments of the present disclosure is shown. Figure 3 As shown, the network structure searching device 300 of this embodiment includes the following modules.

[0088] The distribution determination module 310 is configured to construct the optimization of the network structure as the maximum a posteriori of the weights of the network structure; and to calculate the optimization of the network structure based on the maximum a posteriori of the weights of the network structure to obtain a better distribution of the weights of the network structure.

[0089] The sampling module 320 is configured to obtain a plurality of weights by sampling according to the distribution of weights of a better network structure.

[0090] The selection module 330 is configured to select a target network structure that meets the performance requirements based on the candidate network structure corresponding to each weight obtained by sampling.

[0091] The distribution determination module 310 includes a construction unit 311 and a distribution determination unit 312 .

[0092] The construction unit 311 is configured to construct the optimization of the network structure as the maximum a posteriori of the weights of the network structure, including: constructing the weights of the network structure as the maximum a posteriori of the weights of the network structure under the conditions of a given data set and a super-network parameter of the network structure, wherein the given data set includes multiple sets of data pairs of input data and their target labels.

[0093] The maximum a posteriori of the weights of the network structure is constructed according to the prior distribution of the weights of the network structure and the weights of the network structure under the optimal discrimination probability model, wherein the discrimination probability model is the discrimination probability of the target label under the conditions of input data, super-network parameters of the network structure and weights of the network structure.

[0094] The distribution determination unit 312 is configured to perform an optimization calculation of the network structure based on the maximum a posteriori of the weights of the network structure to obtain a more optimal distribution of the weights of the network structure, including: performing variational inference learning on the posteriori of the weights of the network structure to obtain an approximation of the posteriori of the weights of the network structure; performing a maximum a posteriori calculation of the weights of the network structure based on the approximation of the posteriori of the weights of the network structure to obtain a more optimal distribution of the weights of the network structure.

[0095] The distribution determination unit 312 performs variational inference learning on the posteriori of the weights of the network structure to obtain an approximation of the posteriori of the weights of the network structure, including: constructing a proxy Gaussian distribution, wherein the proxy Gaussian distribution is a Gaussian distribution with mathematical expectation and standard deviation as Gaussian distribution parameters and the weights of the network structure as variables; calculating target Gaussian distribution parameters that minimize the distance between the proxy Gaussian distribution and the posteriori of the weights of the network structure; and using the proxy Gaussian distribution under the target Gaussian distribution parameters as an approximation of the posteriori of the weights of the network structure.

[0096] The distribution determination unit 312 calculates the target Gaussian distribution parameters that minimize the distance between the proxy Gaussian distribution and the posterior of the weight of the network structure, including: constructing the posterior of the weight of the network structure according to the prior distribution of the weight of the network structure and the discriminant probability model, wherein the discriminant probability model is the discriminant probability of the target label under the conditions of input data, super network parameters of the network structure and the weight of the network structure; calculating the target Gaussian distribution parameters that minimize the difference between the distance between the prior distribution of the weight of the proxy Gaussian distribution and the network structure and the mathematical expectation, wherein the mathematical expectation is the mean of the discriminant probability model based on the proxy Gaussian distribution.

[0097] The distribution determination unit 312 calculates the target Gaussian distribution parameters that minimize the difference between the distance and the mathematical expectation between the proxy Gaussian distribution and the prior distribution of the weight of the network structure, including: extracting each Monte Carlo sample from the proxy Gaussian distribution as each posterior sample of the weight of the network structure; based on each posterior sample of the weight of the network structure, calculating the target Gaussian distribution parameters that minimize the target difference, the target difference is determined according to the total difference between the first information minus the second information minus the third information under each posterior sample, the first information is determined according to the proxy Gaussian distribution under each posterior sample, the second information is determined according to the prior distribution of the weight of the network structure under each posterior sample, and the third information is determined according to the discriminant probability model under each posterior sample.

[0098] The distribution determination unit 312 extracts each Monte Carlo sample from the proxy Gaussian distribution as each posterior sample of the weight of the network structure, including: sampling based on the standard Gaussian distribution, shifting using the mathematical expectation of the proxy Gaussian distribution, scaling using the standard deviation of the proxy Gaussian distribution, and introducing random variable noise to obtain the posterior sample of the weight of the network structure.

[0099] The selection module 330 includes an evaluation unit 331 and a selection unit 332 .

[0100] The evaluation unit 331 is configured to perform performance evaluation on the candidate network structure corresponding to each weight obtained by sampling. The evaluation unit 331 is, for example, a network structure performance evaluator, for example, including at least one of a Jacob covariance evaluator and a network synaptic flow evaluator.

[0101] The selection unit 332 is configured to select a target network structure that meets the performance requirements from multiple candidate network structures according to the performance evaluation result.

[0102] Figure 4 A schematic diagram of the structure of a network structure search device according to some embodiments of the present disclosure is shown.

[0103] like Figure 4 As shown, the network structure search device 400 of this embodiment includes: a memory 410 and a processor 420 coupled to the memory 410, and the processor 420 is configured to execute the network structure search method in any of the aforementioned embodiments based on the instructions stored in the memory 410.

[0104] The memory 410 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0105] The processor 420 may be implemented by a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors and other discrete hardware components.

[0106] The device 400 may also include an input / output interface 430, a network interface 440, a storage interface 450, etc. These interfaces 430, 440, 450 and the memory 410 and the processor 420 may be connected, for example, via a bus 460. Among them, the input / output interface 430 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 440 provides a connection interface for various networked devices. The storage interface 450 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 460 may use any of a variety of bus structures. For example, the bus structure includes, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0107] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the steps of a network structure search method when executed by a processor.

[0108] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more non-transient computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0109] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A network structure search method, characterized in that: include: Constructing the optimization of the network structure as the maximum a posteriori of the weights of the network structure, including: constructing the weights of the network structure as the maximum a posteriori of the weights of the network structure under the conditions of a given data set and a hyper-network parameter of the network structure, wherein each set of data pairs of input data and their target labels in the given data set includes an input image and its category label; The network structure is optimized by calculating the maximum a posteriori of the weight of the network structure to obtain the distribution of the weight of the better network structure; Sampling is performed according to the distribution of weights of a better network structure to obtain multiple weights; According to the candidate network structure for completing the image classification task corresponding to each weight obtained by sampling, the target network structure for completing the image classification task that meets the performance requirements is selected.

2. The method according to claim 1, characterized in that The maximum a posteriori of the weights of the network structure is constructed according to the prior distribution of the weights of the network structure and the weights of the network structure under the optimal discrimination probability model, wherein the discrimination probability model is the discrimination probability of the target label under the conditions of input data, super-network parameters of the network structure and weights of the network structure.

3. The method according to claim 1, characterized in that The calculation of the network structure optimization based on the maximum a posteriori of the network structure weights obtains the distribution of the weights of the better network structure, including: Perform variational inference learning on the posterior of the weights of the network structure to obtain an approximation of the posterior of the weights of the network structure; The maximum a posteriori weight of the network structure is calculated based on the posterior approximation of the weight of the network structure to obtain a better distribution of the weight of the network structure.

4. The method according to claim 3, characterized in that Variational inference learning is performed on the posterior of the weights of the network structure to obtain the approximation of the posterior of the weights of the network structure, including: Constructing a proxy Gaussian distribution, wherein the proxy Gaussian distribution has a mathematical expectation and a standard deviation as Gaussian distribution parameters and a weight of a network structure as a variable; Calculating target Gaussian distribution parameters that minimize the distance between the proxy Gaussian distribution and the posterior of the weight of the network structure; The proxy Gaussian distribution under the target Gaussian distribution parameters is used as an approximation of the posterior of the weights of the network structure.

5. The method according to claim 4, characterized in that Calculating the target Gaussian distribution parameters that minimize the distance between the proxy Gaussian distribution and the posterior of the weight of the network structure includes: Constructing the posterior of the weights of the network structure according to the prior distribution of the weights of the network structure and the discriminant probability model, wherein the discriminant probability model is the discriminant probability of the target label under the conditions of the input data, the hypernetwork parameters of the network structure and the weights of the network structure; Calculate target Gaussian distribution parameters that minimize the difference between the distance between the proxy Gaussian distribution and the prior distribution of the weight of the network structure and the mathematical expectation, wherein the mathematical expectation is the mean of the discriminant probability model based on the proxy Gaussian distribution.

6. The method according to claim 5, characterized in that Calculating the target Gaussian distribution parameters that minimize the difference between the distance between the agent Gaussian distribution and the prior distribution of the weight of the network structure and the mathematical expectation includes: Extracting each Monte Carlo sample from the proxy Gaussian distribution as each posterior sample of the weight of the network structure; Based on each posterior sample of the weight of the network structure, the target Gaussian distribution parameters that minimize the target difference are calculated, and the target difference is determined according to the total difference between the first information and the second information and the third information under each posterior sample. The first information is determined according to the proxy Gaussian distribution under each posterior sample, the second information is determined according to the prior distribution of the weight of the network structure under each posterior sample, and the third information is determined according to the discriminant probability model under each posterior sample.

7. The method according to claim 6, characterized in that Extracting each Monte Carlo sample from the proxy Gaussian distribution as each posterior sample of the weight of the network structure includes: Based on the sampling of the standard Gaussian distribution, the mathematical expectation of the proxy Gaussian distribution is used for shifting, the standard deviation of the proxy Gaussian distribution is used for scaling, and random variable noise is introduced to obtain the posterior samples of the weights of the network structure.

8. The method according to claim 2 or 5, characterized in that: The prior distribution of the weights of the network structure is determined according to the proportional mixture distribution of two preset Gaussian distributions.

9. The method according to claim 1, characterized in that: According to the candidate network structures corresponding to each weight obtained by sampling, the target network structures that meet the performance requirements are selected, including: Perform performance evaluation on the candidate network structure corresponding to each weight obtained by sampling; According to the performance evaluation results, a target network structure that meets the performance requirements is selected from multiple candidate network structures.

10. The method according to claim 9, characterized in that A network structure performance evaluator is used to perform performance evaluation on the candidate network structure corresponding to each weight obtained by sampling, wherein the network structure performance evaluator includes at least one of a Jacob covariance evaluator and a network synaptic flow evaluator.

11. The method according to claim 1, characterized in that: Also includes: According to business needs, the selected multiple target network structures are deployed on different types of hardware devices.

12. A network structure search device, comprising: Memory; And a processor coupled to the memory, wherein the processor is configured to execute the network structure search method according to any one of claims 1 to 11 based on instructions stored in the memory.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the network structure search method according to any one of claims 1 to 11.

14. A network structure search device, characterized in that: include: The distribution determination module is configured to construct the optimization of the network structure as the maximum a posteriori of the weights of the network structure, including: constructing the weights of the network structure as the maximum a posteriori of the weights of the network structure under the conditions of a given data set and a super network parameter of the network structure, wherein each set of data pairs of input data and its target label in the given data set includes an input image and its category label; performing the optimization calculation of the network structure based on the maximum a posteriori of the weights of the network structure to obtain a distribution of the weights of a better network structure; A sampling module is configured to obtain a plurality of weights by sampling according to the distribution of weights of a better network structure; The selection module is configured to select a target network structure for completing the image classification task that meets the performance requirements based on the candidate network structure for completing the image classification task corresponding to each weight obtained by sampling.

Images