Method, apparatus and computer storage medium for searching neural network architecture

Abstract

The application discloses a neural network architecture searching method and device and a computer storage medium. The method comprises the following steps: obtaining a candidate neural network architecture; training the candidate neural network architecture; if the candidate neural network architecture is trained, searching a target neural network architecture based on the trained candidate neural network architecture, wherein if a loss function of the candidate neural network architecture meets a training completion condition, it is determined that the candidate neural network architecture is trained, and the loss function comprises a function item corresponding to a first hardware parameter, and the first hardware parameter is used for indicating a hardware parameter of a target hardware platform running the candidate neural network architecture. It can be seen that the application can realize the technical effect that the searched target neural network architecture can be compatible with inference efficiency and inference accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a neural network architecture searching method, device and computer storage medium. BACKGROUND

[0002] At present, neural networks are widely used in the field of artificial intelligence. In recent years, neural network architecture searching methods have emerged one after another, which can automatically design neural network models without requiring a large amount of expert experience. Most of the current neural network architecture searching methods are committed to guaranteeing the inference efficiency of the model in a way of limiting the model computation or memory access amount.

[0003] However, it is found in practice that guaranteeing the inference efficiency of the model in a way of limiting the model computation or memory access amount can result in insufficient efficiency of the neural network model. SUMMARY

[0004] The technical problem to be solved by the present application is that, since different hardware platforms have different hardware characteristics, merely guaranteeing the inference efficiency of the model in a way of limiting the model computation or memory access amount cannot effectively exert the hardware computing capability of the hardware platform on which the neural network model runs, thereby resulting in insufficient efficiency of the neural network model. Therefore, the present application provides a neural network architecture searching method, device and computer storage medium, which can search for a target neural network architecture that is compatible with inference efficiency and inference accuracy.

[0005] To solve the above technical problem, the present application discloses a neural network architecture searching method in the first aspect, which comprises:

[0006] obtaining a candidate neural network architecture;

[0007] training the candidate neural network architecture;

[0008] if the candidate neural network architecture is trained, searching for a target neural network architecture based on the trained candidate neural network architecture, wherein, in the case that a loss function of the candidate neural network architecture meets a training completion condition, it is determined that the candidate neural network architecture is trained, and the loss function comprises a function item corresponding to a first hardware parameter, the first hardware parameter being used to indicate a hardware parameter of a target hardware platform running the candidate neural network architecture.

[0009] As an optional implementation manner, in the first aspect of the present application, the obtaining of the candidate neural network architecture comprises:

[0010] obtaining an initial neural network architecture, the initial neural network architecture comprising N layers of neural network layers, each layer of the neural network layers comprising at least one initial neural network module, and N being a natural number greater than or equal to 1.

[0011] determining a second hardware parameter of each of the initial neural network modules, the second hardware parameter being used to indicate a hardware parameter of the target hardware platform running the initial neural network module;

[0012] determining an evaluation index of each of the initial neural network modules according to the second hardware parameter of each of the initial neural network modules;

[0013] selecting a candidate neural network module of each of the neural network layers from at least one of the initial neural network modules of each of the neural network layers according to the evaluation index of each of the initial neural network modules, to obtain the obtained candidate neural network architecture, wherein the neural network layers of the candidate neural network architecture comprise the candidate neural network modules.

[0014] As an optional implementation form, in the first aspect of the present application, the second hardware parameter comprises at least one of an operation number, a second hardware delay and a theoretical computing power, and the operation number comprises at least one of a multiplication operation number and an addition operation number.

[0015] As an optional implementation form, in the first aspect of the present application, the determining of the evaluation index of each of the initial neural network modules according to the second hardware parameter of each of the initial neural network modules comprises:

[0016] determining a hardware utilization rate of each of the initial neural network modules according to the second hardware parameter of each of the initial neural network modules, wherein the hardware utilization rate of the initial neural network module is positively correlated with the operation number of the initial neural network module, and the hardware utilization rate of the initial neural network module is negatively correlated with the second hardware delay and the theoretical computing power of the initial neural network module respectively;

[0017] determining a parameter quantity of each of the initial neural network modules;

[0018] determining a score of each of the initial neural network modules according to the hardware utilization rate and the parameter quantity of each of the initial neural network modules, and taking the score of the initial neural network module as the evaluation index of the initial neural network module, wherein the score of the initial neural network module is positively correlated with the hardware utilization rate and the parameter quantity of the initial neural network module respectively.

[0019] As an optional implementation form, in the first aspect of the present application, the initial neural network architecture is a convolutional neural network architecture, and the parameter quantity comprises a convolution kernel quantity of the convolutional neural network architecture.

[0020] As an optional implementation, in the first aspect of the present application, the score of the initial neural network module is determined by a first formula, and the first formula comprises:

[0021] S ij = k * u ij * p ij ;

[0022] wherein S ij represents the score of the initial neural network module, u ij represents the hardware utilization rate, p ij represents the parameter amount, k represents a constant coefficient, i represents a certain neural network layer, and j represents a certain initial neural network module;

[0023] and / or,

[0024] The hardware utilization rate of the initial neural network module is determined by a second formula, and the second formula comprises:

[0025] u ij = F ij / t ij *C);

[0026] wherein F ij represents the operation amount, t ij represents the second hardware delay, and C represents the theoretical computing power.

[0027] As an optional implementation, in the first aspect of the present application, the candidate neural network module of each layer of the neural network layer is selected from at least one initial neural network module of each layer of the neural network layer according to the evaluation index of each initial neural network module, comprising:

[0028] If the score of the initial neural network module is greater than the score threshold, the initial neural network module is taken as the candidate neural network module;

[0029] wherein the score threshold corresponding to the initial neural network module is related to the scores of all the initial neural network modules of the neural network layer where the initial neural network module is located, or the score threshold corresponding to the initial neural network module is related to the scores of all the initial neural network modules of the initial neural network architecture.

[0030] As an optional implementation, in the first aspect of the present application, the candidate neural network architecture comprises N layers of neural network layers, and the N layers of neural network layers are pre-divided into at least two parts of neural network layers to be trained, the at least two parts of neural network layers to be trained comprising L layers of first neural network layers and M layers of second neural network layers, wherein N is a natural number greater than or equal to 2, L and M are natural numbers greater than or equal to 1, and L+M≤N.

[0031] training the candidate neural network architecture comprises:

[0032] keeping the M layers of the second neural network layers fixed and training the L layers of the first neural network layers;

[0033] in a case where the training of the L layers of the first neural network layers is completed, keeping the L layers of the first neural network layers fixed and training the M layers of the second neural network layers;

[0034] wherein the loss function of the candidate neural network architecture satisfying the training completion condition comprises that the loss function of the L layers of the first neural network layers satisfies the training completion condition and the loss function of the M layers of the second neural network layers satisfies the training completion condition.

[0035] As an optional implementation, in the first aspect of the present application, the first hardware parameter comprises at least one of a first hardware delay and a memory occupation.

[0036] As an optional implementation, in the first aspect of the present application, the loss function comprises:

[0037] L(a, w a )=CE(a, w a )+α*LAT(a) β +γ*MEM(a) δ ;

[0038] LAT(a)=∑ i ∑ j m ij *LAT(block ij );

[0039] MEM(a)=∑ i ∑ j m ij *MEM(block ij );

[0040] wherein L(a, w a ) represents the loss function, a represents the candidate neural network architecture, w a represents the weight parameter of the candidate neural network architecture, CE(a, w a ) represents the cross-entropy loss term of the candidate neural network architecture, LAT(a) represents the second hardware delay, MEM(a) represents the memory occupation, the hyperparameter α represents the penalty coefficient of the second hardware delay, the hyperparameter γ represents the penalty coefficient of the memory occupation term, the hyperparameter β is used to adjust the numerical value of the function term corresponding to the second hardware delay, the hyperparameter δ is used to adjust the numerical value of the function term corresponding to the memory occupation, and block ijthe jth candidate neural network module of the ith layer, m ij a mask value generated by a Gumbel-Softmax function configured for a softmax layer in the candidate neural network architecture, m ij a sampling probability of a candidate neural network module of the candidate neural network architecture.

[0041] As an optional implementation, in the first aspect of the present application, the candidate neural network architecture includes N layers of neural network layers, at least one of the neural network layers is a neural network layer participating in training, and N is a natural number greater than or equal to 2;

[0042] The searching of the target neural network architecture based on the trained candidate neural network architecture includes:

[0043] obtaining a sampling probability of each candidate neural network module of each neural network layer participating in training, the sampling probability of the candidate neural network module being determined during training of the candidate neural network architecture;

[0044] retaining the candidate neural network module corresponding to the maximum sampling probability in each neural network layer participating in training to obtain the target neural network architecture.

[0045] The second aspect of the present application discloses a device for searching a neural network architecture, the device including:

[0046] an obtaining module for obtaining a candidate neural network architecture;

[0047] a training module for training the candidate neural network architecture;

[0048] an architecture searching module for searching a target neural network architecture based on the trained candidate neural network architecture if the candidate neural network architecture is trained, wherein the candidate neural network architecture is determined to be trained if a loss function of the candidate neural network architecture meets a training completion condition, the loss function including a function item corresponding to a first hardware parameter, the first hardware parameter being used to indicate a hardware parameter of a target hardware platform running the candidate neural network architecture.

[0049] As an optional implementation, in the second aspect of the present application, the obtaining module includes:

[0050] a first obtaining unit for obtaining an initial neural network architecture, the initial neural network architecture including N layers of neural network layers, each of the neural network layers including at least one initial neural network module, and N being a natural number greater than or equal to 1;

[0051] The first parameter determination unit is configured to determine a second hardware parameter of each of the initial neural network modules, where the second hardware parameter is used to indicate a hardware parameter of the target hardware platform for running the initial neural network module.

[0052] The evaluation index determination unit is configured to determine an evaluation index of each of the initial neural network modules according to the second hardware parameter of each of the initial neural network modules.

[0053] The second acquisition unit is configured to filter out a candidate neural network module of each of the neural network layers from at least one of the initial neural network modules of each of the neural network layers according to the evaluation index of each of the initial neural network modules, so as to obtain the obtained candidate neural network architecture, where the neural network layers of the candidate neural network architecture include the candidate neural network modules.

[0054] As an optional implementation, in the second aspect of the present application, the second hardware parameter includes at least one of an operation number, a second hardware delay and a theoretical computing power, and the operation number includes at least one of a multiplication operation number and an addition operation number.

[0055] As an optional implementation, in the second aspect of the present application, the evaluation index determination unit is configured to determine a hardware utilization rate of each of the initial neural network modules according to the second hardware parameter of each of the initial neural network modules, where the hardware utilization rate of the initial neural network module is positively correlated with the operation number of the initial neural network module, and the hardware utilization rate of the initial neural network module is negatively correlated with the second hardware delay and the theoretical computing power of the initial neural network module respectively.

[0056] The parameter quantity of each of the initial neural network modules is determined.

[0057] The score of each of the initial neural network modules is determined according to the hardware utilization rate and the parameter quantity of each of the initial neural network modules, and the score of the initial neural network module is taken as the evaluation index of the initial neural network module, and the score of the initial neural network module is positively correlated with the hardware utilization rate and the parameter quantity of the initial neural network module respectively.

[0058] As an optional implementation, in the second aspect of the present application, the initial neural network architecture is a convolutional neural network architecture, and the parameter quantity includes a convolution kernel number of the convolutional neural network architecture.

[0059] As an optional implementation, in the second aspect of the present application, the score of the initial neural network module is determined by the evaluation index determination unit through a first formula, and the first formula includes:

[0060] Sij = k * u ij * p ij ;

[0061] wherein S ij represents a score of an initial neural network module, u ij represents a hardware utilization rate, p ij represents a parameter amount, and k represents a constant coefficient, i represents a certain neural network layer, and j represents a certain initial neural network module;

[0062] and / or,

[0063] The hardware utilization rate of the initial neural network module is determined by the evaluation index determination unit through a second formula, and the second formula includes:

[0064] u ij = F ij / (t ij * C) ;

[0065] wherein F ij represents an operation amount, t ij represents a second hardware delay, and C represents a theoretical computing power.

[0066] As an optional implementation, in the second aspect of the present application, the second acquisition unit is configured to, if the score of the initial neural network module is greater than a score threshold, take the initial neural network module as the candidate neural network module.

[0067] wherein the score threshold corresponding to the initial neural network module is related to the scores of all the initial neural network modules of the neural network layer where the initial neural network module is located, or the score threshold corresponding to the initial neural network module is related to the scores of all the initial neural network modules of the initial neural network architecture.

[0068] As an optional implementation, in the second aspect of the present application, the candidate neural network architecture includes N layers of neural network layers, and the N layers of neural network layers are pre-divided into at least two parts of neural network layers to be trained, the at least two parts of neural network layers to be trained include L layers of first neural network layers and M layers of second neural network layers, wherein N is a natural number greater than or equal to 2, L and M are natural numbers greater than or equal to 1, and L+M≤N.

[0069] The training module is configured to keep the M layers of second neural network layers fixed and train the L layers of first neural network layers.

[0070] In the case where the L layers of first neural network layers are trained, the L layers of first neural network layers are kept fixed and the M layers of second neural network layers are trained.

[0071] The loss function of the candidate neural network architecture satisfies the training completion condition, including the loss function of the L-layer first neural network layer and the loss function of the M-layer second neural network layer.

[0072] As an optional implementation, in a second aspect of the present invention, the first hardware parameter includes at least one of a first hardware latency and memory usage.

[0073] As an optional implementation, in a second aspect of the invention, the loss function includes:

[0074] L(a,w a )=CE(a,w a )+α*LAT(a) β +γ*MEM(a) δ ;

[0075] LAT(a)=∑ i ∑ j m ij *LAT(block ij );

[0076] MEM(a)=∑ i ∑ j m ij *MEM(block ij );

[0077] Wherein, L(a,w a ) represents the loss function, a represents the candidate neural network architecture, and w a CE(a,w) represents the weight parameters of the candidate neural network architecture. a The cross-entropy loss term of the candidate neural network architecture is represented by ), LAT(a) represents the second hardware latency, MEM(a) represents memory usage, hyperparameter α represents the penalty coefficient for the second hardware latency, hyperparameter γ represents the penalty coefficient for the memory usage term, hyperparameter β is used to adjust the value of the function term corresponding to the second hardware latency, and hyperparameter δ is used to adjust the value of the function term corresponding to the memory usage term. ij Let m represent the j-th candidate neural network module in the i-th layer. ij The mask value generated by the Gumbel-Softmax function configured for the softmax layer in the candidate neural network architecture, m ij This represents the sampling probability of the candidate neural network module in the candidate neural network architecture.

[0078] As an optional implementation, in the second aspect of the present application, the candidate neural network architecture comprises N layers of neural network layers, at least one of the N layers of neural network layers is a neural network layer participating in training, and N is a natural number greater than or equal to 2;

[0079] The architecture search module is configured to obtain a sampling probability of each candidate neural network module of each layer of the neural network layer participating in training, the sampling probability of the candidate neural network module being determined by the candidate neural network architecture during training;

[0080] The candidate neural network module corresponding to the maximum sampling probability is reserved in each layer of the neural network layer participating in training, so as to obtain the target neural network architecture.

[0081] The third aspect of the present application discloses another neural network architecture search device, which comprises:

[0082] A memory storing executable program codes;

[0083] A processor coupled with the memory;

[0084] The processor invokes the executable program codes stored in the memory to execute the neural network architecture search method disclosed in the first aspect of the present application.

[0085] The fourth aspect of the present application discloses a computer storage medium, which stores computer instructions, and the computer instructions are used to execute the neural network architecture search method disclosed in the first aspect of the present application when invoked.

[0086] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0087] In the embodiment of the present application, the candidate neural network architecture is acquired, the candidate neural network architecture is trained, and if the candidate neural network architecture is trained, the target neural network architecture is searched based on the trained candidate neural network architecture. In the case that the loss function of the candidate neural network architecture meets the training completion condition, it is determined that the candidate neural network architecture is trained. The loss function includes a function item corresponding to the first hardware parameter, and the first hardware parameter is used to indicate the hardware parameter of the target hardware platform running the candidate neural network architecture. That is, in the training process of the candidate neural network model, the hardware operation capability of the target hardware platform running the neural network model is considered, that is, the different hardware characteristics of different hardware platforms are considered. Therefore, the target neural network architecture is searched based on the trained candidate neural network architecture, which can effectively exert the hardware operation capability of the hardware platform running the neural network model, and exert the maximum hardware operation capability of the target hardware platform as much as possible. Compared with the scheme of limiting the model calculation amount or the memory amount to ensure the inference efficiency of the model, the technical scheme of the embodiment can realize the technical effect that the searched target neural network architecture can be compatible with the inference efficiency and the inference accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0088] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0089] Figure 1 is a flow diagram of a neural network architecture search method disclosed by the embodiment of the present application;

[0090] Figure 2 is a structural diagram of a neural network architecture search device disclosed by the embodiment of the present application;

[0091] Figure 3 is a structural diagram of another neural network architecture search device disclosed by the embodiment of the present application. DETAILED DESCRIPTION

[0092] In order to make the person skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0093] The terms "first", "second", and the like in the description and in the claims of the present application and above figures are used to distinguish different objects, and are not used to describe a particular sequential order. Furthermore, the terms "comprises", "comprising", "includes", "including" and the like are to be construed open- ended, allowing for instances where there are equivalents to the listed steps or units, and that additional steps or units are added in the future. For example, a process, method, article, or apparatus that comprises a list of steps or units is not limited to only those steps or units which are recited, but can also include additional steps or units that are not expressly listed or inherent to such process, method, article, or apparatus.

[0094] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the other embodiments. It is explicitly contemplated that embodiments described herein can be combined with each other.

[0095] The application discloses a neural network architecture search method, device and computer storage medium, which can be compatible with inference efficiency and inference accuracy. The following will be described in detail.

[0096] Embodiment one

[0097] Please refer to Figure 1 , Figure 1 is a flowchart of a neural network architecture search method disclosed by the embodiment of the application. Wherein, Figure 1The search method of the neural network architecture described can be applied to a device of the neural network architecture, which can be applied to a terminal. The terminal can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart television, a smart vehicle, and the like, but is not limited thereto. In addition, the device can also be applied to a server. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, and the like, which are not limited herein. In addition, the device can also be applied to a processor with data processing capability, such as a GPU (graphics processing unit) chip, a CPU (central processing unit) chip, and an AI (Artificial Intelligence) accelerator, and the like, which are not limited herein.

[0098] As shown in Figure 1 The search method of the neural network architecture can include:

[0099] 110, obtaining a candidate neural network architecture.

[0100] The candidate neural network architecture is used to reflect the composition and structure of the neural network model. In this embodiment, the neural network model includes but is not limited to a convolutional neural network model, a recurrent network model, a recurrent neural network model, a deep belief network model, and a generative adversarial network model, and the like, which are not limited herein.

[0101] 120, training the candidate neural network architecture.

[0102] In this embodiment, the candidate neural network architecture can be trained by inputting training samples. For example, if the candidate neural network architecture is a convolutional neural network architecture, an image training sample is input to train the convolutional neural network architecture.

[0103] 130, if the candidate neural network architecture is trained, a target neural network architecture is searched based on the trained candidate neural network architecture, wherein the candidate neural network architecture is determined to be trained if a loss function of the candidate neural network architecture satisfies a training completion condition, and the loss function includes a function item corresponding to a first hardware parameter, and the first hardware parameter is used to indicate a hardware parameter of a target hardware platform running the candidate neural network architecture.

[0104] The training completion condition is used to indicate that the training of the candidate neural network architecture is completed. In the embodiment, the training completion condition includes that the value of the loss function is less than a preset threshold and / or the value of the loss function tends to be stable, which is not limited herein. That is, if the value of the loss function of the candidate neural network architecture is less than the preset threshold and / or the value of the loss function tends to be stable, it indicates that the training of the candidate neural network model is completed. The value tends to be stable can be that the difference between the continuous values of the loss function is less than a difference threshold, which is not limited herein. The target hardware platform refers to a hardware platform on which the candidate neural network model needs to run, that is, the neural network architecture is run through the target hardware platform. The hardware parameter is used to reflect the hardware operation capability of the target hardware platform.

[0105] In the embodiment, specifically, in the training process of the candidate neural network architecture, the loss function is determined whether to meet the training completion condition every time the training is performed. If the loss function does not meet the training completion condition, the candidate neural network architecture is continuously trained. If the loss function meets the training completion condition, it is determined that the training of the candidate neural network architecture is completed.

[0106] It can be understood that the loss function includes a function item corresponding to the first hardware parameter, that is, in the training process of the candidate neural network model, the hardware operation capability of the target hardware platform on which the neural network model runs is considered.

[0107] Since different hardware platforms have different hardware characteristics, merely ensuring the inference efficiency of the model in a manner of limiting the calculation amount or the memory access amount of the model cannot effectively exert the maximum hardware operation capability of the hardware platform on which the neural network model runs. The technical solution of the embodiment can obtain a candidate neural network architecture, train the candidate neural network architecture, and if the candidate neural network architecture is trained, search for a target neural network architecture based on the trained candidate neural network architecture. In the case where a loss function of the candidate neural network architecture meets a training completion condition, it is determined that the candidate neural network architecture is trained, and the loss function includes a function item corresponding to a first hardware parameter, which is used to indicate a hardware parameter of the target hardware platform running the candidate neural network architecture. That is, in the training process of the candidate neural network model, the hardware operation capability of the target hardware platform on which the neural network model runs is considered, that is, the different hardware characteristics of different hardware platforms are considered. Therefore, searching for the target neural network architecture based on the trained candidate neural network architecture can effectively exert the hardware operation capability of the hardware platform on which the neural network model runs, and exert the maximum hardware operation capability of the target hardware platform as much as possible. Compared with the scheme of limiting the calculation amount or the memory access amount of the model to ensure the inference efficiency of the model, the technical solution of the embodiment can achieve the technical effect that the searched target neural network architecture is compatible with the inference efficiency and the inference accuracy.

[0108] It should be noted that the candidate neural network architecture can be a designed original neural network architecture, or a neural network architecture obtained by processing the designed original neural network architecture, which is not limited herein.

[0109] In a possible implementation, the candidate neural network architecture is obtained, including:

[0110] An initial neural network architecture is obtained, and the initial neural network architecture includes N neural network layers, each neural network layer includes at least one initial neural network module, and N is a natural number greater than or equal to 1;

[0111] A second hardware parameter of each initial neural network module is determined, and the second hardware parameter is used to indicate a hardware parameter of the target hardware platform running the initial neural network module;

[0112] An evaluation index of each initial neural network module is determined according to the second hardware parameter of each initial neural network module;

[0113] The candidate neural network module of each layer of the neural network layer is screened from at least one of the initial neural network modules of each layer of the neural network layer according to the evaluation index of each initial neural network module, so as to obtain the candidate neural network architecture, wherein the neural network layer of the candidate neural network architecture comprises the candidate neural network module.

[0114] The initial neural network architecture can be a neural network architecture designed by a designer in advance. The evaluation index is used to reflect the hardware capability utilization rate of the initial neural network module when running on the target hardware platform.

[0115] In this embodiment, specifically, the evaluation index of each initial neural network module is determined through the second hardware parameter of each initial neural network module, so that the candidate neural network module of each layer of the neural network layer is screened from at least one of the initial neural network modules of each layer of the neural network layer according to the evaluation index of each initial neural network module, that is, the evaluation index can be determined in combination with the second hardware parameter of the initial neural network module in this embodiment, and then the candidate neural network module with better evaluation index is screened out in each layer of the neural network layer by using the evaluation index. Compared with the initial neural network architecture, the candidate neural network architecture has fewer components but similar precision, that is, compared with the initial neural network architecture, the candidate neural network architecture not only ensures the potential expression capability of the network model, but also considers the efficiency of the model when reasoning on the target hardware platform. While ensuring the diversity of the candidate neural network architecture, the size of the search space of the neural network architecture is further reduced, so that the time consumption and the calculation resource consumption of subsequent architecture search are reduced.

[0116] In a possible implementation, the second hardware parameter comprises at least one of an operation number, a second hardware delay, and a theoretical computing power, and the operation number comprises at least one of a multiplication operation number and an addition operation number.

[0117] The second hardware delay can be the time consumption of the initial neural network module when running on the target hardware platform. The theoretical computing power can be the maximum computing power theoretically possessed by the target hardware platform. The operation number comprises at least one of a multiplication operation number and an addition operation number, which can comprise a multiplication operation number, can comprise an addition operation number, or can comprise both a multiplication operation number and an addition operation number, without limitation.

[0118] It should be noted that if the operation number comprises a multiplication operation number and an addition operation number, the size of the operation number is the sum of the multiplication operation number and the addition operation number, without limitation.

[0119] In a possible implementation, the evaluation index of each initial neural network module is determined according to a second hardware parameter of each initial neural network module, including:

[0120] The hardware utilization rate of each initial neural network module is determined according to the second hardware parameter of each initial neural network module, wherein the hardware utilization rate of the initial neural network module is positively correlated with the operation number of the initial neural network module, and the hardware utilization rate of the initial neural network module is negatively correlated with the second hardware delay and the theoretical computing power of the initial neural network module, respectively.

[0121] The parameter quantity of each initial neural network module is determined.

[0122] The score of each initial neural network module is determined according to the hardware utilization rate and the parameter quantity of each initial neural network module, and the score of the initial neural network module is taken as the evaluation index of the initial neural network module, and the score of the initial neural network module is positively correlated with the hardware utilization rate and the parameter quantity of the initial neural network module, respectively.

[0123] Wherein, positive correlation refers to the growth of one variable, and the growth of another variable. The two variables change in the same direction, and when one variable changes from large to small or from small to large, the other variable also changes from large to small or from small to large. Negative correlation refers to the decrease (increase) of the dependent variable with the increase (decrease) of the independent variable. In this case, the correlation coefficient of the dependent variable and the independent variable is negative, that is, negative correlation.

[0124] It should be noted that if the initial neural network architecture is a convolutional neural network architecture, the parameter quantity includes the number of convolutional kernels of the convolutional neural network architecture.

[0125] In a possible implementation, the score of the initial neural network module is determined by a first formula, and the first formula includes:

[0126] S ij =k*u ij *p ij ;

[0127] Wherein, S ij represents the score of the initial neural network module, u ij represents the hardware utilization rate, p ij represents the parameter quantity, k represents a constant coefficient, i represents a neural network layer, and j represents an initial neural network module.

[0128] Specifically, S ij represents the score of the jth initial neural network module in the ith neural network layer, u ijrepresents the hardware utilization of the jthinitial neural network module in the ithlayer of neural network layers. ij represents the parameter quantity of the jthinitial neural network module in the ithlayer of neural network layers.

[0129] In a possible implementation, the hardware utilization of the initial neural network module is determined by a second formula, and the second formula includes:

[0130] u ij = F ij / (t ij *C);

[0131] wherein F ij represents the operation quantity, t ij represents the second hardware delay, and C represents the theoretical computing power.

[0132] Specifically, F ij represents the operation quantity of the jthinitial neural network module in the ithlayer of neural network layers, t ij represents the second hardware delay of the jthinitial neural network module in the ithlayer of neural network layers, and C represents the theoretical computing power of the target hardware platform.

[0133] In a possible implementation, the candidate neural network module of each layer of neural network layers is selected from at least one initial neural network module of each layer of neural network layers according to the evaluation index of each initial neural network module.

[0134] If the score of the initial neural network module is greater than the score threshold, the initial neural network module is taken as the candidate neural network module.

[0135] For example, it is assumed that the ithlayer of neural network layers includes an initial neural network module A, an initial neural network module B, and an initial neural network module C. If the score of the initial neural network module B and the score of the initial neural network module C are greater than the score threshold, and the score of the initial neural network module A is less than or equal to the score threshold, the initial neural network module B and the initial neural network module are taken as the candidate neural network module.

[0136] It should be noted that the score threshold can be an empirical value, that is, an empirical value determined through experiments. In addition, the score threshold corresponding to the initial neural network module is related to the scores of all initial neural network modules of the neural network layer where the initial neural network module is located, or the score threshold corresponding to the initial neural network module is related to the scores of all initial neural network modules of the initial neural network architecture.

[0137] For example, assuming that the first layer of neural network layers includes initial neural network module A, initial neural network module B and initial neural network module C, the second layer of neural network layers includes initial neural network module D and initial neural network module E, the score of initial neural network module A is A, the score of initial neural network module B is B, the score of initial neural network module C is C, the score of initial neural network module D is D, and the score of initial neural network module E is E, the score threshold of the first layer of neural network layers and the second layer of neural network layers can be the same, and the same threshold is (A+B+C+D+E) / 5. In addition, the threshold of the first layer of neural network layers can be (A+B+C) / 3, and the threshold of the second layer of neural network layers can be (D+E) / 2.

[0138] It can be understood that the score threshold corresponding to the initial neural network module is related to the scores of all initial neural network modules in the neural network layer where the initial neural network module is located, or the score threshold corresponding to the initial neural network module is related to the scores of all initial neural network modules in the initial neural network architecture, and then the corresponding score threshold can be adaptively determined according to the actual situation of the initial neural network architecture, the screening accuracy of the candidate neural network module can be improved, and the inference efficiency and inference accuracy of the determined target neural network architecture can be further improved.

[0139] In a possible implementation, all neural network layers in the candidate neural network architecture can be trained, or part of the neural network layers to be trained in the candidate neural network architecture can be trained, which is not limited herein. In addition, training part of the neural network layers to be trained in the candidate neural network architecture can be simultaneous training of all part of the neural network layers to be trained, or batch training of all part of the neural network layers to be trained, which is not limited herein.

[0140] In a possible implementation, the candidate neural network architecture includes N layers of neural network layers, and the N layers of neural network layers are divided into at least two parts of neural network layers in advance, the at least two parts of neural network layers include L layers of first neural network layers and M layers of second neural network layers, wherein N is a natural number greater than or equal to 2, L and M are natural numbers greater than or equal to 1, and L+M≤N.

[0141] The training of the candidate neural network architecture includes:

[0142] The L layers of first neural network layers are trained while the M layers of second neural network layers are kept unchanged.

[0143] In the case that the L layers of first neural network layers are trained, the M layers of second neural network layers are trained while the L layers of first neural network layers are kept unchanged.

[0144] The loss function of the candidate neural network architecture satisfies the training completion condition, including that the loss function of the L-layer first neural network layer satisfies the training completion condition and the loss function of the M-layer second neural network layer satisfies the training completion condition.

[0145] In this embodiment, specifically, the M-layer second neural network layer is first maintained, and the L-layer first neural network layer is trained, that is, in the process of training the L-layer first neural network layer, the M-layer second neural network layer is not trained. If the loss function of the L-layer first neural network layer satisfies the training completion condition, it is indicated that the L-layer first neural network layer is trained. Then, the L-layer first neural network layer is maintained and fixed, and the M-layer second neural network layer is trained, that is, in the process of training the M-layer second neural network layer, the L-layer first neural network layer is not trained. If the loss function of the M-layer second neural network layer satisfies the training completion condition, it is considered that the candidate neural network architecture is trained.

[0146] The technical scheme of this embodiment divides the N-layer neural network layer into at least two parts of neural network layers to be trained in advance, the at least two parts of neural network layers to be trained include an L-layer first neural network layer and an M-layer second neural network layer, the M-layer second neural network layer is first maintained and fixed, and the L-layer first neural network layer is trained. In the case where the L-layer first neural network layer is trained, the L-layer first neural network layer is maintained and fixed, and the M-layer second neural network layer is trained. Through the training in stages, the number of neural network layers to be trained at the same time is reduced, and the computing power required for training is reduced. The architecture search time is further reduced while the network model accuracy and efficiency are ensured.

[0147] Optionally, if L is a natural number greater than or equal to 2, the L-layer first neural network layer is a continuous first neural network layer. If M is a natural number greater than or equal to 2, the M-layer second neural network layer is a continuous second neural network layer.

[0148] In a possible implementation, the L-layer first neural network layer is far away from an input layer of the candidate neural network architecture, and the M-layer second neural network layer is close to the input layer of the candidate neural network architecture.

[0149] Taking a convolutional neural network as an example, a low-level feature such as color and texture is extracted from original input image data in a network low layer, and a large amount of calculation is required, and a large amount of training data is required. A high-level feature related to a target task is learned in a network high layer, and a small amount of calculation is required. Therefore, the N-layer neural network layer to be searched is divided into at least two parts, the high-layer neural network layer is first trained, and the low-layer neural network layer is trained after the high-layer neural network layer is trained.

[0150] In one possible implementation, the first hardware parameter includes at least one of a first hardware latency and memory usage.

[0151] The first hardware latency can be the time consumption of the candidate neural network architecture running on the target hardware platform. Optionally, the first hardware latency can be determined by the second hardware latency of each initial neural network module. Specifically, the sum of the second hardware latencies of the initial neural network modules retained in the candidate neural network architecture can be used as the first hardware latency. Memory usage refers to the runtime overhead of this process. In this embodiment, memory usage can be the runtime overhead of the candidate neural network architecture. In one possible implementation, the loss function includes:

[0152] L(a,w a )=CE(a,w a )+α*LAT(a) β +γ*MEM(a) δ ;

[0153] LAT(a)=∑ i ∑ j m ij *LAT(block ij );

[0154] MEM(a)=∑ i ∑ j m ij *MEM(block ij );

[0155] Wherein, L(a,w a ) represents the loss function, a represents the candidate neural network architecture, and w a CE(a,w) represents the weight parameters of the candidate neural network architecture. a The cross-entropy loss term of the candidate neural network architecture is represented by ), LAT(a) represents the second hardware latency, MEM(a) represents memory usage, hyperparameter α represents the penalty coefficient for the second hardware latency, hyperparameter γ represents the penalty coefficient for the memory usage term, hyperparameter β is used to adjust the value of the function term corresponding to the second hardware latency, and hyperparameter δ is used to adjust the value of the function term corresponding to the memory usage term. ij Let m represent the j-th candidate neural network module in the i-th layer. ij The mask value generated by the Gumbel-Softmax function configured for the softmax layer in the candidate neural network architecture, m ij This represents the sampling probability of the candidate neural network module in the candidate neural network architecture.

[0156] The technical scheme of the embodiment comprises a loss function, which comprises a cross-entropy loss term, a function term corresponding to the second hardware delay, and a function term of memory occupation, so as to balance the inference efficiency and inference accuracy of the model.

[0157] It should be noted that, based on the trained candidate neural network architecture, the target neural network architecture can be obtained by taking the trained candidate neural network architecture as the target neural network architecture, or by screening the trained candidate neural network architecture to obtain the target neural network architecture, which is not limited herein.

[0158] In a possible implementation, the candidate neural network architecture comprises N layers of neural network layers, at least one of the N layers of neural network layers is a neural network layer participating in training, and N is a natural number greater than or equal to 2;

[0159] The target neural network architecture is searched based on the trained candidate neural network architecture, comprising:

[0160] The sampling probability of each candidate neural network module of each layer of the neural network layer participating in training is obtained, and the sampling probability of the candidate neural network module is determined in the training process of the candidate neural network architecture;

[0161] The candidate neural network module corresponding to the maximum sampling probability is retained in each layer of the neural network layer participating in training, so as to obtain the target neural network architecture.

[0162] The technical scheme of the embodiment comprises the sampling probability of the candidate neural network module determined in the training process of the candidate neural network architecture, and the candidate neural network module corresponding to the maximum sampling probability is retained in each layer of the neural network layer participating in training, so that the inference efficiency of the obtained target neural network architecture can be further improved.

[0163] The following embodiments take the neural network architecture as a convolutional neural network architecture and take an AI accelerator as an example to illustrate the scheme.

[0164] The method of the embodiment comprises:

[0165] Step A: design a search space of the convolutional neural network architecture.

[0166] Step B: count the second hardware delay and memory occupation of each candidate convolutional neural network module in the search space.

[0167] Step C: train the super network (candidate convolutional neural network architecture) in stages, sample the super network, and obtain a sub-network model structure.

[0168] Step D: train the sub-network model to obtain a final network model.

[0169] Step A includes:

[0170] Step A-1: ​​Construct the set of basic unit modules (initial convolutional neural network modules).

[0171] Step A-2: Initialize the search space of the network architecture to obtain the initial convolutional neural network architecture.

[0172] Step A-3: Calculate the score of each initial convolutional neural network module in the search space.

[0173] Step A-4: Select the initial convolutional neural network modules of each layer that meet the requirements, determine the search space of the final network architecture, and obtain the candidate convolutional neural network architecture.

[0174] In step A-1, taking into account the hardware characteristics of the AI ​​accelerator, classic network models such as ResNet, MobileNet, and SENet are used as basic convolutional neural network architectures. Suitable and representative basic unit modules, such as residual modules and attention modules, are selected from these to construct a set of basic unit modules. These basic unit modules have different structures, kernel sizes, expansion rates, and channel grouping numbers, ensuring diversity of modules in the search space.

[0175] In step A-2, this invention chooses to construct the search space of the network model based on ResNet50. The ResNet50 network architecture determines the total number of network layers and the size of the input and output feature maps of each layer. The parameters of the first three and last three layers are fixed, and the remaining layers are searched from the sets of basic unit modules in step A-1 to initialize the network search space. In the network model, the number of filters in each layer to be searched is determined manually; typically, lower layers have fewer filters, while higher layers have more.

[0176] In step A-3, the initialized search space is first obtained based on step A-2, and the parameter count p of the basic unit modules of each layer of the network is counted. ij Then, calculate the hardware utilization u of each basic unit module during inference on the target AI accelerator platform. ij The score S of this basic unit module is then obtained. ij The calculation is shown in equation (1):

[0177] S ij =k*u ij *p ij (1)

[0178] Wherein, subscript i represents the position index of the layer in the network, subscript j represents the position index of the current base unit module in the unit module set, and k is a constant coefficient. The utilization rate u of the base unit module when reasoning on the AI hardware accelerator ij The calculation is as shown in formula (2):

[0179] u ij =F ij / (t ij *C) (2)

[0180] Wherein, F ij represents the number of multiplication and addition operations of the module, t ij represents the second hardware delay of the module on the AI accelerator, and C represents the theoretical computing power of the AI accelerator.

[0181] In step A-4, the score threshold of the base unit module is set empirically, and the base module of the layer search space is filtered according to the threshold, and the base unit module with a score greater than the threshold is retained in each layer to constitute the search space of the entire network model. The retained base unit module is used as a candidate convolutional neural network module. The filtered search space not only guarantees the potential expression ability of the network model, but also considers the efficiency of the model when reasoning on the AI accelerator; while ensuring the diversity of the search space, it further reduces the size of the search space, thereby reducing the time consumption and computing resource consumption of subsequent architecture search.

[0182] In step B, the memory occupation and the second hardware delay of each base unit module in the network search space when running on the target AI accelerator hardware platform are counted. The hardware delay can be directly obtained from step A-3, and the memory occupation when running can be obtained from the software tool matched with the AI accelerator.

[0183] For step C, step C includes:

[0184] Step C-1: Prepare data samples for training of the candidate convolutional neural network model.

[0185] Step C-2: Design a loss function for training of the candidate convolutional neural network model.

[0186] Step C-3: Fix the low-layer network architecture and train to search the high-layer network architecture.

[0187] Step C-4: Fix the high-layer network architecture and train to search the low-layer network architecture.

[0188] Step C-5: Sample the network architecture to obtain the final target convolutional neural network architecture.

[0189] In step C-1, CIFAR-10 is selected as the training dataset of the network model. The dataset has a total of 60,000 color pictures, 50,000 pictures for training, and 10,000 pictures for testing, and the picture content mainly involves 10 categories such as different vehicles and animals. Among the 50,000 training pictures, 40,000 pictures are randomly selected as training samples for model training; 10,000 pictures are used as verification samples for model training verification.

[0190] In step C-2, the loss function of network model training is designed in combination with the consideration of the target AI acceleration hardware characteristics. The loss function considers the model classification accuracy, as well as the hardware delay and memory occupation of the model when running on the target hardware platform. The specific expression is shown in equation (3):

[0191] L(a, w a )=CE(a, w a )+α*LAT(a) β +γ*MEM(a) δ (3)

[0192] Where a represents the candidate convolutional neural network architecture, w a represents the weight parameters of the model, CE(a, w a ) is the cross-entropy loss term of the model, LAT(a) represents the first hardware delay of the candidate convolutional neural network architecture when running on the target hardware platform, and MEM(a) represents the memory occupation of the candidate convolutional neural network architecture when running on the hardware platform. The hyperparameters α and γ are the penalty coefficients of the first hardware delay term and the memory occupation term, respectively, and the hyperparameters β and δ are used to adjust the numerical value of the hardware delay term and the memory occupation term, respectively.

[0193] The candidate convolutional neural network architecture a can be sampled from the search space with a probability P(θ). θ is the sampling parameter generated by the softmax function, representing the probability of sampling the candidate convolutional neural network architecture from the search space. Given a high-dimensional discrete search space, the problem of searching for the optimal network architecture from the search space can be converted into an optimization problem, that is, by optimizing the sampling probability P(θ) of the super network model containing the layer search module set, to minimize the expected loss value, as shown in equation (4).

[0194] min θ min wa E a～P(θ) {L(a,w a )}(4)

[0195] In order to be able to use the gradient descent algorithm to optimize the above formula, the Gumbel-Softmax function is used in the discrete unit module search process, and a continuous distribution about the sampling probability θ is introduced. Combined with the hardware delay of each module obtained in step B, the first hardware delay term of the candidate convolutional neural network architecture in formula (3) can be expressed as shown in formula (5):

[0196] LAT(a) =∑ i ∑ j m ij *LAT(block ij ) (5)

[0197] Where block ij is the jth basic unit module of the ith layer in the candidate convolutional neural network architecture, LAT(block ij ) is the time consumption of the basic unit module when running on the hardware platform, m ij is the mask value generated by the Gumbel-Softmax function, which represents the sampling probability of the candidate convolutional neural network architecture. This formula assumes that the running time of each operator is independent on the target AI acceleration hardware platform. Similarly, the memory occupation MEM(a) calculation is shown in formula (6):

[0198] MEM(a) =∑ i ∑ j m ij *MEM(block ij ) (6)

[0199] Where MEM(block ij ) is the memory occupation of module block ij running on the hardware platform. Through the above steps, the search problem of the candidate convolutional neural network architecture in the high-dimensional discrete space is converted into an optimization problem that can use the gradient descent algorithm.

[0200] In step C-3, in addition to the first three layers of the candidate convolutional neural network architecture directly processing the original input image and the last three layers related to classification being fixed, there are N layers in the network that need to search for the most suitable basic unit module in training. The low-level features such as color and texture are extracted from the original input image data in the low layer of the network, and the calculation amount is usually large, and more training data is needed; the high-level features related to the target task are learned in the high layer of the network, and the calculation amount is small. Therefore, the N layers to be searched in the network are divided into two parts: M layers close to the low layer of the network and N-M layers close to the high layer of the network. When the super network model starts training, the network model weight is randomly initialized, the M layers close to the low layer of the network are fixed, and the N-M layers close to the high layer of the network are trained and searched. When the loss function tends to be stable and does not decrease, the network training is stopped.

[0201] In step C-4, the N-M layers of the network trained in step C-3 are fixed, and the M layers close to the front are continuously trained and searched. When the loss function tends to be stable and does not decrease, the network training is stopped. By using the method of training the model in stages, the architecture search time can be further reduced while ensuring the accuracy and efficiency of the network model.

[0202] In step C-5, according to the sampling probability of each layer basic unit module in the candidate convolutional neural network architecture, the basic unit module in the candidate convolutional neural network architecture is sampled to obtain a sub-network architecture.

[0203] In step D, the weight parameters of the sub-network architecture are randomly initialized, the cross-entropy classification loss function is used, the sub-network model is continuously trained on the CIFAR-10 dataset, and the model with the highest accuracy on the validation set is selected as the final target convolutional neural network model suitable for the AI accelerator.

[0204] In order to prove that the present scheme is feasible, the present application takes ResNet50 as the basic network and performs convolutional neural network architecture search experiments on the CIFAR-10 dataset. Before performing network architecture search training, the hardware utilization rate, time consumption and memory occupation of each candidate module in the search space are counted on an AI calculator platform. 40,000 images in the dataset are selected as training samples for model training, 10,000 images are selected as a validation set to screen the model, and the remaining 10,000 images are used for performance evaluation of the searched model. The Top3 classification accuracy, forward inference time consumption and memory occupation are selected as the performance evaluation standard, and the experimental results are as follows:

[0205]

[0206] From the results in the above table, compared with the original ResNet50 model, the TOP3 recognition rate of the model NAS-Model searched by the application on the CIFAR-10 only decreases by 0.5%, the time consumption in running is reduced by 55%, and the memory occupation is reduced by 49.22%. The experimental results show that the architecture search method of the application can ensure the model recognition performance while greatly reducing the hardware delay and memory occupation of the model in running on the AI accelerator platform.

[0207] Embodiment Two

[0208] Please refer to Figure 2 , Figure 2 is a structural schematic diagram of a neural network architecture searching device disclosed by an embodiment of the application. The device can be applied to a terminal. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart television, and a smart vehicle, but is not limited thereto. In addition, the device can also be applied to a server. The server can be a physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, and the like, which are not limited in the application. In addition, the device can also be applied to a processor with data processing capability, such as a GPU (graphics processing unit) chip, a CPU (central processing unit) chip, and an AI (Artificial Intelligence) accelerator, and the like, which are not limited herein.

[0209] As Figure 2 described, the neural network architecture searching device of the embodiment can include:

[0210] The obtaining module 210 is configured to obtain a candidate neural network architecture.

[0211] The training module 220 is configured to train the candidate neural network architecture.

[0212] The architecture search module 230 is configured to search for the target neural network architecture based on the trained candidate neural network architecture if the training of the candidate neural network architecture is completed. The training of the candidate neural network architecture is determined to be completed if a loss function of the candidate neural network architecture satisfies a training completion condition. The loss function includes a function item corresponding to a first hardware parameter, and the first hardware parameter is used to indicate a hardware parameter of the target hardware platform running the candidate neural network architecture.

[0213] In a possible implementation, the obtaining module 210 includes:

[0214] The first obtaining unit is configured to obtain an initial neural network architecture, where the initial neural network architecture includes N layers of neural network layers, each layer of the neural network layers includes at least one initial neural network module, and N is a natural number greater than or equal to 1.

[0215] The first parameter determination unit is configured to determine a second hardware parameter of each initial neural network module, where the second hardware parameter is used to indicate a hardware parameter of the target hardware platform running the initial neural network module.

[0216] The evaluation index determination unit is configured to determine an evaluation index of each initial neural network module according to the second hardware parameter of the initial neural network module.

[0217] The second obtaining unit is configured to select, according to the evaluation index of each initial neural network module, a candidate neural network module of each layer of the neural network layers from at least one initial neural network module of each layer of the neural network layers, to obtain the candidate neural network architecture, where the neural network layers of the candidate neural network architecture include the candidate neural network modules.

[0218] In a possible implementation, the second hardware parameter includes at least one of an operation quantity, a second hardware delay, and theoretical computing power, and the operation quantity includes at least one of a multiplication operation quantity and an addition operation quantity.

[0219] In a possible implementation, the evaluation index determination unit is configured to determine a hardware utilization rate of each initial neural network module according to the second hardware parameter of the initial neural network module, where the hardware utilization rate of the initial neural network module is positively correlated with the operation quantity of the initial neural network module, and the hardware utilization rate of the initial neural network module is negatively correlated with the second hardware delay and the theoretical computing power of the initial neural network module, respectively.

[0220] The parameter quantity of each initial neural network module is determined.

[0221] The score of each initial neural network module is determined according to the hardware utilization rate and the parameter quantity of each initial neural network module, and the score of the initial neural network module is used as an evaluation index of the initial neural network module, and the score of the initial neural network module is positively correlated with the hardware utilization rate and the parameter quantity of the initial neural network module respectively.

[0222] In a possible implementation, the initial neural network architecture is a convolutional neural network architecture, and the parameter quantity includes a number of convolution kernels of the convolutional neural network architecture.

[0223] In a possible implementation, the score of the initial neural network module is determined by the evaluation index determination unit through a first formula, and the first formula includes:

[0224] S ij = k * u ij * p ij ;

[0225] wherein S ij represents the score of the initial neural network module, u ij represents the hardware utilization rate, p ij represents the parameter quantity, k represents a constant coefficient, i represents a certain neural network layer, and j represents a certain initial neural network module.

[0226] and / or,

[0227] The hardware utilization rate of the initial neural network module is determined by the evaluation index determination unit through a second formula, and the second formula includes:

[0228] u ij = F ij / (t ij * C) ;

[0229] wherein F ij represents an operation quantity, t ij represents a second hardware delay, and C represents a theoretical computing power.

[0230] In a possible implementation, the second obtaining unit is configured to, if the score of the initial neural network module is greater than a score threshold, take the initial neural network module as the candidate neural network module.

[0231] The score threshold corresponding to the initial neural network module is related to the scores of all the initial neural network modules of the neural network layer where the initial neural network module is located, or the score threshold corresponding to the initial neural network module is related to the scores of all the initial neural network modules of the initial neural network architecture.

[0232] In a possible implementation, the candidate neural network architecture includes N layers of neural network layers, and the N layers of neural network layers are pre-divided into at least two parts of neural network layers to be trained, the at least two parts of neural network layers including L layers of first neural network layers and M layers of second neural network layers, where N is a natural number greater than or equal to 2, L and M are natural numbers greater than or equal to 1, and L+M≤N.

[0233] The training module 220 is configured to keep the M layers of second neural network layers fixed, and train the L layers of first neural network layers.

[0234] In a case where the training of the L layers of first neural network layers is completed, the L layers of first neural network layers are kept fixed, and the M layers of second neural network layers are trained.

[0235] The loss function of the candidate neural network architecture satisfies the training completion condition includes that the loss function of the L layers of first neural network layers satisfies the training completion condition and the loss function of the M layers of second neural network layers satisfies the training completion condition.

[0236] In a possible implementation, the first hardware parameter includes at least one of a first hardware delay and memory occupation.

[0237] In a possible implementation, the loss function includes:

[0238] L(a, w a ) = CE(a, w ij ) + α * LAT(a) ij + γ * MEM(a) ij ;

[0239] LAT(a) = ∑ ∑

[0001] m * LAT(block ) ;

[0240] MEM(a) = ∑

[0002] ∑ m

[0003] * MEM(block ) ;

[0241] where L(a, w ) represents the loss function, a represents the candidate neural network architecture, w

[0004] represents the weight parameter of the candidate neural network architecture, CE(a, w ) represents a cross-entropy loss function, α represents a first coefficient, LAT(a) represents a first hardware delay loss function, γ represents a second coefficient, and MEM(a) represents a memory occupation loss function.) represents a cross-entropy loss term of a candidate neural network architecture, LAT(a) represents a second hardware delay, MEM(a) represents a memory occupation, a hyperparameter represents a penalty coefficient of the second hardware delay, a hyperparameter represents a penalty coefficient of the memory occupation term, a hyperparameter is used to adjust the numerical size of the function term corresponding to the second hardware delay, a hyperparameter is used to adjust the numerical size of the function term corresponding to the memory occupation, block ij represents the jth candidate neural network module of the ith layer, m ij is a mask value generated by a Gumbel-Softmax function configured for a softmax layer in the candidate neural network architecture, m ij represents a sampling probability of a candidate neural network module of the candidate neural network architecture.

[0242] In a possible implementation, the candidate neural network architecture includes N layers of neural network layers, at least one of the N layers of neural network layers is a neural network layer participating in training, and N is a natural number greater than or equal to 2.

[0243] The architecture search module 230 is configured to obtain a sampling probability of each candidate neural network module of each layer of neural network layer participating in training, the sampling probability of the candidate neural network module being determined in a training process of the candidate neural network architecture.

[0244] The candidate neural network module corresponding to the maximum sampling probability is retained in each layer of neural network layer participating in training, to obtain the target neural network architecture.

[0245] It can be understood that the functions of the device of the embodiment and the beneficial effects thereof can refer to the descriptions of any of the above embodiments, which are not limited herein.

[0246] Embodiment three

[0247] Please refer to Figure 3 , Figure 3 is a structural schematic diagram of another neural network architecture search device disclosed in an embodiment of the present application. As shown in the figure, Figure 3 The neural network architecture search device can include:

[0248] a memory 301 storing executable program codes;

[0249] a processor 302 coupled with the memory 301;

[0250] The processor 302 invokes the executable program codes stored in the memory 301 to execute the steps in the neural network architecture search method described in the embodiment one of the present application.

[0251] Embodiment four

[0252] The embodiment of the present application discloses a computer storage medium, which stores computer instructions, and the computer instructions are used to execute the steps in the search method of the neural network architecture described in the embodiment one of the present application when called.

[0253] Embodiment five

[0254] The embodiment of the present application discloses a computer program product, which comprises a non-transitory computer storage medium storing a computer program, and the computer program is operable to make a computer execute the steps in the search method of the neural network architecture described in the embodiment one.

[0255] The above-described apparatus embodiments are only schematic, wherein the modules described as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0256] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, and the computer software product can be stored in a computer storage medium, and the storage medium includes a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a programmable read-only memory (Programmable Read-only Memory, PROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory, EEPROM), a one-time programmable read-only memory (One-time Programmable Read-Only Memory, OTPROM), an electrically erasable programmable read-only memory (Electrically-Erasable Programmable Read-Only Memory, EEPROM), a compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM) or other optical disk storage, a magnetic disk storage, a magnetic tape storage, or any other computer readable medium that can be used to carry or store data.

[0257] It should be noted that the neural network architecture searching method, the device and the computer storage medium disclosed by the embodiments of the present application are only the preferred embodiments of the present application, and are used to illustrate the technical solutions of the present application, but not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones. The modification or replacement does not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for searching a neural network architecture, characterized in that, The method comprises: obtaining a candidate neural network architecture; training the candidate neural network architecture; if the candidate neural network architecture is trained, searching a target neural network architecture based on the trained candidate neural network architecture, wherein the candidate neural network architecture is determined to be trained if a loss function of the candidate neural network architecture meets a training completion condition, the loss function comprising a function item corresponding to a first hardware parameter, the first hardware parameter being used to indicate a hardware parameter of a target hardware platform running the candidate neural network architecture; wherein the obtaining of the candidate neural network architecture comprises: obtaining an initial neural network architecture, the initial neural network architecture comprising N layers of neural network layers, each layer of the neural network layer comprising at least one initial neural network module, N being a natural number greater than or equal to 1; determining a second hardware parameter of each initial neural network module, the second hardware parameter being used to indicate a hardware parameter of the target hardware platform running the initial neural network module; determining an evaluation index of each initial neural network module according to the second hardware parameter of each initial neural network module, the second hardware parameter comprising at least one of an operation number, a second hardware delay and a theoretical computing power, the operation number comprising at least one of a multiplication operation number and an addition operation number; selecting a candidate neural network module of each layer of the neural network layer from at least one initial neural network module of each layer of the neural network layer according to the evaluation index of each initial neural network module to obtain the candidate neural network architecture, wherein the neural network layer of the candidate neural network architecture comprises the candidate neural network module; wherein the determining of the evaluation index of each initial neural network module according to the second hardware parameter of each initial neural network module comprises: determining a hardware utilization rate of each initial neural network module according to the second hardware parameter of each initial neural network module, wherein the hardware utilization rate of the initial neural network module is positively correlated with the operation number of the initial neural network module, and the hardware utilization rate of the initial neural network module is negatively correlated with the second hardware delay and the theoretical computing power of the initial neural network module, respectively; determining a parameter quantity of each initial neural network module; determining a score of each initial neural network module according to the hardware utilization rate and the parameter quantity of each initial neural network module, and taking the score of the initial neural network module as the evaluation index of the initial neural network module, the score of the initial neural network module being positively correlated with the hardware utilization rate and the parameter quantity of the initial neural network module, respectively.

2. The method of claim 1, wherein, The score of the initial neural network module is determined by a first formula, the first formula comprising: ； wherein, denotes a fraction of initial neural network modules, denotes hardware utilization, denotes parameter quantity, k denotes constant coefficient, i denotes a certain neural network layer, and j denotes a certain initial neural network module. and / or The hardware utilization rate of the initial neural network module is determined by a second formula, the second formula comprising: ； wherein, represents the number of operations, represents a second hardware delay, C represents a theoretical computing power.

3. The method of claim 1, wherein, The candidate neural network architecture comprises N layers of neural network layers, and the N layers of neural network layers are divided into at least two parts of neural network layers to be trained, the at least two parts of neural network layers comprise L layers of first neural network layers and M layers of second neural network layers, wherein N is a natural number greater than or equal to 2, L and M are natural numbers greater than or equal to 1, and L+M≤N; The training of the candidate neural network architecture comprises: keeping the M layers of second neural network layers fixed and training the L layers of first neural network layers; in the case where the training of the L layers of first neural network layers is completed, keeping the L layers of first neural network layers fixed and training the M layers of second neural network layers; wherein the loss function of the candidate neural network architecture meeting the training completion condition comprises the loss function of the L layers of first neural network layers meeting the training completion condition and the loss function of the M layers of second neural network layers meeting the training completion condition.

4. The method according to any one of claims 1 to 3, characterized in that, The first hardware parameter comprises at least one of a first hardware delay and memory occupation; The loss function comprises: ； ； ； wherein, represents a candidate neural network architecture, represents a weight parameter of the candidate neural network architecture, represents a cross-entropy loss term of the candidate neural network architecture, represents a second hardware delay, represents a memory footprint, a hyperparameter represents a penalty coefficient of the second hardware delay, a hyperparameter represents a penalty coefficient of the memory footprint term, a hyperparameter for adjusting a numerical size of a function term corresponding to the second hardware delay, a hyperparameter for a numerical size of a function term corresponding to the memory footprint, represents a jth candidate neural network module of an ith layer, a mask value generated by a Gumbel-Softmax function configured for a softmax layer in the candidate neural network architecture, represents a sampling probability of a candidate neural network module of the candidate neural network architecture.

5. The method according to any one of claims 1 to 3, characterized in that, The candidate neural network architecture comprises N layers of neural network layers, and at least one of the N layers of neural network layers is a neural network layer participating in training, and N is a natural number greater than or equal to 2; The searching of the target neural network architecture based on the trained candidate neural network architecture comprises: obtaining a sampling probability of each candidate neural network module of each layer of neural network layer participating in training, the sampling probability of the candidate neural network module being determined in the training process of the candidate neural network architecture; retaining the candidate neural network module corresponding to the maximum sampling probability in each layer of neural network layer participating in training to obtain the target neural network architecture.

6. An apparatus for searching a neural network architecture, the apparatus comprising: The device is used to execute the searching method of the neural network architecture according to any one of claims 1-5; and the device comprises: an obtaining module configured to obtain a candidate neural network architecture; a training module configured to train the candidate neural network architecture; an architecture searching module configured to search a target neural network architecture based on the trained candidate neural network architecture if the training of the candidate neural network architecture is completed, wherein the training of the candidate neural network architecture is determined to be completed if a loss function of the candidate neural network architecture meets a training completion condition, and the loss function comprises a function item corresponding to a first hardware parameter, the first hardware parameter being used to indicate a hardware parameter of a target hardware platform running the candidate neural network architecture.

7. An apparatus for searching a neural network architecture, the apparatus comprising: The device comprises: a memory storing executable program codes; a processor coupled with the memory; the processor invokes the executable program codes stored in the memory to execute the searching method of the neural network architecture according to any one of claims 1-5.

8. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, and the computer instructions are used to execute the searching method of the neural network architecture according to any one of claims 1-5 when invoked.

Images