Method, device and computer readable storage medium for generating a neural network

Abstract

The application discloses a neural network generation method, device and computer readable storage medium, wherein the neural network generation method acquires an optimal micro unit, constructs a first network by using the optimal micro unit, makes the first network have a powerful enough performance, and can meet actual application requirements; a preset training data set is used to train the first network, a second network is obtained, and a third network is established, wherein the number of micro units of the third network is less than that of the second network, each micro unit of all the third networks is trained by using the second network, and a neural network model can be constructed according to the trained micro units of the third network, so that the functions of all the micro units of the third networks correspond to the function of the second network, compared with a traditional method of training micro units one by one, the training quantity can be reduced, the computing power requirement can be effectively reduced, and thus the cost of generating the neural network model is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a neural network generation method, device and computer readable storage medium. BACKGROUND

[0002] In recent years, deep learning technology has made great progress under the support of stronger computing power and has achieved great success in computer vision, speech recognition, natural language processing and other fields. Researchers have found that among the many elements that make up a deep learning model, the structural differences of artificial neural networks have a huge impact on the final performance of the model. Generally speaking, designing a suitable neural network for a specific problem requires experienced algorithm engineers to spend a lot of time and effort to repeatedly adjust and verify the model, which is inefficient and difficult to guarantee the effect. Therefore, neural network architecture search technology, which can automatically determine the structure of neural networks for target problems, has become a research hotspot in the field of deep learning in recent years.

[0003] However, the existing neural network search technology has a specific deployment problem in actual application. The neural network that solves a specific problem ultimately needs to be deployed on various terminals with different computing power, such as mobile phones, tablets, laptops, desktops, etc. Therefore, it is necessary to provide a set of neural network models of various sizes for various inference terminals, but the computing power cost required to generate such a set of models is also very large. The current scheme for generating a set of neural network models is roughly as follows: (1) build a micro-unit network and train it one by one; (2) define and train a super network containing a large number of third networks. It can be seen that both schemes require a large amount of computing power. Therefore, it is of great significance to propose an efficient neural network model construction method for the deployment of neural network search in actual application. SUMMARY

[0004] The present application aims to at least solve one of the problems in the prior art. To this end, the present application proposes a neural network generation method that can effectively reduce the computing power requirement for building a neural network model and improve the generation efficiency.

[0005] The present application also proposes a device and a computer readable storage medium.

[0006] In a first aspect, the present application provides a neural network generation method, which comprises:

[0007] obtaining an optimal micro-unit, and constructing a first network according to the optimal micro-unit;

[0008] training the first network using a preset training data set to obtain a second network;

[0009] construct a third network, and train each micro unit of the third network by using the second network, wherein the number of micro units of the third network is less than the optimal number of micro units of the first network;

[0010] obtain a neural network model according to the trained micro units in the third network.

[0011] In a second aspect, an apparatus is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method for generating a neural network according to the first aspect.

[0012] In a third aspect, a computer readable storage medium is provided, which stores computer executable instructions for implementing the method for generating a neural network according to the first aspect.

[0013] The embodiments of the present application include: obtaining an optimal micro unit, constructing a first network by using the optimal micro unit, making the first network have a strong enough performance to meet the actual application requirements, training the first network by using a preset training data set to obtain a second network, and establishing a third network, wherein the number of micro units of the third network is less than the optimal number of micro units of the first network, training each micro unit of all the third networks by using the second network, and constructing a neural network model according to the trained micro units of the third network. In this way, the functions of all the micro units of the third network correspond to the functions of the second network. Compared with the traditional method of training micro units one by one, the number of training can be reduced, the computing power requirement can be effectively reduced, the cost of generating a neural network model is reduced, and all the micro units of the third network can be trained in parallel, so that the efficiency of constructing a neural network model is effectively improved.

[0014] Other features and advantages of the present application will be further described in the following description, and will become apparent from the description, or will be learned through practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structure particularly pointed out in the description, claims, and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are included to provide a further understanding of the technical solutions of the present application, and constitute a part of the specification, and are used to explain the technical solutions of the present application together with the embodiments of the present application, and do not constitute a limitation on the technical solutions of the present application.

[0016] Figure 1 is a schematic diagram of a system architecture platform provided by an embodiment of the present application;

[0017] Figure 2is a flowchart of a method for generating a neural network provided by an embodiment of the present application;

[0018] Figure 3 is a flowchart of a method for generating a neural network provided by another embodiment of the present application;

[0019] Figure 4 is a flowchart of a method for generating a neural network provided by another embodiment of the present application;

[0020] Figure 5 is a flowchart of a method for generating a neural network provided by another embodiment of the present application;

[0021] Figure 6 is a flowchart of a method for generating a neural network provided by another embodiment of the present application;

[0022] Figure 7 is a flowchart of a method for generating a neural network provided by another embodiment of the present application;

[0023] Figure 8 is a flowchart of a method for generating a neural network provided by another embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0025] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0026] Current neural network architecture search (NAS) focuses on the research of architecture search strategy, which can be roughly divided into two categories: macro search and micro search. Macro search refers to searching algorithm regarding neural network as a whole, and the search strategy selectively specifies one or more layers to perform alternative operations (change layer type or parameter, insert new layer). The disadvantage of this method is that the search space is too large, and it is difficult to determine how much time is needed to search for a suitable neural network architecture for a complex problem, so the current research is less popular. Micro search learns from the experience of manually designed neural network excellent cases (such as ResNet, InceptionNet, etc.), and regards the search space as a stack of micro units. The search strategy only needs to search for the optimal network micro architecture in a relatively limited search space. In specific applications, a number of micro units can be stacked to match problems with different complexity. Micro search NAS algorithm has high research popularity because of its reasonable search space size and flexible application method.

[0027] However, the existing micro search NAS algorithm has a specific deployment problem in actual application: the neural network solving a specific problem needs to be finally deployed on various terminals with different computing power (such as various models of mobile phones, tablet computers, notebook computers, desktop computers, etc.). Therefore, the NAS algorithm needs to provide a set of neural network models with various sizes for various inference terminals, but the cost of computing power required to generate such a set of models is also very large, which consumes a large amount of computing power.

[0028] Based on the above situation, the present application provides a neural network generation method, device and computer readable storage medium, which obtains an optimal micro unit, uses the optimal micro unit to construct a first network, so that the first network has sufficient strong performance and can meet the actual application requirements. The first network is trained using a predetermined training data set to obtain a second network, and a third network is established, wherein the number of micro units of the third network is less than the number of micro units of the second network. The second network is used to train each micro unit of all third networks, and a neural network model can be constructed according to the trained micro units of the third network. In this way, the functions of all micro units of the third network correspond to the functions of the second network. Compared with the traditional method of training micro units one by one, the number of training can be reduced, the computing power requirement can be effectively reduced, thereby reducing the cost of generating a neural network model, and all micro units of the third network can be trained in parallel, which can effectively improve the efficiency of constructing a neural network model.

[0029] The technical solutions of the present application will be described in detail below with reference to the accompanying drawings. Obviously, the following described embodiments are part of the embodiments of the present application, not all embodiments.

[0030] Referring toFigure 1 As shown, Figure 1 is a schematic diagram of a system architecture platform 100 for implementing a neural network generation method according to an embodiment of the present application.

[0031] In Figure 1 the embodiment shown, the system architecture platform 100 includes a memory 120 and a processor 110, wherein the memory 120 and the processor 110 can be connected by a bus or other means, Figure 1 for example, by a bus connection.

[0032] The memory 120, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 120 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 120 can optionally include a memory disposed remotely with respect to the processor, and these remote memories 120 can be connected to the system architecture platform 100 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The system architecture platform 100 and application scenarios described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, as the system architecture platform 100 evolves and new application scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0034] Those skilled in the art can understand that, Figure 1 the system architecture platform 100 shown in the above-mentioned embodiment does not constitute a limitation on the embodiments of the present application, and can include more or fewer components than those shown, or combine certain components, or different component arrangements.

[0035] In Figure 1 the system architecture platform 100 shown, the processor 110 can invoke a data processing program stored in the memory 120, thereby implementing the neural network generation method.

[0036] Based on the above-mentioned system architecture platform 100, the following proposes various embodiments of the neural network generation method of the present application.

[0037] Referring to Figure 2 as shown, Figure 2 is a flowchart of a neural network generation method according to an embodiment of the present application, which includes but is not limited to steps S100, S200, S300 and S400.

[0038] Step S100, obtaining an optimal micro unit, and constructing a first network according to the optimal micro unit.

[0039] In an embodiment, the neural network architecture is composed of a large number of micro units, and the neural network search generally defines a search space first, and the search space includes basic building blocks that define the neural network architecture. It can be understood that the larger the search space is, the more iterations the search algorithm needs to run, and the more GPU (Graphics Processing Unit) computing power is required. An optimized algorithm can automatically search for the optimal micro unit network structure. For example, an early algorithm is evolutionary search, which is based on fitness and is constantly trained and mutated to achieve good validation indicators. For another example, a search algorithm based on reinforcement learning allows an agent to discover the optimal configuration of the network architecture. Such an agent is usually implemented using a recurrent neural network, and its goal is to find a neural architecture that can achieve higher returns. In this embodiment, a micro search NAS algorithm is used to find the optimal micro unit from the search space.

[0040] In an embodiment, a first network is constructed according to the obtained optimal micro unit. The first network can be understood as a high-performance unit network composed of the optimal micro unit. The size of the unit network can be customized according to actual needs, so that the first network has sufficient performance and can meet the actual application processing. For example, the optimal micro unit is stacked to form the first network, which facilitates the rapid construction of the unit network. It can be understood that the more the number of stacked optimal micro units is, the larger the size of the first network is, and the more powerful the performance is. Conversely, the less the number of stacked optimal micro units is, the smaller the size of the first network is. Therefore, the first network is constructed according to actual application requirements, which can match problems with different complexities.

[0041] It should be noted that before step S100 is performed, a user can construct a search space of a micro unit network structure according to actual conditions, initialize the search space, and then obtain an optimal micro unit based on the search space, i.e., find the optimal micro unit from the limited search space. A reasonable search space size is conducive to reducing computing power and improving processing efficiency.

[0042] Step S200, training the first network using a preset training data set to obtain a second network.

[0043] In an embodiment, the first network is trained by using a preset training data set, which can be understood as a data set for training the micro unit. In step S100, the first network constructed by the optimal micro unit is a network model with undefined parameters, and the parameters of the network model need to be trained by using the corresponding training data, so that the first network has corresponding operation capability, thereby obtaining the second network, i.e., the trained first network.

[0044] Taking deep learning as an example, the deep learning model is trained by using the training data set, so that the learning model meets a specific performance index. With the increase of the training data, the accuracy of the deep learning model can be improved. It can be understood that as long as there are sufficient training data resources, the accuracy improvement brought by the scale improvement of the training model can be expected.

[0045] The preset training data set can be understood as selecting different types of training data sets according to different model objects, and selecting different amounts of training data according to different performance requirements of the model. For example, when the training object is an image recognition model, a large number of pictures can be selected as the preset training data set. For another example, when the training object is a speech recognition model, a large amount of voice data can be selected as the preset training data set.

[0046] In step S300, the third network is constructed, and each micro unit of the third network is trained by using the second network, wherein the number of micro units of the third network is less than the optimal number of micro units of the first network.

[0047] In an embodiment, the third network is a network model with undefined parameters, and the number of micro units of the constructed third network is less than the optimal number of micro units of the first network. It can be understood that a plurality of third networks with different numbers of micro units can be constructed, and the number of micro units of the obtained third network is less than the optimal number of micro units in the first network. In order to facilitate the computer to perform operation, the number of micro units of the third network and the optimal number of micro units of the first network can be set as a multiple, a power relationship, etc. For example, the optimal number of micro units of the first network can be 20, 40, 120, etc., and the number of micro units of the third network can be set to 2, 4, 10, etc. The third networks with the numbers of micro units of 2, 4, 10, etc. can be constructed.

[0048] The third network is constructed, and the second network is used to train all the third networks. Specifically, the second network trains each micro unit in the third network. It can be understood that the number of micro units of each third network is less than the number of micro units of the second network. The training speed of the network model with a smaller number of micro units is faster than that of the network model with a larger number of micro units. Moreover, the micro units of the third network can be trained in parallel, that is, multiple micro units are trained at the same time, and the training efficiency is higher, and the requirement for computing power can be greatly reduced. The trained third network has the same function as the second network. For example, when the number of micro units of the second network is 32 and the number of micro units of the third network is 4, the four micro units of the third network can have the same performance as the 32 micro units of the second network after being trained respectively.

[0049] In the embodiment, the second network can be defined as a teacher network, and the third network can be defined as a student network. The teacher network can train the micro units of all the student networks, and the micro units of all the student networks can be trained in parallel. Therefore, the training time can be reduced, and the training efficiency can be improved.

[0050] In step S400, a neural network model is obtained according to the trained micro units in the third network.

[0051] In an embodiment, each micro unit of all the third networks is trained according to the above step S300. After being trained, the micro units of different third networks are arranged to obtain different neural network models. For example, the number of micro units of the third network is four, and the four micro units are trained. The four trained micro units of the third network are arranged to obtain a neural network model with four micro units. The trained micro units of different third networks can also be combined to obtain a neural network model. For example, two trained micro units in one third network and one trained micro unit in another third network are combined to obtain a neural network model with three micro units. In this way, after step S400 is completed, multiple neural network models are obtained. In actual application, a suitable neural network model can be selected and deployed on different terminals, and the consumption of computing power can be effectively reduced.

[0052] The neural network generation method of the embodiment includes steps S100, S200, S300 and S400. According to actual application requirements, a neural network model suitable for different computing power terminals is obtained. Compared with the traditional neural network training method, the training quantity can be effectively reduced, the requirement for computing power can be reduced, and the cost of generating the neural network model can be reduced.

[0053] In an embodiment, in step S100, the process of constructing the first network according to the optimal micro unit includes but is not limited to the following steps:

[0054] Step S110, according to the predefined size, stack the optimal micro unit N times to generate the first network, wherein N is an integer power of 2.

[0055] Wherein, the predefined size can be understood as the number of micro units of the first network constructed according to the actual application requirements, the optimal micro unit is combined in a stacking manner, the more the number of optimal micro unit stacks, the larger the size of the first network, and the higher the performance. In the embodiment, the optimal micro unit is stacked N times to generate the first network, N is the number of micro units required to constitute a network with sufficient fitting performance, and N is set to an integer power of 2, such as 32, 64, 128, etc. For example, a task of processing image recognition is required, and according to the actual calculation, a network model with 32 micro units needs to be constructed to match, so the number of optimal micro units N of the first network can be selected as 32, or greater than 32, for example, N is 64, so that a network model with sufficient performance can be obtained, which can match problems with different complexity.

[0056] In an embodiment, specifically, the optimal micro unit can be stacked according to the topology principle, which can be understood as that the topology structure is constructed only by considering the positional relationship between objects without considering their shape and size, and the topology can be understood as regarding the optimal micro unit as a point regardless of its size and shape. The topology can be a network macro topology or a sequential topology, the number of optimal micro units is set according to the topology, and all optimal micro units are connected to constitute the first network.

[0057] In an embodiment, in step S100, the process of obtaining the optimal micro unit includes but is not limited to the following steps:

[0058] Step S120, according to the NAS algorithm, the network structure of the optimal micro unit is obtained by using the preset search space.

[0059] In the embodiment, the NAS algorithm has the characteristics of automatically finding the optimal neural network architecture for a specific problem, and the optimal micro unit can be quickly obtained by using the NAS algorithm. The NAS algorithm is given a candidate neural network structure set called search space, and the optimal network structure is searched from the search space by using the search strategy. The specific principle of the NAS algorithm is not described here.

[0060] Referring to Figure 3 In an embodiment, in step S300, the process of constructing the third network can include but is not limited to the following steps:

[0061] Step S310, constructing a third network with a number of micro units satisfying N / M and greater than 1, wherein M is an integer power of 2;

[0062] Step S320, initializing all third networks.

[0063] The third network is defined as a student network, and the number of micro units of the third network is determined according to the optimal number of micro units N of the first network, N is an integer power of 2, the number of micro units of the third network satisfies N / M and is greater than 1, M is an integer power of 2, it can be understood that the value range of N and M is an integer power of 2, N / M>1 indicates that the value of N is greater than M, under the condition of satisfying N / M>1, a plurality of third networks with different numbers of micro units can be constructed, that is, student networks with numbers of micro units of N / 2, N / 4, N / 8...2 can be constructed. For example, when the optimal number of micro units N of the first network is 32, the value of M can be 2, 4, 8, 16, so that the student networks with numbers of micro units of 32 / 2=16, 32 / 4=8, 32 / 8=4, 32 / 16=2 are obtained, a total of 4 student networks, and then all the student networks are initialized to assign default values to the model parameters of each student network, thereby constructing student networks with different numbers of micro units.

[0064] It can be understood that the number of micro units n of the student network and the optimal number of micro units N of the first network are in logarithmic relationship, that is, n=logN, so that when the teacher network is used to train the micro units of all student networks, the training quantity is reduced from N network structures to logN network structures, which can effectively reduce the time required for training and the training speed will be faster, and the micro units of the third network can be trained in parallel, that is, multiple micro units are trained at the same time, compared with the traditional micro unit network training mode, the training efficiency is higher, and the computing power requirement can be greatly reduced.

[0065] In an embodiment, in step S400, the neural network model is obtained according to the trained micro units in the third network, which can include but is not limited to the following steps:

[0066] In step S410, the trained micro units are combined to obtain neural network models with different numbers of micro units, and the number of neural network models ranges from 2 to N.

[0067] The teacher network is used to train each micro unit of the student network, and the trained micro units are arranged and combined, specifically, the arrangement and combination mode is determined according to the number of trained micro units, for example, the number of micro units of the student network is 4 and all the micro units are trained, the 4 trained micro units of the student network are arranged to obtain a neural network model with 4 micro units; for another example, 2 trained micro units in one student network are combined with 1 trained micro unit in another student network to obtain a neural network model with 3 micro units. According to the combination of all the micro units of the student network, a complete set of neural network model groups can be obtained, which can be understood as a set of all neural network models with different numbers of micro units.

[0068] Since the functions of the micro-units of all the student networks trained correspond to certain local morphisms of the teacher network, the combined network can be directly deployed or deployed after a small amount of fine-tuning on the terminal.

[0069] It can be understood that, taking the first network composed of the optimal micro-units N times as an example, the first network after training obtains the teacher network with the number of micro-units N, and at the same time, student networks with the number of micro-units N / 2, N / 4, N / 8...2 can be constructed, and all the student networks are trained by the teacher network, and the trained micro-units are combined to obtain 2-N neural network models. For example, when N is 32, student networks with the number of micro-units 2, 4, 8 and 16 are constructed to obtain 4 student networks, the teacher network has 32 micro-units, the teacher network trains the student network with 2 micro-units to obtain a neural network model with 2 micro-units, and the functions of the 2 micro-units correspond to the teacher network. It can be understood that, the teacher network maps the functions of the micro-units of the teacher network to the micro-units of the student network by training the student network; for another example, the teacher network trains the student network with 4 micro-units to obtain a neural network model with 4 micro-units, and by combining the micro-units of the student network, different neural network models can be obtained.

[0070] In an embodiment, training each micro-unit of the third network by the second network can include but is not limited to the following steps:

[0071] Step S330, training each micro-unit of the third network by the second network with a training data set and through the local input and output mode of the second network.

[0072] During training, the teacher network trains each micro-unit of each student network with a training data set, which is the training data for training the teacher network, and specifically, each micro-unit of each student network is trained by the local input and output training mode of the teacher network.

[0073] Taking a specific example for description, the optimal number of micro-units of the first network is N, and for the convenience of display, N is taken as 8, and in actual application, N should be set to 32, 64 or a larger value. For example, the first network is a-a-a-a-a-a-a-a, the first network after training obtains the teacher network a1-a2-a3-a4-a5-a6-a7-a8, and then student networks with the number of micro-units 2 and 4 are constructed, and the two student networks are trained by the teacher network. Taking the student network with the number of micro-units 4 a’1-a’2-a’3-a’4 as an example, for a certain image data sample S, the parameter gradient of the student network micro-unit a’1 can be calculated by the input tensor of the teacher network a1 unit and the output tensor of the a2 unit:

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] wherein, represents the input of the teacher network unit a1, S represents a specific data sample, represents the output of the teacher network micro-unit a2, is the model parameter of the micro-unit a1, and the activation function, and similarly, and is the model parameter of the micro-unit a2, a'1, and the activation function, is the loss function calculated by the micro-unit a'1 according to the data sample S, and E represents the loss function calculation method, is the model parameter gradient of the micro-unit a'1 calculated by the partial derivative of the loss function , and η represents the learning rate of the model parameter updated this time.

[0080] Similarly, the parameter gradient calculation methods of the remaining three units a'2, a'3, and a'4 in the student network are consistent with the above calculation methods. It can be understood that after training, the local morphism a1-a2 of the teacher network is replaced by a'1, the local morphism a3-a4 of the teacher network is replaced by a'2, the local morphism a5-a6 of the teacher network is replaced by a'3, and the local morphism a 7 -a 8 of the teacher network is replaced by a'4, and the obtained neural network model is a'1-a'2-a'3-a'4.

[0081] a1-a3

[0082] It can be understood that the student network a"1-a'2, with the number of units being 2, replaces the local morphism a2-a4 of the teacher network with a"1, replaces the local morphism a

[0083] a5-a7

[0084] of the teacher network with a'2, and replaces the local morphism a6-a8 of the teacher network with a'3, and the obtained neural network model is a"1-a"2. In this way, the micro-units of the student network are trained by the local input and output of the teacher network, thereby completing the training of all micro-units of the student network, and all micro-units of the student network can be trained in parallel, so that the required time can be greatly reduced, and the training efficiency is effectively improved.

[0085] In an embodiment, the trained micro-units are combined in step S410 to obtain 2-N neural network models, i.e., the trained micro-units are arranged and combined to obtain a complete set of size neural network model groups. As an example of step S330 described above, the following full-size neural network models can be obtained:

[0086] 2 micro-units: a"1-a"2;

[0087] 3 micro-units: a"1-a'3-a'4 or a'1-a'2-a'2,

[0088] 4 micro-units: a'1-a'2-a'3-a'4;

[0089] 5 micro-units: x'1-a'2-a'3-a7-a8;

[0090] 6 micro-units: a'1-a'2-a5-a6-a7-a8;

[0091] 7 micro-units: a'1-a3-a4-a5-a6-a7-a8;

[0092] 8 micro-units: a1-a2-a3-a4-a5-a6-a7-a8.

[0093] The combination of micro-units with quantities of 5, 6, and 7 is not limited to the models shown above, and is not listed one by one here.

[0094] Referring to Figure 4 In an embodiment, the flow of the neural network generation method further includes, but is not limited to, the following steps:

[0095] Step S500, test or evaluate each neural network model, and select a neural network model that meets the preset condition as a target neural network.

[0096] After the steps S100, S200, S300, and S400 of the embodiment are completed, a complete set of size neural network models is obtained. Since the neural network model ultimately needs to be deployed on various terminals with large differences in computing power, according to the performance requirements of the neural network model and the inference time delay requirements of the terminal to be deployed, etc. Limiting conditions, select a suitable neural network model from all combined networks, and then deploy the suitable neural network model as a target neural network to the terminal.

[0097] Specifically, the neural network model meeting the preset condition is selected as the target neural network by testing or evaluating each neural network model. The testing method can test all the neural network models by using a test data set to obtain an optimal neural network model. The evaluation method can evaluate the floating point operations per second (FLOPs) of each neural network model offline, for example, a neural network model with FLOPs not exceeding 400M is selected as the target neural network.

[0098] Referring to Figure 5 In an embodiment, each neural network model is tested, and a neural network model meeting a preset condition is selected as a target neural network, including:

[0099] In step S510, each neural network model is tested by using a preset test data set to obtain a test accuracy of each neural network model.

[0100] In step S520, each neural network model is used to perform the same task to obtain a time delay of each neural network model.

[0101] In step S530, if the time delay of the target neural network model is lower than a preset value and the test accuracy is the best value, the target neural network model is determined as the target neural network.

[0102] In step S510, the preset test data set can be understood as a data set for testing the neural network model, and the corresponding test data is selected according to different terminal performance requirements. By testing each neural network model, the test accuracy of each neural network model can be obtained. It can be understood that the higher the test accuracy, the higher the performance index, and the neural network model with high test accuracy is preferentially selected.

[0103] Step S520 is performed on the target terminal, each neural network model is used to perform the same task to obtain a time delay of each neural network model. This step is used to test the time delay index of each neural network model on the terminal. In general, the transmission time delay and the propagation time delay are the main indexes. The higher the time delay, the larger the time interval of data processing, and the lower the execution efficiency. Therefore, the neural network model with low execution time delay is preferentially selected.

[0104] In this way, each neural network model is tested according to the time delay and the test accuracy. If the time delay of the tested neural network model is lower than a preset value and the test accuracy is the best value, the neural network model is determined as the target neural network. In the embodiment, the limit condition of the time delay can be set as not higher than 500ms, or lower than 400ms or other preset values. The test set accuracy can be selected to reach more than 95%, and in the embodiment, the preferred value is 96.3%.

[0105] To more clearly illustrate the specific steps and processes of the neural network generation methods in the above embodiments, the following three embodiments will be used for explanation.

[0106] Example 1:

[0107] See Figure 6 As shown, the search space of the NAS algorithm is defined as a micro-unit space, where each micro-unit is a single-input, single-output structure, and the macro-unit topology is a sequential structure. A network model composed of multiple micro-units is used to solve the image classification problem, taking multi-channel image data as input and outputting the actual classification of image content. The neural network generation method includes the following steps:

[0108] Step S610: Initialize the search space of the micro-unit network structure, apply the micro-search NAS algorithm, and obtain the optimal micro-unit;

[0109] Step S620: Set the number of micro-units in the sequential topology and form a first network of a predefined size.

[0110] Step S630: Use the image data from the training dataset and the classification information corresponding to each image to train the first network, and obtain the teacher network a1-a2-a3-a4-a5-a6-a7-a8;

[0111] Step S640: Construct the student network and use the teacher network in conjunction with the training dataset to train each micro-unit of the student network with 4 and 2 micro-units respectively;

[0112] Step S650: After training, the micro-units of all student networks are arranged and combined to obtain a complete set of neural network model groups.

[0113] The specific steps for teacher-network training of student networks are detailed in step S330 of the embodiment, and will not be repeated here. In this embodiment, the optimal micro-unit is a single-input single-output structure.

[0114] Example 2:

[0115] See Figure 7 As shown, the search space of the NAS algorithm is defined as a micro-unit space, where each micro-unit is a two-input, single-output structure. A network model composed of multiple micro-units is used to solve the image classification problem, taking multi-channel image data as input and outputting the actual classification of the image content. The neural network generation method includes the following steps:

[0116] Step S710: Initialize the search space of the micro-unit network structure, apply the micro-search NAS algorithm, and obtain the optimal micro-unit structure;

[0117] aaaa

[0118] Step S720: Set the number of micro units of the complex topology network and form the first network a-a-a-a of the pre-defined size;

[0119] Step S730: Train the model using the picture data in the training data set and the classification information corresponding to each image to obtain

[0120] a1-a3-a5-a7

[0121] Teacher network a2-a4-a6-a8;

[0122] Step S740: Build the student network and train each micro unit of the student network with a unit number of 4 and 2 using the teacher network and the training data set;

[0123] Step S750: After the training of all the micro units of the student network, arrange and combine to obtain a complete set of size neural network model groups.

[0124] Taking the student network a'1-a'2-a'3-a'4 with a micro unit number of 4 as an example, for a certain image data sample S, the parameter gradient of the student network micro unit a'2 can be calculated from the output tensor of the teacher network a1, a2 unit and the merged tensor output by a3, a4 unit:

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131] wherein, represents the input of the teacher network a1 unit, S represents a specific data sample, and represent the outputs of the teacher network micro units a1, a2, a3, a4, is the model parameter and activation function of the micro unit a1, and similarly, and are the model parameters and activation functions of the micro units a2, a3, a4, a'2, is the loss function calculated by the micro unit a'2 according to the sample S, and E represents the loss function calculation method, represents the splicing of the outputs of two micro units, is the micro-unit a'2 according to the loss function The partial derivative calculation is the model parameter gradient of the current unit, and η represents the learning rate of the updated model parameter.

[0132] Similarly, the parameter gradient calculation methods of the remaining three units a'1, a'3, and a'4 in the student network are consistent with the above calculation method; the student network a"1-a"2 with two micro-units also has the same calculation method.

[0133] In step S750, taking the combination of three micro-unit networks as an example, there are two combination methods:

[0134] a1-a3 a5 a7

[0135] 1. a"1-a'3-a'4: replace a"1 with the local homomorphism a2-a4 of the teacher network, replace a'3 with a6, and replace a'4 with a8;

[0136] a5-a7 a1 a3

[0137] 2. a'1-a'2-a"2: replace a"2 with the local homomorphism a6-a8 of the teacher network, replace a'1 with a2, and replace a'2 with a4.

[0138] In this embodiment, the optimal micro-unit is a double-input single-output structure.

[0139] Embodiment Three

[0140] Referring to FIG. 8, Figure 8 As shown in FIG. 8, after obtaining the full-size neural network model, the deployment terminal has the restriction condition that the online inference delay should not be greater than 500 ms, and the neural network model test process includes the following steps:

[0141] Step S810: Evaluate the prediction accuracy of all neural network models with different numbers of micro-units on the test data set;

[0142] Step S820: Apply all neural network models with different numbers of micro-units to perform the same inference task on the target terminal, and record the delay index of the terminal when performing the inference task;

[0143] Step S830: According to the prediction accuracy and the delay index, select the model with the highest inference accuracy under the pre-set inference delay limit condition and deploy it to the terminal.

[0144] In this embodiment, the terminal runs a classification network composed of more than 5 micro-units, and the inference latency is greater than 500ms, which does not meet the restriction conditions. Therefore, the best-performing neural network model can be selected from networks composed of 4 or fewer micro-units. For example, the network with the best accuracy among the models that meet the conditions can achieve a test set accuracy of 96.3%. Therefore, this neural network model should be selected first when deploying to the terminal.

[0145] Additionally, one embodiment of the present invention provides an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor and the memory may be connected via a bus or other means.

[0146] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0147] It should be noted that the terminal in this embodiment may include, for example, Figure 1 The system architecture platform 100 in the illustrated embodiment, the terminal in this embodiment, and so on Figure 1 The system architecture platform 100 in the illustrated embodiments belongs to the same inventive concept, therefore these embodiments have the same implementation principle and technical effect, which will not be described in detail here.

[0148] The non-transient software program and instructions required to implement the neural network generation method of the above embodiments are stored in memory. When executed by a processor, the neural network generation method of the above embodiments is executed, for example, the method described above is executed. Figure 2 Method steps S100 to S400 in the text Figure 3 Method steps S310 to S320 in the text Figure 4 Method steps S500, Figure 5 Method steps S510 to S530 Figure 6 Method steps S610 to S650 Figure 7 Method steps S710 to S750 Figure 8 Method steps S810 to S830.

[0149] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the aforementioned terminal embodiment, causing the processor to perform the neural network generation method described above, for example, performing the above-described... Figure 2 Method steps S100 to S400 in the text Figure 3 Method steps S310 to S320 in the text Figure 4 Method steps S500, Figure 5 Method steps S510 to S530 Figure 6 Method steps S610 to S650 Figure 7 Method steps S710 to S750 Figure 8 Method steps S810 to S830.

[0151] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0152] The above describes the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A method for generating a neural network, executed by a computer, comprising: obtaining an optimal micro-unit, and constructing a first network according to the optimal micro-unit; training the first network using a preset training data set to obtain a second network; the training data set comprises at least one of picture and voice data; constructing a plurality of third networks, and training each micro-unit of the third networks using the second network, wherein the number of micro-units of the third networks is less than the number of optimal micro-units of the first network; each micro-unit of the third networks is trained in parallel; each micro-unit of the third networks corresponds to a different local morphism on the second network; combining the trained micro-units and the optimal micro-units in multiple ways to obtain full-size neural network models; testing each of the full-size neural network models using a preset test data set to obtain a test accuracy of each of the neural network models; executing the same inference task on a target terminal using each of the neural network models of different sizes to obtain a time delay of each of the neural network models; if the time delay of a target neural network model is lower than a preset value and the test accuracy is the best value, determining the target neural network model as the target neural network.

2. The method of claim 1, wherein, the constructing a first network according to the optimal micro-unit comprises: stacking the optimal micro-unit N times according to a predefined size to generate the first network, wherein N is an integer power of 2.

3. The method of claim 2, wherein, the constructing a third network comprises: constructing the third network with a number of micro-units satisfying N / M and being greater than 1, wherein M is an integer power of 2; initializing all the third networks.

4. The method of claim 3, wherein, the number of neural network models ranges from 2 to N.

5. The method of claim 1 to 4, wherein, the training each micro-unit of the third networks using the second network comprises: training each micro-unit of the third networks using the second network in combination with the training data set, and through a local input and output mode of the second network.

6. The method of claim 1, wherein, the obtaining an optimal micro-unit comprises: obtaining a network structure of the optimal micro-unit using a preset search space according to a neural network architecture search NAS algorithm.

7. An apparatus comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, the processor executes the computer program to implement the method for generating a neural network according to any one of claims 1 to 6.

8. A computer-readable storage medium storing computer-executable instructions, the computer-executable instructions comprising: the computer executable instructions are used to execute the method for generating a neural network according to any one of claims 1 to 6.

Images