Neural network structure search method, image processing method and device
By determining the search network for the target task and updating the parameter space in the neural network structure search, and maintaining the set of sub-networks, the problem that neural network structures cannot adapt to multiple scenarios in traditional methods is solved, and efficient search and adaptability of multiple target neural network models are achieved.
Patent Information
- Application Number
- CN201910834158.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-09-04
AI Technical Summary
Traditional neural network architecture search methods can only be applied in specific scenarios, cannot meet differentiated needs, and cannot obtain multiple target neural network models suitable for different resource constraints in a single search.
By defining the search network for the target task, including the structure space and parameter space, and using training data to update the parameter space and sub-network set, a sub-network set is maintained to obtain multiple target neural network models, adapting to the resource constraints of different application scenarios.
This method enables the acquisition of multiple target neural network models in a single search, meeting the needs of different computing and storage resource constraints and expanding the applicability and search efficiency of neural network structures.
Smart Images

Figure CN112445823B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence, and more specifically, to a neural network structure search method, image processing method and device. Background Art
[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.
[0003] With the rapid development of artificial intelligence technology, neural networks (e.g., deep neural networks) have made great achievements in the processing and analysis of various media signals such as images, videos, and voices in recent years. A neural network with excellent performance often has a sophisticated network structure, which requires human experts with superb skills and rich experience to spend a lot of effort to build. In order to better build neural networks, people have proposed to build neural networks through neural architecture search (NAS) methods, which automatically search for neural network structures to obtain neural network structures with excellent performance.
[0004] In the traditional neural network structure search method, it is first necessary to define a search space that can represent all network architectures in principle, then find candidate network structures from the search space through search strategies and evaluate them, and conduct the next round of search based on the feedback results. After multiple rounds of search, only one sub-network structure is retained. However, for different application scenarios, resource constraints are different. A sub-network structure obtained by the traditional neural network structure search method can only be applied to scenarios that match the sub-network structure, and cannot meet differentiated needs. Summary of the invention
[0005] The present application provides a neural network structure search method, an image processing method and a device, which can obtain multiple target neural network models through a single search, so that users can select a suitable model for application according to the resource constraints in the application scenario.
[0006] In a first aspect, a method for searching a neural network structure is provided, the method comprising: determining a search network according to a target task, the search network comprising a structure space and a parameter space, the structure space comprising a plurality of sub-network structures, the parameter space comprising a plurality of parameters, wherein each sub-network structure corresponds to at least one parameter in the parameter space; updating the parameter space according to training data of the target task to obtain an updated parameter space; determining a sub-network set from the search network, the sub-network set comprising a plurality of first sub-network structures among the plurality of sub-network structures; updating the sub-network set to obtain an updated sub-network set, the updated sub-network set comprising a plurality of second sub-network structures among the plurality of sub-network structures; and determining a plurality of target neural networks corresponding to the target task according to the updated parameter space and the updated sub-network set.
[0007] The above search network is determined according to the application requirements of the target neural network corresponding to the target task.
[0008] That is to say, the search network can be determined according to the application requirements of the target neural network. Specifically, the search network can be determined according to the data type of data that the target neural network needs to process.
[0009] The search network includes a variety of connection methods and different operations. Generally speaking, the types and number of operations included in the search network should match the application requirements of the target neural network. When the above-mentioned target neural network is used to process image data, the types and number of operations included in the above-mentioned search network should be adapted to the processing of image data. When the above-mentioned target neural network is used to process voice data, the types and number of operations included in the above-mentioned search network should be adapted to the processing of voice data.
[0010] For example, when the target neural network is a neural network for processing image data, the search network may include a long short-term memory unit or a long short-term memory network (LSTM), etc. Specifically, the search network may include a convolution operation, a pooling operation, a skip-connect operation, an activation function, etc.
[0011] For another example, when the target neural network is a neural network for processing speech data, the search network may include a long short term memory unit or a long short term memory network (LSTM). Specifically, the search network may include an activation function (such as ReLU, Tanh), etc.
[0012] The above search space is determined based on the application requirements of the target neural network and the video memory resource conditions of the device that performs the neural network structure search.
[0013] Among them, the graphics memory resource condition of the device that performs the neural network structure search may refer to the graphics memory resource size of the device that performs the neural network structure search.
[0014] That is to say, in the present application, the above-mentioned search space can be comprehensively determined based on the application requirements of the target neural network and the video memory resource conditions of the device that performs the neural network structure search.
[0015] Specifically, the types and number of operations included in the search space can be determined based on the application requirements of the target neural network, and then the types and number of operations included in the search space can be adjusted based on the graphics memory resource conditions of the device that performs the neural network structure search to determine the types and number of operations ultimately included in the search space.
[0016] For example, after determining the types and number of operations included in the search space according to the application requirements of the target neural network, if the video memory resources of the device performing the neural network structure search are small, some less important operations in the search space can be deleted. If the video memory resources of the device performing the neural network structure search are sufficient, the types and number of operations included in the search space can be maintained, or the types and number of operations included in the search space can be increased.
[0017] The search network in the embodiment of the present application may include basic modules for building a neural network, such as convolution, pre-set basic operations or combinations of basic operations that are compatible with the application requirements of the target neural network, wherein these basic operations or combinations of basic operations may be collectively referred to as basic operations. The size, depth, width, connection mode, etc. of the search network may be manually designed.
[0018] The search network includes a structure space and a parameter space, the structure space includes multiple sub-network structures, and the parameter space includes multiple parameters, wherein each sub-network structure corresponds to at least one parameter in the parameter space.
[0019] The structural space in the embodiment of the present application includes all network architectures that can be characterized in principle, and can also be understood as the structural space being a complete set of network structures, where each possible network structure can be called a sub-network structure. The sub-network structure includes all or part of the basic modules and all or part of the basic operations in the search network. The types and quantities of the basic modules and basic operations included in different sub-network structures are different, but the parameters of the basic modules and basic operations included in different sub-network structures are shared. That is, the parameter space in the embodiment of the present application includes the parameters of the basic modules and basic operations in the search network, such as the parameters involved in the separation convolution with a convolution kernel size of 3×3, the parameters involved in the dilated convolution with a convolution kernel size of 5×5 and a dilated rate of 2, and the parameters involved in the jump connection of the basic operation. Different sub-network structures can contain different types and quantities of basic modules and basic operations, and different sub-network structures can have different connection methods, but different sub-network structures share the parameters in the parameter space. For example, if different sub-network structures contain the same basic operation, then different sub-network structures share the parameters corresponding to the same basic operation included in the parameter space. Each basic module can correspond to one or more parameters, each basic operation can correspond to one or more parameters, each sub-network structure can include one or more basic modules, and can include one or more basic operations. Therefore, each sub-network structure in the structural space in the embodiment of the present application corresponds to at least one parameter in the parameter space, for example, each sub-network structure corresponds to one or more parameters in the parameter space.
[0020] In order to better understand the structural space and sub-network structure of the embodiments of the present application, for example, assuming that the number of layers of the search network is 22, and each layer includes 5 basic operations (the 5 basic operations are expressed by numbers 1-5 respectively), then a 22-dimensional vector can be used to express a sub-network structure, for example (1,2,4,5,3,5,1...3,2), different vectors express different sub-network structures, and the sub-network structures expressed by all 22-dimensional vectors can constitute the structural space of the embodiments of the present application.
[0021] Optionally, when determining the search network corresponding to the target task, the parameter values in the parameter space may be randomly set, or default initialized values, or manually set values, which is not specifically limited in the embodiments of the present application.
[0022] It should be understood that in the embodiments of the present application, updating the parameter space may be understood as updating the parameters in the parameter space, or as updating the values of the parameters in the parameter space.
[0023] Optionally, updating the parameter space according to the training data of the target task may be updating part of the parameters in the parameter space, or may be updating all the parameters in the parameter space.
[0024] In the embodiment of the present application, the training data of the target task can be a part of the data that the target neural network needs to process, or all of the data that the target neural network needs to process, or other data of the same data type as the data that the target neural network needs to process, and the implementation of the present application does not make specific limitations. For example, when the above-mentioned target neural network is used to process image data, the training data of the above-mentioned target task can be image data such as pictures; when the above-mentioned target neural network is used to process voice data, the training data of the above-mentioned target task can be voice data such as voice fragments; when the above-mentioned target neural network is used to process text data, the training data of the above-mentioned target task can be text data such as text sequences, etc.
[0025] In an embodiment of the present application, a subnetwork set includes multiple subnetworks, wherein the subnetwork includes a subnetwork structure and parameters corresponding to the subnetwork structure, the subnetwork structure belongs to the structural space, and the parameters corresponding to the subnetwork structure belong to the parameter space. In other words, a subnetwork can be expressed by a subnetwork structure and the parameters corresponding to the subnetwork structure. The parameters corresponding to the subnetwork structure can be understood as parameters corresponding to the basic modules and basic operations included in the subnetwork structure. In an embodiment of the present application, the subnetwork set includes multiple first subnetwork structures among the multiple subnetwork structures included in the structural space. It should be understood that the subnetwork set also includes parameters corresponding to each first subnetwork structure in the multiple first subnetwork structures.
[0026] It should be understood that in the embodiments of the present application, updating the subnetwork set can be understood as replacing the subnetwork structure included in the subnetwork set, or can be understood as re-determining or reselecting the subnetwork structure included in the subnetwork set.
[0027] In an embodiment of the present application, the updated subnetwork set includes multiple second subnetwork structures among the multiple subnetwork structures included in the structure space, wherein the first subnetwork structure and the second subnetwork structure may be the same or different. In other words, the updated subnetwork set may include part or all of the multiple first subnetwork structures in the subnetwork set before the update, or the multiple second subnetwork structures included in the updated subnetwork set are completely different from the multiple first subnetwork structures included in the subnetwork set before the update.
[0028] The method for searching the neural network structure of the embodiment of the present application maintains a sub-network set during the search process, and multiple target neural networks can be obtained through one search. The multiple sub-network structures included in the sub-network set share parameters in the parameter space, which can speed up the search and improve the search efficiency, thereby efficiently obtaining multiple target neural networks from one search.
[0029] Furthermore, the multiple target neural networks may have different properties, providing models that can meet different computing and / or storage resource constraints, allowing users to select corresponding target neural networks based on resource constraints in actual applications.
[0030] In combination with the first aspect, in a possible implementation method, the parameter space is updated according to the training data of the target task to obtain an updated parameter space, including: inputting the training data of the target task; updating the parameter space based on a gradient back propagation algorithm, wherein the gradient of the parameter space is the average value of the parameter gradients corresponding to some of the first sub-network structures among the multiple first sub-network structures.
[0031] In an embodiment of the present application, directly summing up the parameter gradients corresponding to all the first sub-network structures in the sub-network set may bring a large computational burden. The parameter gradients corresponding to some first sub-network structures in a small batch (mini-batch) can be summed up, which can reduce the computational burden and speed up the update of the parameter space, thereby speeding up the search speed of the neural network structure and improving the efficiency of the neural network structure search.
[0032] In combination with the first aspect, in a possible implementation method, the updating of the sub-network set to obtain an updated sub-network set includes: processing the multiple first sub-network structures based on any one of a particle swarm algorithm, a genetic algorithm and a fireworks algorithm to obtain a next-generation sub-network structure; based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a first metric target to obtain a first grade division result; based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a second metric target to obtain a second grade division result; the first grade division result and the sub-network structures included in the same level in the second grade division result are respectively merged to obtain a merged grade division result; and multiple sub-network structures are selected from the merged grade division result as the multiple second sub-network structures.
[0033] It should be understood that the particle swarm algorithm, genetic algorithm and fireworks algorithm are all types of evolutionary algorithms. The embodiments of the present application may also use other evolutionary algorithms to execute the same method as the above-mentioned particle swarm algorithm, genetic algorithm and fireworks algorithm.
[0034] In combination with the first aspect, in a possible implementation method, the sub-network set is updated to obtain an updated sub-network set, including: based on a non-dominated sorting algorithm, a plurality of first sub-network structures and a next-generation sub-network structure are graded according to a first metric target to obtain a first grade division result; based on a non-dominated sorting algorithm, a plurality of first sub-network structures and a next-generation sub-network structure are graded according to a second metric target to obtain a second grade division result; and a plurality of sub-network structures are selected from the top levels of the first grade division result and the second grade division result as the plurality of second sub-network structures.
[0035] The embodiment of the present application performs two non-dominated sorting algorithm levels on the current generation sub-network structure and the next generation sub-network structure according to different metric objectives, and then selects multiple second sub-network structures based on the two level division results. This can protect some models that temporarily have poor performance but may achieve better performance in the future, avoid these models being dominated in the early stage of the search and thus being abandoned prematurely, thereby expanding the coverage of the sub-network set.
[0036] In combination with the first aspect, in a possible implementation method, the first measurement target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy; the second measurement target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and growth rate of accuracy.
[0037] In combination with the first aspect, in a possible implementation method, determining multiple target neural networks corresponding to the target task based on the update parameter space and the update sub-network set includes: selecting part or all of the second sub-network structures from the update sub-network set as the multiple target neural networks, wherein the parameters corresponding to the part or all of the second sub-network structures belong to the update parameter space.
[0038] In combination with the first aspect, in a possible implementation method, it also includes: obtaining target task data to be processed; selecting a first target neural network from the multiple target neural networks based on the computing resources and / or storage resources of the terminal device; processing the target task data to be processed according to the first target neural network to obtain a processing result corresponding to the target task.
[0039] In an embodiment of the present application, a target neural network that matches the resource constraints of the terminal device can be selected from the models of different sizes and precisions provided above according to the computing resources and / or storage resources of the terminal device, and applicable terminal devices and application scenarios are more extensive.
[0040] In a second aspect, a method for searching a neural network structure is provided, the method comprising: determining a search network according to a target task, the search network comprising a structure space, the structure space comprising multiple sub-network structures; determining a sub-network set from the search network, the sub-network set comprising multiple first sub-network structures among the multiple sub-network structures; updating the sub-network set to obtain an updated sub-network set, the updated sub-network set comprising multiple second sub-network structures among the multiple sub-network structures; and determining multiple target neural networks corresponding to the target task according to the updated sub-network set.
[0041] The method for searching for a neural network structure in an embodiment of the present application maintains a sub-network set during the search process, and can provide multiple target neural networks after one search. The multiple target neural networks can have different sizes and accuracies and can meet different computing and / or storage resource constraints, so that users can select corresponding target neural networks according to resource constraints in actual applications.
[0042] In combination with the second aspect, in a possible implementation method, the updating of the sub-network set to obtain an updated sub-network set includes: processing the multiple first sub-network structures based on any one of a particle swarm algorithm, a genetic algorithm and a fireworks algorithm to obtain a next-generation sub-network structure; based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a first metric target to obtain a first grade division result; based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a second metric target to obtain a second grade division result; the first grade division result and the sub-network structures included in the same level in the second grade division result are respectively merged to obtain a merged grade division result; and multiple sub-network structures are selected from the merged grade division result as the multiple second sub-network structures.
[0043] In combination with the second aspect, in a possible implementation method, the first measurement target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy; the second measurement target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and growth rate of accuracy.
[0044] The embodiment of the present application performs two non-dominated sorting algorithm levels on the current-generation sub-network structure and the next-generation sub-network structure according to different metric objectives, and then selects multiple second sub-network structures based on the two level division results. In this way, it is possible to protect some models that temporarily have poor performance but may achieve better performance in the future, avoid these models being dominated in the early stages of the search and thus being abandoned prematurely, thereby expanding the coverage of the sub-network set.
[0045] In combination with the second aspect, in a possible implementation method, determining multiple target neural networks corresponding to the target task based on the updated sub-network set includes: selecting part or all of the second sub-network structures from the updated sub-network set as the multiple target neural networks.
[0046] In a third aspect, an image processing method is provided, which is applied to a terminal device, and the method includes: obtaining an image to be processed; selecting a first target neural network from multiple target neural networks based on the computing resources and / or storage resources of the terminal device; classifying the image to be processed based on the first target neural network to obtain a classification result of the image to be processed; wherein the multiple target neural networks are obtained based on an updated parameter space and an updated subnetwork set, and the updated parameter space is obtained by updating the parameter space included in the search network based on the training data of the image classification task, and the search network corresponds to the image classification task, and the parameter space includes multiple parameters, and the search network also includes a structure space, and the structure space includes multiple subnetwork structures, wherein each subnetwork structure corresponds to at least one parameter in the parameter space, and the updated subnetwork set is obtained by updating the subnetwork set determined from the search network, wherein the subnetwork set includes multiple first subnetwork structures among the multiple subnetwork structures, and the updated subnetwork set includes multiple second subnetwork structures among the multiple subnetwork structures.
[0047] Multiple target neural networks are based on maintaining a sub-network set, and multiple sub-network structures in the sub-network set share parameters in the parameter space, which are determined by updating the parameter space and the sub-network set. The training process of the sub-network structure is not to train from scratch after randomly initializing the parameters, but to inherit the parameters that have been trained before, that is, to update the parameters in the parameter space, which can effectively improve the search efficiency, and multiple target neural networks can be obtained in one search. The multiple target neural networks can have different properties, and a model that can meet different computing and / or storage resource constraints is provided, so that users can select the corresponding target neural network according to the resource constraints in actual applications.
[0048] In combination with the third aspect, in a possible implementation method, the updated parameter space is obtained by updating the parameter space based on a gradient back-propagation algorithm, wherein the gradient of the parameter space is the average value of the parameter gradients corresponding to some of the first sub-network structures among the multiple first sub-network structures.
[0049] In combination with the third aspect, in a possible implementation method, the updated sub-network set is obtained by processing the multiple first sub-network structures based on any one of the particle swarm algorithm, genetic algorithm and fireworks algorithm to obtain the next-generation sub-network structure, and then selecting multiple sub-network structures from the merged hierarchical division results as the multiple second sub-network structures, wherein the merged hierarchical division result is obtained by respectively merging the sub-network structures included in the same level of the first hierarchical division result and the second hierarchical division result, the first hierarchical division result is based on a non-dominated sorting algorithm, and the multiple first sub-network structures and the next-generation sub-network structures are graded according to a first metric target, and the second hierarchical division result is based on a non-dominated sorting algorithm, and the multiple first sub-network structures and the next-generation sub-network structures are graded according to a second metric target.
[0050] In combination with the third aspect, in a possible implementation method, the first measurement target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy; the second measurement target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and growth rate of accuracy.
[0051] In a fourth aspect, an image processing method is provided, the method comprising: acquiring an image to be processed; processing the image to be processed according to a target neural network to obtain a processing result of the image to be processed.
[0052] The above-mentioned processing of the image may refer to recognition, classification, detection, etc. of the image.
[0053] In a fifth aspect, an image processing method is provided, which includes: acquiring a road picture; performing convolution processing on the road picture according to a target neural network to obtain multiple convolution feature maps of the road picture; performing deconvolution processing on the multiple convolution feature maps of the road picture according to the target neural network to obtain a semantic segmentation result of the road picture.
[0054] Among them, the above-mentioned target neural network is a target neural network searched according to the first aspect or any possible implementation method of the first aspect.
[0055] In a sixth aspect, an image processing method is provided, the method comprising: obtaining a facial image; performing convolution processing on the facial image according to a target neural network to obtain a convolution feature map of the facial image; comparing the convolution feature map of the facial image with the convolution feature map of an identity document image to obtain a verification result of the facial image.
[0056] The convolution feature map of the ID card image may be obtained in advance and stored in a corresponding database. For example, the ID card image is convoluted in advance and the obtained convolution feature map is stored in the database.
[0057] In addition, the above-mentioned target neural network is a target neural network obtained by searching according to the first aspect or any possible implementation method of the first aspect.
[0058] It should be understood that the expansion, limitation, explanation and description of the relevant contents in the above-mentioned first aspect also apply to the same contents in the second aspect, the third aspect, the fourth aspect, the fifth aspect and the sixth aspect.
[0059] In the seventh aspect, a neural network structure search device is provided, comprising a module for executing the method in the above-mentioned first aspect or any possible implementation of the first aspect, or comprising a module for executing the method in the above-mentioned second aspect or any possible implementation of the second aspect.
[0060] In an eighth aspect, a neural network structure search device is provided, the device comprising: a memory for storing programs; a processor for executing the programs stored in the memory, when the program stored in the memory is executed, the processor is used to execute the method in the first aspect or any possible implementation of the first aspect, or execute the method in the second aspect or any possible implementation of the second aspect.
[0061] In a ninth aspect, an image processing device is provided, comprising: a memory for storing programs; and a processor for executing the programs stored in the memory, wherein when the programs stored in the memory are executed, the processor is used to execute the method in any possible implementation of the third to sixth aspects.
[0062] In a tenth aspect, a computer-readable medium is provided, which stores a program code for execution by a device, wherein the program code includes a method for executing any one of the implementations of the first to sixth aspects.
[0063] In an eleventh aspect, a computer program product comprising instructions is provided, and when the computer program product is run on a computer, the computer executes a method in any one of the implementation modes of the first to sixth aspects above.
[0064] In the twelfth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute a method in any one of the implementations of the first to sixth aspects above.
[0065] Optionally, as a possible implementation, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in any one of the implementations of the first to sixth aspects. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided by an embodiment of the present application;
[0067] Figure 2 A schematic diagram of a specific application provided by an embodiment of the present application;
[0068] Figure 3 A schematic diagram of a specific application provided by an embodiment of the present application;
[0069] Figure 4 A schematic diagram of a specific application provided by an embodiment of the present application;
[0070] Figure 5 A schematic diagram of the structure of the system architecture provided in the embodiment of the present application;
[0071] Figure 6 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;
[0072] Figure 7 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;
[0073] Figure 8 A schematic diagram of the hardware structure of a chip provided in an embodiment of the present application;
[0074] Fig. 9 A schematic diagram of a system architecture provided for an embodiment of the present application;
[0075] Fig.10 is a schematic flow chart of a method for searching a neural network structure according to an embodiment of the present application;
[0076] Fig.11 is a schematic diagram of a method for searching a neural network structure according to an embodiment of the present application;
[0077] Fig.12 is a schematic diagram of the effect of the search method for the neural network structure of an embodiment of the present application;
[0078] Fig.13 is a schematic diagram of the effect of the search method for the neural network structure of an embodiment of the present application;
[0079] Fig.14 is a schematic flow chart of an image processing method according to an embodiment of the present application;
[0080] Fig.15 is a schematic block diagram of a neural network structure search device according to an embodiment of the present application;
[0081] Fig.16 is a schematic block diagram of an image processing device according to an embodiment of the present application;
[0082] Fig.17 It is a schematic block diagram of a neural network training device according to an embodiment of the present application. DETAILED DESCRIPTION
[0083] The technical solution in this application will be described below in conjunction with the accompanying drawings.
[0084] Figure 1 A schematic diagram of an artificial intelligence main framework is shown, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field requirements.
[0085] The following is a detailed explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "information technology (IT) value chain" (vertical axis).
[0086] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom".
[0087] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of artificial intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.
[0088] (1) Infrastructure:
[0089] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and provides support through a basic platform.
[0090] The infrastructure can communicate with the outside world through sensors, and the computing power of the infrastructure can be provided by smart chips.
[0091] The smart chips here can be hardware acceleration chips such as central processing unit (CPU), neural-network processing unit (NPU), graphics processing unit (GPU), application specific integrated circuit (ASIC) and field programmable gate array (FPGA).
[0092] The basic platform of the infrastructure can include related platform guarantees and support such as distributed computing frameworks and networks, and can include cloud storage and computing, interconnected networks, etc.
[0093] For example, for infrastructure, data can be obtained through sensors and external communication, and then provided to the smart chips in the distributed computing system provided by the basic platform for calculation.
[0094] (2) Data:
[0095] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. This data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.
[0096] (3) Data processing:
[0097] The above data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other processing methods.
[0098] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
[0099] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
[0100] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.
[0101] (4) General abilities:
[0102] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0103] (5) Smart products and industry applications:
[0104] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.
[0105] The embodiments of the present application can be applied in many fields of artificial intelligence, such as smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities and other fields.
[0106] Specifically, the embodiments of the present application can be applied in fields such as image classification, image retrieval, image semantic segmentation, image super-resolution and natural language processing that require the use of (deep) neural networks. The embodiments of the present application can also be applied to cloud services.
[0107] In order to better understand the solution of the embodiment of the present application, Figures 2 to 4 The possible application scenarios of the embodiments of the present application are briefly introduced.
[0108] Cloud Services:
[0109] The embodiments of the present application can be deployed on cloud services to provide users with automatic search services for neural network structures. Users upload training data and specify the required indicators (such as accuracy, model size, etc.), and the cloud service can automatically search for a series of different models for users to choose from.
[0110] For example, Figure 2 As shown, the search method for the neural network structure of the embodiment of the present application can be deployed on a server, and the user can upload training images to the training image library in the server and specify the indicators of the required neural network structure. Using the search method for the neural network structure of the embodiment of the present application, multiple neural network structures suitable for image recognition can be obtained, and then the neural network can be trained according to the training images in the training image library to obtain multiple image recognition neural networks. The multiple image recognition neural networks can have different model sizes and accuracy rates. The user can select a matching image recognition neural network based on the computing resources and / or storage resources of the terminal and deploy it on the terminal. For example Figure 2The server can search for image recognition neural network structures of size 2.7M, 3.2M, 3.7M, etc. through a neural network structure search system (corresponding to the neural network structure search method of the embodiment of the present application). If the computing resources of the terminal device are less than 3.0M, the image recognition neural network structure of size 2.7M can be selected from the multiple image recognition neural network structures searched by the server. The image recognition neural network structure deployed on the terminal can recognize the image to be processed, such as Figure 2 As shown, the image recognition neural network processes the input image and obtains the recognition result that the image is a bird. The cloud service of the embodiment of the present application can search for a series of neural network structures with different model sizes and precisions. Users can select the appropriate neural network structure for deployment according to the application scenario and resource constraints in actual applications.
[0111] Album picture categories:
[0112] Specifically, when a user stores a large number of pictures on a terminal device (eg, a mobile phone) or a cloud disk, identifying images in the album can facilitate the user or the system to classify and manage the album, thereby improving the user experience.
[0113] The search method of the neural network structure of the embodiment of the present application can search for multiple neural network structures suitable for album classification, and then train the neural network according to the training pictures in the training picture library to obtain multiple album classification neural networks. Select an album classification neural network that matches the computing resources of the current terminal device from the multiple album classification neural networks, and then use the selected album classification neural network to classify the pictures, so as to label the pictures of different categories, which is convenient for users to view and find. In addition, the classification labels of these pictures can also be provided to the album management system for classification management, saving the user's management time, improving the efficiency of album management, and enhancing the user experience.
[0114] For example, Figure 3 As shown, multiple neural networks suitable for album classification can be searched and obtained through the search method of the neural network structure of the embodiment of the present application. After obtaining multiple neural networks suitable for album classification, the neural network can be trained according to the training pictures to obtain multiple corresponding album classification neural networks. Select an album classification neural network that matches the computing resources and / or storage resources of the terminal device from the multiple album classification neural networks, and then use the selected album classification neural network to classify the images to be processed. Figure 3As shown, the album classification neural network processes the input picture and obtains the category of the picture as tulip. Alternatively, the album classification neural network classifies the input pictures, for example, classifying pictures of birds together for centralized display, and classifying pictures of swimming together for centralized display.
[0115] Object detection and segmentation in autonomous driving scenarios:
[0116] In autonomous driving, detecting and segmenting pedestrians, vehicles and other objects on the street is crucial for the vehicle to make driving decisions. Figure 4 As shown, the neural network structure provided by the search method of the neural network structure of the embodiment of the present application can be embedded in the target detection and segmentation framework. The target detection and segmentation framework including the neural network structure can process the input road image, accurately detect, locate and segment targets on the street, and output the detection and segmentation results.
[0117] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.
[0118] (1) Neural Network
[0119] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit with the intercept 1 as input can be:
[0120]
[0121] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.
[0122] (2) Deep Neural Networks
[0123] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.
[0124] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Taking the coefficient W as an example, assuming that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.
[0125] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as
[0126] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).
[0127] (3) Convolutional Neural Network
[0128] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.
[0129] (4) Recurrent neural networks (RNN) are used to process sequence data. In traditional neural network models, the layers are fully connected from the input layer to the hidden layer and then to the output layer, while the nodes within each layer are disconnected. Although this common neural network has solved many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN.
[0130] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future. Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN is designed to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.
[0131] (5) Loss Function
[0132] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.
[0133] (6) Back propagation algorithm
[0134] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.
[0135] (7) Genetic algorithm (GA)
[0136] Genetic algorithms follow the principle of "survival of the fittest" in nature and are a type of randomized search algorithm that draws on the natural selection and natural genetic mechanisms of the biological world. The basic idea of genetic algorithms is to imitate the natural evolution process, generate new populations by performing genetic operations on individuals with a certain structural form in the population, and gradually approach the optimal solution. In the solution process, a fixed-size population is set. Each individual in the population represents a possible solution to the problem. The degree of individual adaptation to the environment is judged by the fitness function. Individuals with poor fitness are eliminated, and individuals with good fitness continue to reproduce. In the process of reproduction, selection, crossover, and mutation may be required to form new populations. This cycle repeats until more and better solutions are obtained. The main steps of genetic algorithms include: encoding, population initialization, calculation of individual fitness, evolutionary calculation, and decoding. In the encoding step, the candidate solutions to the problem are represented by chromosomes to realize the mapping process from the solution space to the encoding space. The genetic algorithm does not directly process the decision variables in the solution space, but converts them into chromosomes composed of genes in a certain structure. In the population initialization step, an initial population (encoding set) representing the possible potential solution set of the problem is generated. In the step of calculating individual fitness, the fitness function is used to calculate the fitness of each individual. In the step of evolutionary computation, a population representing a new solution set is generated through selection, crossover, and mutation. In the decoding step, the best individual in the last generation of the population is decoded to realize the mapping from the encoding space to the solution space, which can be used as an approximate optimal solution to the problem.
[0137] In addition, in the evolutionary computing step, selection is based on the size of individual fitness and the principle of survival of the fittest to eliminate unreasonable individuals; crossover is the crossover recombination of coding, similar to the crossover recombination of chromosomes; mutation is the change of coding caused by small probability perturbations, similar to gene mutation.
[0138] Fitness: It can be essentially understood as a cost function or a rule. By calculating the fitness of individuals in the initial population, we can get a measure of whether the individuals in the initial population are good or bad.
[0139] (8) Evolutionary algorithms (EAS)
[0140] Evolutionary algorithm is not a specific algorithm, but a "cluster of algorithms". The inspiration for the generation of evolutionary algorithms draws on the evolutionary operations of organisms in nature. It generally includes basic operations such as gene encoding, population initialization, crossover mutation operators, and management retention mechanisms. The basic framework of evolutionary algorithms is still the framework described by simple genetic algorithms, but there are great differences in the way of evolution, with many changes in selection, crossover, mutation, population control, etc. Compared with traditional optimization algorithms such as calculus-based methods and exhaustive methods, evolutionary computation is a mature global optimization method with high robustness and wide applicability. It has the characteristics of self-organization, self-adaptation, and self-learning. It can effectively handle complex problems that are difficult to solve with traditional optimization algorithms without being restricted by the nature of the problem.
[0141] (9) Particle swarm optimization (PSO)
[0142] Particle swarm optimization is also known as particle swarm optimization algorithm, particle swarm optimization algorithm, particle swarm optimization algorithm, or bird foraging algorithm.
[0143] Particle swarm optimization algorithm is an evolutionary computation technology, which originated from the study of the behavior of bird flocks. The basic idea of particle swarm optimization algorithm is to find the optimal solution through cooperation and information sharing between individuals in the group. In layman's terms, imagine a scene like this: a group of birds are randomly searching for food. There is only one piece of food in this area. All birds don't know where the food is. But they know how far away the current position is from the food. So what is the optimal strategy to find food? The simplest and most effective way is to search the area around the bird that is currently closest to the food. During the entire search process, the flock of birds transmits their own information to each other to let other birds know their position. Through such cooperation, they can judge whether they have found the optimal solution, and also transmit the information of the optimal solution to the entire flock of birds. Finally, the entire flock of birds can gather around the food source, that is, the optimal solution is found. In PSO, the solution of each optimization problem is a bird in the search space, called a "particle". All particles have a fitness value determined by the function to be optimized, and each particle also has a speed that determines the direction and distance of their flight, and then the particles follow the current optimal particle to search in the solution space. PSO is initialized as a group of random particles (random solutions), and then finds the optimal solution through iteration. In each iteration, the particle updates itself by tracking two "extreme values". The first is the optimal solution found by the particle itself, which is called the individual extreme value pBest. The other extreme value is the optimal solution currently found by the entire population, which is the global extreme value gBest. In addition, you can also use only a part of the population as the particle's neighbors instead of the entire population. In this case, the extreme value among all neighbors is the local extreme value. PSO searches for the global optimum by following the currently searched optimal value. This algorithm has the advantages of easy implementation, high accuracy, and fast convergence, and has demonstrated its superiority in solving practical problems.
[0144] (10) Fireworks algorithm (FWA)
[0145] The fireworks algorithm is a swarm intelligence algorithm inspired by the explosion of fireworks in the night sky. The implementation of the fireworks algorithm includes starting iterations, using explosion operators, mutation operators, mapping rules and selection strategies in turn, until the termination condition is reached, that is, the accuracy requirements of the problem are met or the maximum number of function evaluations is reached. The implementation of the fireworks algorithm can include the following steps: 1) Randomly generate some fireworks in a specific solution space, and each firework represents a solution in the solution space. 2) Calculate the fitness value of each firework according to the fitness function, and generate sparks according to the fitness value. The number of sparks is calculated based on the idea of immune concentration in immunology, that is, the better the fitness value, the more sparks a firework generates. 3) According to the properties of fireworks in reality and combined with the actual situation of the search problem, sparks are generated in the radiation space of the fireworks. (The size of the explosion amplitude of a certain firework is determined by the fitness value of the firework on the function. The larger the fitness value, the larger the explosion amplitude, and vice versa). Each spark represents a solution in the solution space. In order to ensure the diversity of the population, it is necessary to mutate the fireworks appropriately, such as Gaussian mutation. 4) Calculate the optimal solution of the population and determine whether it meets the requirements. If it does, stop searching. If it does not, continue iterating. The initial value of the iteration is the best solution obtained in this cycle and the other solutions selected.
[0146] (11) Non-dominated sorting genetic algorithms (NSGA)
[0147] The main difference between NSGA and simple genetic algorithms is that the algorithm stratifies according to the dominance relationship between individuals before the selection operator is executed. Its selection operator, crossover operator and mutation operator are no different from simple genetic algorithms. Before the selection operation is executed, the population is sorted according to the dominance and non-domination relationship between individuals: first, all non-dominated individuals in the population are found and assigned a shared virtual fitness value. The first non-dominated optimal layer is obtained; then, this group of stratified individuals is ignored, and the other individuals in the population continue to be stratified according to the dominance and non-domination relationship, and a new virtual fitness value is assigned to them, which is smaller than the value of the previous layer, and the above operation is continued for the remaining individuals until all individuals in the population are stratified.
[0148] For ease of understanding and explanation, the target neural network model is described as a CNN model (or CNN network) in the embodiments of the present application. However, it should be understood that the type of the target neural network model in the embodiments of the present application is not limited to this, and can also be any one of the models introduced above and not shown.
[0149] refer to Figure 5 , the embodiment of the present application provides a system architecture 100. Figure 5In the embodiment, the data acquisition device 160 is used to collect training data. For the image processing method of the embodiment of the present application, the training data may include training images and classification results corresponding to the training images, wherein the results of the training images may be manually pre-labeled results.
[0150] After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130 .
[0151] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input original image and compares the output image with the original image until the difference between the image output by the training device 120 and the original image is less than a certain threshold, thereby completing the training of the target model / rule 101.
[0152] The above-mentioned target model / rule 101 can be used to implement the image processing method of the embodiment of the present application. The target model / rule 101 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130, and it is also possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.
[0153] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 5 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 5 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include an image to be processed input by the client device in the embodiment of the present application.
[0154] The preprocessing module 113 and / or the preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I / O interface 112. In the embodiment of the present application, the preprocessing module 113 and the preprocessing module 114 may be omitted (or only one of the preprocessing modules may be present), and the computing module 111 may be used directly to process the input data.
[0155] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.
[0156] Finally, the I / O interface 112 returns the processing result, such as the image classification result obtained above, to the client device 140 for providing to the user.
[0157] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.
[0158] exist Figure 5 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.
[0159] It is worth noting that Figure 5 This is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 5In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.
[0160] like Figure 5 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 in the embodiment of the present application can be the neural network in the present application. Specifically, the neural network provided in the embodiment of the present application can be CNN, deep convolutional neural network (DCNN), recurrent neural network (RNN), etc.
[0161] Since CNN is a very common neural network, Figure 6 The focus is on a detailed introduction to the structure of CNN. As mentioned in the basic concept introduction above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.
[0162] The structure of the neural network specifically used in the image processing method of the embodiment of the present application can be as follows: Figure 6 As shown. Figure 6 In the example, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230. The input layer 210 may obtain an image to be processed, and the obtained image to be processed is passed to the convolutional layer / pooling layer 220 and the subsequent neural network layer 230 for processing, and the image processing result may be obtained. Figure 6 The internal layer structure of CNN 200 is introduced in detail.
[0163] Convolutional layer / pooling layer 220:
[0164] Convolutional Layer:
[0165] like Figure 6The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.
[0166] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.
[0167] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter for extracting specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on an image, the weight matrix is usually processed one pixel after another (or two pixels after two pixels... depending on the value of the stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" described above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is used to extract image edge information, another weight matrix is used to extract specific colors of the image, and another weight matrix is used to blur unnecessary noise points in the image, etc. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple convolution feature maps of the same size are then merged to form the output of the convolution operation.
[0168] The weight values in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values obtained through training can extract information from the input image, so that the convolutional neural network 200 can make correct predictions.
[0169] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.
[0170] Pooling layer:
[0171] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 6 Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.
[0172] Neural Network Layer 230:
[0173] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 6 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.
[0174] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 6 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 6 The propagation from 240 to 210 is called back propagation) and then the weight values and biases of the aforementioned layers will begin to be updated to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.
[0175] The structure of the neural network specifically used in the image processing method of the embodiment of the present application can be as follows: Figure 7 As shown. Figure 7 In FIG. 1 , a convolutional neural network (CNN) 300 may include an input layer 310, a convolutional layer / pooling layer 320 (where the pooling layer is optional), and a neural network layer 330. Figure 6 compared to, Figure 7 The multiple convolutional layers / pooling layers in the convolutional layer / pooling layer 320 are operated in parallel, and the features extracted respectively are input to the full neural network layer 330 for processing.
[0176] It should be noted that Figure 6 and Figure 7 The convolutional neural network shown is only an example of two possible convolutional neural networks for the image processing method of an embodiment of the present application. In specific applications, the convolutional neural network adopted by the image processing method of the embodiment of the present application may also exist in the form of other network models.
[0177] In addition, the structure of the convolutional neural network obtained by the search method for the neural network structure of the embodiment of the present application can be as follows: Figure 6 and Figure 7 The convolutional neural network structure is shown in .
[0178] Figure 8 A schematic diagram of the hardware structure of a chip provided in an embodiment of the present application. The chip includes a neural processing unit (NPU) 40. The chip can be arranged in Figure 5 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 5 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 6 and Figure 7 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 8is implemented in the chip shown.
[0179] The following is a brief introduction to each module and unit in the NPU 40.
[0180] The neural network processor NPU 40 can be mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU 40 is the operation circuit 403. When the NPU 40 is working, the controller 404 in the NPU 40 can control the operation circuit 403 to extract data from the memory (weight memory or input memory) and perform operations.
[0181] In some implementations, the operation circuit 403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 403 is a two-dimensional systolic array. The operation circuit 403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 403 is a general-purpose matrix processor.
[0182] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 401 and performs matrix operation with matrix B, and the partial result or final result of the matrix is stored in the accumulator 408.
[0183] The vector calculation unit 407 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 407 can be used for network calculations of non-convolutional / non-fully connected layers (FC) layers in a neural network, such as pooling, batch normalization, local response normalization, etc.
[0184] In some implementations, the vector calculation unit 407 can store the processed output vector to the unified buffer 406. For example, the vector calculation unit 407 can apply a nonlinear function to the output of the operation circuit 403, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 407 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 403, for example, for use in a subsequent layer in a neural network.
[0185] The unified memory 406 is used to store input data and output data.
[0186] The weight data is directly transferred from the external memory to the input memory 401 and / or the unified memory 406 through the direct memory access controller 405 (DMAC), the weight data in the external memory is stored in the weight memory 402, and the data in the unified memory 406 is stored in the external memory.
[0187] The bus interface unit (BIU) 410 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 409 through the bus.
[0188] An instruction fetch buffer 409 connected to the controller 404 and used to store instructions used by the controller 404;
[0189] The controller 404 is used to call the instructions cached in the memory 409 to control the working process of the computing accelerator.
[0190] Generally, unified memory 406, input memory 401, weight memory 402 and instruction fetch memory 409 can all be on-chip memories. The external memory of NPU can be a memory outside the NPU, and the external memory can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.
[0191] in, Figure 6 or Figure 7 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 403 or the vector calculation unit 407.
[0192] Introduced above Figure 5 The execution device 110 in the embodiment of the present application can execute each step of the image processing method. Figure 6 and Figure 7 The CNN model shown and Figure 8 The chip shown can also be used to execute each step of the image processing method of the embodiment of the present application. The following is a detailed introduction to the search method of the neural network structure of the embodiment of the present application and the image processing method of the embodiment of the present application in conjunction with the accompanying drawings.
[0193] like Fig. 9 As shown, the embodiment of the present application provides a system architecture 500. The system architecture includes a local device 501, a local device 502, an execution device 510 and a data storage system 550, wherein the local device 501 and the local device 502 are connected to the execution device 510 via a communication network.
[0194] The execution device 510 can be implemented by one or more servers. Optionally, the execution device 510 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, and other devices. The execution device 510 can be arranged at one physical site, or distributed at multiple physical sites. The execution device 510 can use the data in the data storage system 550, or call the program code in the data storage system 550 to implement the search method of the neural network structure of the embodiment of the present application.
[0195] Specifically, the execution device 510 may perform the following process:
[0196] A search network is determined according to a target task, wherein the search network includes a structure space and a parameter space, wherein the structure space includes multiple sub-network structures, and the parameter space includes multiple parameters, wherein each sub-network structure corresponds to at least one parameter in the parameter space; the parameter space is updated according to training data of the target task to obtain an updated parameter space; a sub-network set is determined from the search network, wherein the sub-network set includes multiple first sub-network structures among the multiple sub-network structures; the sub-network set is updated to obtain an updated sub-network set, wherein the updated sub-network set includes multiple second sub-network structures among the multiple sub-network structures; and multiple target neural networks corresponding to the target task are determined according to the updated parameter space and the updated sub-network set.
[0197] Through the above process execution device 510 can search out multiple target neural networks, and the multiple target neural networks can be used for image classification or image processing, etc.
[0198] Users can operate their respective user devices (e.g., local device 501 and local device 502) to interact with execution device 510. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.
[0199] The local device of each user can interact with the execution device 510 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.
[0200] In one implementation, the local device 501 and the local device 502 obtain relevant parameters of multiple target neural networks from the execution device 510, select a target neural network that matches the resource constraints of the local device from the multiple target neural networks, deploy the selected target neural network on the local device 501 and the local device 502, and use the selected target neural network to perform image classification or image processing, etc.
[0201] In another implementation, the execution device 510 can select a target neural network that matches the resource constraints of the execution device from multiple target neural networks and directly deploy the target neural network. The execution device 510 obtains the image to be processed from the local device 501 and the local device 502, and classifies the image to be processed or performs other types of image processing according to the selected and deployed target neural network.
[0202] The execution device 510 may also be referred to as a cloud device, and in this case the execution device 510 is generally deployed in the cloud.
[0203] Let's combine Fig.10 The search method for the neural network structure of the embodiment of the present application is introduced in detail. Fig.10 The method shown can be performed by a neural network structure search device, which can be a computer, server, cloud device or other device with computing power sufficient to realize the search of neural network structure.
[0204] Fig.10 The method shown includes steps 1001 to 1005, each of which is described in detail below.
[0205] 1001. Determine a search network according to a target task, wherein the search network includes a structure space and a parameter space, wherein the structure space includes a plurality of sub-network structures, and the parameter space includes a plurality of parameters, wherein each sub-network structure corresponds to at least one parameter in the parameter space.
[0206] There are several ways to determine the search network based on the target task.
[0207] As a possible implementation, the above search network is determined according to the application requirements of the target neural network corresponding to the target task.
[0208] That is to say, the search network can be determined according to the application requirements of the target neural network. Specifically, the search network can be determined according to the data type of data that the target neural network needs to process.
[0209] The search network includes a variety of connection methods and different operations. Generally speaking, the types and number of operations included in the search network should match the application requirements of the target neural network. When the above-mentioned target neural network is used to process image data, the types and number of operations included in the above-mentioned search network should be adapted to the processing of image data. When the above-mentioned target neural network is used to process voice data, the types and number of operations included in the above-mentioned search network should be adapted to the processing of voice data.
[0210] For example, when the target neural network is a neural network for processing image data, the search network may include a long short-term memory unit or a long short-term memory network (LSTM), etc. Specifically, the search network may include a convolution operation, a pooling operation, a skip-connect operation, an activation function, etc.
[0211] For another example, when the target neural network is a neural network for processing speech data, the search network may include a long short term memory unit or a long short term memory network (LSTM), etc. Specifically, the search network may include an activation function (such as ReLU, Tanh), etc.
[0212] As another possible implementation, the above search space is determined based on the application requirements of the target neural network and the video memory resource conditions of the device that performs the neural network structure search.
[0213] Among them, the graphics memory resource condition of the device that performs the neural network structure search may refer to the graphics memory resource size of the device that performs the neural network structure search.
[0214] That is to say, in the present application, the above-mentioned search space can be comprehensively determined based on the application requirements of the target neural network and the video memory resource conditions of the device that performs the neural network structure search.
[0215] Specifically, the types and number of operations included in the search space can be determined based on the application requirements of the target neural network, and then the types and number of operations included in the search space can be adjusted based on the graphics memory resource conditions of the device that performs the neural network structure search to determine the types and number of operations ultimately included in the search space.
[0216] For example, after determining the types and number of operations included in the search space according to the application requirements of the target neural network, if the video memory resources of the device performing the neural network structure search are small, some less important operations in the search space can be deleted. If the video memory resources of the device performing the neural network structure search are sufficient, the types and number of operations included in the search space can be maintained, or the types and number of operations included in the search space can be increased.
[0217] As another possible implementation, the above search space is determined according to the application requirements of the target neural network and the size of the model storage space.
[0218] That is to say, in the present application, the above-mentioned search space can be comprehensively determined according to the application requirements of the target neural network and the size of the model storage space. Specifically, the types and number of operations contained in the search space can be determined according to the application requirements of the target neural network, and then the types and number of operations contained in the search space can be adjusted according to the size of the model storage space to determine the types and number of operations ultimately contained in the search space. For example, after determining the types and number of operations contained in the search space according to the application requirements of the target neural network, if the model storage space is small, some less important operations in the search space can be deleted. If the model storage space is large, the types and number of operations contained in the search space can be maintained, or the types and number of operations contained in the search space can be increased.
[0219] The search network in the embodiment of the present application may include basic modules for building a neural network, such as convolution, pre-set basic operations or combinations of basic operations that are compatible with the application requirements of the target neural network, wherein these basic operations or combinations of basic operations may be collectively referred to as basic operations. The size, depth, width, connection mode, etc. of the search network may be manually designed.
[0220] The search network includes a structure space and a parameter space, the structure space includes multiple sub-network structures, and the parameter space includes multiple parameters, wherein each sub-network structure corresponds to at least one parameter in the parameter space.
[0221] The structural space in the embodiment of the present application includes all network architectures that can be characterized in principle, and can also be understood as the structural space being a complete set of network structures, where each possible network structure can be called a sub-network structure. The sub-network structure includes all or part of the basic modules and all or part of the basic operations in the search network. The types and quantities of the basic modules and basic operations included in different sub-network structures are different, but the parameters of the basic modules and basic operations included in different sub-network structures are shared. That is, the parameter space in the embodiment of the present application includes the parameters of the basic modules and basic operations in the search network, such as the parameters involved in the separation convolution with a convolution kernel size of 3×3, the parameters involved in the dilated convolution with a convolution kernel size of 5×5 and a dilated rate of 2, and the parameters involved in the jump connection of the basic operation. Different sub-network structures can contain different types and quantities of basic modules and basic operations, and different sub-network structures can have different connection methods, but different sub-network structures share the parameters in the parameter space. For example, if different sub-network structures contain the same basic operation, then different sub-network structures share the parameters corresponding to the same basic operation included in the parameter space. Each basic module can correspond to one or more parameters, each basic operation can correspond to one or more parameters, each sub-network structure can include one or more basic modules, and can include one or more basic operations. Therefore, each sub-network structure in the structural space in the embodiment of the present application corresponds to at least one parameter in the parameter space, for example, each sub-network structure corresponds to one or more parameters in the parameter space.
[0222] In order to better understand the structural space and sub-network structure of the embodiments of the present application, for example, assuming that the number of layers of the search network is 22, and each layer includes 5 basic operations (the 5 basic operations are expressed by numbers 1-5 respectively), then a 22-dimensional vector can be used to express a sub-network structure, for example (1,2,4,5,3,5,1...3,2), different vectors express different sub-network structures, and the sub-network structures expressed by all 22-dimensional vectors can constitute the structural space of the embodiments of the present application.
[0223] Optionally, when determining the search network corresponding to the target task, the parameter values in the parameter space may be randomly set, or default initialized values, or manually set values, which is not specifically limited in the embodiments of the present application.
[0224] 1002. Update the parameter space according to the training data of the target task to obtain an updated parameter space.
[0225] It should be understood that in the embodiments of the present application, updating the parameter space may be understood as updating the parameters in the parameter space, or as updating the values of the parameters in the parameter space.
[0226] Optionally, updating the parameter space according to the training data of the target task may be updating part of the parameters in the parameter space, or may be updating all the parameters in the parameter space.
[0227] In the embodiment of the present application, the training data of the target task can be a part of the data that the target neural network needs to process, or all of the data that the target neural network needs to process, or other data of the same data type as the data that the target neural network needs to process, and the implementation of the present application does not make specific limitations. For example, when the above-mentioned target neural network is used to process image data, the training data of the above-mentioned target task can be image data such as pictures; when the above-mentioned target neural network is used to process voice data, the training data of the above-mentioned target task can be voice data such as voice fragments; when the above-mentioned target neural network is used to process text data, the training data of the above-mentioned target task can be text data such as text sequences, etc.
[0228] Optionally, for a sub-network structure, the training data of the target task can be used as the input of the sub-network structure, the output of the sub-network structure is compared with the correct result, and the parameter value corresponding to the sub-network structure in the parameter space is adjusted according to the difference between the output of the sub-network structure and the correct result, so that the difference between the output of the sub-network structure and the correct result meets a preset condition, for example, the difference between the output of the sub-network structure and the correct result is less than a certain threshold, or the difference between the output of the sub-network structure and the correct result remains unchanged or no longer decreases.
[0229] 1003. Determine a sub-network set from the search network, where the sub-network set includes multiple first sub-network structures among the multiple sub-network structures.
[0230] In an embodiment of the present application, a subnetwork set includes multiple subnetworks, wherein the subnetwork includes a subnetwork structure and parameters corresponding to the subnetwork structure, the subnetwork structure belongs to the structural space, and the parameters corresponding to the subnetwork structure belong to the parameter space. In other words, a subnetwork can be expressed by a subnetwork structure and parameters corresponding to the subnetwork structure. The parameters corresponding to the subnetwork structure can be understood as parameters corresponding to the basic modules and basic operations included in the subnetwork structure. In an embodiment of the present application, the subnetwork set includes multiple first subnetwork structures among the multiple subnetwork structures included in the structural space. It should be understood that the subnetwork set also includes parameters corresponding to each first subnetwork structure in the multiple first subnetwork structures.
[0231] It should be understood that the sub-network set in this step can be understood as the sub-network set before updating.
[0232] 1004. Update the sub-network set to obtain an updated sub-network set, where the updated sub-network set includes a plurality of second sub-network structures among the plurality of sub-network structures.
[0233] It should be understood that in the embodiments of the present application, updating the subnetwork set can be understood as replacing the subnetwork structure included in the subnetwork set, or can be understood as re-determining or reselecting the subnetwork structure included in the subnetwork set.
[0234] In an embodiment of the present application, the updated subnetwork set includes multiple second subnetwork structures among the multiple subnetwork structures included in the structure space, wherein the first subnetwork structure and the second subnetwork structure may be the same or different. In other words, the updated subnetwork set may include part or all of the multiple first subnetwork structures in the subnetwork set before the update, or the multiple second subnetwork structures included in the updated subnetwork set are completely different from the multiple first subnetwork structures included in the subnetwork set before the update.
[0235] It should be noted that step 1002 can be understood as a process of updating the parameter space, and steps 1003 and 1004 can be understood as a process of updating the sub-network set (or understood as a process of updating the sub-network structure). In the embodiment of the present application, the process of updating the parameter space is relatively independent from the process of updating the sub-network set. Fig.10 The sequence numbers of the steps do not limit the execution order of the steps. The execution order of steps 1002, 1003, and 1004 can be determined according to actual applications. In some embodiments, steps 1002, 1003, and 1004 can be executed multiple times. For example, the process of updating the parameter space (step 1002) and the process of updating the sub-network set (steps 1003 and 1004) are executed alternately until the preset conditions are met, and then step 1005 is executed.
[0236] In step 1003, determining a sub-network set from the search network may include an initialization process of the sub-network set and an update process of the sub-network set.
[0237] When the subnetwork set is determined from the search network in this step as the initialization process of the subnetwork set, the subnetworks included in the subnetwork set (the subnetworks include the subnetwork structure and the parameters corresponding to the subnetwork structure) can be sampled from the search network. Specifically, the multiple first subnetwork structures included in the subnetwork set can be sampled from the structure space, wherein the parameters corresponding to each first subnetwork structure in the multiple first subnetwork structures can be obtained from the parameter space. Optionally, the parameter space may not have been updated yet, and the parameters in the parameter space may be initialized values or randomly set values, that is, the parameters corresponding to each first subnetwork structure in the multiple first subnetwork structures are initialized values or randomly set values. It should be noted that the method of sampling the subnetwork from the search network can be random sampling according to the existing method, and the embodiments of the present application are not limited thereto.
[0238] When the subnetwork set is determined from the search network in this step as the update process of the subnetwork set, the updated subnetwork set obtained in the previous round of the subnetwork set update process can be determined as the subnetwork set in the current round of the subnetwork set update process (i.e., the subnetwork set before the update of the current round of the update process). Alternatively, some subnetwork structures can be selected from the multiple subnetwork structures included in the updated subnetwork set obtained in the previous round of the subnetwork set update process to form the subnetwork set in the current round of the subnetwork set update process (i.e., the subnetwork set before the update of the current round of the update process), that is, the subnetwork set in the current round of the subnetwork set update process includes some subnetwork structures in the updated subnetwork set obtained in the previous round of the subnetwork set update process.
[0239] For ease of understanding, the description of the following process is based on the premise that the search network has been determined and the sub-network set has been initialized. The update process of the parameter space and the update process of the sub-network set are performed alternately.
[0240] Update of parameter space
[0241] In this round of parameter space updating process, step 1002, the parameter space is updated according to the training data of the target task to obtain an updated parameter space, including: according to the training data of the target task and the sub-network set (i.e., the sub-network set determined in this round of sub-network set updating process in step 1003), the parameter space is updated to obtain an updated parameter space.
[0242] Optionally, the parameter space can be updated based on the gradient back propagation algorithm. Specifically, the process of updating the parameter space based on the gradient back propagation algorithm consists of a forward propagation process and a back propagation process. In the forward propagation process, the training data of the target task is input into each sub-network structure included in the sub-network set, and the training data of the target task passes through the input layer and the hidden layer, and is processed layer by layer and transmitted to the output layer. If the expected output value is not obtained in the output layer, an objective function is taken (the selection of the objective function corresponds to the target task. For example, when the target task is a classification task, the cross entropy of the output and the expected error can be taken as the objective function. In the regression problem, the square error and loss function of the output and the expected error can be taken as the objective function), and then back propagation is started. The partial derivatives of the objective function with respect to each neuron weight (each neuron weight belongs to the parameter in the parameter space) are obtained layer by layer to form the gradient of the objective function with respect to the weight vector, that is, the parameter gradient corresponding to each sub-network structure. The gradient of the parameter space can be obtained according to the parameter gradient corresponding to each sub-network structure, which is used as the basis for modifying the weight (that is, as the basis for updating the parameter space). The learning of the network is completed in the process of modifying the weight. When the error reaches the expected value or reaches a certain number of iterations, the network learning ends, the weight modification process ends, and the updating of the parameter space ends.
[0243] In the embodiment of the present application, there are multiple ways to determine the gradient of the parameter space of the search network.
[0244] As a possible implementation, the gradient of the parameter space may be the average value of the parameter gradients corresponding to the multiple first sub-network structures included in the sub-network set. In other words, the parameter gradients corresponding to all the first sub-network structures in the multiple first sub-networks included in the sub-network set are summed and then averaged, which can be used as the gradient of the parameter space.
[0245] It should be understood that in the above possible implementations, when the parameter space is updated, the parameters corresponding to the multiple first sub-network structures included in the sub-network set in the parameter space are updated.
[0246] As another possible implementation, the gradient of the parameter space may be the average value of the parameter gradients corresponding to some of the first sub-network structures in the multiple first sub-network structures included in the sub-network set. That is, some of the first sub-network structures are selected from the multiple first sub-networks included in the sub-network set, and the parameter gradients corresponding to the some of the first sub-network structures are summed and then averaged, which can be used as the gradient of the parameter space.
[0247] It should be understood that in the above possible implementations, when the parameter space is updated, the parameters corresponding to some of the first sub-network structures in the multiple first sub-network structures included in the sub-network set in the parameter space are updated.
[0248] It should be noted that in the embodiment of the present application, the parameter gradient corresponding to the sub-network structure has the same meaning as the gradient expression of the sub-network, where the sub-network includes the sub-network structure and the parameters corresponding to the sub-network structure.
[0249] In an embodiment of the present application, directly summing up the parameter gradients corresponding to all the first sub-network structures in the sub-network set may bring a large computational burden. The parameter gradients corresponding to some first sub-network structures in a small batch (mini-batch) can be summed up, which can reduce the computational burden and speed up the update of the parameter space, thereby speeding up the search speed of the neural network structure and improving the efficiency of the neural network structure search.
[0250] Update of subnetwork collection
[0251] In the embodiment of the present application, updating the subnetwork set can be understood as replacing the subnetwork structure included in the subnetwork set, or can be understood as re-determining or selecting the subnetwork structure included in the subnetwork set. In the embodiment of the present application, the subnetwork set is maintained, so updating the subnetwork set can also be understood as optimizing the subnetwork structure in the subnetwork set.
[0252] In the process of updating the sub-network set in this round, after the sub-network set is determined in step 1003, the parameter space of this round is updated as described above according to the sub-network set. Therefore, when updating the sub-network set in this round, the parameter space has been updated, and the updated parameter space is obtained.
[0253] Optionally, the multiple first sub-network structures included in the sub-network set can be processed based on any one of the particle swarm algorithm, genetic algorithm and fireworks algorithm to obtain the next generation sub-network structure, wherein the multiple first sub-network structures included in the sub-network set can be called the parent sub-network structure or the current generation sub-network structure, and then multiple sub-network structures can be selected from the current generation sub-network structure and the next generation sub-network structure to form the updated sub-network set, that is, the multiple second sub-network structures included in the updated sub-network set are selected from the current generation sub-network structure and the next generation sub-network structure. It should be understood that the current generation sub-network structure and the next generation sub-network structure both belong to the structural space determined in step 1001, that is, the multiple second sub-network structures included in the updated sub-network set belong to the structural space determined in step 1001. It should be understood that the next generation sub-network structure includes at least one sub-network structure.
[0254] It should be understood that the embodiments of the present application can also obtain the next generation sub-network structure based on other evolutionary algorithms, and the embodiments of the present application will not be listed one by one.
[0255] Specifically, taking any evolutionary algorithm as an example, crossover and mutation operations can be performed on multiple first sub-network structures included in the sub-network set to generate a next-generation sub-network structure, and the multiple first sub-network structures are the current-generation sub-network structures; the fitness of the current-generation sub-network structure and the next-generation sub-network structure is evaluated according to the verification data of the target task, such as the number of parameters, recognition rate, etc.; based on the evaluation results, multiple sub-network structures are selected from the current-generation sub-network structure and the next-generation sub-network structure to form an updated sub-network set, and the multiple second sub-network structures included in the updated sub-network set are selected from the current-generation sub-network structure and the next-generation sub-network structure.
[0256] There are many methods for selecting the plurality of second sub-network structures.
[0257] As a possible implementation method, a sub-network structure that performs better on the training data of the target task can be selected from the current generation sub-network structure and the next generation sub-network structure according to any one of the measurement targets such as parameter quantity, network running time, network required running memory, network computing quantity, energy consumption, number of floating point operations, accuracy, etc. For example, a sub-network structure with high accuracy can be selected from the current generation sub-network structure and the next generation sub-network structure; or a sub-network structure with less parameters can be selected from the current generation sub-network structure and the next generation sub-network structure.
[0258] As another possible implementation, multiple second sub-network structures can be selected from the current generation sub-network structure and the next generation sub-network structure based on a non-dominated sorting algorithm such as the non-dominated sorting algorithm NSGA-Ⅲ. Specifically, based on the non-dominated sorting algorithm NSGA-Ⅲ, the current generation sub-network structure and the next generation sub-network structure can be graded (or layered) according to the dominance relationship between individuals according to multiple measurement targets. Specifically, all non-dominated individuals in the population (i.e., the current generation sub-network structure and the next generation sub-network structure in the embodiment of the present application) are found, and a shared virtual fitness value is assigned to them to obtain the first level (or the first non-dominated optimal layer), and then, this group of layered individuals is ignored, and other individuals in the population continue to be layered according to the dominance and non-domination relationship, and a new virtual fitness value is assigned to them, which is smaller than the value of the previous layer, and the above operation is continued for the remaining individuals until all individuals in the population are layered. The sub-network structures in the front levels are not dominated by the back levels. For example, the sub-network structures in the first level are not dominated by other levels, and the sub-network structures in the second level are not dominated by the third level and below. In other words, there will be no sub-network structures in the back levels that perform better than the sub-network structures in the front levels. For example, considering the accuracy and the number of parameters, there will be no sub-network structures in the back levels that have the same accuracy as a sub-network structure in the front level but with fewer parameters. Therefore, multiple second sub-network structures can be selected from the front levels.
[0259] Since the parameters in the parameter space of the search network are not fully trained, and the performance of the sub-network structure during the search process is not completely consistent with the performance of the final network structure, the small model with higher accuracy may dominate the large model with temporarily lower accuracy. Even though these large models may eventually achieve higher performance, they may be abandoned prematurely during the search process. Therefore, in order to avoid the large model being eliminated prematurely, the large model can be protected during the search process.
[0260] As another possible implementation, multiple second sub-network structures can be selected from the current generation sub-network structure and the next generation sub-network structure based on a non-dominated sorting algorithm, such as the improved non-dominated sorting algorithm of the embodiment of the present application (for the convenience of description, the embodiment of the present application is represented as pNSGA-Ⅲ) .
[0261] Specifically, based on a non-dominated sorting algorithm, multiple first sub-network structures and next-generation sub-network structures are graded according to a first metric target to obtain a first grade division result; based on a non-dominated sorting algorithm, multiple first sub-network structures and next-generation sub-network structures are graded according to a second metric target to obtain a second grade division result; the sub-network structures included in the same grade in the first grade division result and the second grade division result are respectively merged to obtain a merged grade division result; and multiple sub-network structures are selected from the merged grade division result as multiple second sub-network structures.
[0262] In other words, based on the non-dominated sorting algorithm, the current generation sub-network structure and the next generation sub-network structure can be graded according to the dominance relationship between individuals according to the first metric target to obtain the first grade division result; based on the non-dominated sorting algorithm, the current generation sub-network structure and the next generation sub-network structure can be graded according to the dominance relationship between individuals according to the second metric target to obtain the second grade division result; the sub-network structure included in the first grade in the first grade division result and the sub-network structure included in the first grade in the second grade division result are merged to obtain the merged first grade, similarly, the sub-network structure included in the second grade in the first grade division result and the sub-network structure included in the second grade in the second grade division result are merged to obtain the merged second grade, and other identical grades are similarly merged to finally obtain a merged grade division result; multiple sub-network structures are selected from the merged grade division result as multiple second sub-network structures, for example, multiple second sub-network structures are selected from the front grade in the merged grade division result.
[0263] Optionally, the first level division results and the second level division results mentioned above may not be merged, and the sub-network structures in the front levels may be directly selected from the first level division results and the second level division results respectively, and the sub-network structures selected from the first level division results and the sub-network structures selected from the second level division results may be used as the multiple second sub-network structures included in the updated sub-network set.
[0264] Specifically, based on a non-dominated sorting algorithm, multiple first sub-network structures and next-generation sub-network structures are graded according to a first metric target to obtain a first grade division result; based on a non-dominated sorting algorithm, multiple first sub-network structures and next-generation sub-network structures are graded according to a second metric target to obtain a second grade division result; and multiple sub-network structures are selected from the top levels of the first grade division result and the second grade division result as multiple second sub-network structures.
[0265] When selecting multiple second sub-network structures from the current-generation sub-network structure and the next-generation sub-network structure through the improved non-dominated sorting algorithm of the embodiment of the present application, it is possible to protect some models that temporarily have poor performance but may achieve better performance in the future, thereby preventing these models from being dominated in the early stage of the search and being abandoned prematurely, thereby expanding the coverage of the sub-network set.
[0266] Optionally, the first metric target may include at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy, and the second metric target may include the growth rate of at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy.
[0267] For ease of understanding, taking the example that the first metric target includes the amount of parameters and the accuracy, and the second metric target includes the growth rate of the amount of parameters and the accuracy, the current generation sub-network structure and the next generation sub-network structure are graded according to the dominance relationship between individuals based on the amount of parameters and the accuracy, and the first grade division result is obtained; based on the non-dominated sorting algorithm, the current generation sub-network structure and the next generation sub-network structure can be graded according to the dominance relationship between individuals based on the growth rate of the amount of parameters and the accuracy, and the second grade division result is obtained; the sub-network structures included in the same level in the first grade division result and the second grade division result are respectively merged to obtain a merged grade division result; the sub-network structure in the front level is selected from the merged grade division result, that is, the sub-network structure with less parameter amount or higher accuracy, and the sub-network structure with more parameter amount or higher accuracy are selected, then there is no sub-network structure in the back level that has both a higher accuracy and fewer parameters than the selected sub-network structure, nor a sub-network structure that has both a higher accuracy and more parameters than the selected sub-network structure. That is, the first metric is used to select networks with fewer parameters and lower precision, as well as networks with medium parameters and slightly higher precision. The second metric is used to select networks with more parameters and lower precision (which may have a lot of room for growth in the future), as well as networks with medium parameters and slightly higher precision.
[0268] Alternatively, taking the example that the first metric target includes the amount of parameters and the accuracy, and the second metric target includes the growth rate of the amount of parameters and the accuracy, the current generation sub-network structure and the next generation sub-network structure are graded according to the dominance relationship between individuals based on the amount of parameters and the accuracy, and the first grade division result is obtained; based on the non-dominated sorting algorithm, the current generation sub-network structure and the next generation sub-network structure can be graded according to the dominance relationship between individuals based on the growth rate of the amount of parameters and the accuracy, and the second grade division result is obtained; from the first grade division result, the sub-network structure in the front grade is selected, that is, the sub-network structure with fewer parameters or higher accuracy is selected. If a sub-network structure with a high accuracy is selected, there is no sub-network structure in the later levels that has both a higher accuracy and fewer parameters than the selected sub-network structure. At the same time, a sub-network structure in the earlier levels is selected from the second level division result, that is, a sub-network structure with more parameters or a higher accuracy is selected. Then, there is no sub-network structure in the later levels that has both a higher accuracy and more parameters than the selected sub-network structure. The sub-network structure selected from the first level division result and the sub-network structure selected from the second level division result are merged as the second sub-network structure in the updated sub-network set. This ensures that during the search process, sub-network structures with fewer parameters or higher accuracy and sub-network structures with more parameters or higher accuracy can be searched, so that among the remaining sub-network structures, there are no sub-network structures that are both more accurate and have fewer parameters than the selected sub-network structure, and no sub-network structures that are both more accurate and have more parameters than the selected sub-network structure. Therefore, the range of covered parameters can be expanded, which is conducive to the final search for sub-network structures with more parameters and higher accuracy and sub-network structures with fewer parameters and higher accuracy, which is conducive to determining the target neural network with higher accuracy and the target neural network with fewer parameters, effectively solving the problem of the "small model trap".
[0269] In addition, in the process of updating the sub-network set, after the next-generation sub-network structure is generated, when evaluating the fitness of the current-generation sub-network structure and the next-generation sub-network structure according to the verification data of the target task, the verification data of the target task needs to be input into the current-generation sub-network structure and the next-generation sub-network structure respectively, and the output result is obtained. As mentioned above, when updating the sub-network set in this round, the parameter space has been updated, and the updated parameter space is obtained. Therefore, the parameters corresponding to the current-generation sub-network structure and the next-generation sub-network structure are the parameters in the updated parameter space, that is, the parameters corresponding to the current-generation sub-network structure have been updated, and the parameters corresponding to the next-generation sub-network structure inherit the parameters trained in the previous round. In this way, the training process of the current-generation sub-network structure and the next-generation sub-network structure is not to train from scratch after randomly initializing the parameters, but to inherit the parameters that have been trained before, which can effectively improve the search efficiency.
[0270] 1005. Determine a plurality of target neural networks corresponding to the target task based on the update parameter space and the update sub-network set.
[0271] In this step, the updated parameter space can be understood as the parameter space updated last time, and the updated sub-network set can be understood as the sub-network set updated last time.
[0272] There are many ways to determine multiple target neural networks corresponding to the target tasks.
[0273] As an example, part or all of the second sub-network structures included in the updated sub-network set can be selected as the target neural network corresponding to the target task, wherein the parameters corresponding to each of the multiple second sub-network structures included in the updated sub-network set belong to the parameters in the updated parameter space.
[0274] As another example, part or all of the second sub-network structures included in the updated sub-network set can be selected to be widened or expanded to obtain a target neural network corresponding to the target task, wherein the parameters corresponding to each of the multiple second sub-network structures included in the updated sub-network set belong to the parameters in the updated parameter space.
[0275] As another example, part or all of the second sub-network structures can be selected from the multiple second sub-network structures included in the update sub-network set, and training data of a larger scale or more target tasks can be input for training to obtain a target neural network corresponding to the target task, wherein the parameters corresponding to each of the multiple second sub-network structures included in the update sub-network set belong to the parameters in the update parameter space.
[0276] The method for searching the neural network structure of the embodiment of the present application maintains a sub-network set during the search process, and multiple target neural networks can be obtained through one search, and the multiple sub-network structures included in the sub-network set share parameters in the parameter space, which can speed up the search speed and improve the search efficiency, thereby efficiently obtaining multiple target neural networks from one search. The multiple target neural networks can have different properties such as parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy, etc., and a model that can meet different computing and / or storage resource constraints is provided, so that users can select the corresponding target neural network according to the resource constraints in actual applications.
[0277] Optionally, after step 1005, the search method for the neural network structure of the embodiment of the present application further includes: obtaining target task data to be processed; selecting a first target neural network from the multiple target neural networks based on the computing resources and / or storage resources of the terminal device; processing the target task data to be processed according to the first target neural network to obtain a processing result corresponding to the target task.
[0278] In an embodiment of the present application, a target neural network that matches the resource constraints of the terminal device can be selected from the models of different sizes and precisions provided above according to the computing resources and / or storage resources of the terminal device, and applicable terminal devices and application scenarios are more extensive.
[0279] In order to better understand the search method of the neural network structure in the embodiment of the present application, Fig.11 The overall process of the search method for the neural network structure provided in the embodiment of the present application is briefly introduced. Taking the target task as image classification as an example, the overall process of the search method for the neural network structure provided in the embodiment of the present application is as follows: Fig.11 shown.
[0280] The method for searching the neural network structure of the embodiment of the present application can be deployed on the computing node of the server, and the target neural network obtained by the method for searching the neural network structure of the embodiment of the present application can be deployed on terminals such as mobile terminals, monitoring equipment, self-driving cars, and the devices described above that can deploy the target neural network.
[0281] At 1101, a search network and training data are determined according to a target task and used as inputs of a neural network structure search device, such as a server.
[0282] The search network in the embodiment of the present application may be a supernet, which includes many connection modes and different operations. The search network includes a structure space and a parameter space. The structure space includes multiple sub-network structures, and the parameter space includes multiple parameters, wherein each sub-network structure corresponds to at least one parameter in the parameter space. Different connection modes and different operations connected thereto may constitute a sub-network structure.
[0283] For ease of understanding, in the embodiment of the present application, the search network is recorded as N, the parameter space is recorded as W, and a sub-network set maintained includes N sub-network structures, and the sub-network set is represented as {N1, N2, ... N N}, wherein during the sub-network set initialization process, the N sub-network structures can be sampled from the search network N, and the N sub-network structures share the parameters in the parameter space W.
[0284] The training data is image data corresponding to the image classification task, such as pictures.
[0285] At 1102, a parameter space of a search network is updated.
[0286] In this step, a gradient back propagation algorithm can be used to update the parameter space W of the search network using the sub-network structure in the sub-network set.
[0287] Specifically, the process of updating the parameter space based on the gradient back propagation algorithm consists of a forward propagation process and a back propagation process. In the forward propagation process, the training data is input into each sub-network structure included in the sub-network set, and the training data is processed layer by layer through the input layer and the hidden layer and transmitted to the output layer. If the expected output value cannot be obtained in the output layer, the cross entropy between the output and the expected value is taken as the objective function, and the back propagation is turned on. The partial derivative of the objective function to each neuron weight (each neuron weight belongs to the parameter in the parameter space W) is obtained layer by layer to form the gradient of the objective function to the weight vector, that is, the parameter gradient corresponding to each sub-network structure. According to the parameter gradient corresponding to each sub-network structure, the gradient of the parameter space W of the search network N can be obtained, which is used as the basis for modifying the weight (that is, as the basis for updating the parameter space). The learning of the network is completed in the weight modification process. When the error reaches the expected value, the network learning ends, the weight modification process ends, and the updating of the parameter space ends.
[0288] Exemplarily, the gradient of the parameter space W may be the average value of the parameter gradients corresponding to the N sub-network structures included in the sub-network set. For example, it may be calculated by the following formula:
[0289] For the sub-network structure N in the N sub-network structures i The corresponding parameters can be expressed as:
[0290]
[0291] Among them, W i N is the sub-network structure i The corresponding parameter, C i is a binary parameter ({0,1}) indicating whether the corresponding connection is retained, and the value range of i is a positive integer from 1 to N.
[0292] For the sub-network structure N i , its output loss is L i , then the sub-network structure N i The corresponding parameter W i The gradient of is:
[0293]
[0294] The gradient of the parameter space W of the search network N is the average of the gradients of the parameters corresponding to the N sub-network structures:
[0295]
[0296] Exemplarily, the gradient of the parameter space W may be the average value of the gradients of the parameters corresponding to the M sub-network structures in the N sub-network structures included in the sub-network set. For example, it may be calculated by the following formula:
[0297]
[0298] Wherein, M is a positive integer greater than or equal to 1 and less than N.
[0299] In an embodiment of the present application, directly summing up the parameter gradients corresponding to the N sub-network structures in the sub-network set may bring a large computational burden. The parameter gradients corresponding to the M first sub-network structures in a mini-batch can be summed up, which can reduce the computational burden and speed up the update of the parameter space, thereby speeding up the search speed of the neural network structure and improving the efficiency of the neural network structure search.
[0300] At 1103, the subnetwork set is updated.
[0301] In this step, the sub-network set is updated by using an evolutionary algorithm as an example. The sub-network set including N sub-network structures is represented as {N1, N2, ... N N}, the P metric targets used in the sub-network set update process are {F1, F2, …F P The P measurement targets may include any one or more of the number of parameters, network running time, network running memory required, network computing amount, energy consumption, number of floating point operations and accuracy.
[0302] Specifically, based on the evolutionary algorithm, the N sub-network structures included in the sub-network set can be crossover and mutation operations to generate the next generation of t×N sub-network structures, where t is greater than 0, and t can be set accordingly according to the target task or selected based on experience.
[0303] The fitness of the parent samples (i.e., N sub-network structures) and the offspring samples (i.e., the next generation t×N sub-network structures) totaling (t+1)×N sub-network structures is evaluated based on the verification data, such as the number of parameters, recognition rate, etc.
[0304] From the (t+1)×N sub-network structures including the parent samples and the child samples, the improved non-dominated sorting algorithm proposed in the embodiment of the present application is used to select N sub-network structures as a new solution set, that is, as an updated sub-network set.
[0305] It should be understood that in this step, other numbers of sub-network structures may be selected from the (t+1)×N sub-network structures including the parent samples and the child samples, and this is not limited in the embodiment of the present application.
[0306] Taking the measurement target including parameter quantity and accuracy as an example, the specific process of applying the improved non-dominated sorting algorithm proposed in the embodiment of the present application can be as follows:
[0307] Based on the non-dominated sorting algorithm, the (t+1)×N sub-network structures containing parent samples and child samples are graded according to the dominance relationship between individuals according to the parameter quantity and accuracy, and the first grade division result is obtained;
[0308] Based on the non-dominated sorting algorithm, the (t+1)×N sub-network structures containing parent samples and child samples are graded according to the dominance relationship between individuals according to the growth rate of the parameter quantity and accuracy, and the second grade division result is obtained;
[0309] Select the sub-network structure in the front level from the first level division result, that is, select the sub-network structure with fewer parameters or higher accuracy, then there is no sub-network structure in the back level that has higher accuracy and fewer parameters than the selected sub-network structure;
[0310] Select the sub-network structure in the front level from the second level division result, that is, select the sub-network structure with more parameters or higher accuracy, then there is no sub-network structure in the back level that has higher accuracy and more parameters than the selected sub-network structure;
[0311] The sub-network structure selected from the first hierarchical division result and the sub-network structure selected from the second hierarchical division result are combined to form a new solution set {N1, N2, ... N N}.
[0312] This ensures that during the search process, sub-network structures with fewer parameters or higher accuracy and sub-network structures with more parameters or higher accuracy can be searched, so that among the remaining sub-network structures, there are no sub-network structures that are both more accurate and have fewer parameters than the selected sub-network structure, and no sub-network structures that are both more accurate and have more parameters than the selected sub-network structure. Therefore, the range of covered parameters can be expanded, which is conducive to the final search for sub-network structures with more parameters and higher accuracy and sub-network structures with fewer parameters and higher accuracy, which is conducive to determining the target neural network with higher accuracy and the target neural network with fewer parameters, effectively solving the problem of the "small model trap".
[0313] After step 1103, step 1102 may be performed again. In the re-executed step 1102, the subnetwork set used when updating the parameter space is the updated subnetwork set obtained in the last step 1103, that is, the new solution set. And the parameters corresponding to the subnetwork structure in the new solution set used are the parameters in the updated parameter space.
[0314] Similarly, after executing step 1102 again, step 1103 may be executed again, and in step 1103 executed again, the parameters corresponding to the subnetwork structure used when updating the subnetwork set are the parameters in the parameter space updated again after executing step 1102 again. Step 1102 and step 1103 are performed alternately, that is, the parameter space update of the search network and the subnetwork set update are performed alternately.
[0315] At 1104, after multiple alternating updates, when the sub-network structure in the updated sub-network set meets the preset conditions, multiple target neural network models can be output, for example, N sub-network structures can be obtained, and the N sub-network structures have different accuracies and properties, such as the number of parameters, network running time, network required running memory, network calculation amount, energy consumption, and number of floating-point operations.
[0316] At 1105, in actual application, a target neural network model can be selected from multiple target neural network models and deployed on the terminal according to hardware resource limitations, and the selected target neural network model can be used to perform the target task, such as image classification, and output the corresponding classification result.
[0317] In order to illustrate the effect of the method for searching the neural network structure of the embodiment of the present application, the method for searching the neural network structure of the embodiment of the present application is compared with the existing solution.
[0318] Fig.12 A schematic diagram of the verification effect of the network structure searched out on the CIFAR-10 dataset when the improved non-dominated sorting algorithm pNSGA-III is used in the search method for the neural network structure of an embodiment of the present application under similar constraints and when the conventional non-dominated sorting algorithm NSGA-III is used in the existing scheme is shown. Table 1 shows the verification result statistics of the network structure searched out by the conventional method and the method of the present application on the CIFAR-10 dataset.
[0319] Table 1
[0320]
[0321] In Table 1, CARS-A to CARS-I are the network structures obtained by searching when the improved non-dominated sorting algorithm pNSGA-Ⅲ is used in this application, and the remaining network structures are the network structures obtained by searching when the traditional non-dominated sorting algorithm NSGA-Ⅲ is used. Fig.12 It can be seen that the network structure obtained by the search method for the neural network structure provided in the embodiment of the present application on CIFA-R10 can cover the range of 2.4M-3.6M parameters, and a search can obtain network structures under different computing constraints for users to choose. Compared with the traditional method, under the condition of the same number of parameters, the method provided by the present application can obtain a network structure with better performance, and the search time is only 0.4 GPU Days, which greatly reduces the search overhead.
[0322] Fig.13 A schematic diagram of the verification effect of the network structure searched by the neural network structure search method in an embodiment of the present application on the dataset ImageNet is shown. Table 2 shows the verification result statistics of the network structure searched by the traditional method and the method of the present application on the dataset ImageNet.
[0323] Table 2
[0324]
[0325] In Table 2, CARS-A to CARS-I are network structures obtained by searching using the neural network structure search method provided by the present application, and the remaining network structures are network structures obtained by searching using traditional methods. Fig.13It can be seen that the network structure obtained by the search method for the neural network structure provided in the embodiment of the present application on the dataset ImageNet can cover the range of 3.7M-5.1M parameters and the range of 430-590 calculations. Compared with the traditional method, under the condition of the same number of parameters, the network structure performance obtained by the method provided in the present application is better.
[0326] The present application also provides a method for searching a neural network structure. The method can be performed by a neural network structure search device, which can be a computer, server, cloud device, or other device with sufficient computing power to search for a neural network structure.
[0327] Step 1: Determine a search network according to a target task, wherein the search network includes a structure space, and the structure space includes multiple sub-network structures.
[0328] In this step, the method of determining the search network according to the target task is similar to the method mentioned in the above step 1001. Please refer to the above description for details, which will not be repeated for brevity.
[0329] The search network in the embodiment of the present application may include basic modules for building a neural network, such as convolution, pre-set basic operations or combinations of basic operations that are compatible with the application requirements of the target neural network, wherein these basic operations or combinations of basic operations may be collectively referred to as basic operations. The size, depth, width, connection mode, etc. of the search network may be manually designed.
[0330] The search network includes a structure space, and the structure space includes multiple sub-network structures. The structure space in the embodiment of the present application includes all network architectures that can be characterized in principle, and it can also be understood that the structure space is a complete set of network structures, where each possible network structure can be called a sub-network structure. The sub-network structure includes all or part of the basic modules and all or part of the basic operations in the search network, and different sub-network structures include different types and quantities of basic modules and basic operations.
[0331] Step 2: Determine a sub-network set from the search network, wherein the sub-network set includes a plurality of first sub-network structures among the plurality of sub-network structures.
[0332] Determining a sub-network set from the search network may include an initialization process of the sub-network set and an update process of the sub-network set.
[0333] When the subnetwork set is determined from the search network in this step as the initialization process of the subnetwork set, the subnetwork structure included in the subnetwork set can be sampled from the structure space. The parameters corresponding to each first subnetwork structure in the multiple first subnetwork structures can be initialized values or randomly set values. It should be noted that the method of sampling the subnetwork from the search network can be randomly sampled according to the existing method, and the embodiment of the present application is not limited.
[0334] When the subnetwork set is determined from the search network in this step as the update process of the subnetwork set, the updated subnetwork set obtained in the previous round of the subnetwork set update process can be determined as the subnetwork set in the current round of the subnetwork set update process (i.e., the subnetwork set before the update of the current round of the update process). Alternatively, some subnetwork structures can be selected from the multiple subnetwork structures included in the updated subnetwork set obtained in the previous round of the subnetwork set update process to form the subnetwork set in the current round of the subnetwork set update process (i.e., the subnetwork set before the update of the current round of the update process), that is, the subnetwork set in the current round of the subnetwork set update process includes some subnetwork structures in the updated subnetwork set obtained in the previous round of the subnetwork set update process.
[0335] Step three: updating the sub-network set to obtain an updated sub-network set, wherein the updated sub-network set includes a plurality of second sub-network structures among the plurality of sub-network structures.
[0336] The process of updating the sub-network set can be similar to the process of updating the sub-network set in step 1004, which can be specifically described above. The difference from the process described above is that in the embodiment of the present application, there is only the process of updating the sub-network set, and there is no related operation of the parameter space. The parameters corresponding to the first sub-network structure in the sub-network set can be the values that are reinitialized during each training.
[0337] Optionally, updating the sub-network set to obtain an updated sub-network set may include:
[0338] Based on an evolutionary algorithm such as any one of a particle swarm algorithm, a genetic algorithm and a fireworks algorithm, the multiple first sub-network structures are processed to obtain a next-generation sub-network structure; based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a first metric target to obtain a first grade division result; based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a second metric target to obtain a second grade division result; the first grade division result and the sub-network structures included in the same grade in the second grade division result are respectively merged to obtain a merged grade division result; and multiple sub-network structures are selected from the merged grade division result as the multiple second sub-network structures.
[0339] Optionally, updating the sub-network set to obtain an updated sub-network set may include:
[0340] Based on the non-dominated sorting algorithm, multiple first sub-network structures and next-generation sub-network structures are graded according to the first metric target to obtain a first grade division result; based on the non-dominated sorting algorithm, multiple first sub-network structures and next-generation sub-network structures are graded according to the second metric target to obtain a second grade division result; and multiple sub-network structures are selected from the top levels of the first grade division result and the second grade division result as multiple second sub-network structures.
[0341] Optionally, the first metric target may include at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy, and the second metric target may include the growth rate of at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy.
[0342] When selecting multiple second sub-network structures from the current-generation sub-network structure and the next-generation sub-network structure through the improved non-dominated sorting algorithm of the embodiment of the present application, it is possible to protect some models that temporarily have poor performance but may achieve better performance in the future, thereby preventing these models from being dominated in the early stage of the search and being abandoned prematurely, thereby expanding the coverage of the sub-network set.
[0343] It should be understood that the above description of the improved non-dominated sorting algorithm is also applicable to the embodiments of the present application, and reference is made to the above description for any details not provided.
[0344] Step 4: Determine multiple target neural networks corresponding to the target task based on the updated sub-network set.
[0345] The method for searching the neural network structure in the embodiment of the present application maintains a sub-network set during the search process. After one search, a model that can meet different computing and / or storage resource constraints can be provided, allowing users to select the corresponding target neural network according to the resource constraints in actual applications.
[0346] Fig.14 is a schematic flow chart of the image processing method of the embodiment of the present application. Fig.11 The relevant content, interpretation and extension of the methods shown also apply to Fig.14 The method shown below is introduced in Fig.14 The repeated descriptions of the methods shown are appropriately omitted. Fig.14 The method shown can be applied to a terminal device, including:
[0347] 2001. Obtain an image to be processed.
[0348] 2002. Select a first target neural network from multiple target neural networks based on computing resources and / or storage resources of the terminal device.
[0349] 2003. Classify the image to be processed according to the first target neural network to obtain a classification result of the image to be processed.
[0350] Wherein the multiple target neural networks are obtained based on an updated parameter space and an updated subnetwork set, wherein the updated parameter space is obtained by updating the parameter space included in the search network based on the training data of the image classification task, the search network corresponds to the image classification task, the parameter space includes multiple parameters, the search network also includes a structure space, the structure space includes multiple subnetwork structures, wherein each subnetwork structure corresponds to at least one parameter in the parameter space, the updated subnetwork set is obtained by updating the subnetwork set determined from the search network, wherein the subnetwork set includes multiple first subnetwork structures among the multiple subnetwork structures, and the updated subnetwork set includes multiple second subnetwork structures among the multiple subnetwork structures.
[0351] It should be understood that Fig.14In the method shown, multiple target neural networks are determined by maintaining a sub-network set, and multiple sub-network structures in the sub-network set share parameters in the parameter space, which are determined by updating the parameter space and the sub-network set. The training process of the sub-network structure is not to train from scratch after randomly initializing the parameters, but to inherit the parameters that have been trained before, that is, to update the parameters in the parameter space. This can effectively improve the search efficiency, and multiple target neural networks can be obtained in one search. The multiple target neural networks can have different properties such as size or number of floating-point operations. A model that can meet different computing and / or storage resource constraints is provided, so that users can select the corresponding target neural network according to the resource constraints in actual applications.
[0352] Fig.15 It is a schematic diagram of the hardware structure of the neural network structure search device provided in an embodiment of the present application. Fig.15 The neural network structure search device 3000 shown (the device 3000 may be a computer device) includes a memory 3001, a processor 3002, a communication interface 3003 and a bus 3004. The memory 3001, the processor 3002 and the communication interface 3003 are connected to each other through the bus 3004.
[0353] The memory 3001 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 3001 may store a program, and when the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 is used to execute each step of the method for searching the neural network structure of the embodiment of the present application.
[0354] Processor 3002 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the search method of the neural network structure of the method embodiment of the present application.
[0355] The processor 3002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the neural network structure search method of the present application may be completed by hardware integrated logic circuits in the processor 3002 or software instructions.
[0356] The processor 3002 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 3001, and the processor 3002 reads the information in the memory 3001, and combines its hardware to complete the functions required to be performed by the units included in the neural network structure search device, or executes the neural network structure search method of the method embodiment of the present application.
[0357] The communication interface 3003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the device 3000 and other devices or communication networks. For example, the information of the target neural network to be determined and the training data required in the process of determining the target neural network can be obtained through the communication interface 3003.
[0358] The bus 3004 may include a path for transmitting information between various components of the device 3000 (eg, the memory 3001 , the processor 3002 , and the communication interface 3003 ).
[0359] Fig.16 It is a schematic diagram of the hardware structure of the image processing device according to an embodiment of the present application. Fig.16 The image processing device 4000 shown includes a memory 4001 , a processor 4002 , a communication interface 4003 and a bus 4004 . The memory 4001 , the processor 4002 , and the communication interface 4003 are connected to each other through the bus 4004 .
[0360] The memory 4001 may be a ROM, a static storage device, or a RAM. The memory 4001 may store a program. When the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute each step of the image processing method of the embodiment of the present application.
[0361] Processor 4002 can be a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits, which are used to execute relevant programs to implement the functions required to be performed by the units in the image processing device of the embodiment of the present application, or to execute the image processing method of the method embodiment of the present application.
[0362] The processor 4002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image processing method of the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 4002 or by instructions in the form of software.
[0363] The processor 4002 may also be a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001, and completes the functions required to be performed by the units included in the image processing device of the embodiment of the present application in combination with its hardware, or executes the image processing method of the method embodiment of the present application.
[0364] The communication interface 4003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 4000 and other devices or a communication network. For example, the image to be processed can be obtained through the communication interface 4003.
[0365] The bus 4004 may include a path for transmitting information between various components of the device 4000 (eg, the memory 4001 , the processor 4002 , and the communication interface 4003 ).
[0366] Fig.17 Schematic diagram of the hardware structure of the neural network training device of the embodiment of the present application. Similar to the above-mentioned device 3000 and device 4000, Fig.17 The neural network training device 5000 shown includes a memory 5001, a processor 5002, a communication interface 5003 and a bus 5004. The memory 5001, the processor 5002 and the communication interface 5003 are connected to each other through the bus 5004.
[0367] In passing Fig.15After the neural network structure search device shown in the figure searches for the neural network, it can be used Fig.17 The neural network training device 5000 shown trains the neural network, and the trained neural network can be used to execute the image processing method of the embodiment of the present application.
[0368] Specifically, Fig.17 The device shown can obtain training data and a neural network to be trained from the outside world through the communication interface 5003, and then the processor trains the neural network to be trained according to the training data.
[0369] It should be noted that although the above-mentioned devices 3000, 4000 and 5000 only show a memory, a processor and a communication interface, in the specific implementation process, those skilled in the art should understand that the devices 3000, 4000 and 5000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the devices 3000, 4000 and 5000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the devices 3000, 4000 and 5000 may also only include the devices necessary for implementing the embodiments of the present application, and do not necessarily include Fig.15 , Fig.16 and Fig.17 All devices shown in .
[0370] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0371] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0372] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0373] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0374] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0375] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.
[0376] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A method for searching a neural network structure, characterized in that: include: Determine a search network according to the target task, the search network comprising a structure space and a parameter space, the structure space comprising a plurality of sub-network structures, the parameter space comprising a plurality of parameters, wherein each sub-network structure corresponds to at least one parameter in the parameter space; Updating the parameter space according to the training data of the target task to obtain an updated parameter space; Determine a sub-network set from the search network, the sub-network set including a plurality of first sub-network structures among the plurality of sub-network structures; Updating the sub-network set to obtain an updated sub-network set, wherein the updated sub-network set includes a plurality of second sub-network structures among the plurality of sub-network structures; Determining a plurality of target neural networks corresponding to the target task according to the update parameter space and the update subnetwork set; The training data of the target task is in the form of images, voice or text.
2. The search method according to claim 1, characterized in that: The updating of the parameter space according to the training data of the target task to obtain the updated parameter space includes: Inputting training data of the target task; The parameter space is updated based on a gradient back-propagation algorithm, wherein the gradient of the parameter space is an average value of parameter gradients corresponding to some of the first sub-network structures in the multiple first sub-network structures.
3. The search method according to claim 1 or 2, characterized in that: The updating of the sub-network set to obtain an updated sub-network set includes: Processing the multiple first sub-network structures based on any one of a particle swarm algorithm, a genetic algorithm, and a fireworks algorithm to obtain a next-generation sub-network structure; Based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a first metric target to obtain a first grade division result; Based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a second metric target to obtain a second grade division result; Merging the subnetwork structures included in the same level in the first level division result and the second level division result respectively to obtain a merged level division result; A plurality of sub-network structures are selected from the merged hierarchical division results as the plurality of second sub-network structures.
4. The search method according to claim 3, characterized in that: The first metric target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy; the second metric target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and growth rate of accuracy.
5. The search method according to any one of claims 1 to 4, characterized in that: Also includes: Obtain target task data to be processed; Selecting a first target neural network from the plurality of target neural networks according to computing resources and / or storage resources of the terminal device; The target task data to be processed is processed according to the first target neural network to obtain a processing result corresponding to the target task.
6. A method for searching a neural network structure, characterized in that: include: Determine a search network according to the target task, wherein the search network includes a structure space, and the structure space includes multiple sub-network structures; Determine a sub-network set from the search network, the sub-network set including a plurality of first sub-network structures among the plurality of sub-network structures; Updating the sub-network set to obtain an updated sub-network set, wherein the updated sub-network set includes a plurality of second sub-network structures among the plurality of sub-network structures; Determining a plurality of target neural networks corresponding to the target task according to the updated sub-network set; The target neural network is a neural network used to process image data, speech data or text data.
7. The search method according to claim 6, characterized in that: The updating of the sub-network set to obtain an updated sub-network set includes: Processing the multiple first sub-network structures based on any one of a particle swarm algorithm, a genetic algorithm, and a fireworks algorithm to obtain a next-generation sub-network structure; Based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a first metric target to obtain a first grade division result; Based on a non-dominated sorting algorithm, the multiple first sub-network structures and the next-generation sub-network structure are graded according to a second metric target to obtain a second grade division result; Merging the subnetwork structures included in the same level of the first level division result and the second level division result respectively to obtain a merged level division result; A plurality of sub-network structures are selected from the merged hierarchical division results as the plurality of second sub-network structures.
8. The search method according to claim 7, characterized in that: The first metric target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy; the second metric target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and growth rate of accuracy.
9. An image processing method, applied to a terminal device, characterized in that: include: Get the image to be processed; Selecting a first target neural network from a plurality of target neural networks according to computing resources and / or storage resources of the terminal device; Classifying the image to be processed according to the first target neural network to obtain a classification result of the image to be processed; Among them, the multiple target neural networks are obtained based on the updated parameter space and the updated sub-network set, the updated parameter space is obtained by updating the parameter space included in the search network according to the training data of the image classification task, the search network corresponds to the image classification task, the parameter space includes multiple parameters, the search network also includes a structure space, the structure space includes multiple sub-network structures, wherein each sub-network structure corresponds to at least one parameter in the parameter space, the updated sub-network set is obtained by updating the sub-network set determined from the search network, wherein the sub-network set includes multiple first sub-network structures among the multiple sub-network structures, and the updated sub-network set includes multiple second sub-network structures among the multiple sub-network structures.
10. The method according to claim 9, characterized in that The updated parameter space is obtained by updating the parameter space based on a gradient back-propagation algorithm, wherein the gradient of the parameter space is an average value of parameter gradients corresponding to some of the first sub-network structures in the multiple first sub-network structures.
11. The method according to claim 9 or 10, characterized in that: The updated subnetwork set is obtained by processing the multiple first subnetwork structures based on any one of particle swarm algorithm, genetic algorithm and fireworks algorithm to obtain the next-generation subnetwork structure, and then selecting multiple subnetwork structures from the merged hierarchical division results as the multiple second subnetwork structures, wherein the merged hierarchical division result is obtained by respectively merging the subnetwork structures included in the same level of the first hierarchical division result and the second hierarchical division result, the first hierarchical division result is obtained by hierarchically dividing the multiple first subnetwork structures and the next-generation subnetwork structure according to a first metric target based on a non-dominated sorting algorithm, and the second hierarchical division result is obtained by hierarchically dividing the multiple first subnetwork structures and the next-generation subnetwork structure according to a second metric target based on a non-dominated sorting algorithm.
12. The method according to claim 11, characterized in that The first metric target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and accuracy; the second metric target includes at least one of parameter quantity, network running time, network required running memory, network calculation amount, energy consumption, number of floating-point operations and growth rate of accuracy.
13. A neural network structure search device, characterized in that: include: Memory, used to store programs; A processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to perform the following process: Determine a search network according to the target task, the search network comprising a structure space and a parameter space, the structure space comprising a plurality of sub-network structures, the parameter space comprising a plurality of parameters, wherein each sub-network structure corresponds to at least one parameter in the parameter space; Updating the parameter space according to the training data of the target task to obtain an updated parameter space; Determine a sub-network set from the search network, the sub-network set including a plurality of first sub-network structures among the plurality of sub-network structures; Updating the sub-network set to obtain an updated sub-network set, wherein the updated sub-network set includes a plurality of second sub-network structures among the plurality of sub-network structures; Determining a plurality of target neural networks corresponding to the target task according to the update parameter space and the update subnetwork set; The training data of the target task is in the form of images, voice or text.
14. An image processing device, characterized in that: include: Memory, used to store programs; A processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to perform the following process: Get the image to be processed; Selecting a first target neural network from a plurality of target neural networks based on computing resources and / or storage resources of a terminal device; Classifying the image to be processed according to the first target neural network to obtain a classification result of the image to be processed; Among them, the multiple target neural networks are obtained based on the updated parameter space and the updated sub-network set, the updated parameter space is obtained by updating the parameter space included in the search network according to the training data of the image classification task, the search network corresponds to the image classification task, the parameter space includes multiple parameters, the search network also includes a structure space, the structure space includes multiple sub-network structures, wherein each sub-network structure corresponds to at least one parameter in the parameter space, the updated sub-network set is obtained by updating the sub-network set determined from the search network, wherein the sub-network set includes multiple first sub-network structures among the multiple sub-network structures, and the updated sub-network set includes multiple second sub-network structures among the multiple sub-network structures.
15. A computer-readable storage medium, characterized in that: The computer-readable medium stores a program code for execution by a device, wherein the program code includes a program for executing the method according to any one of claims 1-8 or 9-12.
16. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the method as described in any one of claims 1-8 or 9-12.
