Neural network structure search method and device

Abstract

A neural network structure search method and device, relating to the field of AI, can determine a neural network structure with excellent performance in a shorter time using fewer computing resources while ensuring consistency between theoretical delay and actual delay. The method includes: obtaining a super network according to a target task, obtaining the delay of each deep learning operator in the super network when running on an electronic device, determining a delay loss function based on the delay of each deep learning operator when running on an electronic device, performing training operations on the super network, updating the model parameters of the super network based on the delay loss function and the network loss function until the updated super network meets the conditions for the target task to run on the electronic device, and determining the target neural network structure based on the updated architectural parameters of each network layer. The super network includes multiple network layers, each network layer includes multiple nodes, and any two nodes in a network layer are connected by a deep learning operator. The model parameters include the architectural parameters of each network layer.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a method and device for searching a neural network structure. Background Art

[0002] With the rapid development of AI technology, a variety of neural network models have emerged. The performance of the neural network structure has a significant impact on the task execution effectiveness of the neural network model. The better the performance of the neural network structure, the better the task execution effect of the neural network model. Therefore, when building a neural network model, how to determine the best performing neural network structure is a hot topic for those skilled in the art.

[0003] Neural architecture search (NAS) technology was developed for this purpose. NAS can automatically search for the optimal neural network architecture within a predefined search space. However, existing techniques for using NAS to search for neural network architectures suffer from high computational resource consumption and an inability to ensure that theoretical latency matches actual latency. Summary of the Invention

[0004] The present application provides a neural network structure search method and device, which can determine a neural network structure with excellent performance in a shorter time using fewer computing resources while ensuring the consistency between theoretical delay and actual delay.

[0005] In the first aspect, the present application provides a neural network structure search method, wherein the neural network structure search device obtains a super network according to a target task, obtains the delay of each deep learning operator in the super network running on an electronic device, and determines the delay loss function of the super network according to the delay of each deep learning operator running on the electronic device, and then performs a training operation on the super network, and updates the model parameters of the super network according to the delay loss function and the network loss function obtained during the training process, until the updated super network meets the conditions for the target task to run on the electronic device, and determines the target neural network structure according to the updated architecture parameters of each network layer. Wherein, the super network includes multiple network layers, each network layer includes multiple nodes, any two nodes of a network layer are connected by a deep learning operator, and the model parameters include the architecture parameters of each network layer in the multiple network layers.

[0006] In this way, the super network obtained according to the target task is a structure including multiple network layers, each network layer including multiple nodes, and any two nodes are connected by a deep learning operator. The super network contains all sub networks that may be used to perform the target task. The embodiment of the present application trains the super network and updates the model parameters of the super network. The model parameters include the architectural parameters of each network layer until the updated super network meets the conditions. Then, the target neural network structure can be determined according to the architectural parameters of each updated network layer, that is, the neural network structure with the best performance is determined. Compared with the prior art that a large number of sub networks are trained to obtain the target neural network structure, the embodiment of the present application only needs to train the super network to obtain the target neural network structure, thus saving a lot of computing resources, shortening the search time, and improving the search efficiency. Moreover, since the delay loss function is referenced when updating the model parameters, the delay loss function is obtained by the actual delay of each deep learning operator running on the electronic device, which can ensure the consistency of the theoretical delay and the actual delay when determining the target neural network structure.

[0007] Optionally, in a possible implementation of the present application, the above-mentioned method of "determining the delay loss function of the super network based on the delay of each deep learning operator running in the electronic device" may include: the neural network structure search device determines the network embedding coefficient corresponding to each deep learning operator based on the pre-stored correspondence between the operator and the network embedding coefficient, determines the product of the delay of each deep learning operator running in the electronic device and the network embedding coefficient corresponding to the deep learning operator, and determines the sum of all products, and then determines the delay loss function based on the sum and the delay consistency coefficient.

[0008] In this way, by constructing the delay corresponding to the discrete deep learning operators into a continuous delay constraint function based on the actual delay of each deep learning operator running on the electronic device and the network embedding coefficient corresponding to each deep learning operator, the delay corresponding to the discrete deep learning operators is ensured to ensure delay consistency.

[0009] Optionally, in another possible implementation of the present application, the architectural parameters of the network layer include the connection weights of each deep learning operator of the network layer. In this case, the method of "determining the target neural network structure based on the updated architectural parameters of each network layer" may include: the neural network structure search device obtains the connection weights whose values ​​meet preset conditions in the updated architectural parameters of each network layer, and determines the target neural network structure based on all the obtained connection weights.

[0010] Since the target neural network structure is determined in the prior art based on the largest connection weight among the architectural parameters of each network layer, while the target neural network structure is determined in the present application based on the connection weights among the architectural parameters of each network layer whose values ​​meet the preset conditions. The number of connection weights for each network layer is not limited in the present application. When the number of connection weights obtained from each network layer is multiple, the more connection weights retained compared to the one in the prior art, the more stable the target neural network structure obtained, and thus the better the task execution effect of the neural network model determined according to the target neural network structure.

[0011] Optionally, in another possible implementation of the present application, the above-mentioned method of "updating the model parameters of the super network based on the delay loss function and the network loss function obtained during the training process" may include: the neural network structure search device determines the overall loss function of the super network based on the delay loss function and the network loss function, and updates the model parameters of the super network based on the overall loss function.

[0012] In this way, by updating the model parameters in the super network according to the delay loss function and the network loss function, it is ensured that the target neural network structure meets the delay consistency and network accuracy requirements.

[0013] Optionally, in another possible implementation of the present application, the method of "updating the model parameters of the hypernetwork based on the overall loss function" may include: the neural network structure search device determines the gradient information of each model parameter based on the overall loss function, and adjusts the model parameter based on the gradient information of each model parameter. The gradient information is used to represent the adjustment coefficient of the corresponding model parameter.

[0014] Implemented the updating of model parameters through gradient.

[0015] In a second aspect, a neural network structure search device is provided, which includes various modules for executing the neural network structure search method of the above-mentioned first aspect or any possible implementation of the above-mentioned first aspect.

[0016] In a third aspect, a neural network structure search device is provided. The neural network structure search device includes a memory and a processor. The memory and the processor are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the neural network structure search device performs the neural network structure search method according to the first aspect and any possible implementation thereof.

[0017] In a fourth aspect, a chip system is provided, which is applied to a neural network structure search device. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via circuits; the interface circuits are configured to receive signals from the memory of the neural network structure search device and send signals to the processors, the signals including computer instructions stored in the memory. When the processors execute the computer instructions, the neural network structure search device performs the neural network structure search method according to the first aspect and any possible implementation thereof.

[0018] In a fifth aspect, a computer-readable storage medium is provided, which includes computer instructions. When the computer instructions are run on a neural network structure search device, the neural network structure search device executes the neural network structure search method as described in the first aspect and any possible implementation thereof.

[0019] In a sixth aspect, the present application provides a computer program product comprising computer instructions. When the computer instructions are executed on a neural network structure search device, the neural network structure search device executes the neural network structure search method of the first aspect and any possible implementation thereof.

[0020] For the specific descriptions of the second to sixth aspects and their various implementations in this application, reference can be made to the detailed descriptions in the first aspect and its various implementations; and for the beneficial effects of the second to sixth aspects and their various implementations, reference can be made to the analysis of the beneficial effects in the first aspect and its various implementations, which will not be repeated here.

[0021] These and other aspects of the present application will become more readily apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the structure of a neural network structure search system provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0024] Figure 3 One of the flow charts of the neural network structure search method provided in the embodiment of the present application;

[0025] Figure 4 A schematic diagram of the structure of a super network provided in an embodiment of the present application;

[0026] Figure 5 The second flowchart of the neural network structure search method provided in the embodiment of the present application;

[0027] Figure 6 The third flowchart of the neural network structure search method provided in the embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of a neural network structure search device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0030] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0031] At present, the process of constructing a neural network model is: constructing a neural network structure, training and evaluating the constructed neural network structure to obtain a neural network structure with excellent performance, and determining the neural network model based on the neural network structure with excellent performance.

[0032] Most existing neural network architectures are manually designed. For example, network structures like ResNet, which has achieved remarkable success in image classification, and Transformer, which has dominated machine translation, were designed by experts in the field. However, these network architectures, derived from extensive experience and extensive experimentation, can be time-consuming, inaccurate, and experience inconsistent latency. Inconsistent latency refers to the discrepancy between the theoretical latency of a neural network model and its actual latency. Actual latency refers to the latency experienced when the neural network model actually runs on an electronic device.

[0033] NAS technology can automatically search for neural network structures with excellent performance within a pre-defined search space, thereby solving the problems existing in manually designed neural network structures.

[0034] In the solution of the first prior art, reinforcement learning technology is used to search for neural network structures. Specifically, the neural network structure search device can use a recurrent neural network (RNN) as a controller and generate a sub-network using controller parameter sampling according to a preset search space. The sub-network is trained to converge to obtain model evaluation indicators, such as the accuracy of the sub-network, the number of floating-point operations per second (FLOPs), etc. The controller parameters can then be updated based on the model evaluation indicators. The neural network structure search device can then repeat the above operations, that is, according to the search space, another sub-network is generated using the updated controller parameter sampling, and the other sub-network is trained to obtain a new model evaluation indicator, and the controller parameters after the last update are updated based on the new model evaluation indicator. The cycle is repeated until a sub-network with excellent performance is obtained, and the sub-network is used as the network structure of the neural network model to be determined.

[0035] However, because the neural network structure search device requires training a large number of subnetworks to obtain the optimal subnetwork, each training step requires initializing the network weights, resulting in a significant consumption of computing resources. Furthermore, because the neural network structure search device uses FLOPs as a reference when updating controller parameters, this FLOPs does not reflect the actual latency of the subnetwork on different electronic devices, making it impossible to ensure that the theoretical latency of the subnetwork is consistent with the actual latency.

[0036] In the second prior art solution, an evolutionary algorithm and reinforcement learning techniques are used to search for neural network structures. This solution, based on the first prior art solution, adds a neural network structure search device that sends a subnetwork to an electronic device and receives back the actual latency of the subnetwork. This allows the neural network structure search device to reference this actual latency rather than FLOPs when updating controller parameters, resolving the latency inconsistency issue in the first prior art solution.

[0037] However, the solution of the second existing technology still has the problem of large consumption of computing resources, and sending the sub-network to the electronic device will also increase a large amount of computing resources, resulting in low search efficiency of the neural network structure.

[0038] In summary, the neural network structure search in the existing technology has the problem of high consumption of computing resources and failure to ensure that the theoretical delay is consistent with the actual delay.

[0039] In order to ensure that the theoretical delay and the actual delay are consistent, and to determine a neural network structure with good performance in a shorter time using fewer computing resources, an embodiment of the present application provides a neural network structure search method, which obtains a super network according to the target task, and determines the delay loss function of the super network according to the delay of each deep learning operator in the super network running on the electronic device. In the process of training the super network, the model parameters of the super network are updated according to the delay loss function and the network loss function until the updated super network meets the conditions for the target task to run on the electronic device, and finally the target neural network structure is determined according to the updated architectural parameters of each network layer. In this way, the super network obtained according to the target task is a structure including multiple network layers, each network layer including multiple nodes, and any two nodes are connected by deep learning operators. The super network contains all sub-networks that may be used to perform the target task. The embodiment of the present application trains a supernetwork and updates the model parameters of the supernetwork, which include the architectural parameters of each network layer. Until the updated supernetwork meets the conditions, the target neural network structure can be determined based on the updated architectural parameters of each network layer, that is, the neural network structure with the best performance can be determined. Compared with the prior art method of training a large number of subnetworks to obtain the target neural network structure, the embodiment of the present application only needs to train the supernetwork to obtain the target neural network structure, thus saving a lot of computing resources, shortening the search time, and improving the search efficiency. Moreover, since the delay loss function is referenced when updating the model parameters, the delay loss function is obtained by the actual delay of each deep learning operator running on the electronic device, which can ensure the consistency of the theoretical delay and the actual delay when determining the target neural network structure.

[0040] The execution entity of the neural network structure search method provided in the embodiment of the present application is a neural network structure search device.

[0041] In one scenario, the neural network structure search device can be an electronic device, which can be a server or a terminal device. That is, the electronic device itself initiates the target task and determines the target neural network structure with the best performance by executing the neural network structure search method provided in the embodiments of the present application, thereby determining the neural network model. The electronic device then runs the neural network model to perform the target task.

[0042] In another scenario, the neural network structure search device can be a server, and the terminal device runs the neural network model. That is, the server determines the target neural network structure with the best performance by executing the neural network structure search method provided in the embodiment of the present application, thereby determining the neural network model and sending the neural network model to the terminal device. The terminal device runs the received neural network model to perform the target task. Specifically, the neural network structure search method provided in the embodiment of the present application can be applied to a neural network structure search system.

[0043] Figure 1 Figure 2 shows a structure of the neural network structure search system. Figure 1 As shown, the neural network structure search system may include: a server 11 and a terminal device 12. The server 11 and the terminal device 12 are connected using a wired communication method or a wireless communication method.

[0044] Server 11 is the main body of the neural network structure search method provided in the embodiment of the present application. It is mainly used to train the super network and update the model parameters in the super network according to the delay loss function and the network loss function until the updated super network meets the conditions for running the target task on the terminal device 12. It is also used to determine the target neural network structure based on the updated architectural parameters of each network layer, thereby determining the neural network model, and sending the neural network model to the terminal device 12.

[0045] In some embodiments, the server 11 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. The present application embodiment does not limit the specific form of the server. Figure 1 A server is used as an example.

[0046] The terminal device 12 is used to run the neural network model from the server 11 to perform the target task.

[0047] In some embodiments, the terminal device 12 may be: a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, an Internet of Things (IOT) device, etc. The embodiments of the present application do not limit the specific form of the terminal device. Figure 1 In the figure, the terminal device 12 is taken as a mobile phone as an example.

[0048] The embodiments of the present application do not limit the specific application scenario of the neural network structure search method.

[0049] The basic hardware structures of the server 11 and the terminal device 12 are similar, both including Figure 2 The computing device shown in FIG. Figure 2 Taking the computing device shown as an example, the hardware structure of the server 11 and the terminal device 12 is introduced.

[0050] like Figure 2 As shown, the computing device may include a processor 21, a memory 22, a communication interface 23, and a bus 24. The processor 21, the memory 22, and the communication interface 23 may be connected via the bus 24.

[0051] Processor 21 is the control center of the computing device and can be a single processor or a collective term for multiple processing elements. For example, processor 21 can be a general-purpose central processing unit (CPU) or other general-purpose processor. A general-purpose processor can be a microprocessor or any conventional processor, such as a graphics processing unit (GPU) or a digital signal processor (DSP).

[0052] As an embodiment, the processor 21 may include one or more CPUs, such as Figure 2 CPU 0 and CPU 1 are shown in Figure 1.

[0053] The memory 22 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0054] In one possible implementation, the memory 22 may exist independently of the processor 21 and may be connected to the processor 21 via a bus 24 for storing instructions or program codes. When the processor 21 calls and executes the instructions or program codes stored in the memory 22, the neural network structure search method provided in the following embodiments of the present application can be implemented.

[0055] In the embodiment of the present application, the server 11 and the terminal device 12 have different software programs stored in the memory 22, so the server 11 and the terminal device 12 implement different functions. The functions performed by each device will be described in conjunction with the following flowchart.

[0056] In another possible implementation, the memory 22 may also be integrated with the processor 21 .

[0057] The communication interface 23 is used to connect the computing device to other devices via a communication network, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 23 may include a receiving unit for receiving data and a sending unit for sending data.

[0058] The bus 24 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0059] It should be pointed out that Figure 2 The structure shown in the figure does not constitute a limitation on the computing device, except Figure 2 In addition to the components shown, the computing device may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0060] Based on the hardware structure of the computing device described above, an embodiment of the present application provides a neural network structure search method. The neural network structure search method provided by the embodiment of the present application is described below with reference to the accompanying drawings. In this embodiment of the present application, the neural network structure search method provided by the embodiment of the present application is introduced using a scenario where a server executes the neural network structure search method and determines a neural network model, and a terminal device receives and runs the neural network model as an example.

[0061] When the neural network architecture search method is applied to Figure 1 When searching for a neural network structure such as Figure 3 As shown, the neural network structure search method may include the following steps 301-305.

[0062] 301. The server obtains a super network according to the target task.

[0063] The target task indicates the neural network model built and executed on the terminal device. The hypernetwork consists of multiple network layers, each of which includes multiple nodes. Any two nodes in a network layer are connected by one or more deep learning operators. These operators can be convolution, separable convolution, dilated convolution, average pooling, and other types. Furthermore, each neural network structure sampled from the hypernetwork can be used to execute the target task.

[0064] Typically, each network layer includes at least two nodes. The more nodes a network layer includes, the more deep learning operators it corresponds to, the more computing resources it requires, and the higher the accuracy of the output results.

[0065] When the server obtains a target task for instructing to construct a neural network model to run on a terminal device, it can first determine a target neural network structure with the best performance for the neural network model. Specifically, the server can first obtain a super network based on the target task.

[0066] It can be understood that the process of the above-mentioned server obtaining a hypernetwork based on the target task is as follows: the server can determine whether there is a historical task that is the same or similar to the target task locally. If so, it means that the server has previously constructed a hypernetwork based on the historical task. At this time, the server can directly obtain the hypernetwork previously constructed based on the historical task from the local. If not, it means that the server has not previously constructed a hypernetwork based on the target task. At this time, the server can construct a hypernetwork based on the target task and the preset search space. Of the two methods of obtaining a hypernetwork mentioned above, the method of the server directly obtaining a hypernetwork from the local can reduce the workload of searching for the target neural network structure, thereby improving search efficiency.

[0067] In addition, the target task may include the output type of the neural network model. For example, the target task may be a face recognition neural network model built and running on a terminal device, used to recognize faces and output the corresponding person's name. Another example is a hand pose estimation model built and running on a terminal device, used to recognize the hand pose of a person in an image.

[0068] For example, Figure 4 This is a schematic diagram of the structure of a super network provided in an embodiment of the present application. Figure 4 As shown, a super network including three network layers is used as an example. The first network layer includes three nodes, and the deep learning operators used to connect these three nodes include: 3×3 standard convolution, 5×5 standard convolution and jump connection operators. The second network layer includes three nodes, and the deep learning operators used to connect these three nodes include: 3×3 standard convolution, 5×5 standard convolution and 3×3 separation convolution. The third network layer includes four nodes, and the deep learning operators used to connect these four nodes include: 3×3 standard convolution, 5×5 separation convolution, 3×3 expansion convolution and jump connection operators. In this way, Figure 4 It can be seen that the super network includes: 3×3 standard convolution, 5×5 standard convolution, skip connection operator, 3×3 separable convolution, 5×5 separable convolution, 3×3 dilated convolution, a total of six deep learning operators. It can be understood that Figure 4 In each network layer shown, the connection between each node is only an exemplary connection. The specific connection between the nodes in each network layer and the deep learning operator used to connect two nodes are not limited in this embodiment of the application.

[0069] 302. The server obtains the latency of each deep learning operator in the super network running on the terminal device.

[0070] After building the hypernetwork, the server can send each deep learning operator in the hypernetwork to the terminal device. The terminal device can run each received deep learning operator and return the latency of running each deep learning operator to the server. In this way, the server can obtain the latency of each deep learning operator running on the terminal device.

[0071] 303. The server determines a delay loss function of the hypernetwork based on the delay of each deep learning operator running on the terminal device.

[0072] After obtaining the latency of each deep learning operator running on the terminal device, the server can determine the latency loss function of the entire hypernetwork based on these latency. For details, please refer to the description of steps 303A-303C below.

[0073] 304. The server performs a training operation on the super network and updates the model parameters of the super network according to the delay loss function and the network loss function obtained during the training process until the updated super network meets the conditions for the target task to be run on the terminal device.

[0074] The network loss function represents the discrepancy between the hypernetwork's predicted output and the data labels. A larger network loss function indicates a greater discrepancy between the predicted output and the data labels. The hypernetwork training process can be understood as minimizing the output values ​​of the delay loss function and the network loss function.

[0075] After the server obtains the hypernetwork in step 301, it can train the hypernetwork and update the model parameters in the hypernetwork based on the delay loss function determined in step 303 and the network loss function obtained during the training process until the updated hypernetwork meets the conditions for running the target task on the terminal device, and then terminate the training process. The model parameters may include architectural parameters of each of the multiple network layers.

[0076] It is understood that the above conditions may include accuracy requirements and latency requirements. For example, the conditions may include that the accuracy of the output results obtained by running the hypernetwork reaches a preset percentage, and the latency of running the hypernetwork is less than a preset time value. These conditions are pre-defined based on the target task and the hardware structure of the terminal device.

[0077] 305. The server determines the target neural network structure based on the updated architectural parameters of each network layer.

[0078] Among them, the target neural network structure is the network structure with the best performance.

[0079] In a specific implementation, the architectural parameters of each network layer may include the connection weights of each deep learning operator in all deep learning operators in that network layer. In this case, the server may determine the target neural network structure based on the updated architectural parameters of each network layer by first obtaining the connection weights whose values ​​meet preset conditions in the updated architectural parameters of each network layer, and then determining the target neural network structure based on all obtained connection weights.

[0080] It can be understood that the above preset conditions can be implemented in multiple ways.

[0081] In one possible implementation, the preset condition may be a preset number of connection weights in each network layer. The connection weights that meet the preset condition are the first preset number of weights after sorting all connection weights in the network layer in descending order. The preset number of connection weights in different network layers may be the same or different.

[0082] In another possible implementation, the preset condition may be: a connection weight greater than a preset weight value. In this way, the server can obtain connection weights greater than the preset weight value from the architectural parameters of each network layer and determine the target neural network structure based on all obtained connection weights. Of course, the preset condition may also be to set a corresponding preset weight value for each network layer. The preset weight values ​​corresponding to different network layers may be the same or different.

[0083] Since the target neural network structure is determined in the prior art based on the largest connection weight among the architectural parameters of each network layer, while the target neural network structure is determined in the present application based on the connection weights among the architectural parameters of each network layer whose values ​​meet the preset conditions. The number of connection weights for each network layer is not limited in the present application. When the number of connection weights obtained from each network layer is multiple, the more connection weights retained compared to the one in the prior art, the more stable the target neural network structure obtained, and thus the better the task execution effect of the neural network model determined according to the target neural network structure.

[0084] The neural network structure search method provided in the embodiment of the present application obtains a super network according to the target task, and determines the delay loss function of the super network according to the delay of each deep learning operator in the super network running on the electronic device. In the process of training the super network, the model parameters of the super network are updated according to the delay loss function and the network loss function until the updated super network meets the conditions for the target task to run on the electronic device, and finally the target neural network structure is determined according to the updated architectural parameters of each network layer. In this way, the super network obtained according to the target task is a structure including multiple network layers, each network layer including multiple nodes, and any two nodes are connected by deep learning operators. The super network contains all sub-networks that may be used to perform the target task. The embodiment of the present application trains a supernetwork and updates the model parameters of the supernetwork, which include the architectural parameters of each network layer. Until the updated supernetwork meets the conditions, the target neural network structure can be determined based on the updated architectural parameters of each network layer, that is, the neural network structure with the best performance can be determined. Compared with the prior art method of training a large number of subnetworks to obtain the target neural network structure, the embodiment of the present application only needs to train the supernetwork to obtain the target neural network structure, thus saving a lot of computing resources, shortening the search time, and improving the search efficiency. Moreover, since the delay loss function is referenced when updating the model parameters, the delay loss function is obtained by the actual delay of each deep learning operator running on the electronic device, which can ensure the consistency of the theoretical delay and the actual delay when determining the target neural network structure.

[0085] For example, suppose the target task is to build a hand posture estimation model that runs on a terminal device, and the hand posture estimation model runs on the GPU of the terminal device. Then, if the target neural network structure needs to be determined within one day, if the solution of the prior art is used to determine the target neural network structure, it may take thousands of GPUs. If the target neural network structure is determined by the neural network structure search method provided in the embodiment of the present application, it may only take 1 to 2 GPUs. It can be seen that the neural network structure search method provided in the embodiment of the present application greatly saves the computing resources required to search for the target neural network structure.

[0086] Optionally, in the embodiment of the present application, based on Figure 3 ,like Figure 5 As shown, the above step 303 may specifically include the following steps 303A-303C.

[0087] 303A. The server determines the network embedding coefficient corresponding to each deep learning operator based on the pre-stored correspondence between the operator and the network embedding coefficient.

[0088] The role of the network embedding coefficient is to ensure that the delay loss function obtained by using the network embedding coefficient is consistent with the meaning of the network loss function.

[0089] 303B. The server determines the product of the latency of each deep learning operator running on the terminal device and the network embedding coefficient corresponding to the deep learning operator, and determines the sum of all products.

[0090] After determining the network embedding coefficient corresponding to each deep learning operator, the server can calculate the product of the latency of each deep learning operator running on the terminal device and the network embedding coefficient corresponding to the deep learning operator, and add all the products to obtain the sum.

[0091] In the specific implementation, it is assumed that the server uses It represents the delay set consisting of the delay of each deep learning operator running on the terminal device among the multiple deep learning operators included in the super network.

[0092]

[0093] Among them, S represents the set of all deep learning operators in the super network, and operator represents the deep learning operators in the S set. Represents the latency of the i-th deep learning operator in the S set on the terminal device.

[0094] Then, the server can calculate the product of the latency of each deep learning operator running on the terminal device and the network embedding coefficient corresponding to the deep learning operator, and calculate the sum of all products. The sum satisfies the following formula:

[0095]

[0096] in, represents the delay of the i-th deep learning operator in the S set on the terminal device, α i represents the network embedding coefficient corresponding to the i-th deep learning operator, Represents the weighted sum of all deep learning operators in the S set, that is, the product of the delay of each deep learning operator and the network embedding coefficient corresponding to the deep learning operator is calculated, and the sum of all products is calculated.

[0097] 303C. The server determines a delay loss function based on the sum value and the delay consistency coefficient.

[0098] The server gets the sum of all products After that, the delay loss function can be determined. The delay loss function satisfies the following formula:

[0099]

[0100] Among them, λ la Indicates the delay consistency coefficient, loss la represents the delay loss function.

[0101] It can be understood that the above λ la It is a variable, which is a matrix composed of the connection weights of each deep learning operator in each network layer in multiple network layers of the hypernetwork, and is continuously updated during the training process.

[0102] In this way, by constructing the delay corresponding to the discrete deep learning operators into a continuous delay constraint function based on the actual delay of each deep learning operator running on the electronic device and the network embedding coefficient corresponding to each deep learning operator, the delay corresponding to the discrete deep learning operators is ensured to ensure delay consistency.

[0103] Optionally, in the embodiment of the present application, based on Figure 5 ,like Figure 6 As shown, the above step 304 may specifically include the following steps 304A-304B.

[0104] 304A. The server performs a training operation on the super network and determines an overall loss function of the super network based on the delay loss function and the network loss function.

[0105] Among them, the delay loss function is to ensure the delay consistency of the target neural network structure, and the network loss function is to ensure the precision requirements of the target neural network structure, that is, the accuracy requirements.

[0106] It is understandable that in order to avoid overfitting of the hypernetwork, the server also needs to take the network regularization term into account when determining the overall loss function.

[0107] Specifically, the server may determine the overall loss function of the hypernetwork. The overall loss function satisfies the following formula:

[0108]

[0109] Among them, loss la Represents the delay loss function, loss mse represents the network loss function, Represents the network regularization term, and loss represents the overall loss function.

[0110] 304B. The server updates the model parameters of the hypernetwork according to the volume loss function until the updated hypernetwork meets the conditions for the target task to be run on the terminal device.

[0111] In a specific implementation, the server can determine the gradient information of each model parameter based on the overall loss function. The gradient information is used to represent the adjustment coefficient of the corresponding model parameter. The server can then adjust the model parameter based on the gradient information of each model parameter.

[0112] In the case where the model parameters include the network parameters and architecture parameters of each network layer, and the network parameters of each network layer include the weights of each deep learning algorithm of the network layer, the server can first update the network parameters of each network layer. The updated network parameters satisfy the following formula:

[0113]

[0114] Among them, w represents the weight of a deep learning operator in the network parameters of a certain network layer, W N 'Indicates the value of the network parameter w after the last training, Represents the gradient information of the network parameter w, W N Represents the updated value of the network parameter w.

[0115] Afterwards, the server can update the architecture parameters of each network layer. The updated architecture parameters satisfy the following formula:

[0116]

[0117] Where a represents the connection weight of a deep learning operator in the architecture parameters of a certain network layer, W A 'Indicates the value of the architecture parameter a after the last training, Represents the gradient information of the architecture parameter a, W A Indicates the updated value of the architecture parameter a.

[0118] In this way, by updating the model parameters in the super network according to the delay loss function and the network loss function, it is ensured that the target neural network structure meets the delay consistency and network accuracy requirements.

[0119] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0120] like Figure 7 FIG. 7 is a schematic diagram of a neural network structure search device 70 provided in an embodiment of the present application. The neural network structure search device 70 is used to perform Figure 3 、 Figure 5 、 Figure 6 The neural network structure search method shown in any of the figures in FIG. The neural network structure search device 70 may include an acquisition unit 71, a determination unit 72, a training unit 73 and an update unit 74.

[0121] The acquisition unit 71 is used to acquire a super network according to the target task, wherein the super network includes multiple network layers, each network layer includes multiple nodes, and any two nodes in a network layer are connected through a deep learning operator; and is also used to acquire the delay of each deep learning operator in the super network running on the electronic device. For example, combined with Figure 3 , the acquisition unit 71 can be used to execute step 301 and step 302. The determination unit 72 is used to determine the delay loss function of the super network according to the delay of each deep learning operator running in the electronic device obtained by the acquisition unit 71. For example, combined with Figure 3 , the determining unit 72 can be used to execute step 303. The training unit 73 is used to perform a training operation on the super network obtained by the obtaining unit 71. For example, in combination with Figure 3 The training unit 73 can be used to perform the training operation on the super network described in step 304. The updating unit 74 is used to update the model parameters of the super network according to the delay loss function determined by the determining unit 72 and the network loss function obtained during the training process of the training unit 73, until the updated super network meets the conditions for the target task to run on the electronic device, and the model parameters include the architecture parameters of each network layer in the multiple network layers. For example, combined with Figure 3 The updating unit 74 can be used to execute the step 304 described above to update the model parameters of the super network according to the delay loss function and the network loss function obtained during the training process. The determining unit 72 is also used to determine the target neural network structure according to the architecture parameters of each network layer updated by the updating unit 74. For example, Figure 3 , the determination unit 72 can be used to execute step 305.

[0122] Optionally, the determination unit 72 is specifically used to: determine the network embedding coefficient corresponding to each deep learning operator based on the correspondence between the pre-stored operator and the network embedding coefficient; determine the product of the delay of each deep learning operator running in the electronic device and the network embedding coefficient corresponding to the deep learning operator, and determine the sum of all products; determine the delay loss function based on the sum and the delay consistency coefficient.

[0123] Optionally, the architectural parameters of the network layer include the connection weights of each deep learning operator of the network layer. The determination unit 72 is specifically used to: obtain the connection weights whose values ​​meet preset conditions in the updated architectural parameters of each network layer; and determine the target neural network structure based on all the obtained connection weights.

[0124] Optionally, the updating unit 74 is specifically configured to: determine an overall loss function of the hypernetwork according to the delay loss function and the network loss function; and update model parameters of the hypernetwork according to the overall loss function.

[0125] Optionally, the updating unit 74 is specifically used to: determine the gradient information of each model parameter according to the overall loss function, where the gradient information is used to represent the adjustment coefficient of the corresponding model parameter; and adjust the model parameter according to the gradient information of each model parameter.

[0126] Of course, the neural network structure search device 70 provided in the embodiment of the present application includes but is not limited to the above modules.

[0127] In actual implementation, the acquisition unit 71, the determination unit 72, the training unit 73 and the update unit 74 can be composed of Figure 2 The specific execution process can be referred to Figure 3 、 Figure 5 or Figure 6 The description of the neural network structure search method shown in FIG is not repeated here.

[0128] Another embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on the neural network structure search device, the neural network structure search device executes each step executed by the neural network structure search device in the method flow shown in the above method embodiment.

[0129] Another embodiment of the present application provides a chip system, which is applied to a neural network structure search device. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via circuits. The interface circuits are used to receive signals from the memory of the neural network structure search device and send signals to the processors, the signals including computer instructions stored in the memory. When the processor executes the computer instructions, the neural network structure search device performs each step performed by the neural network structure search device in the method flow shown in the above method embodiment.

[0130] In another embodiment of the present application, a computer program product is also provided. The computer program product includes computer instructions. When the computer instructions are run on a neural network structure search device, the neural network structure search device executes each step executed by the neural network structure search device in the method flow shown in the above method embodiment.

[0131] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer execution instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more media that can be integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0132] The above is only a specific embodiment of the present application. Those skilled in the art may conceive of changes or substitutions based on the specific embodiment provided in this application, and all such changes or substitutions shall fall within the scope of protection of this application.

Claims

1. A neural network structure search method, characterized in that: Applications in image processing include: A supernetwork is obtained according to a target task, wherein the supernetwork includes multiple network layers, each network layer includes multiple nodes, and any two nodes in a network layer are connected via a deep learning operator; the target task is used to instruct a neural network model to be constructed and run on a terminal device, wherein the neural network model includes a hand posture estimation model, and the hand posture estimation model is used to recognize the hand posture of a person in an image; Obtaining the latency of each deep learning operator in the super network running on the electronic device; Determining a delay loss function of the hypernetwork based on the delay of each deep learning operator running on the electronic device; Performing a training operation on the supernetwork, and updating model parameters of the supernetwork according to the delay loss function and the network loss function obtained during the training process, until the updated supernetwork meets the conditions for running the target task on the electronic device; the model parameters include architectural parameters of each network layer in the multiple network layers; The target neural network structure is determined based on the updated architectural parameters of each network layer.

2. The neural network structure search method according to claim 1, characterized in that: Determining the delay loss function of the super network according to the delay of each deep learning operator running on the electronic device includes: Determine the network embedding coefficient corresponding to each deep learning operator based on the pre-stored correspondence between operators and network embedding coefficients; Determining the product of the time delay of each deep learning operator running on the electronic device and the network embedding coefficient corresponding to the deep learning operator, and determining the sum of all products; The delay loss function is determined according to the sum value and the delay consistency coefficient.

3. The neural network structure search method according to claim 1 or 2, characterized in that: The architecture parameters of the network layer include the connection weights of each deep learning operator of the network layer, and determining the target neural network structure according to the updated architecture parameters of each network layer includes: Obtain the connection weights of the updated architecture parameters of each network layer whose values ​​meet the preset conditions; The target neural network structure is determined based on all the acquired connection weights.

4. The neural network structure search method according to claim 1 or 2, characterized in that: The updating of the model parameters of the hypernetwork according to the delay loss function and the network loss function obtained during the training process includes: Determining an overall loss function of the supernetwork based on the delay loss function and the network loss function; The model parameters of the hypernetwork are updated according to the overall loss function.

5. The neural network structure search method according to claim 4, characterized in that: The updating of the model parameters of the hypernetwork according to the overall loss function comprises: Determine the gradient information of each model parameter according to the overall loss function, wherein the gradient information is used to represent the adjustment coefficient of the corresponding model parameter; Adjust the model parameters based on the gradient information of each model parameter.

6. A neural network structure search device, characterized in that: Applications in image processing include: An acquisition unit is configured to acquire a supernetwork according to a target task, the supernetwork comprising multiple network layers, each network layer comprising multiple nodes, and any two nodes in a network layer being connected via a deep learning operator; and to acquire a time delay of each deep learning operator in the supernetwork running on an electronic device; the target task is used to instruct a neural network model to be constructed and run on a terminal device, the neural network model comprising a hand posture estimation model, the hand posture estimation model being used to recognize a person's hand posture in an image; a determining unit, configured to determine a delay loss function of the supernetwork according to the delay of each deep learning operator acquired by the acquiring unit when running on the electronic device; a training unit, configured to perform a training operation on the supernetwork acquired by the acquisition unit; an updating unit, configured to update model parameters of the supernetwork according to the delay loss function determined by the determining unit and the network loss function obtained during the training process of the training unit, until the updated supernetwork meets the conditions for running the target task on the electronic device; the model parameters include architectural parameters of each network layer in the multiple network layers; The determining unit is further used to determine the target neural network structure based on the architectural parameters of each network layer updated by the updating unit.

7. The neural network structure search device according to claim 6, characterized in that: The determining unit is specifically configured to: Determine the network embedding coefficient corresponding to each deep learning operator based on the pre-stored correspondence between operators and network embedding coefficients; Determining the product of the time delay of each deep learning operator running on the electronic device and the network embedding coefficient corresponding to the deep learning operator, and determining the sum of all products; The delay loss function is determined according to the sum value and the delay consistency coefficient.

8. The neural network structure search device according to claim 6 or 7, characterized in that: The architecture parameters of the network layer include the connection weight of each deep learning operator of the network layer, and the determining unit is specifically configured to: Obtain the connection weights of the updated architecture parameters of each network layer whose values ​​meet the preset conditions; The target neural network structure is determined based on all the acquired connection weights.

9. The neural network structure search device according to claim 6 or 7, characterized in that: The updating unit is specifically configured to: Determining an overall loss function of the supernetwork based on the delay loss function and the network loss function; The model parameters of the hypernetwork are updated according to the overall loss function.

10. The neural network structure search device according to claim 9, characterized in that: The updating unit is specifically configured to: Determine the gradient information of each model parameter according to the overall loss function, wherein the gradient information is used to represent the adjustment coefficient of the corresponding model parameter; Adjust the model parameters based on the gradient information of each model parameter.

11. A neural network structure search device, characterized in that: The neural network structure search device includes a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the neural network structure search device executes the neural network structure search method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on a neural network structure search device, enable the neural network structure search device to execute the neural network structure search method according to any one of claims 1 to 5.

Images