Adaptive search method and device for neural network

Abstract

The present application provides an adaptive search method and device for a neural network, which relates to the field of neural networks and can realize adaptive neural network search for hardware platforms. The method includes: receiving a search condition set, including target hardware platform information, network structure information of a source neural network and one or more evaluation indicators. The network structure information of the source neural network is used to characterize the source neural network. A training process is performed on a super network to be trained based on a training data set to obtain a trained super network. The super network to be trained is expanded based on the network structure of the source neural network. A sub-network search process is performed on the trained super network based on one or more evaluation indicators to obtain the network structure information of the target neural network, and the network structure information of the target neural network is used to characterize the target neural network. The evaluation result of the target neural network running on the target hardware platform is better than the evaluation result of the source neural network running on the target hardware platform.

Description

Technical Field

[0001] The present application relates to the field of neural networks, and in particular to a method and device for adaptively searching a neural network. Background Art

[0002] In recent years, neural networks have made rapid progress in computer vision applications. As the accuracy of neural networks gradually improves, the network structure of neural networks becomes more and more complex, which increases the computing resource requirements of the device, resulting in low device operation speed and high time overhead.

[0003] At present, the neural network search method in the prior art generally searches for neural network models based on floating-point operations per second (FLOPs). Among them, FLOPs is often used to estimate the operating efficiency of the hardware platform (also known as model overhead). Then, based on FLOPs, the neural network model search may search for the same target neural network for different hardware platforms, and the same target neural network has the same FLOPS on different hardware platforms (for example, the amount of addition and multiplication calculations of the target neural network on different hardware platforms). However, it does not mean that the lower the FLOPS, the shorter the reasoning time of the neural network model. For example, there are three existing hardware platforms, hardware platform 1 (CPU1), hardware platform 2 (CPU2), and hardware platform 3 (GPU). Using the prior art, based on FLOPs, the neural network search may search for the same target neural network, and the target neural network has a lower FLOPs on the three hardware platforms. However, in the actual operation process, due to the different structures of the three hardware platforms, the target neural network may have a better running effect on hardware platform 1, a general running effect on hardware platform 2, and a poor running effect on hardware platform 3.

[0004] Alternatively, when searching for a neural network model based on FLOPs, a target neural network may be found for a certain hardware platform. The FLOPs of the target neural network on the hardware platform are lower (for example, the addition and multiplication computations of the target neural network are smaller). However, in reality, the operating efficiency of the target neural network on the hardware platform is poor (also known as a large model overhead).

[0005] It can be seen that how to search for a target neural network that can run efficiently on a specific hardware platform has become an urgent problem to be solved. Summary of the invention

[0006] The adaptive neural network search method provided in the present application can search for a target neural network that is suitable for a target hardware platform under the premise of high search flexibility.

[0007] In order to achieve the above objectives, this application adopts the following technical solutions:

[0008] In a first aspect, the present application provides an adaptive search method for a neural network, which may include: receiving a search condition set, the search condition set including: target hardware platform information, network structure information of a source neural network and one or more evaluation indicators, the network structure information of the source neural network is used to characterize the source neural network. A training process is performed on a super network to be trained based on a training data set to obtain a trained super network, wherein the super network to be trained is obtained by expanding based on the network structure of the source neural network. A sub-network search process is performed on the trained super network based on one or more evaluation indicators to obtain the network structure information of the target neural network, wherein the network structure information of the target neural network is used to characterize the target neural network, and the evaluation result of the target neural network running on the target hardware platform is better than the evaluation result of the source neural network running on the target hardware platform.

[0009] The number of target neural networks searched in the trained super network can be one or more, for example, based on the target hardware platform information and one or more evaluation indicators, a neural network with the best evaluation result is used as the target neural network. Alternatively, multiple neural networks with the best ranking are sorted according to the evaluation results and used as the target neural network.

[0010] The target hardware platform includes a platform for carrying a neural network. The target hardware platform information obtained may include, for example, attribute information of the hardware platform (e.g., graphics processing unit (GPU), central processing unit (CPU), digital signal processor (DSP), etc.), model, manufacturer, etc. In this way, different hardware platforms can obtain different implementations and optimizations of different operators and / or operator sets, thereby realizing adaptive neural network search.

[0011] Among them, the evaluation index can also be called search conditions, constraints, etc. The evaluation index is used to ensure that the neural network finally searched meets the requirements. For example, the accuracy and time cost of the neural network searched need to meet the requirements.

[0012] Among them, the network structure information of the neural network may, for example, include information about operators and / or operator sets contained in the neural network, the arrangement of operators and / or operator sets (for example, a neural network with a tree structure, a neural network with a mesh structure, etc.), hierarchical information, parameters of operators and / or operator sets, etc.

[0013] Thus, compared with the existing method of searching for neural networks based on FLOPs, the adaptive search method for neural networks provided by the present application is to search for neural networks in a trained super network based on a search condition set, wherein the super network is obtained by expanding the network structure of the source neural network, so the super network can cover or support different hardware platforms for different implementations of operators / channels. Since the search condition set can be flexibly set, and the search condition set at least includes target hardware platform information, therefore, by adopting the method provided by the present application, under the premise of high search flexibility, the different operator / channel overheads caused by different implementations of operators / channels on different hardware platforms can be considered, and the target application network for the target hardware platform can be obtained by adaptive search, wherein the running effect of the target neural network on the target hardware platform is better than the running effect of the source neural network on the target hardware platform. For example, suppose operator A in the source neural network is a 1×1 convolution algorithm that requires 10 million multiplications, and operator B is a 3×3 convolution algorithm that requires 15 million operations. The source neural network runs on the target hardware platform, operator A requires 2μs, and operator B requires 3μs. Operator A in the target neural network needs 2μs to run on the target hardware platform, and operator B' (for example, operator B is optimized) only requires 1.5μs.

[0014] In a possible implementation, one or more evaluation indicators include any one or more of the following: an accuracy evaluation indicator of a neural network model, a time overhead evaluation indicator of a neural network model, a storage space evaluation indicator of a neural network model, a power consumption evaluation indicator of a neural network model, a utilization evaluation indicator of a tensor computing unit of a neural network model, and a memory read / write rate evaluation indicator of a neural network model. Among them, the accuracy evaluation indicator of a neural network model includes that the accuracy of a target neural network is not less than an accuracy threshold. The time overhead evaluation indicator of a neural network model includes that the time overhead of a target neural network is not greater than a time overhead threshold. The storage space evaluation indicator of a neural network model includes that the storage space occupied by a target neural network is not greater than a storage space threshold. The power consumption evaluation indicator of a neural network model includes that the power consumption of a target neural network is not greater than a power consumption threshold. The utilization evaluation indicator of a tensor computing unit of a neural network model includes that the utilization of a tensor computing unit of a target neural network is not less than a utilization threshold of a tensor computing unit. The memory read / write rate evaluation indicator of a neural network model includes that the memory read / write rate of a target neural network is not less than a memory read / write rate threshold.

[0015] In this way, one or more combinations of evaluation indicators can be set according to different user needs, different application scenarios, or different tasks. For example, the combination of evaluation indicators is determined based on the hardware indicators of the hardware platform itself, the different tasks of the neural network, etc. Exemplarily, two evaluation indicators can be configured, the accuracy evaluation indicator and the time overhead evaluation indicator, then the accuracy threshold and the time overhead threshold need to be set at the same time. For example, in some scenarios, it is necessary to obtain recognition results for a large number of images in a short time, but the accuracy requirements for image recognition are not high. Then, a smaller accuracy threshold and a smaller time overhead threshold can be set so that the searched target neural network has a faster recognition speed than the source neural network.

[0016] In a possible implementation, a sub-network search process is performed on a trained super network based on one or more evaluation indicators to obtain network structure information of a target neural network, including: using a first knowledge base to determine one or more candidate sub-networks to be evaluated within a target search range from the trained super network. A sub-network search process is performed on one or more candidate sub-networks to be evaluated within a target search range based on one or more evaluation indicators to obtain network structure information of the target neural network.

[0017] The first knowledge base may include, for example, expert prior knowledge. The first knowledge base includes network receptive field constraints of neural networks determined based on expert experience, as well as optimization conditions of operators and / or operator sets for specific hardware platforms. The optimization conditions may include, for example, optimizing the (7×7 convolution operator + pooling) input for a specific hardware platform, so that although its computational amount is greater than that of (3×3 convolution operator + pooling) or (5×5 convolution operator + pooling), its time overhead on the current hardware platform is less than that of (3×3 convolution operator + pooling) or (5×5 convolution operator + pooling). In this way, based on the first knowledge base, it can be directly determined to select a neural network including (7×7 convolution operator + pooling) with lower time overhead as a candidate subnetwork, thereby narrowing the target search range and improving the search efficiency of the neural network.

[0018] In this way, the first knowledge base can be used to determine the target search range in the super network based on the target hardware platform information, and the sub-network search process can be executed within the target search range, which is conducive to narrowing the search range of the sub-network, reducing the search time, and improving the efficiency of the neural network search.

[0019] In a possible implementation, a sub-network search process is performed on a trained super network based on one or more evaluation indicators to obtain network structure information of a target neural network, including: obtaining target hardware platform information, network structures of M neural networks, N evaluation indicators, and corresponding relationship information between P evaluation results, wherein M, N, and P are all positive integers; the network structures of the M neural networks are included in the network structure of the trained super network. Based on the one or more evaluation indicators and the corresponding relationship information, the evaluation results of one or more candidate sub-networks to be evaluated included in the trained super network are determined. Based on the evaluation results of one or more candidate sub-networks to be evaluated, a search process is performed on the trained super network to obtain network structure information of the target neural network.

[0020] Among them, the M neural networks are sub-networks extracted from the trained super network, that is, the candidate sub-networks to be evaluated. Obviously, the network structure of the M neural networks is included in the network structure of the trained super network. When a candidate sub-network runs on the target hardware platform, it corresponds to N evaluation results based on N evaluation indicators. When the M candidate sub-networks run on the target hardware platform, they correspond to M*N evaluation results based on N evaluation indicators, that is, P evaluation results. Then, based on the P evaluation results, one or more candidate sub-networks are determined as the target neural network.

[0021] In this way, during the search process, by selecting any one or more evaluation indicators, the evaluation results of the candidate sub-network can be determined according to the corresponding relationship, and then the network structure information of the target neural network can be determined.

[0022] For example, on target hardware platform 1, the evaluation index is accuracy, the evaluation result of candidate subnetwork 1 is A, and the evaluation result of candidate subnetwork 2 is B, and A is better than B. If the number of target neural networks is 1, candidate subnetwork 1 is determined to be the target neural network based on one or more evaluation indicators and corresponding relationship information.

[0023] In one possible implementation, one or more evaluation indicators include a time overhead evaluation indicator of a neural network model, and the time overhead evaluation indicator of the neural network model includes that the time overhead of a target neural network is not greater than a time overhead threshold. Accordingly, a sub-network search process is performed on a trained super network based on the one or more evaluation indicators, including: performing a sub-network evaluation on one or more candidate sub-networks included in the trained super network based on the time overhead evaluation indicator of the neural network model to obtain one or more target neural networks whose time overhead of the neural network model is less than the time overhead threshold.

[0024] For example, a neural network runs on a hardware platform that processes images at an original speed of 100 images per second. Now it is necessary to increase the speed of the hardware platform to process images, for example, to make the optimized neural network process images at a speed of 150 images per second on the hardware platform. Then, a time cost evaluation index can be set, and a time cost threshold can be used to ensure that the speed of the searched target neural network to process images can meet the requirements.

[0025] In a possible implementation, the method further includes: constructing a super network to be trained by horizontal and / or vertical expansion according to the network structure of the source neural network, the expansion rules in the second knowledge base, and the target hardware platform information.

[0026] Among them, the second knowledge base can be the same as or different from the above-mentioned first knowledge base, and the extension rules in the second knowledge base include different corresponding relationships between the operator set in the source neural network (i.e., the first operator set hereinafter) and the replaceable operator set corresponding to the operator set (i.e., the derived second operator set hereinafter) based on different hardware platforms. It can be understood that the operator set in the source neural network and the corresponding replaceable operator set have the same or similar functions. Among them, the corresponding relationship includes the corresponding relationship between the operator set in the source neural network based on the target hardware platform and the replaceable operator set. Moreover, the corresponding relationship is established based on experimental data, expert prior knowledge and the design of the target hardware platform, so that the extension rules can be used to limit the network scale of the super network. For example, if a certain operator set in the source neural network does not perform well on the target hardware platform, the operator set can be replaced (not retained) based on the second knowledge base, thereby reducing the network scale of the generated super network.

[0027] Among them, the horizontal expansion method includes expanding the operator set contained in one level of the source neural network into one or more operator sets contained in one level of the super network based on the network structure information of the source neural network. The vertical expansion method includes vertically expanding one level of the source neural network into multiple levels based on the network structure information of the source neural network, and then obtaining multiple operator sets contained in multiple levels of the super network according to the operator sets contained in the multiple levels after the vertical expansion.

[0028] Among them, the operator set contained in the super network to be trained has the same function as the corresponding operator set in the source neural network, and the target neural network searched in the super network can perform the same tasks as the source neural network, such as image classification tasks in image recognition.

[0029] In a possible implementation, the third operator set in the super network to be trained includes: the first operator set and / or a derived second operator set, wherein the first operator set is included in the source neural network.

[0030] Among them, each third operator set in the super network can correspond to a level of the source neural network, and the first operator set in each level can derive a second operator set based on the knowledge base, and the third operator set includes the first operator set and the derived second operator set. Alternatively, based on the knowledge base, it is determined that the derived second operator set in a certain level can replace the first operator set, so that the corresponding third operator set only includes the derived second operator set. Of course, it is also possible to determine based on the knowledge base that there is no derived second operator set in a certain level, and the corresponding third operator set only includes the first operator set.

[0031] In a possible implementation, based on one or more evaluation indicators and corresponding relationship information, determining the evaluation results of one or more candidate sub-networks to be evaluated included in the trained super network includes: based on one or more evaluation indicators and corresponding relationship information, determining the evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set in the trained super network, wherein the first operator set is included in the source neural network. Based on the proportional relationship of the N evaluation indicators and the evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set, obtaining the evaluation results of one or more candidate sub-networks to be evaluated.

[0032] Among them, the proportional relationship can be determined based on the target task performed by the target hardware platform or the properties of the target hardware platform. It can be understood that the proportional relationship of the N evaluation indicators reflects the relative importance of the N evaluation indicators. Obviously, the higher the proportional evaluation indicator, the better the evaluation effect of the searched target neural network based on the evaluation indicator on the target hardware platform. For example, the evaluation indicators include accuracy and time overhead. The target hardware platform is a hardware platform for processing suburban video surveillance images. The amount of information contained in the surveillance images collected per unit time is small, and a high image processing accuracy needs to be guaranteed, but the processing speed requirement is low. In this way, the accuracy ratio can be set higher in the proportional relationship. In this way, based on the proportional relationship and the evaluation results, a candidate subnetwork with better running performance on the target hardware platform can be obtained, and then a better target neural network can be searched.

[0033] In a possible implementation, based on the proportional relationship of N evaluation indicators and the evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set, obtaining the evaluation results of one or more candidate subnetworks to be evaluated includes: receiving r proportional relationships of N evaluation indicators, and performing a pre-search for each proportional relationship of N evaluation indicators; wherein r is a positive integer. In a pre-search process, the weights of the evaluation results of each first operator set and / or each derived second operator set are set, and based on the N evaluation indicators and the proportional relationship of the N evaluation indicators, the trained super network is trained again, and in the training process, the weights corresponding to the evaluation results of each first operator set and each derived second operator set are adjusted. According to the weights corresponding to the evaluation results of each first operator set and each derived second operator set after adjustment, the candidate subnetwork to be evaluated and the evaluation results of the candidate subnetwork to be evaluated are determined. The evaluation results of one or more candidate subnetworks to be evaluated are determined according to the r pre-search processes.

[0034] Among them, the pre-search process is to narrow the target search range. First, some candidate sub-networks are determined based on different proportional relationships, and then one or more evaluation indicators are used to evaluate the determined partial candidate sub-networks, and then the network structure information of the target neural network is determined.

[0035] Among them, in the pre-search process, the weight ratio is adjusted to determine the first operator set or the derived second operator set that is more suitable for the target hardware platform in each third operator set. In this way, in each pre-search process, a candidate subnetwork with the best performance under the current ratio relationship and its evaluation result can be determined. Then, under multiple ratio relationships, multiple candidate subnetworks can be obtained. In this way, the target neural network can be determined from multiple candidate subnetworks, and then the network structure information of the target neural network can be determined.

[0036] In one possible implementation, the N evaluation indicators include an accuracy evaluation indicator of the neural network model and a time overhead evaluation indicator of the neural network model; accordingly, the proportional relationship among the N evaluation indicators includes: the proportional relationship between the accuracy evaluation indicator and the time overhead evaluation indicator.

[0037] For example, the user inputs the first ratio and the second ratio of the accuracy evaluation index and the time cost evaluation index. In this way, it can be ensured that the accuracy of the target neural network is not lower than the accuracy of the source neural network, or the accuracy is not much different from the source target neural network. In addition, while ensuring the accuracy of the target neural network, the time cost is reduced, that is, the running speed of the target neural network on the target hardware platform is accelerated to improve work efficiency.

[0038] In a possible implementation, a search process is performed on the trained super network based on the evaluation results of one or more candidate subnetworks to be evaluated to obtain the network structure information of the target neural network, and also includes: sorting the evaluation results of one or more candidate subnetworks to be evaluated to obtain K first target candidate subnetworks corresponding to the K evaluation results that are sorted first. According to the trained super network and the K first target candidate subnetworks, S second target candidate subnetworks after the nth round of hybridization and / or mutation are obtained; wherein n is the preset number of screening times, n≥1; K and S are positive integers, S>K. Based on one or more evaluation indicators and corresponding relationship information, the evaluation results of the S second target candidate subnetworks are obtained and sorted to obtain K second target candidate subnetworks corresponding to the K evaluation results that are sorted first. The network structure information of one or more target neural networks is obtained from the K*(n+1) target candidate subnetworks.

[0039] In the present application, hybridization refers to exchanging a partial set of operators contained in a corresponding level of some of the K first target candidate subnetworks to obtain a second target candidate subnetwork. Mutation refers to replacing a partial set of operators contained in a partial set of operators contained in some of the K first target candidate subnetworks with a partial set of operators of a corresponding level in a trained super network to obtain a second target candidate subnetwork.

[0040] For example, after (n+1) rounds of iterative training, that is, after n rounds of hybridization and / or mutation, K*(n+1) target candidate sub-networks are obtained. The K*(n+1) target candidate sub-networks can be sorted again according to one or more evaluation indicators to obtain the network structure information of one or more target neural networks.

[0041] In this way, the evaluation results of one or more candidate subnetworks to be evaluated are sorted first, and a better partial candidate subnetwork is obtained to narrow the target search range. Afterwards, more candidate subnetworks are generated based on the better partial candidate subnetworks through hybridization and / or mutation methods. In the iterative process, a better partial candidate subnetwork is obtained in each iteration process, and one or more target subnetworks that meet one or more evaluation indicators and have better performance are determined.

[0042] In one possible implementation, the source neural network is a neural network running on a source hardware platform, or the target neural network is a neural network running on a target hardware platform, and the source hardware platform is different from the target hardware platform.

[0043] The source hardware platform is a hardware platform different from the target hardware platform, and the source hardware platform and the target hardware platform include processors with different attributes, or processors of different models, or processors of the same attributes but different models. The source hardware platform will be replaced with the target hardware platform. The target neural network required for the target hardware platform is searched based on the source neural network running on the source hardware platform.

[0044] For example, the source neural network is neural network 1 running on the GPU. Now the GPU needs to be switched to chip A. Then, the target neural network needs to be searched based on neural network 1 and chip A. The running effect of the target neural network on chip A is better than the running effect of neural network 1 on chip A.

[0045] In one possible implementation, the source neural network and the target neural network are neural networks running on a target hardware platform in different time domains.

[0046] For example, the source neural network is neural network 1 running on the GPU. Now it is necessary to improve the GPU image processing speed, so neural network 1 needs to be optimized, and the searched target neural network is the optimized neural network 1. Neural network 1 and the target neural network run successively in time or are mounted on the GPU (target hardware platform).

[0047] In a possible implementation, the method further includes: constructing a target neural network based on the network structure information of the target neural network, and performing a training process on the target neural network based on the second training data set to obtain a trained target neural network.

[0048] The second training data set may be the same as or different from the first training data set, and is used to participate in the training process of the neural network. By searching the network structure information of the target neural network through the above method, the target neural network can be constructed, and the target neural network can be trained using the training data set, and the parameter information in the target neural network can be adjusted to obtain a neural network that can run on the target hardware platform to perform the target task.

[0049] In one possible implementation, the trained target neural network runs on a target hardware platform to perform target detection tasks and image classification tasks.

[0050] In this way, the trained target neural network can perform the target task on the target hardware platform. For example, it can perform traffic light recognition tasks, pedestrian and vehicle detection tasks, etc. Moreover, its operation effect on the target hardware platform is better. For example, the time cost of the source neural network to perform the traffic light recognition task on the target hardware platform is 0.2ms, and the time cost of the trained target neural network to perform the traffic light recognition task on the target hardware platform is 0.1ms. Under the premise of ensuring that the accuracy remains basically unchanged, the time cost is reduced by 50%.

[0051] In a second aspect, the present application provides an adaptive search device for a neural network, which may include: a receiving unit, a training unit, and a search unit. The receiving unit is used to receive a search condition set, which includes: target hardware platform information, network structure information of a source neural network, and one or more evaluation indicators, wherein the network structure information of the source neural network is used to characterize the source neural network. The training unit is used to perform a training process on a super network to be trained based on a training data set to obtain a trained super network, wherein the super network to be trained is expanded based on the network structure of the source neural network. The search unit is used to perform a sub-network search process on the trained super network based on one or more evaluation indicators to obtain the network structure information of the target neural network, wherein the network structure information of the target neural network is used to characterize the target neural network, and the evaluation result of the target neural network running on the target hardware platform is better than the evaluation result of the source neural network running on the source hardware platform.

[0052] In a possible implementation, one or more evaluation indicators include any one or more of the following: an accuracy evaluation indicator of a neural network model, a time overhead evaluation indicator of a neural network model, a storage space evaluation indicator of a neural network model, a power consumption evaluation indicator of a neural network model, a utilization evaluation indicator of a tensor computing unit of a neural network model, and a memory read / write rate evaluation indicator of a neural network model. Among them, the accuracy evaluation indicator of a neural network model includes that the accuracy of a target neural network is not less than an accuracy threshold. The time overhead evaluation indicator of a neural network model includes that the time overhead of a target neural network is not greater than a time overhead threshold. The storage space evaluation indicator of a neural network model includes that the storage space occupied by a target neural network is not greater than a storage space threshold. The power consumption evaluation indicator of a neural network model includes that the power consumption of a target neural network is not greater than a power consumption threshold. The utilization evaluation indicator of a tensor computing unit of a neural network model includes that the utilization of a tensor computing unit of a target neural network is not less than a utilization threshold of a tensor computing unit. The memory read / write rate evaluation indicator of a neural network model includes that the memory read / write rate of a target neural network is not less than a memory read / write rate threshold.

[0053] In one possible implementation, the search unit is specifically used to use the first knowledge base to determine one or more candidate sub-networks to be evaluated within the target search range from the trained super network; based on one or more evaluation indicators, a sub-network search process is performed on the one or more candidate sub-networks to be evaluated within the target search range to obtain network structure information of the target neural network.

[0054] In one possible implementation, the search unit is specifically used to obtain the correspondence information between the target hardware platform, the network structure of M neural networks, N evaluation indicators, and P evaluation results, wherein M, N and P are all positive integers; the network structure of the M neural networks is included in the network structure of a trained super network; based on one or more evaluation indicators and the correspondence information, the evaluation results of one or more candidate sub-networks to be evaluated included in the trained super network are determined; based on the evaluation results of one or more candidate sub-networks to be evaluated, a search process is performed on the trained super network to obtain the network structure information of the target neural network.

[0055] In one possible implementation, one or more evaluation indicators include a time overhead evaluation indicator of a neural network model, and the time overhead evaluation indicator of a neural network model includes that the time overhead of a target neural network is not greater than a time overhead threshold. Accordingly, a search unit is specifically used to perform a sub-network evaluation on one or more candidate sub-networks included in a trained super network based on the time overhead evaluation indicator of the neural network model, so as to obtain one or more target neural networks whose time overhead of the neural network model is less than the time overhead threshold.

[0056] In a possible implementation, the device also includes: a construction unit, which is used to construct the super network to be trained by horizontal and / or vertical expansion according to the network structure of the source neural network, the expansion rules in the second knowledge base and the target hardware platform information.

[0057] In a possible implementation, the third operator set in the super network to be trained includes: the first operator set and / or a derived second operator set, wherein the first operator set is included in the source neural network.

[0058] In a possible implementation, the search unit is specifically used to determine the evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set in a trained super network based on one or more evaluation indicators and corresponding relationship information, wherein the first operator set is included in the source neural network; based on the proportional relationship of N evaluation indicators and the evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set, obtain the evaluation results of one or more candidate sub-networks to be evaluated; based on the evaluation results of one or more candidate sub-networks to be evaluated, perform a search process on the trained super network to obtain the network structure information of the target neural network.

[0059] In a possible implementation, the search unit is specifically used to receive a proportional relationship of r N evaluation indicators, and perform a pre-search for each proportional relationship of the N evaluation indicators; wherein r is a positive integer; in a pre-search process, the weights of the evaluation results of each first operator set and / or each derived second operator set are set, and based on the N evaluation indicators and the proportional relationship of the N evaluation indicators, the trained super network is trained again, and in the training process, the weights corresponding to the evaluation results of each first operator set and each derived second operator set are adjusted; according to the weights corresponding to the evaluation results of each first operator set and each derived second operator set after adjustment, the candidate subnetwork to be evaluated and the evaluation results of the candidate subnetwork to be evaluated are determined; and the evaluation results of one or more candidate subnetworks to be evaluated are determined according to the r pre-search processes.

[0060] In one possible implementation, the N evaluation indicators include an accuracy evaluation indicator of the neural network model and a time overhead evaluation indicator of the neural network model; accordingly, the proportional relationship among the N evaluation indicators includes: the proportional relationship between the accuracy evaluation indicator and the time overhead evaluation indicator.

[0061] In one possible implementation, the search unit is specifically used to sort the evaluation results of one or more candidate subnetworks to be evaluated, and obtain K first target candidate subnetworks corresponding to the K evaluation results that are sorted first; according to the trained super network and the K first target candidate subnetworks, S second target candidate subnetworks after the nth round of hybridization and / or mutation are obtained; wherein n is a preset number of screening times, n≥1; K and S are positive integers, S>K; based on one or more evaluation indicators and corresponding relationship information, the evaluation results of the S second target candidate subnetworks are obtained and sorted, and K second target candidate subnetworks corresponding to the K evaluation results that are sorted first are obtained; and the network structure information of one or more target neural networks is obtained from the K*(n+1) target candidate subnetworks.

[0062] In one possible implementation, the source neural network is a neural network running on a source hardware platform, or the target neural network is a neural network running on a target hardware platform, and the source hardware platform is different from the target hardware platform.

[0063] In one possible implementation, the source neural network and the target neural network are neural networks running on a target hardware platform in different time domains.

[0064] In a possible implementation, the construction unit is further used to construct the target neural network based on the network structure information of the target neural network. The training unit is further used to perform a training process on the target neural network based on the second training data set to obtain a trained target neural network.

[0065] In one possible implementation, the trained target neural network runs on a target hardware platform to perform target detection tasks and image classification tasks.

[0066] In a third aspect, the present application provides a server, which may include: one or more processors, a memory, and one or more computer programs. The one or more computer programs are stored in the memory, and the one or more computer programs include instructions. When the instructions are executed by the server, the server performs the adaptive search method of the neural network as described in the first aspect and any possible implementation thereof.

[0067] In a fourth aspect, the present application provides a device having the function of implementing the adaptive search method of a neural network as described in the first aspect and any possible implementation thereof. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0068] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on a server, the server executes the adaptive search method of a neural network as described in the first aspect and any possible implementation thereof.

[0069] In a sixth aspect, the present application provides a computer program product, which, when executed on a server, enables the server to execute the adaptive search method of a neural network as described in the first aspect and any possible implementation thereof.

[0070] In a seventh aspect, a circuit system is provided, the circuit system comprising a processing circuit, the processing circuit being configured to execute the adaptive search method of a neural network as described in the first aspect and any possible implementation thereof.

[0071] In an eighth aspect, an embodiment of the present application provides a communication system comprising at least one server and at least one terminal device, wherein at least one server executes the adaptive search method for a neural network as described in the first aspect and any possible implementation thereof, and at least one terminal device sends a search condition set to at least one server, wherein the search condition set includes: target hardware platform information, network structure information of a source neural network, and one or more evaluation indicators, wherein the network structure information of the source neural network is used to characterize the source neural network; and at least one terminal device receives a trained target neural network sent by at least one server.

[0072] In one possible implementation, at least one terminal device is further used to confirm whether to send the trained target neural network to a user device including a target hardware platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a schematic diagram of an application scenario of an adaptive search method of a neural network provided in an embodiment of the present application;

[0074] Figure 2 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0075] Figure 3 is a schematic diagram of the hardware structure of a chip provided in an embodiment of the present application;

[0076] Figure 4 This is a schematic diagram of a process of an adaptive search method of a neural network provided in an embodiment of the present application. Figure 1 ;

[0077] Figure 5 This is a schematic diagram of the network structure of the source neural network provided in the embodiment of the present application. Figure 1 ;

[0078] Figure 6 Schematic diagram of the network structure of the source neural network provided in the embodiment of the present application Figure 2 ;

[0079] Figure 7 Schematic diagram of the construction process of the super network provided in the embodiment of the present application Figure 1 ;

[0080] Figure 8 Schematic diagram of the construction process of the super network provided in the embodiment of the present application Figure 2 ;

[0081] Fig. 9 A schematic diagram of the network structure of a super network provided in an embodiment of the present application;

[0082] Fig.10This is a schematic diagram of a process of an adaptive search method of a neural network provided in an embodiment of the present application. Figure 2 ;

[0083] Fig.11 A schematic diagram of a process for obtaining a candidate subnetwork to be evaluated provided in an embodiment of the present application;

[0084] Fig.12 A schematic diagram of a process for acquiring a target candidate subnetwork provided in an embodiment of the present application;

[0085] Fig.13 This is a schematic diagram of a process of an adaptive search method of a neural network provided in an embodiment of the present application. Figure 3 ;

[0086] Fig.14 A schematic diagram of the structure of the device provided in the embodiment of the present application;

[0087] Fig.15 A schematic diagram of the hardware structure of the device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] The following is a detailed description of the adaptive search method and device for a neural network provided in an embodiment of the present application in conjunction with the accompanying drawings.

[0089] Figure 1 A neural network adaptive search system is shown, and the neural network adaptive search system includes a data processing device and a terminal device. The data processing device and the terminal device can be connected via a wired network or a wireless network. The embodiment of the present application does not specifically limit the connection method between the devices.

[0090] The above-mentioned terminal device refers to a client device, which can be a terminal device operated by a developer, and provide a relevant human-computer interaction interface so that the user can input the relevant parameters required in the adaptive search process of the neural network, such as a search condition set. Exemplarily, the search condition set may include target hardware platform information (such as the manufacturer, attributes, model, type of target task to be performed, etc. of the target hardware platform), network structure information of the source neural network, one or more evaluation indicators (such as accuracy threshold, time overhead threshold, storage space threshold, etc.), etc. Exemplarily, the terminal device can be a desktop, laptop, handheld computer, notebook computer, etc. The embodiment of the present application does not impose any special restrictions on the specific form of the terminal device.

[0091] The hardware platform may include one or more of the following: graphics processing unit (GPU), central processing unit (CPU), image signal processor (ISP), neural-network processing units (NPU), digital signal processor (DSP), etc. The applications that can be realized by running or carrying neural networks include but are not limited to: image classification, object recognition, action recognition, pose estimation, neural style transfer and other computer vision applications, and can also be natural language processing (NLP) applications.

[0092] In one scenario, when a neural network (i.e., source neural network) in a terminal device needs to be optimized, the terminal device can input the network structure information of the source neural network and the specific hardware platform (i.e., target hardware platform) information of the source neural network into a data processing device for processing to obtain an optimized neural network (i.e., target neural network). That is, the source neural network and the target neural network are neural networks running on the same hardware platform in different time domains.

[0093] In another scenario, when the hardware platform of the source neural network in the terminal device is to be replaced with other hardware platforms (including hardware platforms with different attributes (such as GPU, CPU, NPU or DSP, etc.), or hardware platforms of different models with the same attributes, or hardware platforms of different manufacturers with the same attributes), the network structure information of the source neural network and the information of the replaced hardware platform (i.e., the target hardware platform) can be input into the data processing device for processing to obtain the neural network after adaptive search (i.e., the target neural network). This is because the design of the replaced hardware platform (i.e., the target hardware platform) may be different from that of the source hardware platform, which will affect the operation effect of the source neural network. Alternatively, some designs in the replaced hardware platform may not be fully utilized by the source neural network, which will lead to a decline in network performance and waste of hardware platform design.

[0094] It is understandable that in other examples, the adaptive search system of the neural network may also include a user device. The terminal device may also confirm whether to send the network structure information of the target neural network to the user device including the target hardware platform, so that the target neural network runs or is mounted on the corresponding target hardware platform in the user device to perform the task. Exemplarily, the user device may be a mobile phone, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, an artificial intelligence (AI) terminal and other terminal devices. The embodiments of the present disclosure do not impose any special restrictions on the specific form of the user device.

[0095] The above-mentioned data processing device can be a cloud server or a network server or other device or server with a neural network search function. The data processing device receives a set of search conditions sent from the terminal device through an interactive interface, including target hardware platform information, network structure information of the source neural network, one or more evaluation indicators, etc., and then based on the knowledge base stored in the memory, the processor constructs and trains the super network and searches for the target neural network. And the search results, i.e., the target neural network, will be sent to the corresponding terminal device. The memory in the data processing device can be a general term, including local storage and a database for storing historical data. The database can be on the data processing device or on other cloud servers.

[0096] It should be noted that the data processing device can be a server, a server cluster consisting of multiple servers, or a cloud computing service center.

[0097] For example, in Figure 1 The data processing device can execute the adaptive search method of the neural network in the embodiment of the present application.

[0098] Figure 2 A system architecture provided by an embodiment of the present application is given as an example.

[0099] like Figure 2As shown in the system architecture in , the data processing device 210 is configured with an input / output (I / O) interface 211 for data interaction with an external device. The user can input data to the I / O interface 211 through the terminal device 220. The input data in the embodiment of the present application may include: a search condition set. The search condition set may include, for example: target hardware platform information (such as the manufacturer, attributes, model, and type of target task to be performed of the target hardware platform), network structure information of the source neural network, one or more evaluation indicators (such as accuracy threshold, time overhead threshold, storage space threshold, etc.), etc.

[0100] The data acquisition device 240 is used to collect training data. For example, in the embodiment of the present application, the training data may be a training data set for training a neural network. After collecting the training data, the data acquisition device 240 stores the training data in the database 230. The database 230 also stores the knowledge base 231 collected by the data acquisition module 240, and the knowledge base 231 can be used to build a super network according to the source neural network. The knowledge base 231 may include matching information between operators and / or operator sets in the neural network and hardware platforms. For example, the knowledge base 231 includes a correspondence between a first operator set based on different hardware platforms and a third operator set. Among them, the third operator set includes a first operator set and / or one or more derived second operator sets. Among them, each derived second operator set has the same function as its corresponding first operator set. That is, the first operator set can be replaced by a derived second operator set. Then, when the source neural network is expanded into a super network using the knowledge base 231, each first operator set contained in the source neural network is replaced by a third operator set corresponding to the first operator set. As shown in Table 1 below, an example of the knowledge base 231 is given.

[0101] Table 1

[0102]

[0103]

[0104] It can be seen that the first operator set may include one or more operators. Each first operator set may correspond to one or more derived second operator sets. The one or more derived second operator sets and / or the first operator set constitute the third operator set corresponding to the first operator set.

[0105] For example, in Table 1 above, a first operator set included in the source neural network is a 3×3 convolution operator. Based on the knowledge base 231, it is known that on a GPU of model A (target hardware platform), a 1×1 convolution operator and a 5×5 convolution operator both have the same or similar functions as the 3×3 convolution operator. Then the derived second operator set includes a 1×1 convolution operator and a 5×5 convolution operator. The corresponding third operator set in the super network includes the first operator set (3×3 convolution operator) and the derived second operator set (1×1 convolution operator and 5×5 convolution operator).

[0106] For example, in Table 1 above, a first operator set included in the source neural network includes a 7×7 convolution operator + pooling layer operator when stride = 1. Based on knowledge base 231, it is known that on a CPU of model B (target hardware platform), a 7×7 convolution operator when stride = 2 and a 7×7 convolution operator + pooling layer operator when stride = 1 have the same or similar functions. Then, the derived second operator set includes a 7×7 convolution operator when stride = 2, and the evaluation result of the derived second operator set is better than that of the first operator set. The first operator set can be replaced by the derived second operator set. The corresponding third operator set in the super network includes the derived second operator set (7×7 convolution operator).

[0107] For example, in Table 1 above, a first operator set included in the source neural network includes a 7×7 convolution operator. Based on the knowledge base 231, it is known that on an NPU (target hardware platform) of model C, three 3×3 convolution operators can have the same or similar functions as a 7×7 convolution operator. Then the derived second operator set includes 3×3 convolution operator + 3×3 convolution operator + 3×3 convolution operator, and the result of the derived second operator set is better than the first operator set. The corresponding third operator set in the super network includes the derived second operator set (3×3 convolution operator + 3×3 convolution operator + 3×3 convolution operator).

[0108] It should be noted that the “+” between operators in Table 1 above is used to indicate combining operator sets at adjacent levels, that is, combining operator sets distributed vertically.

[0109] It should be noted that when establishing the knowledge base 231, the corresponding relationship between the first operator set and the derived second operator set can be determined based on the data of each operator set actually running on different hardware platforms, or based on the operation status of each operator set predicted by the prior knowledge of experts. Of course, the knowledge base can also be updated according to the upgrade status of different hardware platforms. The embodiment of the present application does not specifically limit the method for establishing the knowledge base 231.

[0110] It should be noted that, in actual applications, the training data maintained in the database 230 may not necessarily come from the data collection device 240 , but may also be received from other devices, such as information sent by the terminal device 220 for expanding the knowledge base 231 .

[0111] The construction module 212 is used to construct the super network 201. Specifically, the construction module 212 constructs the super network 201 extended by the source neural network based on the network structure information of the source neural network in the search condition set and using the knowledge base 231 obtained from the database 230.

[0112] The training module 213 is used to train the super network 201 to obtain the trained super network 202. Specifically, the super network 201 sent by the construction module 212 is received, and the super network 201 is trained based on the training data set obtained from the database 230. The trained super network 202 is sent to the search module 215 to perform the neural network search process.

[0113] The evaluation result acquisition module 214 is used to obtain the evaluation results of each first operator set and / or the derived second operator set on the target hardware platform based on one or more evaluation indicators, and obtain the evaluation results of the candidate sub-network searched by the search module 215 on the target hardware platform.

[0114] The search module 215 is used to search for the target neural network 203. Specifically, the trained super network 202 sent by the training module 213 is received, and the target neural network 203 is searched from the trained super network 202 in combination with the neural network evaluation result obtained by the evaluation result acquisition module 214. The target neural network 203 is output to the terminal device 220 through the I / O interface 211.

[0115] It should also be noted that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 2 In the embodiment, the database 230 is an external memory relative to the data processing device 210. In other cases, the database 230 may also be placed in the data processing device 210.

[0116] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0117] (1) Neural Network

[0118] A neural network can be composed of neural units. A neural unit can be a sThe output of the operation unit with the intercept 1 as input can be:

[0119]

[0120] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0121] (2) Deep Neural Networks

[0122] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0123] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0124] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0125] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0126] (3) Loss Function

[0127] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0128] (4) Back propagation algorithm

[0129] The neural network can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0130] The technical solution provided in the embodiment of the present application can retain or replace some operators and / or operator sets based on the specific structure of the convolutional neural network model and combined with the functions and evaluation results of the operators and / or operator sets in each layer contained in the hidden layer of the source neural network, thereby obtaining a target neural network that is more matched with the target hardware platform.

[0131] Figure 3 The hardware structure of a chip provided in an embodiment of the present application includes a neural-network processing unit (NPU) 300. The chip can be arranged in Figure 2 The data processing device 210 shown in FIG. Figure 2 It can also be used to complete all or part of the work of the construction module 212 (e.g., constructing the super network 201), all or part of the work of the training module 213 (e.g., training the super network 201), all or part of the work of the search module 215 (e.g., searching for the target neural network 203), and all or part of the work of the evaluation structure acquisition module 214 (e.g., evaluating the searched neural network based on one or more evaluation indicators).

[0132] The neural network processor NPU300 is mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of NPU 300 is the operation circuit 303, and the controller 304 controls the operation circuit 303 to extract data from the memory (weight memory or input memory) and perform operations.

[0133] In some implementations, the operation circuit 303 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array. The operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0134] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit 303 takes the corresponding data of the matrix B from the weight memory 302 and caches it on each PE in the operation circuit 303. The operation circuit 303 takes the matrix A data from the input memory 301 and performs a matrix operation with the matrix B, and the partial result or the final result of the matrix is ​​stored in the accumulator 308 (accumulator).

[0135] The vector calculation unit 307 can further process the output of the operation circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0136] In some implementations, the vector calculation unit 307 can store the vector of processed outputs to the unified memory 306. For example, the vector calculation unit 307 can apply a non-linear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value.

[0137] In some implementations, the vector calculation unit 307 generates normalized values, merged values, or both.

[0138] In some implementations, the vector of processed outputs can be used as activation input to the computational circuit 303, e.g., for use in subsequent layers in a neural network.

[0139] The unified memory 306 is used to store input data and output data. The weight data is directly stored in the input data in the external memory to the input memory 301 and / or the unified memory 306 through the storage unit access controller 305 (direct memory access controller, DMAC), the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0140] The bus interface unit 310 (BIU) is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 309 through the bus.

[0141] The instruction fetch buffer 309 connected to the controller 304 is used to store instructions used by the controller 304. The controller 304 is used to call the instructions cached in the instruction fetch buffer 309 to control the working process of the computing accelerator.

[0142] Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0143] At present, the neural network search method in the prior art generally searches for neural network models based on floating-point operations per second (FLOPs) to compress the source neural network and obtain the network structure information of the target neural network. Among them, FLOPs is often used to estimate the operating efficiency of the hardware platform. However, since different hardware platforms have different implementations and optimizations for different operators, FLOPs does not always reflect the actual time cost (i.e. time consumption).

[0144] For example, suppose that operator A and operator B can achieve the same function. In other words, the neural network can use operator A or operator B for operation. Here, suppose operator A is a 1×1 convolution algorithm, which contains 10 million multiplications; operator B is a 3×3 convolution algorithm, which contains 15 million multiplications. It can be seen that the FLOPs of operator A is less than the FLOPs of operator B. It is generally estimated that the time consumption of operator A is less than that of operator B.

[0145] In actual applications, there may be such a situation: when running on hardware platform 1, operator A actually takes 2μs, and operator B actually takes 3μs. So, when the neural network is deployed on hardware platform 1, if the time consumption is the main consideration, it is best to use operator A. When running on hardware platform 2, since hardware platform 2 has made specific optimizations to operator B, the actual time consumption of operator B is reduced to 1.5μs, while the actual time consumption of operator A is still 2μs. So, when the neural network is deployed on hardware platform 2, if the time consumption is the main consideration, it is best to use operator B.

[0146] Therefore, the evaluation results of the target neural network on the target hardware platform (such as the evaluation results of time overhead) obtained by searching the neural network model based on FLOPs are not necessarily better than the evaluation results of the source neural network on the target hardware platform.

[0147] In view of this, an embodiment of the present application provides an adaptive search method for a neural network, which can perform an adaptive neural network search on a super network extended by a source neural network based on the evaluation indicators of the target hardware platform. Under the premise of high search flexibility, a target neural network adapted to the target hardware platform can be searched and obtained, so that the evaluation result of the searched target neural network running on the target hardware platform is better than the evaluation result of the source neural network running on the target hardware platform.

[0148] The following is a detailed description of the adaptive search method of the neural network of the embodiment of the present application in conjunction with the accompanying drawings. The adaptive search method of the neural network of the embodiment of the present application can be performed by Figure 1 Data processing equipment in Figure 2 The data processing device 210 and other devices are executed.

[0149] like Figure 4 As shown, a flowchart of an adaptive search method of a neural network is provided in an embodiment of the present application. The method may include S101-S103:

[0150] S101: Receive a search condition set.

[0151] As mentioned above, when a neural network (i.e., a source neural network) running (or carried) on a hardware platform on a terminal device needs to be optimized, or when a hardware platform on which a neural network (i.e., a source neural network) is running (or carried) on a terminal device is to be replaced with another hardware device, the network structure of the source neural network needs to be optimized based on the target hardware platform. Generally speaking, the network structure information of the source neural network is first expanded horizontally and / or vertically to form a super network. Then, based on one or more evaluation indicators, a neural network suitable for the target hardware platform is searched from the super network as a target neural network. The target neural network can be run or carried on the target hardware platform. Moreover, the evaluation result of the target neural network on the target hardware platform is better than that of the source neural network running on the target hardware platform.

[0152] First, the user needs to enter a set of search conditions, which includes: target hardware platform information, network structure information of the source neural network, and one or more evaluation indicators.

[0153] The above-mentioned target hardware platform information may include, for example, the attributes of the target hardware platform, the model of the target hardware platform, the manufacturer of the target hardware platform, the tasks performed by the target hardware platform, and other information. In this way, a target neural network that is more suitable for the current target hardware platform can be searched based on the target hardware platform information. The hardware designs of hardware platforms with the same attributes from different manufacturers, or hardware platforms with different attributes from the same manufacturer, or hardware platforms with different models of the same attributes may be different, and the advantages and disadvantages of running the same operator or the same set of operators may also be different, so different hardware platforms need to be considered in the search process. In addition, when the tasks corresponding to the target neural network running on the target hardware platform are different, the evaluation indicators considered may be different, so the tasks corresponding to the target neural network running on the hardware platform also need to be considered in the search process. For example, the image detection task (detection) is to detect objects in the image and select the location of the object. The image segmentation task (segmentation) is to refine the bounding box on the basis of the detection task, and segment the image of the identified object and the precise boundary. Obviously, the image segmentation task accuracy evaluation index is higher than the image detection task accuracy evaluation.

[0154] It is understandable that in the scenario of optimizing the neural network of a hardware platform on the terminal device, the hardware platform is the target hardware platform. In the scenario of replacing a hardware platform on the terminal device, the replaced hardware platform is the target hardware platform.

[0155] The network structure information of the above-mentioned source neural network is used to characterize the source neural network. For example, the network structure information of the source neural network may include each first operator set contained in the source neural network. Each first operator set may include one or more operators. The source neural network in the embodiment of the present application may be a convolutional neural network (CNN), a recurrent neural network (RNN), a perceptron, a feedforward neural network (FNN), a deep belief network (DBN), a generative adversarial network (GAN), etc.

[0156] Taking the source neural network as a convolutional neural network as an example, the network structure information of the source neural network is explained.

[0157] like Figure 5(a) in the figure is a schematic diagram of the network structure of a convolutional neural network. Generally speaking, a convolutional neural network includes an input layer, a hidden layer, and an output layer. The hidden layer of a convolutional neural network includes one or more convolutional layers, one or more pooling layers, and one or more fully-connected layers. Usually, one or more convolutional layers are connected to a pooling layer. Figure 5 As shown in (a) in , for example, a convolution layer may include one or more convolution operators, one convolution operator may correspond to a first operator set, or multiple convolution operators may correspond to a first operator set. A pooling layer may include one or more pooling operators, which may correspond to one or more first operator sets. A fully connected layer may include one or more fully connected layer operators, which may correspond to one or more first operator sets. For another example, Figure 5 As shown in (b) in FIG. 5 , multiple operators included in a convolutional layer and a pooling layer can constitute a first operator set.

[0158] Exemplarily, if each layer in the source neural network corresponds to a first operator set, and an operator set contains a first operator, the network structure of the source neural network can be expressed as follows: Figure 6 The network structure shown in (a) in FIG. 1 includes an input layer, a hidden layer, and an output layer. Each level in the hidden layer corresponds to a first operator set containing a single first operator. For another example, if multiple levels in the source neural network correspond to a first operator set, such as Figure 6 The network structure shown in (b) includes an input layer, a hidden layer and an output layer. The hidden layer includes two Block structures (combinations of multiple operators). Each Block structure consists of three convolution operators of a convolution layer and a summation operator of a pooling layer.

[0159] The above evaluation index may include the requirements for the network performance that the target neural network needs to have in the process of searching for the neural network model. One or more evaluation indicators include any one or more of the following: an accuracy evaluation indicator of the neural network model, a time overhead evaluation indicator of the neural network model, a storage space evaluation indicator of the neural network model, a power consumption evaluation indicator of the neural network model, an utilization evaluation indicator of the tensor computing unit of the neural network model, and a memory read and write rate evaluation indicator of the neural network model.

[0160] Among them, the accuracy evaluation indicators of the neural network model include that the accuracy of the target neural network is not less than the accuracy threshold; the time overhead evaluation indicators of the neural network model include that the time overhead of the target neural network is not greater than the time overhead threshold; the storage space evaluation indicators of the neural network model include that the storage space occupied by the target neural network is not greater than the storage space threshold; the power consumption evaluation indicators of the neural network model include that the power consumption of the target neural network is not greater than the power consumption threshold; the utilization evaluation indicators of the tensor computing unit of the neural network model include that the utilization of the tensor computing unit of the target neural network is not less than the utilization threshold of the tensor computing unit; the memory read and write rate evaluation indicators of the neural network model include that the memory read and write rate of the target neural network is not less than the memory read and write rate threshold.

[0161] It should be noted that one or more evaluation indicators can be set based on factors such as different user needs, different application scenarios, or different tasks.

[0162] Exemplarily, an accuracy threshold may be set to ensure that the accuracy of the searched target neural network is not lower than the accuracy of the source neural network or is not lower than a certain interval of the accuracy of the source neural network (for example, not lower than 99% of the accuracy of the source neural network).

[0163] As another example, the time cost threshold can be set to be less than the time cost threshold of the target neural network to increase the processing speed of the searched target neural network. Generally speaking, the smaller the time cost of the target neural network, the faster the processing speed of the target neural network. For example, the time cost threshold can be set within a certain interval of the time cost of the source neural network (for example, within 80% of the time cost of the source neural network).

[0164] For example, the image processing module in the monitor on a busy road has a large amount of information contained in the monitoring images collected per unit time, and a large number of calculations are required. In this case, the neural network installed on it needs to have a smaller time overhead to process the image more quickly. In this way, the time overhead threshold needs to be set smaller so that the searched target neural network can process a large number of monitoring images more quickly. For another example, the image processing module in the monitor in the wild with few people has a small amount of information contained in the monitoring images collected per unit time, and the calculations are small. In this case, the neural network installed on it is allowed to have a slightly higher time overhead, and the time overhead threshold can be set slightly higher.

[0165] As another example, the storage space threshold can be set according to the storage space size of the target hardware platform to avoid the searched target neural network requiring too much storage space and being unable to be stored in the target hardware platform. Alternatively, the power consumption threshold can be set according to the power consumption requirements of the target hardware platform to reduce the power consumption generated by the target neural network running in the target hardware platform and meet the overall power consumption of the target hardware platform. Alternatively, the utilization threshold of the tensor computing unit and the memory read and write rate threshold are used to measure the degree of match between the searched target neural network and the target hardware platform. The higher the utilization rate of the tensor computing unit of the searched target neural network on the target hardware platform, the higher the match between the searched target neural network and the target hardware platform, and the faster the overall processing speed. For example, when performing image processing, the higher the utilization rate of the tensor computing unit of the target hardware platform, the faster the image processing speed will be.

[0166] As another example, two or more evaluation indicators can be set at the same time. For example, the accuracy threshold and the time cost threshold can be set at the same time. For example, in some scenarios, it is necessary to obtain recognition results of a large number of images in a short time, but the accuracy of image recognition is not required. Then, a smaller accuracy threshold and a smaller time cost threshold can be set so that the searched target neural network can recognize faster than the source neural network.

[0167] S102, performing a training process on the super network to be trained based on the training data set to obtain a trained super network, wherein the super network to be trained is obtained by expanding the network structure of the source neural network.

[0168] Exemplarily, according to the network structure of the source neural network, the expansion rules in the knowledge base, and the target hardware platform information, a horizontal and / or vertical expansion method is adopted to construct a super network to be trained. Among them, the expansion rules in the knowledge base include that based on different hardware platforms, the first operator set in the source neural network and the derived second operator set have different corresponding relationships. It can be understood that the first operator set and the corresponding derived second operator set have the same or similar functions.

[0169] Specifically, based on the expansion rules of the knowledge base, find the derived second operator sets corresponding to the target hardware platform for each first operator set included in the source neural network to run (or be carried on). For example, when expanding the super network according to the knowledge base, it can be determined to retain certain first operator sets of the source neural network, or replace certain first operator sets in the source neural network with second operator sets, or skip certain levels corresponding to the first operator sets.

[0170] For example, suppose that the source neural network includes three adjacent layers, and each layer contains a first operator set of 3×3 convolution operators, that is, these three layers can be represented as 3×3 convolution operators + 3×3 convolution operators + 3×3 convolution operators. If the 5×5 convolution operator can achieve the same or similar functions as the 3×3 convolution operator on the target hardware platform, some of the 3×3 convolution operators in the three adjacent layers can be replaced with 5×5 convolution operators, and some of the 3×3 convolution operators can be replaced with skip. In this way, while reducing the overall time overhead of the three layers, it is also ensured that the replaced operator set can complete the same task as the corresponding first operator set in the source neural network. For example, the derived second operator set can be skip+3×3 convolution operator+5×5 convolution operator. In other words, in the knowledge base, the 3×3 convolution operator can correspond to the 5×5 convolution operator based on the target hardware platform. Then, when constructing a super network of the source neural network, the 3×3 convolution operator in the source neural network can be replaced by a 5×5 convolution operator.

[0171] For another example, the source neural network includes a first operator set (A+B+C), where A, B and C are different operators. According to the prior art, there are three operator sets (A1+B1+C1), (A2+B2+C2) and (A3+B3+C3) all of which have the same or similar functions as (A+B+C). However, based on expert prior knowledge, it is known that the implementation effect of (A3+B3+C3) on the target hardware platform is poor. Therefore, when establishing the knowledge base, the correspondence between the first operator set (A+B+C) and the second operator set (A3+B3+C3) on the target hardware platform is not established. Therefore, when constructing the super network of the source neural network, based on the knowledge base, the first operator set (A+B+C) can be replaced by the second operator set (A1+B1+C1) and (A2+B2+C2).

[0172] It should be noted that the expansion rules in the knowledge base include horizontal expansion and / or vertical expansion. Among them, horizontal expansion can be understood as that the first operator set and the corresponding derived second operator set have the same level, but the first operator set corresponds to multiple derived second operator sets. For example, Figure 7 As shown in (a) in the figure, the first operator set 7×7 convolution operators can derive two second operator sets at the current level, namely 3×3 convolution operators and 5×5 convolution operators, respectively. Figure 7The arrangement of the corresponding layers shown in (b) in . Among them, vertical expansion can be understood as vertically expanding certain layers in the source neural network when constructing a super network. For example, the first operator set contained in the 4 adjacent layers in the source neural network is 3×3 convolution operator (64 channels) + 3×3 convolution operator (64 channels) + 3×3 convolution operator (128 channels) + 3×3 convolution operator (128 channels). Since it is not possible to directly replace operator sets with the same convolution kernel size but different number of channels, in order to reduce the overall time overhead, the source neural network can be expanded vertically to achieve the construction of a super network. For example, the operator sets of the above four adjacent levels are vertically expanded to operator sets of six adjacent levels, such as 3×3 convolution operator (64 channels) + 3×3 convolution operator (64 channels) + 3×3 convolution operator (64 channels) + 3×3 convolution operator (128 channels) + 3×3 convolution operator (128 channels) + 3×3 convolution operator (128 channels), then the corresponding derived second operator set may include 3×3 convolution operator (64 channels) + 3×3 convolution operator (64 channels) + 3×3 convolution operator (64 channels) + 3×3 convolution operator (128 channels) + skip + skip.

[0173] The following provides two examples of expanding source neural networks into super networks to illustrate the difference between the expansion method in the prior art and the method provided in the embodiments of the present invention.

[0174] Example 1: The source neural network is a chain neural network composed of operators of smaller granularity (such as convolution operators, pooling operators, activation functions). That is, the first operator set included in the source neural network includes only one first operator.

[0175] See also Figure 7 (a) in FIG. 1 is an example of a structure of a source neural network. The structural information of the source neural network includes multiple first operator sets, and the first operator sets are: 7×7 convolution operator, pooling layer operator, 3×3 convolution operator, prelu activation function, etc.

[0176] like Figure 7 As shown in (b), it is a super network constructed using existing technology. It can be seen that all the second operator sets with the same or similar functions as the first operator set will be added to the third operator set corresponding to the position of the first operator set. For example, the 7×7 convolution operator in the source neural network is replaced by a 3×3 convolution operator, a 7×7 convolution operator and a 5×5 convolution operator. The 3×3 convolution operator in the source neural network is replaced by a 1×1 convolution operator, a 3×3 convolution operator and a 5×5 convolution operator. The prelu activation function in the source neural network is replaced by a prelu activation layer and a relu activation layer. In this way, when performing a neural network search based on a super network, the search space size is 3×3×2=18.

[0177] like Figure 7 As shown in (c), it is a super network constructed by the method provided in an embodiment of the present application, that is, the set of operators suitable for the target hardware platform is determined based on the knowledge base. For example, if it is determined that the 7×7 convolution operator in the source neural network, the corresponding 3×3 convolution operator and 5×5 convolution operator have poor operating effects on the target hardware platform, then in the knowledge base, the third operator set corresponding to the 7×7 convolution operator only includes the 7×7 convolution operator. For another example, if it is determined that the prelu activation function in the source neural network has poor operating effect on the target hardware platform, then in the knowledge base, the third operator set corresponding to the prelu activation function only includes the relu activation layer. Then, when constructing a super network based on the source neural network, it is constructed as follows. Figure 7 The super network shown in (c) in the figure. When performing a neural network search based on the super network, the search space size is 1×3×1=3.

[0178] It can be seen that in the embodiment of the present application, the knowledge base can be used to exclude the derivative operator set that is not suitable for the target hardware platform, which is conducive to reducing the scale of the super network and accelerating the training process of the super network. In addition, since the scale of the super network is reduced, the search range of the target neural network is small, which can also shorten the search process and speed up the search for the target neural network.

[0179] Example 2: The source neural network is a chain neural network composed of block structures of larger granularity, that is, the first operator set is one or more block structures, and each block structure includes multiple operators.

[0180] See also Figure 8 (a) in FIG. 1 is an example of a structure of a source neural network. The source neural network includes multiple first operator sets, each of which is a Block structure including a 1×1 convolution operator, a 3×3 convolution operator, a 1×1 convolution operator, and a summation operator, and is recorded as Block structure 1.

[0181] like Figure 8As shown in (b), it is a super network constructed by the method provided in the embodiment of the present application. For example, if it is determined based on the knowledge base that the source neural network Block structure 1 has the same or similar functions as Block structure 2 (for example, a Block structure including a 1×1 convolution operator, a 5×5 convolution operator, a 1×1 convolution operator and a summation operator) and Block structure 3 (for example, a Block structure including a 1×1 convolution operator, a 7×7 convolution operator, a 1×1 convolution operator and a summation operator), and is suitable for the target hardware platform. Then, when constructing a super network based on the source neural network, Block structure 1 can be replaced with Block structure 2 or Block structure 3, or Block structure 1 can be retained, that is, a super network as shown in FIG. Figure 8 The super network shown in (b).

[0182] like Fig. 9 As shown, an example is given of a super network that is expanded based on the target hardware platform according to the source neural network. The expanded super network can achieve the same function as the source neural network, but at this time, the accuracy may be low due to the unadjusted internal parameters, which cannot meet the user's needs, so it is a super network to be trained. Furthermore, the training data set can be used to train the super network to be trained to obtain a trained super network, so that the accuracy of the trained super network is close to the accuracy threshold, and the trained super network is used as the basis for subsequent search of the target neural network.

[0183] In some examples, when obtaining a training data set, it is also necessary to obtain a data reading method of the training data set, such as the arrangement method of the images contained in the training data set, the reading method, etc. In other examples, it is also necessary to obtain the hyperparameter setting values ​​required in the process of training the super network, such as: the preset number of iterations of training in the training process, the training duration, the target accuracy threshold of the training super network, etc. In some other examples, it is also necessary to obtain the hyperparameter setting values ​​required in the process of searching for the target neural network from the trained super network, such as the number of iterations required in the search process, the number of sub-networks required to be obtained, etc.

[0184] The following describes in detail two methods for training a super network provided in embodiments of the present application.

[0185] Method 1: Use weighted average method to train the super network.

[0186] In some examples, such as Fig. 9The extended super network shown includes four third operator sets, namely, third operator set 1 to third operator set 4, that is, all operator sets contained in a dotted box constitute a third operator set. Among them, the third operator set 1 includes a first operator set 1 and two second operator sets (second operator set 1 and second operator set 2). Weights are set for the first operator set 1, the second operator set 1, and the second operator set 2, respectively, and the sum of the weights of these three operator sets is 1. In some examples, the weights of the first operator set 1, the second operator set 1, and the second operator set 2 can be set to be the same, for example, about 0.33. By analogy, the weights of the first operator set and / or the second operator set in other third operator sets are set.

[0187] After that, the training data set is input into the expanded super network to train the super network. For example, the training data set is input into the third operator set 1, and all the operator sets contained in the third operator set 1 will participate in the calculation of the training data set. The calculation results of all the operator sets contained in the third operator set 1 are weighted averaged to obtain the feature map output by the third operator set 1. The feature map output by the calculated third operator set 1 is input into the third operator set 2, and all the operator sets contained in the third operator set 2 continue to perform weighted average calculation. And so on, after obtaining the calculation result of the last third operator set, the final feature map is output.

[0188] After that, the final feature map is input into the loss function, and the expanded super network is iteratively trained again. During the iterative training process, the internal parameters of the expanded super network are adjusted.

[0189] Finally, when the preset number of iterations, training duration, or target accuracy threshold requirement of the training super network is reached, the training is stopped. At this time, the super network is a trained super network, and the search process of the target neural network can be performed from the trained super network.

[0190] Method 2: Use onehot sampling method to train the super network.

[0191] In some examples, a first operator set or a derived second operator set is obtained from each third operator set in the expanded super network using a onehot sampling method to form a sub-network to be trained. The training data set is input into the sub-network to be trained to train the sub-network.

[0192] For example, Fig. 9The extended super network shown includes four third operator sets, namely, the third operator set 1 to the third operator set 4, that is, all operator sets contained in a dotted box constitute a third operator set. Among them, the third operator set 1 includes a first operator set 1 and two second operator sets (second operator set 1 and second operator set 2). The third operator set 2 includes a first operator set 2 and two second operator sets (second operator set 3 and second operator set 4). For example, using the onehot sampling method, the first operator set 1 in the third operator set 1 is extracted, and the second operator set 3 in the third operator set 2 is extracted. The training data set is input into the first operator set 1 in the third operator set 1 for calculation, and the feature map is output to the second operator set 3 in the third operator set 2 for calculation. By analogy, after obtaining the calculation result of the last third operator set, the final feature map is output and input into the loss function. The subnetwork is iteratively trained, and the internal parameters of the subnetwork to be trained are adjusted during the iterative training process.

[0193] Through the above method, after each sampling using the onehot sampling method, a sub-network to be trained can be obtained for training and its internal parameters can be adjusted.

[0194] Finally, when the preset number of sampling times, the preset number of iterative training times, the training time, or the target accuracy threshold requirement of the training super network is reached, the training is stopped. At this time, the super network is a trained super network, and the search process of the target neural network can be executed from the trained super network.

[0195] It should be noted that the expanded super network can be trained by any of the above methods to obtain a trained super network. The expanded super network can also be trained by the above method one first, and then trained again by the training method in method two after the training is completed to obtain a trained super network.

[0196] S103: Perform a sub-network search process on the trained super network based on one or more evaluation indicators to obtain network structure information of the target neural network, wherein the network structure information of the target neural network is used to characterize the target neural network.

[0197] In some embodiments, after obtaining a trained super network, it is also necessary to obtain the evaluation results of the first operator set and / or the derived second operator set in each third operator set in the trained super network on the target hardware platform based on one or more evaluation indicators. A first operator set or a derived second operator set is extracted from each third operator set in the trained super network to form a candidate subnetwork to be evaluated. Multiple extractions are performed to obtain multiple candidate subnetworks to be evaluated. Then, based on the evaluation results of the first operator set and / or the derived second operator set contained in the candidate subnetwork to be evaluated on the target hardware platform based on one or more evaluation indicators, one or more target neural networks are determined. When the one or more target neural networks are running on the target hardware platform, the overall evaluation results based on one or more evaluation indicators are better than the overall evaluation results of the source neural network when it is running on the target hardware platform.

[0198] In other words, after obtaining the trained super network, the corresponding relationship information between the target hardware platform, the network structure of the M neural networks, the N evaluation indicators, and the P evaluation results is obtained. Wherein, M, N and P are all positive integers, and P = M*N. The M neural networks are subnetworks extracted from the trained super network, that is, the candidate subnetworks to be evaluated. Obviously, the network structure of the M neural networks is included in the network structure of the trained super network. In this way, the evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set in the trained super network can be determined by the above-mentioned corresponding relationship (the evaluation results corresponding to the N evaluation indicators respectively), and the evaluation results of the candidate subnetwork to be evaluated can be obtained by adding the evaluation results of the first operator set and / or each derived second operator set included in each candidate subnetwork to be evaluated. Obviously, one candidate subnetwork corresponds to N evaluation results based on N evaluation indicators, and M candidate subnetworks correspond to M*N evaluation results based on N evaluation indicators, that is, P evaluation results. Then, based on the P evaluation results, one or more candidate sub-networks are determined as the target neural network.

[0199] For example, when N evaluation indicators are time cost evaluation indicators of a neural network model (ie, N=1), the time cost evaluation indicators of the neural network model include that the time cost of the target neural network is not greater than a time cost threshold.

[0200] Correspondingly, a subnetwork evaluation is performed on the M candidate subnetworks included in the trained super network based on the time overhead evaluation index of the neural network model: first, the time overhead of each first operator set and / or derived second operator set contained in the M candidate subnetworks running on the target hardware platform is obtained, that is, the evaluation result under the time overhead index. Then, based on the network structure of each candidate subnetwork in the M candidate subnetworks, the time overhead of each candidate subnetwork running on the target hardware platform is determined, that is, the evaluation result of each candidate subnetwork based on the time overhead index. Based on the time overhead evaluation index and the time overhead of the M candidate subnetworks running on the target hardware platform, one or more candidate subnetworks are determined from the M candidate subnetworks as the target neural network. Among them, the time overhead of the target neural network is not greater than the time overhead threshold.

[0201] The time overhead of each candidate sub-network running on the target hardware platform is the sum of the time overhead of each first operator set and / or derived second operator included in each candidate sub-network running on the target hardware platform.

[0202] For another example, when the N evaluation indicators are the accuracy evaluation indicator and the time cost evaluation indicator of the neural network model (i.e., N=2), the accuracy evaluation indicator of the neural network model includes that the accuracy of the target neural network is not less than the accuracy threshold. The time cost evaluation indicator of the neural network model includes that the time cost of the target neural network is not greater than the time cost threshold.

[0203] Correspondingly, the M candidate subnetworks included in the trained super network are evaluated based on the accuracy evaluation index and time cost evaluation index of the neural network model: first, the accuracy and time cost of each first operator set and / or derived second operator set contained in the M candidate subnetworks running on the target hardware platform are obtained, that is, the accuracy evaluation result under the accuracy evaluation index and the time cost evaluation result under the time cost evaluation index. Then, according to the network structure of each candidate subnetwork in the M candidate subnetworks, the accuracy and time cost of each candidate subnetwork running on the target hardware platform are determined, that is, the accuracy evaluation result under the accuracy evaluation index and the time cost evaluation result under the time cost index of each candidate subnetwork. Based on the accuracy evaluation index, the time cost evaluation index, the accuracy of the M candidate subnetworks running on the target hardware platform, and the time cost of the M candidate subnetworks running on the target hardware platform, one or more candidate subnetworks are determined from the M candidate subnetworks as the target neural network. Among them, the accuracy of the target neural network is not less than the accuracy threshold, and the time cost is not greater than the time cost threshold.

[0204] In the above, obtaining the evaluation results of each first operator set or each derived second operator set on the target hardware platform based on one or more evaluation indicators may adopt an actual measurement method or a prediction method.

[0205] In some examples of this embodiment, the trained super network can be sent to a terminal device where a target neural network needs to be deployed, and the trained super network can be run on a target hardware platform of the terminal device to obtain evaluation results of each first operator set and / or derived second operator set on the target hardware platform, that is, to obtain measured evaluation results.

[0206] For example, during actual measurement, one first operator set or one second operator set can be selected from each third operator set in the trained super network to form a sub-network. Multiple extractions are performed to extract multiple sub-networks from the trained super network. Multiple sub-networks are respectively run on the target hardware platform to obtain evaluation results of the multiple sub-networks on the target hardware platform. Based on the evaluation results of the multiple sub-networks on the target hardware platform, and the first operator set and / or the second operator set contained in the multiple sub-networks, the evaluation results of each first operator set and each second operator set on the target hardware platform are calculated.

[0207] In other examples of this embodiment, the first operator set and / or the derived second operator set in each third operator set in the trained super network can be input into a prediction model of the evaluation result (the prediction model can be deployed on a data processing device or on other devices), and the evaluation results of each first operator set and / or the derived second operator set can be directly obtained, that is, the predicted evaluation result can be obtained. Among them, the establishment of the prediction model can be obtained by training based on the evaluation results of a large number of operators and / or operator sets on the target hardware platform. The training process of the prediction model can refer to the prior art, and the embodiments of this application will not be repeated.

[0208] It should be noted that, after the super network to be trained is obtained through the above step S102, the above step of obtaining the evaluation results of the first operator set and / or the derived second operator set on the target hardware platform based on one or more evaluation indicators can be performed. In this way, after the super network is trained, the evaluation results of each first operator set and / or the derived second operator set on the target hardware platform based on one or more evaluation indicators can be directly used to determine the network structure information of the target neural network, which is conducive to speeding up the process of searching for the target neural network. In other words, the embodiment of the present application does not limit the timing of executing the evaluation results of obtaining the first operator set and / or the derived second operator set on the target hardware platform based on one or more evaluation indicators.

[0209] Below, we describe in detail the process of performing sub-network search on a trained super network based on one or more evaluation metrics.

[0210] like Fig.10 As shown, it is a flow chart of another adaptive search method of a neural network provided by an embodiment of the present application. The method comprises: receiving a search condition set, including network structure information of a source neural network, target hardware platform information, one or more evaluation indicators, etc. Based on the network structure information and knowledge base of the source neural network, a super network is constructed and trained. For related content, please refer to the description of the related steps above, which will not be repeated here.

[0211] In some embodiments, when performing a sub-network search on a trained super network, a knowledge base can be used to narrow the search scope of the target neural network. For example, the network structure of some candidate sub-networks is first determined based on the knowledge base to have a better implementation effect on the target hardware platform, and then the candidate sub-network is selected for evaluation based on one or more evaluation indicators, saving unnecessary time for obtaining evaluation results of the candidate sub-network.

[0212] In other embodiments, in the process of performing a subnetwork search from a trained super network based on one or more evaluation indicators, the whole or part of the structure of the super network can be evaluated based on one or more evaluation indicators to obtain an evaluation result, and one or more subnetworks can be searched from the trained super network based on the evaluation result to determine the target neural network. Alternatively, several subnetworks can be extracted from the super network first, and the extracted subnetworks can be evaluated based on one or more evaluation indicators to obtain an evaluation result. Based on the obtained evaluation result, one or more of the several subnetworks are determined to be the target neural network. Alternatively, several subnetworks can be extracted from the super network first, and the extracted subnetworks can be evaluated based on one or more evaluation indicators to obtain an evaluation result, and then new subnetworks can be extracted from the trained super network based on the evaluation result. The newly extracted subnetworks are evaluated based on one or more evaluation indicators to obtain an evaluation result. Then, based on the evaluation result at this time, one or more network structures of the target neural network are determined in the re-acquired subnetworks. That is to say, the embodiment of the present application does not limit the method of extracting subnetworks from a trained super network based on one or more evaluation indicators.

[0213] In the following, two methods for performing a sub-network search process from a trained super network provided in embodiments of the present application are described in detail.

[0214] Method 1: Use differentiable methods to perform the sub-network search process.

[0215] In some examples, a super network trained by a weighted average method is obtained, and a sub-network search process is performed in the trained super network using a differentiable method.

[0216] Exemplarily, when the user inputs N evaluation indicators, he also inputs one or more proportional relationships of these N evaluation indicators. It is understandable that the proportional relationship of the N evaluation indicators reflects the relative importance of the N evaluation indicators. Obviously, the higher the proportion of the evaluation indicator, the better the evaluation effect of the searched target neural network based on the evaluation indicator on the target hardware platform. Exemplarily, if the search is applicable to the target neural network on the target hardware platform with higher precision requirements, the proportion of the precision indicator can be increased. For example, the evaluation indicators include precision indicators and time overhead indicators, then the proportion corresponding to the precision indicator is greater than the proportion corresponding to the time overhead indicator, such as the ratio between the precision indicator and the time overhead indicator is 6:4.

[0217] If the data processing device receives r proportional relationships about N evaluation indicators input by the user, the data processing device performs a pre-search for each proportional relationship, wherein r is a positive integer.

[0218] In a pre-search process, a weight is set for each evaluation result of the first operator set and / or the derived second operator set in each third operator set in the trained super network. And based on N evaluation indicators and the proportional relationship of the N evaluation indicators, the trained super network is trained again. During the training process, the weights corresponding to the evaluation results of each first operator set and each derived second operator set are adjusted. According to the weights corresponding to the evaluation results of each first operator set and each derived second operator set after adjustment, the candidate sub-network to be evaluated and the evaluation results of the candidate sub-network to be evaluated are determined.

[0219] The evaluation results of one or more candidate subnetworks to be evaluated are determined according to the r pre-search processes, and one or more target neural networks are determined according to the evaluation results of the one or more candidate subnetworks to be evaluated. In one example, the evaluation results of the multiple candidate subnetworks to be evaluated can be sorted from best to worst, and the one or more candidate subnetworks ranked first are obtained as the target neural network.

[0220] For example, assume that the N evaluation indicators include an accuracy evaluation indicator and a time cost evaluation indicator (ie, N=2). Accordingly, the proportional relationship of the N evaluation indicators includes: the proportional relationship between the accuracy evaluation indicator and the time cost evaluation indicator. For example, the user inputs a first ratio and a second ratio of the accuracy evaluation indicator and the time cost evaluation indicator.

[0221] like Fig.11As shown in (a), it is a trained super network. Corresponding weights are set for the first operator set and / or one or more derived second operator sets in each third operator set in the super network. The sum of the weights of the first operator set and / or one or more derived second operator sets in a third operator set is 1. For example, Fig.11 The super network shown in (a) includes four third operator sets, that is, all operator sets contained in a dotted box constitute a third operator set, which are third operator set 1 to third operator set 4. Among them, the third operator set 1 includes a first operator set 1 and two second operator sets (second operator set 1 and second operator set 2). Weights are set for the first operator set 1, the second operator set 1, and the second operator set 2, respectively, and the sum of the weights of these three operator sets is 1. In some examples, the weights of the first operator set 1, the second operator set 1, and the second operator set 2 can be set to the same, for example, about 0.33. By analogy, the weights of the first operator set and / or the second operator set in other third operator sets are set.

[0222] During the training process, the training data set is input into the trained super network for training. For example, the training data set is input into the third operator set 1, and the first operator set and / or the derived second operator set in the third operator set 1 will calculate the training data set to obtain the feature map output by the third operator set 1. In addition, it is also necessary to calculate the evaluation results of the third operator set 1 based on N evaluation indicators. Among them, the evaluation result of the third operator set 1 based on a certain evaluation indicator (such as time overhead) = the evaluation result of the first operator set 1 based on the evaluation indicator * W1 + the evaluation result of the second operator set 1 based on the evaluation indicator * W2 + the evaluation result of the second operator set 2 based on the evaluation indicator * W3. The feature map output by the third operator set 1 after calculation and the evaluation result of the third operator set 1 based on N evaluation indicators are input into the third operator set 2, and the calculation continues. Similarly, the evaluation result of the third operator set 2 based on N evaluation indicators = the evaluation result of the first operator set 2 based on N evaluation indicators * W4 + the evaluation result of the second operator set 3 based on N evaluation indicators * W5 + the evaluation result of the second operator set 4 based on N evaluation indicators * W6. And so on, after obtaining the calculation result of the last third operator set, the final feature map and the accumulated evaluation results based on N evaluation indicators (i.e., the overall evaluation result of the super network) are output.

[0223] Afterwards, the final feature map and the accumulated evaluation results based on N evaluation indicators are input into the loss function, and the trained super network is iteratively trained again. During the iterative training process, according to the accumulated evaluation results based on N evaluation indicators, in the back propagation process, the weights corresponding to the evaluation results of one or more first operator sets and / or derived second operator sets in each third set are adjusted. The training is stopped until the result of the loss function no longer changes or the preset number of iterative training is reached to obtain a retrained super network. A first operator set or a derived second operator set with the largest weight is selected from each third operator in the neural network to form a candidate subnetwork.

[0224] For example, Fig.11 As shown in (b) of FIG. 1 , it is a trained super network that has been iteratively trained by the above method and has reached the training stop condition under the ratio 1. For the super network at this time, the first operator set with the largest weight or the derived second operator set is extracted from each third operator set to form a candidate subnetwork. For example, in Fig.11 In (b), the multiple second operator sets and the first operator set selected by the solid line boxes connected by arrows constitute a candidate subnetwork, which is recorded as candidate subnetwork 1 to be evaluated. Similarly, Fig.11 As shown in (c) in FIG. 2 , it is a trained super network that has been iteratively trained by the above method and has reached the condition for stopping training under the ratio relationship 2. Fig.11 In (c), the first operator set and multiple second operator sets selected by the solid box connected by arrows constitute a candidate sub-network, which is recorded as candidate sub-network 2 to be evaluated.

[0225] A neural network with a better evaluation result is selected from candidate sub-network 1 to be evaluated and candidate sub-network 2 to be evaluated as the searched target neural network.

[0226] Method 2: Use evolutionary methods to perform the target neural network search process.

[0227] In some examples, a super network trained by a onehot sampling method is obtained, and a sub-network search process is performed in the trained super network using an evolutionary method. Alternatively, a super network first trained by a weighted average method and then trained by a onehot sampling method is obtained, and a sub-network search process is performed in the trained super network using an evolutionary method.

[0228] Exemplarily, one or more candidate subnetworks to be evaluated are extracted from a trained super network based on preset rules, and the evaluation results of each candidate subnetwork to be evaluated based on N evaluation indicators are determined according to the network structure of each candidate subnetwork to be evaluated, and the evaluation results of the first operator set and / or the second operator set in each third operator set based on N evaluation indicators. The evaluation results of each candidate subnetwork to be evaluated based on the N evaluation indicators are sorted from best to worst to obtain K first target candidate subnetworks corresponding to the K evaluation results that are sorted first. Among them, the preset rule can be random extraction, or it can be to give priority to the first operator set or the second operator set with small FLOPs, etc. This embodiment of the present application is not limited to this.

[0229] For example, if Fig.12 As shown in (a) in FIG. 1 , the trained super network includes, for example, four third operator sets, that is, all operator sets contained in a dotted box constitute a third operator set, which are third operator set 1 to third operator set 4. Third operator set 1 includes first operator set 1, second operator set 1 and second operator set 2. First, a first operator set or a derived second operator set is randomly extracted from each third operator set to form a target candidate subnetwork. Multiple extractions are performed to obtain multiple target candidate subnetworks. For example, Fig.12 As shown in (b) of FIG. 1 , the neural network composed of multiple second operator sets and one first operator set selected by the solid line box connected by the arrow is a candidate sub-network to be evaluated. For another example: Fig.12 As shown in (c), the network consisting of a first operator set and multiple second operator sets selected by the solid line box connected by arrows is another extracted candidate sub-network to be evaluated.

[0230] Then, according to the trained super network and K first target candidate sub-networks, S second target candidate sub-networks after the nth round of hybridization and / or mutation are obtained; wherein n is the preset number of screening times, n≥1; K and S are positive integers, S>K.

[0231] For example, in Fig.12 In (b), the neural network composed of the first operator set and the second operator set connected by the arrow is a first target candidate subnetwork, denoted as first target candidate subnetwork 1. Fig.12In (c), the neural network composed of the first operator set and the second operator set connected by the arrow is another first target candidate subnetwork, denoted as the first target candidate subnetwork 2. Moreover, the first target candidate subnetwork 1 and the first target candidate subnetwork 2 are the two first target candidate subnetworks ranked first among the K first target candidate subnetworks. In this way, the first target candidate subnetwork 1 and the first target candidate subnetwork 2 are hybridized. For example, the second operator set 1 in the third operator set 1 in the first target candidate subnetwork 1 can be replaced with the first operator set 1 in the third operator set 1 in the first target candidate subnetwork 2 to obtain a second target candidate subnetwork (including the first operator set 1-the second operator set 4-the second operator set 6-the first operator set 3).

[0232] The above-mentioned variation refers to replacing the first operator set or the second operator set included in the third operator set in the K first target candidate sub-networks with any first operator set or second operator set in the corresponding third operator set in the trained super network to obtain the second target candidate sub-network.

[0233] For example, in Fig.12 In (b), the neural network composed of the first operator set and the second operator set connected by the arrow is a first target candidate subnetwork, denoted as first target candidate subnetwork 1. And the first target candidate subnetwork 1 is the first target candidate subnetwork ranked first among the K first target candidate subnetworks. In this way, the first target candidate subnetwork 1 is based on Fig.12 The trained super network shown in (a) is mutated. For example, the second operator set 1 in the third operator set 1 in the first target candidate subnetwork 1 is replaced with the second operator set 2 included in the third operator set 1 in the trained super network, and a second target candidate subnetwork (including the second operator set 2-the second operator set 4-the second operator set 6-the first operator set 3) is obtained.

[0234] It is understandable that K first target candidate subnetworks may be hybridized to obtain S second target candidate subnetworks. Alternatively, K first target candidate subnetworks may be mutated to obtain S second target candidate subnetworks. Alternatively, K first target candidate subnetworks may be hybridized and mutated to obtain S second target candidate subnetworks. The embodiment of the present application does not specifically limit the ratio of the number of hybridizations or mutations.

[0235] Next, according to the network structure of the second target candidate subnetwork and the evaluation results of the first operator set and / or the second operator set in each third operator set based on N evaluation indicators, the evaluation results of the S second target candidate subnetworks are obtained and sorted from best to worst, and K second target candidate subnetworks corresponding to the K evaluation results that are sorted first are obtained.

[0236] Finally, after (n+1) rounds of iterative training, that is, after n rounds of hybridization and / or mutation, K*(n+1) target candidate sub-networks are obtained. The K*(n+1) target candidate sub-networks can be sorted again according to one or more evaluation indicators to obtain the network structure information of one or more target neural networks.

[0237] For example, if S=10, K=5, then 10 first target candidate subnetworks will be extracted from the trained super network for the first time, and 10 target candidate subnetworks will be obtained after each round of hybridization and / or mutation, and the best 5 target candidate subnetworks will be retained for each screening and training. Since hybridization and / or mutation may cause the target candidate subnetwork after the previous round of screening to disappear, it is possible that during the training process, a target candidate subnetwork that disappears is the neural network with the best network performance. Therefore, all target candidate subnetworks ranked first in the evaluation results obtained each time will eventually be sorted again, and one or more better target candidate subnetworks will be selected as the target neural network. Assuming n=2, that is, iterative training is performed 3 times and after 2 rounds of hybridization and / or mutation, the network structure information of the target neural network is finally determined among 5×(2+1)=15 target candidate subnetworks.

[0238] Therefore, the adaptive neural network search method provided in the embodiment of the present application can perform an adaptive neural network search on the super network extended by the source neural network based on the evaluation indicators of the target hardware platform, and can search for the target neural network adapted to the target hardware platform under the premise of high search flexibility, so that the evaluation result of the searched target neural network running on the target hardware platform is better than the evaluation result of the source neural network running on the target hardware platform.

[0239] In some embodiments, after the above step S103, Fig.13 As shown, the adaptive search method of the neural network may further include S201-S202:

[0240] S201. Construct a target neural network based on the network structure information of the target neural network.

[0241] Exemplarily, as described above, the network structure information of the target neural network used to characterize the target neural network is obtained, such as the information of the first operator set and / or the derived second operator set, the arrangement of the first operator set and / or the derived second operator set, and the number of layers of the target neural network, etc. According to the network structure information of the target neural network, the target neural network is constructed.

[0242] S202: Execute a training process on the target neural network based on the second training data set to obtain a trained target neural network.

[0243] The second training data set may be the same as or different from the training data set used to train the super network in step S102.

[0244] In some embodiments, the constructed target neural network is a neural network searched based on the trained super network, which can achieve the same function as the source neural network, and the evaluation result on the target hardware platform is better than the evaluation result of the source neural network on the target hardware platform. However, at this time, since its internal parameters have not been adjusted to the optimal state, problems may occur in the actual application process, so it is necessary to use the second training data set to train the target neural network, adjust the parameter information in the target neural network, and obtain a trained target neural network that can be actually used to perform the target task.

[0245] For example, preset conditions can be set according to the evaluation index, and the target neural network can be iteratively trained using the second training data set. During the iterative training process, the internal parameters of the target neural network are adjusted. The neural network that reaches the preset number of training times or when the loss function does not decrease is regarded as a trained target neural network that can be actually applied to the target hardware platform.

[0246] For example, the trained target neural network can perform target tasks in the field of computer vision (CV) on the target hardware platform, such as target detection tasks and image classification tasks. Specifically, target detection tasks may include, for example, portrait detection, window face detection, human face detection, image pattern detection, machine-non-human detection, license plate detection, etc. Image classification tasks may include, for example, traffic light recognition, vehicle model recognition, special vehicle model recognition, etc.

[0247] In this way, a target neural network can be obtained that can actually execute the target task on the target hardware platform.

[0248] An embodiment of the present application provides a communication system, characterized in that it includes at least one server and at least one terminal device.

[0249] At least one server executes Figure 4 Steps S101 to S103 in the above embodiment, and executing Fig.13 Steps S201-S202 in, and / or other processes for the technology described herein. At least one terminal device sends a search condition set to at least one server, the search condition set including: target hardware platform information, network structure information of the source neural network and one or more evaluation indicators, the network structure information of the source neural network is used to characterize the source neural network. At least one terminal device receives a trained target neural network sent by at least one server.

[0250] Optionally, at least one terminal device is also used to confirm whether to send the trained target neural network to a user device including a target hardware platform. The user device is used to apply the target neural network, and the user device can be a mobile phone, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, an artificial intelligence (AI) terminal, and other terminal devices. The embodiments of the present disclosure do not impose any special restrictions on the specific form of the user device.

[0251] Fig.14 A possible structural diagram of the adaptive search device for a neural network involved in the above embodiment is shown. The device includes: a receiving unit 1401, a construction unit 1402, a training unit 1403 and a search unit 1404.

[0252] The receiving unit 1401 is used to support the device to execute Figure 4 Step S101 in and / or other processes for the technology described herein.

[0253] Construction unit 1402, used to support the device to execute Figure 4 In step S102, Fig.13 In step S201, Figure 2 Steps performed by the building block 212 in, and / or other processes for the techniques described herein.

[0254] Training unit 1403, used to support the device to perform Figure 4 In step S102, Fig.13 In step S202, Figure 2 The training module 213 in, and / or other processes for the techniques described herein.

[0255] Search unit 1404 is used to support the device to perform Figure 4 In step S103, Figure 2 Steps performed by the search module 215 in, and / or other processes for the technology described herein.

[0256] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional unit, and will not be repeated here.

[0257] Fig.15 The hardware structure diagram of the device provided in the embodiment of the present application is shown in FIG. The device includes at least one processor 1501, a communication line 1502, a memory 1503 and at least one communication interface 1504. The memory 1503 may also be included in the processor 1501.

[0258] Processor 1501 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0259] The communication link 1502 may include a pathway to transmit information between the above-mentioned components.

[0260] Communication interface 1504, used to communicate with other devices. In the embodiment of the present application, the communication interface can be a module, a circuit, a bus, an interface, a transceiver or other device capable of implementing a communication function, used to communicate with other devices. Optionally, when the communication interface is a transceiver, the transceiver can be an independently arranged transmitter, which can be used to send information to other devices, and the transceiver can also be an independently arranged receiver, which is used to receive information from other devices. The transceiver can also be a component that integrates the functions of sending and receiving information, and the embodiment of the present application does not limit the specific implementation of the transceiver.

[0261] The memory 1503 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via the communication line 202. The memory may also be integrated with the processor.

[0262] The memory 1503 is used to store computer-executable instructions for implementing the solution of the present application, and the execution is controlled by the processor 1501. The processor 1501 is used to execute the computer-executable instructions stored in the memory 1503, thereby implementing the neural network adaptive search method provided in the following embodiments of the present application.

[0263] Optionally, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, instructions, computer program or other names, which are not specifically limited in the embodiments of the present application.

[0264] In a specific implementation, as an embodiment, the processor 1501 may include one or more CPUs, such as Fig.15 CPU0 and CPU1 in.

[0265] In a specific implementation, as an embodiment, the device may include multiple processors, such as Fig.15 1501 and 1505 in the embodiment of the present invention. Each of these processors may be a single-CPU processor or a multi-CPU processor. The processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0266] It should be noted that the above-mentioned device can be a general device or a special device, and the embodiment of the present application does not limit the type of the device. The structure illustrated in the embodiment of the present application does not constitute a specific limitation on the device. In other embodiments of the present application, the device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0267] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a server, the server executes the above-mentioned related method steps to implement the adaptive search method of the neural network in the above-mentioned embodiment.

[0268] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement the adaptive search method of the neural network in the above-mentioned embodiment.

[0269] In addition, an embodiment of the present application also provides a device, which may specifically be a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor may execute the computer-executable instructions stored in the memory, so that the device executes the adaptive search method of the neural network in the above-mentioned method embodiments.

[0270] Among them, the device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0271] Through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0272] In the several embodiments provided in the present application, it should be understood that the disclosed method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of modules or units, which can be electrical, mechanical or other forms.

[0273] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0274] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0275] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk and other media that can store program instructions.

[0276] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A neural network adaptive search method, characterized in that: The method comprises: Receive a search condition set, the search condition set comprising: target hardware platform information, network structure information of a source neural network, and one or more evaluation indicators, the network structure information of the source neural network being used to characterize the source neural network; The source neural network is expanded according to the network structure of the source neural network, the expansion rules in the knowledge base and the target hardware platform information to obtain a super network to be trained; wherein the expansion rules in the knowledge base include: based on different hardware platforms, the first operator set in the source neural network and the derived second operator set have different corresponding relationships; based on the training data set, a training process is performed on the super network to be trained to obtain a trained super network; A sub-network search process is performed on the trained super network based on the one or more evaluation indicators to obtain network structure information of the target neural network, wherein the network structure information of the target neural network is used to characterize the target neural network, and an evaluation result of the target neural network running on the target hardware platform is better than an evaluation result of the source neural network running on the target hardware platform; the target neural network runs on the target hardware platform to perform target detection tasks and / or image classification tasks.

2. The method according to claim 1, characterized in that The one or more evaluation indicators include any one or more of the following: The accuracy evaluation index of the neural network model, the time overhead evaluation index of the neural network model, the storage space evaluation index of the neural network model, the power consumption evaluation index of the neural network model, the utilization evaluation index of the tensor computing unit of the neural network model, and the memory read and write rate evaluation index of the neural network model; Wherein, the accuracy evaluation index of the neural network model includes that the accuracy of the target neural network is not less than the accuracy threshold; The time cost evaluation index of the neural network model includes that the time cost of the target neural network is not greater than the time cost threshold; The storage space evaluation index of the neural network model includes that the storage space occupied by the target neural network is not greater than the storage space threshold; The power consumption evaluation index of the neural network model includes that the power consumption of the target neural network is not greater than the power consumption threshold; The utilization evaluation index of the tensor computing unit of the neural network model includes that the utilization of the tensor computing unit of the target neural network is not less than the utilization threshold of the tensor computing unit; The memory read and write rate evaluation index of the neural network model includes that the memory read and write rate of the target neural network is not greater than the memory read and write rate threshold.

3. The method according to claim 1 or 2, characterized in that: The performing a sub-network search process on the trained super network based on the one or more evaluation indicators to obtain network structure information of the target neural network includes: Using the first knowledge base, determining one or more candidate sub-networks to be evaluated within a target search range from the trained super network; Based on the one or more evaluation indicators, a sub-network search process is performed on one or more candidate sub-networks to be evaluated within the target search range to obtain network structure information of the target neural network.

4. The method according to claim 1 or 2, characterized in that: The performing a sub-network search process on the trained super network based on the one or more evaluation indicators to obtain network structure information of the target neural network includes: Obtaining the corresponding relationship information among the target hardware platform, the network structures of the M neural networks, the N evaluation indicators, and the P evaluation results, wherein M, N, and P are all positive integers; the network structures of the M neural networks are included in the network structure of the trained super network; Determine, based on the one or more evaluation indicators and the corresponding relationship information, an evaluation result of one or more candidate sub-networks to be evaluated included in the trained super network; A search process is performed on the trained super network based on the evaluation results of the one or more candidate sub-networks to be evaluated to obtain network structure information of the target neural network.

5. The method according to claim 1 or 2, characterized in that: The one or more evaluation indicators include a time cost evaluation indicator of a neural network model, wherein the time cost evaluation indicator of the neural network model includes that the time cost of the target neural network is not greater than a time cost threshold, Accordingly, the performing of a sub-network search process on the trained super network based on the one or more evaluation indicators includes: Based on the time cost evaluation index of the neural network model, a sub-network evaluation is performed on one or more candidate sub-networks included in the trained super network to obtain one or more target neural networks whose time cost of the neural network model is less than the time cost threshold.

6. The method according to claim 1 or 2, characterized in that: The method further comprises: According to the network structure of the source neural network, the expansion rules in the second knowledge base and the target hardware platform information, the super network to be trained is constructed by adopting a horizontal and / or vertical expansion method.

7. The method according to claim 1 or 2, characterized in that: The third operator set in the super network to be trained includes: a first operator set and / or a derived second operator set, wherein the first operator set is included in the source neural network; and the derived second operator set is an operator set having the same function as the first operator set.

8. The method according to claim 4, characterized in that The step of determining the evaluation results of the one or more candidate sub-networks to be evaluated included in the trained super network based on the one or more evaluation indicators and the corresponding relationship information includes: Based on the one or more evaluation indicators and the corresponding relationship information, determining evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set in the trained super network, wherein the first operator set is included in the source neural network; Based on the proportional relationship among the N evaluation indicators and the evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set, the evaluation results of the one or more candidate sub-networks to be evaluated are obtained, wherein the proportional relationship is determined based on a target task performed by a target hardware platform or a property of a target hardware platform.

9. The method according to claim 8, characterized in that The obtaining, based on the proportional relationship of the N evaluation indicators and the evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set, the evaluation results of the one or more candidate sub-networks to be evaluated includes: Receiving r proportional relationships of the N evaluation indicators, and performing a pre-search for each proportional relationship of the N evaluation indicators; wherein r is a positive integer; In a pre-search process, weights of evaluation results of each first operator set and / or each derived second operator set are set, and based on the N evaluation indicators and the proportional relationship of the N evaluation indicators, the trained super network is trained again, and during the training process, the weights corresponding to the evaluation results of each first operator set and each derived second operator set are adjusted; Determining a candidate subnetwork to be evaluated and an evaluation result of the candidate subnetwork to be evaluated according to the adjusted weights corresponding to the evaluation results of each of the first operator sets and each of the derived second operator sets; The evaluation results of the one or more candidate sub-networks to be evaluated are determined according to the r pre-search processes.

10. The method according to claim 8 or 9, characterized in that: The N evaluation indicators include an accuracy evaluation indicator of the neural network model and a time overhead evaluation indicator of the neural network model; accordingly, the proportional relationship between the N evaluation indicators includes: the proportional relationship between the accuracy evaluation indicator and the time overhead evaluation indicator.

11. The method according to claim 4, characterized in that Performing a search process on the trained super network based on the evaluation results of the one or more candidate sub-networks to be evaluated to obtain network structure information of the target neural network, including: Sorting the evaluation results of the one or more candidate sub-networks to be evaluated in descending order to obtain K first target candidate sub-networks corresponding to the K evaluation results that are sorted first; According to the trained super network and the K first target candidate sub-networks, S second target candidate sub-networks after the nth round of hybridization and / or mutation are obtained; wherein n is a preset number of screening times, n≥1; K and S are positive integers, S>K; Based on the one or more evaluation indicators and the corresponding relationship information, obtaining evaluation results of the S second target candidate subnetworks and sorting them in descending order to obtain K second target candidate subnetworks corresponding to the K evaluation results that are sorted first; Obtain network structure information of one or more target neural networks from K*(n+1) target candidate sub-networks.

12. The method according to claim 1 or 2, characterized in that: The source neural network is a neural network running on a source hardware platform, or the target neural network is a neural network running on the target hardware platform, and the source hardware platform is different from the target hardware platform.

13. The method according to claim 1 or 2, characterized in that: The source neural network and the target neural network are neural networks running on the target hardware platform in different time domains.

14. The method according to claim 1 or 2, characterized in that: The method further comprises: Based on the network structure information of the target neural network, construct a target neural network; A training process is performed on the target neural network based on the second training data set to obtain a trained target neural network.

15. An adaptive search device for a neural network, characterized in that: The device comprises: a receiving unit, a training unit and a searching unit; The receiving unit is used to receive a search condition set, wherein the search condition set includes: target hardware platform information, network structure information of a source neural network, and one or more evaluation indicators, wherein the network structure information of the source neural network is used to characterize the source neural network; The training unit is used to expand the source neural network according to the network structure of the source neural network, the expansion rules in the knowledge base and the target hardware platform information to obtain the super network to be trained; wherein the expansion rules in the knowledge base include: based on different hardware platforms, the first operator set in the source neural network and the derived second operator set have different corresponding relationships; based on the training data set, the training process is performed on the super network to be trained to obtain a trained super network; The search unit is used to perform a sub-network search process on the trained super network based on the one or more evaluation indicators to obtain network structure information of the target neural network, wherein the network structure information of the target neural network is used to characterize the target neural network, and an evaluation result of the target neural network running on the target hardware platform is better than an evaluation result of the target neural network running on the source hardware platform; the target neural network runs on the target hardware platform to perform target detection tasks and / or image classification tasks.

16. The device according to claim 15, characterized in that The one or more evaluation indicators include any one or more of the following: The accuracy evaluation index of the neural network model, the time overhead evaluation index of the neural network model, the storage space evaluation index of the neural network model, the power consumption evaluation index of the neural network model, the utilization evaluation index of the tensor computing unit of the neural network model, and the memory read and write rate evaluation index of the neural network model; Wherein, the accuracy evaluation index of the neural network model includes that the accuracy of the target neural network is not less than the accuracy threshold; The time cost evaluation index of the neural network model includes that the time cost of the target neural network is not greater than the time cost threshold; The storage space evaluation index of the neural network model includes that the storage space occupied by the target neural network is not greater than the storage space threshold; The power consumption evaluation index of the neural network model includes that the power consumption of the target neural network is not greater than the power consumption threshold; The utilization evaluation index of the tensor computing unit of the neural network model includes that the utilization of the tensor computing unit of the target neural network is not less than the utilization threshold of the tensor computing unit; The memory read and write rate evaluation index of the neural network model includes that the memory read and write rate of the target neural network is not greater than the memory read and write rate threshold.

17. The device according to claim 15 or 16, characterized in that The search unit is specifically used for: Using the first knowledge base, one or more candidate sub-networks to be evaluated within the target search range are determined from the trained super network; based on the one or more evaluation indicators, a sub-network search process is performed on the one or more candidate sub-networks to be evaluated within the target search range to obtain network structure information of the target neural network.

18. The device according to claim 15 or 16, characterized in that The search unit is specifically used for: Obtaining the correspondence information between the target hardware platform, the network structures of M neural networks, N evaluation indicators, and P evaluation results, wherein M, N, and P are all positive integers; the network structures of the M neural networks are included in the network structure of the trained super network; based on the one or more evaluation indicators and the correspondence information, determining the evaluation results of one or more candidate sub-networks to be evaluated included in the trained super network; performing a search process on the trained super network based on the evaluation results of the one or more candidate sub-networks to be evaluated to obtain the network structure information of the target neural network.

19. The device according to claim 15 or 16, characterized in that The one or more evaluation indicators include a time cost evaluation indicator of a neural network model, wherein the time cost evaluation indicator of the neural network model includes that the time cost of the target neural network is not greater than a time cost threshold, Correspondingly, the search unit is specifically used to perform sub-network evaluation on one or more candidate sub-networks included in the trained super network based on the time overhead evaluation index of the neural network model, so as to obtain one or more target neural networks whose time overhead of the neural network model is less than the time overhead threshold.

20. The device according to claim 15 or 16, characterized in that The device also includes: A construction unit is used to construct a super network to be trained by horizontal and / or vertical expansion according to the network structure of the source neural network, the expansion rules in the second knowledge base and the target hardware platform information.

21. The device according to claim 15 or 16, characterized in that The third operator set in the super network to be trained includes: a first operator set and / or a derived second operator set, wherein the first operator set is included in the source neural network; and the derived second operator set is an operator set having the same function as the first operator set.

22. The device according to claim 18, characterized in that The search unit is specifically used for: Based on the one or more evaluation indicators and the corresponding relationship information, determining evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set in the trained super network, wherein the first operator set is included in the source neural network; Obtaining evaluation results of the one or more candidate subnetworks to be evaluated based on the proportional relationship of the N evaluation indicators and the evaluation results corresponding to each first operator set and / or each derived second operator set in each third operator set; Based on the evaluation results of the one or more candidate sub-networks to be evaluated, a search process is performed on the trained super network to obtain network structure information of the target neural network, wherein the proportional relationship is determined based on a target task performed by a target hardware platform or an attribute of a target hardware platform.

23. The device according to claim 22, characterized in that The search unit is also used for: Receive r proportional relationships of the N evaluation indicators, and perform a pre-search for each proportional relationship of the N evaluation indicators; wherein r is a positive integer; in one pre-search process, set the weights of the evaluation results of each first operator set and / or each derived second operator set, and re-train the trained super network based on the N evaluation indicators and the proportional relationship of the N evaluation indicators, and in the training process, adjust the weights corresponding to the evaluation results of each first operator set and each derived second operator set; determine the candidate subnetwork to be evaluated and the evaluation results of the candidate subnetwork to be evaluated according to the adjusted weights corresponding to the evaluation results of each first operator set and each derived second operator set; determine the evaluation results of one or more candidate subnetworks to be evaluated according to the r pre-search processes.

24. The device according to claim 22 or 23, characterized in that The N evaluation indicators include an accuracy evaluation indicator of the neural network model and a time overhead evaluation indicator of the neural network model; accordingly, the proportional relationship between the N evaluation indicators includes: the proportional relationship between the accuracy evaluation indicator and the time overhead evaluation indicator.

25. The device according to claim 18, characterized in that The search unit is also used for: The evaluation results of the one or more candidate subnetworks to be evaluated are sorted in descending order to obtain K first target candidate subnetworks corresponding to the K evaluation results that are sorted first; based on the trained super network and the K first target candidate subnetworks, S second target candidate subnetworks after the nth round of hybridization and / or mutation are obtained; wherein n is the preset number of screening times, n≥1; K and S are positive integers, S>K; based on the one or more evaluation indicators and the corresponding relationship information, the evaluation results of the S second target candidate subnetworks are obtained and sorted in descending order to obtain K second target candidate subnetworks corresponding to the K evaluation results that are sorted first; and the network structure information of one or more target neural networks is obtained from the K*(n+1) target candidate subnetworks.

26. The device according to claim 15 or 16, characterized in that The source neural network is a neural network running on a source hardware platform, or the target neural network is a neural network running on the target hardware platform, and the source hardware platform is different from the target hardware platform.

27. The device according to claim 15 or 16, characterized in that The source neural network and the target neural network are neural networks running on the target hardware platform in different time domains.

28. The device according to claim 15 or 16, characterized in that The construction unit is also used to construct a target neural network based on the network structure information of the target neural network; The training unit is also used to perform a training process on the target neural network based on a second training data set to obtain a trained target neural network.

29. A server, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions; when the instructions are executed by the server, the server executes the adaptive search method of the neural network according to any one of claims 1 to 14.

30. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on a server, enable the server to execute the adaptive search method of a neural network according to any one of claims 1 to 14.

31. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the adaptive search method for a neural network according to any one of claims 1 to 14.

32. A communication system, characterized in that: The method comprises at least one server and at least one terminal device, wherein the at least one server executes the adaptive search method of the neural network according to any one of claims 1 to 14; The at least one terminal device sends a search condition set to the at least one server, the search condition set comprising: target hardware platform information, network structure information of a source neural network, and one or more evaluation indicators, the network structure information of the source neural network being used to characterize the source neural network; The at least one terminal device receives the trained target neural network sent by the at least one server.

Images