Neural network architecture search method, image processing method, device and storage medium

Abstract

The present application provides a search method, an image processing method, a device and a storage medium for a neural network architecture. It relates to the field of artificial intelligence, and specifically to the field of computer vision. The method includes: determining a search space and a plurality of building blocks, and stacking the plurality of building blocks to obtain an initial neural network architecture of the first stage, and then optimizing the initial neural network architecture of the first stage until convergence; after obtaining the optimized initial neural network architecture of the first stage, optimizing the initial neural network architecture of the second stage until convergence to obtain the optimized building blocks, and finally building a target neural network based on the optimized building blocks. Among them, each edge in the initial neural network architecture of the first stage and the initial neural network architecture of the second stage corresponds to a mixed operator consisting of a class of operations and multiple operations, respectively, and the present application can search for a target neural network with better performance.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a search method, image processing method, device and storage medium for a neural network architecture. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] With the rapid development of artificial intelligence technology, neural networks (e.g., deep neural networks) have made great achievements in the processing and analysis of various media signals such as images, videos, and voices in recent years. A neural network with excellent performance often has a sophisticated network structure, which requires human experts with superb skills and rich experience to spend a lot of effort to build. In order to better build neural networks, people have proposed to build neural networks through neural architecture search (NAS) methods, and automatically search for neural network architectures to obtain neural network architectures with excellent performance. Therefore, how to obtain a neural network architecture with better performance through neural network architecture search is a relatively important issue. Summary of the invention

[0004] The present application provides a neural network architecture search method, image processing method, device and storage medium to search for a neural network with better performance.

[0005] In a first aspect, a method for searching a neural network architecture is provided, the method comprising: determining a search space and a plurality of building units; stacking the plurality of building units to obtain an initial neural network architecture of a first stage; then, optimizing the initial neural network architecture of the first stage until convergence to obtain the optimized initial neural network architecture of the first stage; obtaining the initial neural network architecture of the second stage, and optimizing the initial neural network architecture of the second stage until convergence to obtain the optimized building units; and finally building a target neural network based on the optimized building units.

[0006] The search space includes multiple groups of candidate operators, each group of candidate operators includes at least one operator, and the operators included in each group of candidate operators are of the same type (at least one operator in each group of operators belongs to the same type of operator).

[0007] Each of the multiple building blocks is a network structure obtained by connecting multiple nodes through basic operators of the neural network, and the connection between the nodes of each of the multiple building blocks forms an edge.

[0008] The structures of the initial neural network architecture of the first stage and the initial neural network architecture of the second stage are the same. Specifically, the types and quantities of building units included in the initial neural network architecture of the first stage are the same as the types and quantities of building units included in the initial neural network architecture of the second stage, and the structure of the i-th building unit in the initial neural network architecture of the first stage is exactly the same as the structure of the i-th building unit in the initial neural network architecture of the second stage, and i is a positive integer.

[0009] The difference between the initial neural network architecture of the first stage and the initial neural network architecture of the second stage is that the alternative operators corresponding to the corresponding edges in the corresponding building units are different.

[0010] Specifically, each edge of each building unit in the initial neural network architecture of the first stage corresponds to a plurality of candidate operators, and each of the plurality of candidate operators corresponds to a group from a plurality of groups of candidate operators.

[0011] Optionally, the search space is composed of M groups of candidate operators (the search space includes M groups of candidate operators in total), and each edge of each building unit in the initial neural network architecture of the first stage corresponds to M candidate operators, and the M candidate operators are respectively from the M groups of candidate operators in the search space.

[0012] That is, one candidate operator is selected from each group of the M groups of candidate operators, thereby obtaining M candidate operators. The above M is an integer greater than 1.

[0013] For example, the above search space includes a total of 4 groups of alternative operators. Then, each edge of each building unit in the initial neural network architecture of the first stage can correspond to 4 alternative operators, and the 4 alternative operators are respectively from the above 4 groups of alternative operators (select 1 alternative operator from each group of alternative operators to obtain the 4 alternative operators).

[0014] The mixed operator corresponding to the jth edge in the ith building unit in the initial neural network architecture of the second stage is composed of all operators in the kth group of candidate operators in the initial neural network architecture optimized in the first stage, and the kth group of candidate operators is a group of candidate operators with the largest weight among multiple candidate operators corresponding to the jth edge in the ith building unit in the initial neural network architecture optimized in the first stage, and i, j and k are all positive integers;

[0015] The above-mentioned optimized building unit can be called an optimal building unit, and the optimized building unit is used to build or stack the required target neural network.

[0016] In the present application, during the process of searching for a neural network architecture, by determining which type of alternative operator should be used for each edge of each building unit in the optimization process of the first stage, and determining which specific alternative operator should be used for each edge of each building unit in the optimization process of the second stage, the problem of multicollinearity can be avoided, and a target neural network with better performance can be built based on the optimized building units.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the plurality of groups of candidate operators include:

[0018] The first group of candidate operators: 3x3 maximum pooling operation, 3x3 average pooling operation;

[0019] The second set of alternative operators: skip operations;

[0020] The third group of alternative operators: 3x3 separation convolution operation, 5x5 separation convolution operation;

[0021] The fourth group of alternative operators: 3x3 dilated separable convolution operation and 5x5 dilated separable convolution operation.

[0022] For example, for the initial neural network architecture of the first stage above, the multiple candidate operators corresponding to each edge of each building unit may include a 3x3 maximum pooling operation, a skip connection operation, a 3x3 separable convolution operation, and a 3x3 dilated separable convolution operation.

[0023] For another example, for the initial neural network architecture after optimization in the first stage, the operator with the largest weight in the j-th edge in the ith building unit is the 3x3 maximum pooling operation. Then, for the initial neural network architecture after optimization in the second stage, the alternative operator corresponding to the j-th edge in the ith building unit is a hybrid operator consisting of a 3x3 maximum pooling operation and a 3x3 average pooling operation.

[0024] Next, in the process of optimizing the initial neural network architecture of the second stage, the weights of the 3x3 maximum pooling operation and the 3x3 average pooling operation on the j-th edge in the ith building unit in the initial neural network architecture of the second stage are determined, and then the operator with the largest weight is selected as the operator on the j-th edge in the ith building unit.

[0025] In combination with the first aspect, in some implementations of the first aspect, the method further includes: clustering the multiple candidate operators in the search space to obtain the multiple groups of candidate operators.

[0026] The clustering process for the multiple candidate operators in the search space may be to divide the multiple candidate operators in the search space into different categories, and the candidate operators in each category constitute a group of candidate operators.

[0027] Optionally, the above-mentioned clustering processing of multiple candidate operators in the search space to obtain multiple groups of candidate operators includes: clustering processing of multiple candidate operators in the search space to obtain correlation relationships between multiple candidate operators in the search space; grouping multiple candidate operators in the search space according to the correlation relationships between multiple candidate operators in the search space to obtain the above-mentioned multiple groups of candidate operators.

[0028] The above correlation may be a linear correlation, which may be represented by a linear correlation degree (which may be a value between 0 and 1). A larger value of the linear correlation degree between two candidate operators indicates a closer relationship between the two candidate operators.

[0029] For example, through cluster analysis, the linear correlation between the 3x3 maximum pooling operation and the 3x3 average pooling operation is 0.9. Therefore, it can be considered that the correlation between the 3x3 maximum pooling operation and the 3x3 average pooling operation is high, and the 3x3 maximum pooling operation and the 3x3 average pooling operation can be divided into one group.

[0030] Through clustering processing, multiple candidate operators in the search space can be divided into multiple groups of candidate operators, which is convenient for subsequent optimization in the search neural network process.

[0031] In combination with the first aspect, in certain implementations of the first aspect, the above method also includes: selecting an operator from each group of candidate operators of multiple groups of candidate operators to obtain multiple candidate operators corresponding to each edge of each building unit in the initial neural network architecture of the first stage.

[0032] In combination with the first aspect, in certain implementations of the first aspect, the above method also includes: determining the operator with the largest weight in each edge of each building unit in the initial neural network architecture of the first stage; and determining a mixed operator consisting of all alternative operators in a group of alternative operators where the operator with the largest weight in the j-th edge in the ith building unit in the initial neural network architecture of the first stage is located as the alternative operator corresponding to the j-th edge in the ith building unit in the initial neural network architecture of the second stage.

[0033] In combination with the first aspect, in certain implementations of the first aspect, the above-mentioned optimization of the initial neural network architecture of the first stage until convergence to obtain an optimized building unit includes: using the same training data to optimize the network architecture parameters and network model parameters of the building unit in the initial neural network architecture of the first stage respectively until convergence to obtain the optimized initial neural network architecture of the first stage; and / or, optimizing the initial neural network architecture of the second stage until convergence to obtain the optimized building unit, including: using the same training data to optimize the network architecture parameters and network model parameters of the building unit in the initial neural network architecture of the second stage respectively until convergence to obtain the optimized building unit.

[0034] By using the same training data to optimize the network architecture parameters and network model parameters, compared with the traditional two-layer optimization method, it is possible to search for a neural network with better performance with the same amount of training data.

[0035] In a second aspect, a method for searching a neural network architecture is provided, the method comprising: determining a search space and a plurality of building units; stacking a plurality of building units to obtain a search network; optimizing the network architecture parameters and network model parameters of the building units in the search network using the same training data in the search space to obtain optimized building units; and building a target neural network based on the optimized building units.

[0036] Each of the above-mentioned multiple building blocks is a network structure obtained by connecting multiple nodes through basic operators of the neural network.

[0037] In this application, by using the same training data to optimize the network architecture parameters and the network model parameters, compared with the traditional two-layer optimization method, it is possible to search for a neural network with better performance with the same amount of training data.

[0038] In conjunction with the second aspect, in some implementations of the second aspect, the network architecture parameters and network model parameters of the building units in the search network are optimized respectively using the same training data in the search space to obtain the optimized building units, including:

[0039] Determine the optimized network architecture parameters and the optimized network model parameters of the building units in the search network using the same training data and formula;

[0040]

[0041]

[0042] Among them, α t and w t Respectively represent the network architecture parameters and network model parameters after the t-th step optimization of the building units in the search network; α t-1 and w t-1 They respectively represent the network architecture parameters and network model parameters after the optimization of the building units in the search network in the t-1th step; η t and δ t They respectively represent the learning rates of the network architecture parameters and the network model parameters when the building units in the search network are optimized in the t-th step; L train (w t-1 ,α t-1 ) represents the loss function value on the test set when the loss function is optimized in step t. represents the gradient of the loss function on the test set with respect to α when optimizing in the tth step, It represents the gradient of the loss function on the test set with respect to w during the optimization of the tth step.

[0043] In addition, the above network architecture parameter α refers to the weight coefficient of each operator, and the α value indicates the importance of the corresponding operator. w refers to the set of all other parameters in the architecture, including parameters in convolution, prediction layer parameters, etc.

[0044] In a third aspect, an image processing method is provided, the method comprising: acquiring an image to be processed; processing the image to be processed according to a target neural network to obtain a processing result of the image to be processed.

[0045] Among them, the target neural network in the above-mentioned third aspect is a neural network constructed according to any one of the implementation methods in the first aspect or the second aspect.

[0046] Since the search method of the neural network architecture in the first aspect can construct a target neural network with better performance, in the third aspect, more accurate image processing results can be obtained when the target neural network is used to process the image to be processed.

[0047] The above-mentioned processing of the image to be processed may refer to identifying, classifying, detecting, etc. the image to be processed.

[0048] In a fourth aspect, an image processing method is provided, the method comprising: acquiring an image to be processed; processing the image to be processed according to a target neural network to obtain a classification result of the image to be processed.

[0049] Among them, the target neural network in the above-mentioned fourth aspect is a target neural network constructed according to any one of the implementation methods in the first aspect or the second aspect.

[0050] In a fifth aspect, an image processing method is provided, which includes: acquiring a road picture; performing convolution processing on the road picture according to a target neural network to obtain multiple convolution feature maps of the road picture; performing deconvolution processing on the multiple convolution feature maps of the road picture according to the target neural network to obtain a semantic segmentation result of the road picture.

[0051] Among them, the target neural network in the above-mentioned fifth aspect is a target neural network constructed according to any one of the implementation methods in the first aspect or the second aspect.

[0052] In a sixth aspect, an image processing method is provided, the method comprising: obtaining a facial image; performing convolution processing on the facial image according to a target neural network to obtain a convolution feature map of the facial image; comparing the convolution feature map of the facial image with the convolution feature map of an identity document image to obtain a verification result of the facial image.

[0053] The convolution feature map of the ID card image may be obtained in advance and stored in a corresponding database. For example, the ID card image is convoluted in advance and the obtained convolution feature map is stored in the database.

[0054] In addition, the target neural network in the sixth aspect is a target neural network constructed according to any one of the implementation methods in the first aspect or the second aspect.

[0055] In the seventh aspect, a neural network architecture search device is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in any one of the implementation modes of the first aspect or the second aspect.

[0056] In an eighth aspect, an image processing device is provided, comprising: a memory for storing programs; and a processor for executing the programs stored in the memory, wherein when the programs stored in the memory are executed, the processor is used to execute the method in any one of the implementation modes of the third aspect to the sixth aspect.

[0057] In a ninth aspect, a computer-readable medium is provided, which stores a program code for execution by a device, wherein the program code includes a method for executing any one of the implementations of the first to sixth aspects.

[0058] In a tenth aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute a method in any one of the implementations of the first to sixth aspects.

[0059] In the eleventh aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes a method in any one of the implementation modes of the first to sixth aspects.

[0060] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in any one of the implementation methods of the first aspect to the sixth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of a specific application provided by an embodiment of the present application;

[0062] Figure 2 A schematic diagram of the structure of the system architecture provided in the embodiment of the present application;

[0063] Figure 3 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0064] Figure 4 A schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the hardware structure of a chip provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0067] Figure 7 is a schematic flow chart of a search method for a neural network architecture according to an embodiment of the present application;

[0068] Figure 8 It is a schematic diagram of the structure of a building unit;

[0069] Fig. 9 It is a schematic diagram of a building block in the initial network architecture of the first phase;

[0070] Fig.10 is a schematic diagram of a building block in the initial neural network architecture after the first stage of optimization;

[0071] Fig.11 is a schematic diagram of a building block in the initial network architecture of the second phase;

[0072] Fig.12 It is a schematic diagram of the structure of the search network;

[0073] Fig.13 is a schematic flow chart of a search method for a neural network architecture according to an embodiment of the present application;

[0074] Fig.14 is a schematic flow chart of a search method for a neural network architecture according to an embodiment of the present application;

[0075] Fig.15 is a schematic flow chart of an image processing method according to an embodiment of the present application;

[0076] Fig.16 is a schematic block diagram of a neural network architecture search device according to an embodiment of the present application;

[0077] Fig.17 is a schematic block diagram of an image processing device according to an embodiment of the present application;

[0078] Fig.18 It is a schematic block diagram of a neural network training device according to an embodiment of the present application. DETAILED DESCRIPTION

[0079] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0080] The solutions of the embodiments of the present application can be applied to many specific fields in the field of artificial intelligence, such as smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities and other fields.

[0081] Specifically, the embodiments of the present application can be applied in fields that require the use of (deep) neural networks, such as image classification, image retrieval, image semantic segmentation, image super-resolution, and natural language processing.

[0082] In the image classification scenario, the neural network searched by the embodiment of the present application (the neural network searched by the search method of the neural network architecture of the embodiment of the present application) can be specifically applied in the classification of album pictures. The application of the embodiment of the present application in the classification of album pictures is described in detail below.

[0083] Album picture categories:

[0084] Specifically, when a user stores a large number of pictures on a terminal device (eg, a mobile phone) or a cloud disk, identifying images in the album can facilitate the user or the system to classify and manage the album, thereby improving the user experience.

[0085] The search method of the neural network architecture of the embodiment of the present application can search for a neural network architecture suitable for album classification, and then train the neural network according to the training pictures in the training picture library to obtain the album classification neural network. Next, the album classification neural network can be used to classify the pictures, so as to label the pictures of different categories for users to view and search. In addition, after obtaining the classification labels of these pictures, the album management system can also be classified and managed according to the classification labels of these pictures, which can save the user's management time, improve the efficiency of album management, and enhance the user experience.

[0086] like Figure 1 As shown, a neural network suitable for album classification can be constructed by a neural network architecture search system (corresponding to the neural network architecture search method of the embodiment of the present application). After obtaining a neural network suitable for album classification, the neural network can be trained based on training pictures to obtain an album classification neural network. Next, the album classification neural network can be used to classify the images to be processed. For example, Figure 1 As shown, the album classification neural network processes the input image and obtains the category of the image as tulip.

[0087] In addition to being used for image classification, the neural network searched by the embodiment of the present application (the neural network searched by the search method of the neural network architecture of the embodiment of the present application) can also be used in autonomous driving scenarios. Specifically, the neural network searched by the embodiment of the present application can be used for object recognition in autonomous driving scenarios.

[0088] Object recognition in autonomous driving scenarios:

[0089] There is a large amount of image information that needs to be processed in autonomous driving, and deep neural networks play an important role in autonomous driving with their powerful capabilities. By adopting the search method of the neural network architecture of the embodiment of the present application, a neural network suitable for processing image information in an autonomous driving scenario can be constructed. Next, the neural network is trained with training data in the autonomous driving scenario (including image information and labels of image information), and a neural network for processing image information in an autonomous driving scenario can be obtained. Finally, the neural network can be used to process the input image information, thereby identifying different objects in the road picture.

[0090] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0091] (1) Neural Network

[0092] A neural network can be composed of neural units. A neural unit can be a s The output of the operation unit is shown in formula (1).

[0093]

[0094] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0095] (2) Deep Neural Networks

[0096] Deep neural network (DNN), also known as multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0097] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Taking the coefficient W as an example, assuming that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0098] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0099] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0100] (3) Convolutional Neural Network

[0101] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0102] (4) Residual Network

[0103] The residual network is a deep convolutional network proposed in 2015. Compared with traditional convolutional neural networks, the residual network is easier to optimize and can improve accuracy by increasing the depth. The core of the residual network is to solve the side effects (degradation problem) brought by increasing the depth, so that the network performance can be improved by simply increasing the network depth. The residual network generally contains many sub-modules with the same structure. Usually, a residual network (ResNet) is used to connect a number to indicate the number of times the sub-module is repeated. For example, ResNet50 means that there are 50 sub-modules in the residual network.

[0104] (6) Classifier

[0105] Many neural network architectures have a classifier at the end to classify objects in the image. The classifier is generally composed of a fully connected layer and a softmax function (which can be called a normalized exponential function), which can output the probability of different categories based on the input.

[0106] (7) Loss Function

[0107] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0108] (8) Back propagation algorithm

[0109] Neural networks can use the error back propagation (BP) algorithm to correct the values ​​of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0110] like Figure 2 As shown, the embodiment of the present application provides a system architecture 100. Figure 2 In the data acquisition device 160, the training data may be collected. For the image processing method of the embodiment of the present application, the training data may include training images and labels of the training images (if the images are classified, the labels may be classification results of the training images), wherein the labels of the training images may be manually pre-labeled.

[0111] After collecting the training data, the data collection device 160 stores the training data in the database 130 , and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130 .

[0112] The following describes the process of the training device 120 obtaining the target model / rule 101 based on the training data. Specifically, the training device 120 processes the input training image to obtain the processing result of the training image, then compares the processing result of the training image with the label of the training image, and continues to train the target model / rule 101 according to the comparison between the processing result of the training image and the label of the training image, until the difference between the processing result of the training image and the label of the training image meets the requirement, thereby completing the training of the target model / rule 101.

[0113] The above-mentioned target model / rule 101 can be used to implement the image processing method of the embodiment of the present application. The target model / rule 101 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130, and it is also possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0114] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 2 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 2 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include an image to be processed input by the client device in the embodiment of the present application.

[0115] Preprocessing module 113 and preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by I / O interface 112. In the embodiment of the present application, preprocessing module 113 and preprocessing module 114 may be omitted (or only one of the preprocessing modules may be present), and the computing module 111 may be used directly to process the input data.

[0116] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0117] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0118] exist Figure 2In the example, the user can manually give input data (the input data can be an image to be processed), and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data input to the I / O interface 112 and the output results of the I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data input to the I / O interface 112 and the output results of the I / O interface 112 as new sample data in the database 130.

[0119] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0120] like Figure 2 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 in the embodiment of the present application can be a neural network in the present application. Specifically, the neural network provided in the embodiment of the present application can be CNN, deep convolutional neural networks (DCNN), recurrent neural networks (RNNS), etc.

[0121] Since CNN is a very common neural network, Figure 3 The focus is on a detailed introduction to the structure of CNN. As mentioned in the basic concept introduction above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.

[0122] The structure of the neural network specifically used in the image processing method of the embodiment of the present application can be as follows: Figure 3 As shown. Figure 3 In the example, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230. The input layer 210 may obtain an image to be processed, and the obtained image to be processed is passed to the convolutional layer / pooling layer 220 and the subsequent neural network layer 230 for processing, and the image processing result may be obtained. Figure 3 The internal layer structure of CNN 200 is introduced in detail.

[0123] Convolutional layer / pooling layer 220:

[0124] Convolutional Layer:

[0125] like Figure 3 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0126] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0127] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image, etc. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple convolution feature maps of the same size are then merged to form the output of the convolution operation.

[0128] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0129] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0130] Pooling layer:

[0131] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 3Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0132] Neural Network Layer 230:

[0133] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 3 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0134] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 3 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 3 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0135] The structure of the neural network specifically used in the image processing method of the embodiment of the present application can be as follows: Figure 4As shown. Figure 4 In the embodiment of the present invention, the convolutional neural network (CNN) 200 may include an input layer 110, a convolutional layer / pooling layer 120 (wherein the pooling layer is optional), and a neural network layer 130. Figure 3 compared to, Figure 4 The multiple convolutional layers / pooling layers in the convolutional layer / pooling layer 120 are operated in parallel, and the features extracted respectively are input to the full neural network layer 130 for processing.

[0136] It should be noted that Figure 3 and Figure 4 The convolutional neural network shown is only an example of two possible convolutional neural networks for the image processing method of an embodiment of the present application. In specific applications, the convolutional neural network adopted by the image processing method of the embodiment of the present application may also exist in the form of other network models.

[0137] In addition, the structure of the convolutional neural network obtained by the search method of the neural network architecture of the embodiment of the present application can be as follows: Figure 3 and Figure 4 The convolutional neural network architecture is shown in Figure 2.

[0138] Figure 5 The hardware structure of a chip provided in an embodiment of the present application includes a neural network processor 50. The chip can be arranged in Figure 2 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 2 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 3 or Figure 4 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 5 is implemented in the chip shown.

[0139] The neural network processor NPU 50 is mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 50, and the controller 504 controls the operation circuit 503 to extract data from the memory (weight memory or input memory) and perform operations.

[0140] In some implementations, the operation circuit 503 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

[0141] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 508.

[0142] The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0143] In some implementations, the vector calculation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, such as for use in a subsequent layer in a neural network.

[0144] The unified memory 506 is used to store input data and output data.

[0145] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0146] The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0147] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0148] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0149] Entrance: According to the actual invention, the data here is descriptive data, such as the detected vehicle speed, obstacle distance, etc.

[0150] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM for short), a high bandwidth memory (HBM) or other readable and writable memory.

[0151] in, Figure 3 or Figure 4 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 303 or the vector calculation unit 307.

[0152] Introduced above Figure 2 The execution device 110 in the embodiment of the present application can execute each step of the image processing method. Figure 3 and Figure 4 The CNN model shown and Figure 5 The chip shown can also be used to execute each step of the image processing method of the embodiment of the present application. The image processing method of the embodiment of the present application and the image processing method of the embodiment of the present application are described in detail below in conjunction with the accompanying drawings.

[0153] like Figure 6 As shown, the embodiment of the present application provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 210 and a data storage system 250, wherein the local device 301 and the local device 302 are connected to the execution device 210 via a communication network.

[0154] The execution device 210 can be implemented by one or more servers. Optionally, the execution device 210 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, and other devices. The execution device 210 can be arranged at one physical site, or distributed at multiple physical sites. The execution device 210 can use the data in the data storage system 250, or call the program code in the data storage system 250 to implement the method for searching the neural network architecture of the embodiment of the present application.

[0155] Specifically, the execution device 210 can perform the following process: determine a search space and multiple building units; stack the multiple building units to obtain a search network, which is a neural network for searching a neural network architecture; optimize the network structure of the building units in the search network within the search space to obtain optimized building units, wherein during the optimization process, the search space gradually decreases and the number of building units gradually increases, and the reduction in the search space and the increase in the number of building units make the video memory consumption generated during the optimization process within a preset range; and build the target neural network according to the optimized building units.

[0156] By executing the above process, the device 210 can build a target neural network, which can be used for image classification or image processing, etc.

[0157] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0158] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0159] In one implementation, the local device 301 and the local device 302 obtain relevant parameters of the target neural network from the execution device 210, deploy the target neural network on the local device 301 and the local device 302, and use the target neural network to perform image classification or image processing, etc.

[0160] In another implementation, the target neural network can be directly deployed on the execution device 210. The execution device 210 obtains the image to be processed from the local device 301 and the local device 302, and classifies the image to be processed or performs other types of image processing according to the target neural network.

[0161] The execution device 210 may also be referred to as a cloud device. In this case, the execution device 210 is generally deployed in the cloud.

[0162] The following is a corresponding analysis of the problems that exist when searching for neural network architectures (also called neural network structures).

[0163] When searching for neural network architectures, a feasible solution is differentiable architecture search (DARTS). However, when using the DARTS solution for neural network search, there is a problem of multicollinearity.

[0164] Specifically, when using the DARTS scheme to search for neural network architectures, when there are operators with high correlation, the weights of each operator determined during the search process may not reflect the true importance of each operator. In the process of selecting operators, truly important operators may be removed, resulting in poor performance of the neural network finally searched.

[0165] For example, in the search process, there are three operators: convolution, maximum pooling, and average pooling (the linear correlation between maximum pooling and average pooling is as high as 0.9). Among them, the weight of the convolution operation is 0.4, and the weights of maximum pooling and average pooling are both 0.3. At this time, the convolution operation will be selected as the final operation according to the principle of the largest weight. However, since the linear correlation between maximum pooling and average pooling is very high, maximum pooling and average pooling can be approximately regarded as a pooling operation. Then, at this time, the weight of the pooling operation is 0.6, and the weight of the convolution operation is 0.4. At this time, the pooling operation should be selected as the final operation, but the existing solution selects the convolution operation as the final operation. The selection of the operator is not accurate enough, resulting in poor performance of the neural network obtained by the final search.

[0166] In order to overcome the above-mentioned multicollinearity problem, when searching for a neural network, the optimization process can be divided into two stages. In the first stage of the optimization process, the type of alternative operation corresponding to each edge of the construction unit is first determined (the type of operator with the largest weight on each edge is determined), and then in the second stage, the specific operator on each edge in the construction unit is determined, thereby avoiding the multicollinearity problem in the process of searching the neural network, and then constructing a target neural network with higher performance.

[0167] The following is a detailed introduction to the search method of the neural network architecture of the embodiment of the present application in conjunction with the accompanying drawings.

[0168] Figure 7It is a schematic flowchart of the search method of the neural network architecture of an embodiment of the present application. Figure 7 The method shown can be executed by the neural network architecture search device of the embodiment of the present application (for example, Figure 7 The method shown can be Fig.16 The neural network architecture search device shown is executed). Figure 7 The method shown includes steps 1001 to 1006, which are described in detail below.

[0169] 1001. Determine a search space and a plurality of building blocks.

[0170] The search space in step 1001 includes multiple groups of candidate operators, each group of candidate operators includes at least one operator, and the operators included in each group of candidate operators are of the same type (at least one operator in each group of operators belongs to the same type of operator).

[0171] Optionally, the search space includes four groups of candidate operators, and the operators included in the four groups of candidate operators are specifically as follows:

[0172] The first group of candidate operators: 3x3 maximum pooling operation (3x3 max pooling or max_pool_3x3), 3x3 average pooling operation (3x3 average pooling or avg_pool_3x3);

[0173] The second group of alternative operators: skip-connect operations (identity or skip-connect);

[0174] The third group of alternative operators: 3x3 separable convolutions (or sep_conv_3x3), 5x5 separable convolutions (or sep_conv_5x5);

[0175] The fourth group of alternative operators: 3x3 dilated separable convolutions, 5x5 dilated separable convolutions.

[0176] Optionally, the above search space is determined according to the application requirements of the target neural network to be constructed.

[0177] Specifically, the above search space can be determined according to the type of processing data of the target neural network.

[0178] When the target neural network is a neural network for processing image data, the types and quantities of operations included in the search space must be adapted to the processing of the image data.

[0179] For example, when the target neural network is a neural network for processing images, the above search space may include convolution operations, pooling operations, skip connection operations, and the like.

[0180] When the target neural network is a neural network for processing speech data, the types and quantities of operations included in the search space should be adapted to the processing of speech data.

[0181] For example, when the target neural network is a neural network for processing speech data, the above search space may include activation functions (such as ReLU, Tanh), etc.

[0182] Optionally, the above search space is based on the application requirements and execution Figure 7 The neural network architecture of the method shown is determined by the video memory resource conditions of the search device.

[0183] The above-mentioned video memory resource condition of the device that performs the neural network architecture search may refer to the video memory resource size of the device that performs the neural network architecture search.

[0184] The types and quantity of operations included in the above search space can be comprehensively determined based on the application requirements of the target neural network and the video memory resource conditions of the device performing the neural network architecture search.

[0185] Specifically, the types and number of operations included in the search space can be determined based on the application requirements of the target neural network, and then the types and number of operations included in the search space can be adjusted in combination with the graphics memory resource conditions of the device that performs the neural network architecture search to determine the types and number of operations ultimately included in the search space.

[0186] For example, after determining the types and number of operations included in the search space based on the application requirements of the target neural network, if the graphics memory resources of the device performing the neural network architecture search are relatively small, some less important operations in the search space can be deleted; if the graphics memory resources of the device performing the neural network architecture search are relatively sufficient, the types and number of operations included in the search space can be maintained, or the types and number of operations included in the search space can be increased.

[0187] In addition, each of the multiple building units in the above step 1001 (also referred to as a cell) is a network structure obtained by connecting multiple nodes through basic operators of the neural network, and the connection between the nodes of each of the multiple building units forms an edge.

[0188] A building block can be considered as a directed acyclic graph (DAG), each of which consists of N (N is an integer greater than 1) ordered nodes connected by directed edges. Each node represents a feature graph, and each directed edge represents an operator used to process the input feature graph. For example, a directed edge (i, j) represents a connection relationship from node i to node j, and the operator o∈O on the directed edge (i, j) is used to transform the feature graph x_i input by node i into the feature graph x_j. Among them, O represents all the alternative operations in the search space.

[0189] like Figure 8 As shown in the figure, the building block consists of 4 nodes (nodes 0, 1, 2 and 3) connected by directed edges, where nodes 0, 1, 2 and 3 represent feature graphs. There are 6 directed edges in the building block, which are: directed edge (0, 1), directed edge (0, 2), directed edge (0, 3), directed edge (1, 2), directed edge (1, 3) and directed edge (2, 3).

[0190] Optionally, the number of the multiple construction units determined in the above step 1001 is determined according to the video memory resource conditions of the device performing the neural network architecture search.

[0191] Specifically, when executing Figure 7 When the video memory resources of the neural network architecture search device of the method shown are small, the number of building units can be smaller, and when executing Figure 7 When the video memory resources of the neural network architecture search device of the method shown are sufficient, the number of construction units can be larger.

[0192] Optionally, the number of the above-mentioned construction units is determined according to the application requirements of the target neural network to be constructed and the video memory resource conditions of the device that performs the neural network architecture search.

[0193] Specifically, the initial number of building units can be determined based on the application requirements of the target neural network, and then the initial number of building units can be further adjusted based on the video memory resources of the device that performs the neural network architecture search to determine the final number of building units.

[0194] For example, after determining the initial number of building units based on the application requirements of the target neural network, if the graphics memory resources of the device executing the target neural network architecture search are relatively small, then the number of building units can be further reduced; if the graphics memory resources of the device executing the target neural network architecture search are relatively sufficient, the initial number of building units can be kept unchanged. At this time, the initial number of building units is the final number of building units.

[0195] 1002. Stack multiple building units to obtain an initial neural network architecture of the first stage.

[0196] For example, in the above step 1002, multiple Figure 8 The building blocks shown are stacked to obtain the initial neural network architecture of the first stage.

[0197] 1003. Optimizing the initial neural network architecture of the first stage until convergence, so as to obtain the optimized initial neural network architecture of the first stage;

[0198] 1004. Obtain an initial neural network architecture for the second stage.

[0199] The structure of the initial neural network architecture of the first stage and the initial neural network architecture of the second stage are the same.

[0200] Specifically, the types and quantities of building units included in the initial neural network architecture of the first stage are the same as the types and quantities of building units included in the initial neural network architecture of the second stage, and the structure of the i-th building unit in the initial neural network architecture of the first stage is exactly the same as the structure of the i-th building unit in the initial neural network architecture of the second stage, and i is a positive integer.

[0201] The difference between the initial neural network architecture of the first stage and the initial neural network architecture of the second stage is that the alternative operators corresponding to the corresponding edges in the corresponding building units are different.

[0202] Specifically, each edge of each building unit in the initial neural network architecture of the first stage corresponds to a plurality of candidate operators, and each of the plurality of candidate operators corresponds to a group from a plurality of groups of candidate operators.

[0203] The mixed operator corresponding to the j-th edge in the ith building unit in the initial neural network architecture of the second stage is composed of all operators in the k-th group of candidate operators in the initial neural network architecture optimized in the first stage. The k-th group of candidate operators is a group of candidate operators with the largest weight among multiple candidate operators corresponding to the j-th edge in the ith building unit in the initial neural network architecture optimized in the first stage, and i, j and k are all positive integers.

[0204] When the neural network architecture is optimized in the above steps 1003 and 1004, optimization methods such as stochastic gradient descent (SGD) can be used for optimization.

[0205] 1005. Optimize the initial neural network architecture of the second stage until convergence to obtain an optimized building block.

[0206] The above-mentioned optimized building unit can be called an optimal building unit, and the optimized building unit is used to build or stack the required target neural network.

[0207] The construction units in the initial network architecture of the first stage and the construction units in the initial network architecture of the second stage are described below with reference to the accompanying drawings.

[0208] For example, a building block in the initial network architecture of the first stage above can be Fig. 9 As shown, Fig. 9 As shown, in the construction unit, the multiple candidate operators corresponding to each edge include operation 1, operation 2 and operation 3. Here, operation 1, operation 2 and operation 3 can be selected from the first group of candidate operators, the third group of candidate operators and the fourth group of candidate operations respectively. Specifically, operation 1 here can be a 3x3 maximum pooling operation in the first group of candidate operators, operation 2 can be a 3x3 separable convolution operation in the third group of candidate operators, and operation 3 can be a 3x3 dilated separable convolution operation in the fourth group of candidate operators.

[0209] It should be understood that for the convenience of description, Fig. 9 Each edge in the construction unit shows only three candidate operations. In this case, the corresponding search space may include only three groups of candidate operations, and the three candidate operations corresponding to each edge are selected from the three groups of candidate operations respectively.

[0210] After optimizing the initial neural network architecture of the first stage in the above step 1003, the optimized initial neural network architecture of the first stage can be obtained.

[0211] For example, a building block in the initial neural network architecture after the first stage of optimization can be Fig.10 As shown, through Fig. 9 By optimizing the building blocks shown in the figure, we can obtain the weights of each candidate operator on each edge, such as Fig.10 As shown, the bold operation on each edge represents the operator with the largest weight on the edge.

[0212] Specifically, in Fig.10 The operators with the largest weights on each edge of the building block are shown in Table 1.

[0213] Table 1

[0214] Directed Edge The operator with the highest weight 0-1 Operation 3 0-2 Operation 1 0-3 Operation 1 1-2 Operation 1 1-3 Operation 2 2-3 Operation 3

[0215] The initial neural network architecture of the second stage can be obtained by replacing the operator with the largest weight in the jth edge in the i-th building unit in the initial neural network architecture optimized in the first stage with a mixed operator consisting of all operators in a set of alternative operators in which the operator with the largest weight is located.

[0216] For example, by Fig.10 The operator with the largest weight in each edge of the building unit shown is replaced by a mixed operator consisting of all operators in a set of alternative operators where the operator with the largest weight belongs, and the following can be obtained: Fig.11 The building blocks shown.

[0217] Specifically, Fig.11 The specific composition of the mixing operation in the building unit can be shown in Table 2.

[0218] Table 2

[0219] Mixed Operation Contains Operator Mixing Operation 1 All operators in the set of candidate operators where operation 1 is located Mixing Operation 2 All operators in the set of alternative operators where operation 2 is located Mixing Operation 3 All operators in the set of alternative operators where operation 3 is located

[0220] When the above operation 1 is a 3x3 maximum pooling operation in the first group of candidate operators, operation 2 is a 3x3 separable convolution operation in the third group of candidate operators, and operation 3 is a 3x3 dilated separable convolution operation in the fourth group of candidate operators, the specific composition of the above hybrid operations 1 to 3 can be shown in Table 3.

[0221] Table 3

[0222] Mixed Operation Contains Operator Mixing Operation 1 3x3 max pooling operation, 3x3 average pooling operation Mixing Operation 2 3x3 separation convolution operation, 5x5 separation convolution operation Mixing Operation 3 3x3 dilated separable convolution operation, 5x5 dilated separable convolution operation

[0223] In the above step 1005, in the process of optimizing the initial neural network architecture of the second stage, the specific operator on each edge of each building unit in the initial neural network architecture of the second stage may be determined.

[0224] A building block in the initial network architecture of the second stage above can be Fig.11 As shown, in the above step 1005, Fig.11 The construction unit shown continues to optimize, determines the operator with the largest weight on each edge in the construction unit, and determines the operator with the largest weight on the edge as the final operator on the edge.

[0225] For example, Fig.11 The operation on the edge from node 1 to node 2 in is hybrid operation 1, which is a hybrid operation composed of a 3x3 maximum pooling operation and a 3x3 average pooling operation. Therefore, in the optimization process of step 1005, it is necessary to determine the weights of the 3x3 maximum pooling operation and the 3x3 average pooling operation, and then determine the operation with the larger weight as the final operation on the edge from node 1 to node 2.

[0226] 1006. Finally, the target neural network is constructed based on the optimized building blocks.

[0227] In the present application, during the process of searching for a neural network architecture, by determining which type of alternative operator should be used for each edge of each building unit in the optimization process of the first stage, and determining which specific alternative operator should be used for each edge of each building unit in the optimization process of the second stage, the problem of multicollinearity can be avoided, and a target neural network with better performance can be built based on the optimized building units.

[0228] Optionally, the multiple building blocks in step 1001 may include first-type building blocks.

[0229] Among them, the first type of building blocks are building blocks whose number (specifically, the number of channels) and size of input feature maps are the same as the number and size of output feature maps.

[0230] For example, the input of a first-class building unit is a feature map of size C×D1×D2 (C is the number of channels, D1 and D2 are width and height respectively), and the size of the output feature map after processing by the first-class building unit is still C×D1×D2.

[0231] The first type of building block mentioned above can specifically be a normal cell.

[0232] Optionally, the multiple building blocks in step 1001 include second-type building blocks.

[0233] Among them, the resolution of the output feature map of the second type of building unit is 1 / M of the input feature map, the number of output feature maps of the second type of building unit is M times the number of input feature maps, and M is a positive integer greater than 1.

[0234] The value of M mentioned above can generally be 2, 4, 6, 8, etc.

[0235] For example, if the input of a second-class building unit is a feature map of size C×D1×D2 (C is the number of channels, D1 and D2 are width and height respectively, and the product of C1 and C2 can represent the resolution of the feature map), then after being processed by the second-class building unit, the resulting feature map is feature map.

[0236] The second type of construction unit mentioned above may specifically be a reduction cell.

[0237] The initial neural network architecture of the first stage and the initial neural network architecture of the second stage can both be referred to as a search network. The search network can be formed by stacking the first building unit and the second building unit. Fig.12 A detailed introduction to the structure of the search network.

[0238] When the search network is composed of the first type of building blocks and the second type of building blocks, the structure of the search network can be as follows: Fig.12 shown.

[0239] like Fig.12 The search network is composed of five building blocks stacked in sequence, among which the first type of building blocks are located at the front and back ends of the search network, and there is a second type of building block between every two first building blocks.

[0240] Fig.12 The first construction unit in the search network can process the input image. After the first type of construction unit processes the image, the processed feature map is input to the second type of construction unit for processing, and the process is transmitted backward in sequence until the last first type of construction unit in the search network outputs the processing result of the image.

[0241] Optionally, Figure 7 The method shown also includes: clustering the multiple candidate operators in the search space to obtain the multiple groups of candidate operators.

[0242] The clustering process for the multiple candidate operators in the search space may be to divide the multiple candidate operators in the search space into different categories, and the candidate operators in each category constitute a group of candidate operators.

[0243] Optionally, the above-mentioned clustering processing of multiple candidate operators in the search space to obtain multiple groups of candidate operators includes: clustering processing of multiple candidate operators in the search space to obtain correlation relationships between multiple candidate operators in the search space; grouping multiple candidate operators in the search space according to the correlation relationships between multiple candidate operators in the search space to obtain the above-mentioned multiple groups of candidate operators.

[0244] The above correlation may be a linear correlation, which may be represented by a linear correlation degree (which may be a value between 0 and 1). A larger value of the linear correlation degree between two candidate operators indicates a closer relationship between the two candidate operators.

[0245] For example, through cluster analysis, the linear correlation between the 3x3 maximum pooling operation and the 3x3 average pooling operation is 0.9. Therefore, it can be considered that the correlation between the 3x3 maximum pooling operation and the 3x3 average pooling operation is high, and the 3x3 maximum pooling operation and the 3x3 average pooling operation can be divided into one group.

[0246] Through clustering processing, multiple candidate operators in the search space can be divided into multiple groups of candidate operators, which is convenient for subsequent optimization in the search neural network process.

[0247] Optionally, the multiple groups of candidate operators in the search space in step 1001 above include:

[0248] The first group of candidate operators: 3x3 maximum pooling operation, 3x3 average pooling operation;

[0249] The second set of alternative operators: skip operations;

[0250] The third group of alternative operators: 3x3 separation convolution operation, 5x5 separation convolution operation;

[0251] The fourth group of alternative operators: 3x3 dilated separable convolution operation and 5x5 dilated separable convolution operation.

[0252] Optionally, Figure 7 The method shown also includes: selecting an operator from each group of candidate operators in the multiple groups of candidate operators to obtain multiple candidate operators corresponding to each edge of each building unit in the initial neural network architecture of the first stage.

[0253] For example, for the initial neural network architecture of the first stage in step 1002 above, the multiple candidate operators corresponding to each edge of each building unit may include a 3x3 maximum pooling operation, a skip connection operation, a 3x3 separable convolution operation, and a 3x3 dilated separable convolution operation.

[0254] Optionally, Figure 7 The method shown also includes: determining the operator with the largest weight in each edge of each building unit in the initial neural network architecture of the first stage; and determining a mixed operator consisting of all alternative operators in a group of alternative operators in which the operator with the largest weight in the j-th edge in the ith building unit in the initial neural network architecture of the first stage is located as the alternative operator corresponding to the j-th edge in the ith building unit in the initial neural network architecture of the second stage.

[0255] For example, for the initial neural network architecture after optimization in the first stage, when the operator with the largest weight in the j-th edge in the ith building unit is the 3x3 maximum pooling operation, then, for the initial neural network architecture after optimization in the second stage, the candidate operator corresponding to the j-th edge in the ith building unit is a mixed operator consisting of a 3x3 maximum pooling operation and a 3x3 average pooling operation.

[0256] Next, in the process of optimizing the initial neural network architecture of the second stage, the weights of the 3x3 maximum pooling operation and the 3x3 average pooling operation on the j-th edge in the ith building unit in the initial neural network architecture of the second stage are determined, and then the operator with the largest weight is selected as the operator on the j-th edge in the ith building unit.

[0257] Optionally, Figure 7 The method shown also includes: the above-mentioned optimization of the initial neural network architecture of the first stage until convergence to obtain an optimized building unit, including: using the same training data to optimize the network architecture parameters and network model parameters of the building unit in the initial neural network architecture of the first stage respectively until convergence to obtain the optimized initial neural network architecture of the first stage; and / or, optimizing the initial neural network architecture of the second stage until convergence to obtain an optimized building unit, including: using the same training data to optimize the network architecture parameters and network model parameters of the building unit in the initial neural network architecture of the second stage respectively until convergence to obtain an optimized building unit.

[0258] By using the same training data to optimize the network architecture parameters and network model parameters, compared with the traditional two-layer optimization method, it is possible to search for a neural network with better performance with the same amount of training data.

[0259] Combine the following Fig.13 The search method for the neural network architecture of the embodiment of the present application is described in detail.

[0260] Fig.13 It is a schematic flowchart of the search method of the neural network architecture of an embodiment of the present application. Fig.13 The method shown can be executed by the neural network architecture search device of the embodiment of the present application (for example, Fig.13 The method shown can be Fig.16 The neural network architecture search device shown is executed).

[0261] Fig.13 The method shown includes steps 2001 to 2013, which are described in detail below.

[0262] 2001. Obtain training data.

[0263] In step 2001, the training data can be obtained by downloading from the Internet or manually collecting. The training data here can specifically be training pictures. After the training pictures are obtained, the training pictures can be preprocessed according to the target tasks to be processed by the searched neural network. The preprocessing here can include labeling picture categories, picture denoising, picture resizing, data enhancement, etc. In addition, the training data can also be divided into a training set and a test set as needed.

[0264] 2002. Determine the search space mother architecture based on alternative operators.

[0265] The above search space mother architecture is equivalent to the initial neural network architecture built based on multiple building blocks.

[0266] Before the above step 2002, the search space may be determined first. Specifically, the continuous search space based on the building blocks may be designed according to the application scenario of the final neural network architecture (eg, image size and image type of the image classification task).

[0267] The search space here may include multiple groups of candidate operators, specifically including the first group of candidate operators, the second group of candidate operators, the third group of candidate operators and the fourth group of candidate operators mentioned above.

[0268] 2003. Select an operation from each category of candidate operators to obtain the first-stage mother architecture.

[0269] The above step 2003 is equivalent to selecting an operation from each group of candidate operators on the basis of the search space mother architecture to obtain the first stage mother architecture.

[0270] The above-mentioned first-stage mother architecture can be equivalent to the initial neural network architecture of the first stage mentioned above.

[0271] 2004. Optimize the first phase mother architecture.

[0272] Among them, when the first-stage mother architecture is optimized, it can be matched with the complexity of the final neural network architecture, so that the first-stage mother architecture matches the complexity of the final neural network architecture as much as possible.

[0273] The optimization process of the first-stage mother architecture in the above step 2001 can refer to the process of optimizing the first-stage initial neural network architecture in the above step 1003.

[0274] 2005. Select all operators in the group of the operator with the largest weight to form a mixed operator to obtain the second-stage mother architecture.

[0275] The above-mentioned second-stage mother architecture can be equivalent to the initial neural network architecture of the second stage mentioned above.

[0276] 2005. The second-stage mother architecture is optimized to obtain the optimized building units.

[0277] Among them, when the second-stage mother architecture is optimized, it can be matched with the complexity of the final neural network architecture, so that the second-stage mother architecture matches the complexity of the final neural network architecture as much as possible.

[0278] The optimization process of the first-stage mother architecture in the above step 2005 can refer to the process of optimizing the second-stage initial neural network architecture in the above step 1005.

[0279] 2006. The optimized building blocks are stacked to obtain the final neural network architecture.

[0280] When the existing DARTS scheme performs a neural network search, a double-layer optimization is performed on the network structure parameters and the network model parameters in the construction unit. Specifically, the existing DARTS scheme divides the training data into two parts, one part of the training data is used to optimize the network architecture parameters of the construction unit in the search network, and the other part of the training data is used to optimize the network model parameters of the construction unit in the search network. The utilization rate of the training data is not high enough, and the performance of the neural network finally searched is limited.

[0281] Based on the above problems, this application proposes a single-layer optimization scheme, which uses the same training data to optimize the network structure parameters and network model parameters in the construction unit respectively, so as to improve the utilization rate of training data. Compared with the two-layer optimization method in the existing DARTS scheme, it can search for a neural network with better performance under the condition of the same amount of training data. Fig.14 This single-layer optimization scheme is described in detail.

[0282] Fig.14 It is a schematic flowchart of the search method of the neural network architecture of an embodiment of the present application. Fig.14 The method shown can be executed by the neural network architecture search device of the embodiment of the present application (for example, Fig.14 The method shown can be Fig.16 The neural network architecture search device shown is executed). Fig.14 The method shown includes steps 3010 to 3040, which are described in detail below.

[0283] 3010. Determine a search space and a plurality of building blocks.

[0284] The search space in step 2001 may include the following operations:

[0285] 3x3 max pooling operation;

[0286] 3x3 average pooling operation;

[0287] Jump operation;

[0288] 3x3 separation convolution operation;

[0289] 5x5 separation convolution operation;

[0290] 3x3 dilated separable convolution operation;

[0291] 5x5 dilated separable convolution operation.

[0292] It should be understood that the alternative operators in the search space in the above step 3010 can also be divided into multiple groups. Specifically, the search space in the above step 3010 can include the above-mentioned first group of alternative operators, second group of alternative operators, third group of alternative operators and fourth group of alternative operators.

[0293] 3020. Stack multiple building blocks to obtain a search network.

[0294] 3030. Using the same training data in the search space, the network architecture parameters and the network model parameters of the building units in the search network are optimized respectively to obtain optimized building units.

[0295] In the above step 3030, the network architecture parameters and network model parameters of the construction unit in the search network are optimized respectively. When the optimized construction unit is obtained, the specific method can be as follows: Figure 7 The method in steps 1002 to 1005 of the method shown adopts two stages for optimization (in this case, the search space in step 2001 includes multiple groups of alternative operators) to obtain an optimized construction unit (the specific process can be found in the relevant content of the above steps 1002 to 1005, which will not be described in detail here).

[0296] 3040. Build a target neural network based on the optimized building blocks.

[0297] Each of the above-mentioned multiple building blocks is a network structure obtained by connecting multiple nodes through basic operators of the neural network.

[0298] In this application, by using the same training data to optimize the network architecture parameters and the network model parameters, compared with the traditional two-layer optimization method, it is possible to search for a neural network with better performance with the same amount of training data.

[0299] Optionally, in the above step 3030, the network architecture parameters and network model parameters of the construction units in the search network are optimized respectively using the same training data in the search space to obtain the optimized construction units, including:

[0300] Determine the optimized network architecture parameters and the optimized network model parameters of the construction unit in the search network according to the same training data and using formula (2) and formula (3);

[0301]

[0302]

[0303] In the above formula (2) and formula (3), the meanings of the various parameters are as follows:

[0304] α t and w t Respectively represent the network architecture parameters and network model parameters after the t-th step optimization of the building units in the search network;

[0305] α t-1 and w t-1 Respectively represent the network architecture parameters and network model parameters after the t-1 step optimization of the building units in the search network;

[0306] η t and δ t They respectively represent the learning rates of the network architecture parameters and the network model parameters when the building units in the search network are optimized in the t-th step;

[0307] L train (w t-1 ,α t-1 ) represents the loss function value on the test set when the loss function is optimized in step t. represents the gradient of the loss function on the test set with respect to α when optimizing in the tth step, It represents the gradient of the loss function on the test set with respect to w during the optimization of the tth step.

[0308] It should be understood that the above network architecture parameter α refers to the weight coefficient of each operator, and the value of α will represent the importance of the corresponding operator. w refers to the set of all other parameters in the architecture, including parameters in convolution, prediction layer parameters, etc.

[0309] Through analysis, it is found that when using the existing DARTS scheme to search for neural network architectures, there is a problem of mismatch between data complexity and the expressive power of alternative parent architectures in the search space. Specifically, due to certain factors (for example, memory size limitations), the depth of the parent architectures stacked during the search process of the DARTS scheme is very different from the depth of the neural network architecture finally built.

[0310] For example, in the DARTS search process, a search mother structure consisting of 8 building blocks is used, but the final neural network architecture is composed of 20 building blocks. The expression ability and optimization difficulty of these two deep neural networks are very different. For a small architecture with only 8 building blocks, the search algorithm tends to choose more complex operators to express data features, while for a large architecture with 20 building blocks actually used, it is unnecessary to use too many complex operators, and it is easy to cause problems such as difficulty in optimization, resulting in limited performance of the final neural network.

[0311] Based on this, the present application proposes a new neural network architecture search scheme, in which the search parent architecture during the search process must match the complexity of the neural network that is ultimately constructed.

[0312] Specifically, compared to the original DARTS, which has seven candidate operators for each hybrid operator, in this solution, each hybrid operator has four candidate operators in the first stage and two candidate operators in the second stage. In this way, we can increase the number of cells to 14 building blocks in the first stage and 20 building blocks in the second stage. This solves the problem of the mismatch between the expression capabilities of the alternative parent architectures in the search space and the final training architecture.

[0313] In addition, in order to reduce the difficulty of optimization, when the architecture of 20 building units is finally used, a skip connection fast channel (annularity tower) will be pulled at the position of 14 building units to directly connect to the end. In this way, the optimization difficulty of this architecture is equivalent to the network architecture with a depth of 14 building units, thereby reducing the difficulty of optimization.

[0314] To verify the effectiveness of the above solution, we calculated the gradient confusion metric to measure the network optimization complexity. We found that using 14 building blocks in the first stage and 20 building blocks in the second stage can match the optimization complexity of the final constructed neural network architecture.

[0315] The effect of the neural network architecture search method of the embodiment of the present application is explained below in combination with specific test results.

[0316] In this section, we specifically take the image classification task as an example to verify the effectiveness of the search method of the neural network architecture of the embodiment of the present application.

[0317] At test time, we conducted the first set of experiments on two public datasets (CIFAR-10 and CIFAR-100), each consisting of 50,000 training images and 10,000 test images.

[0318] In the architecture search phase of the neural network, the training set was randomly divided into two subsets, one subset containing 45,000 images for simultaneous training of α and w, and the other subset containing 5,000 images as a training set for selecting the architecture parameter α that can obtain the highest verification accuracy during training. In the performance evaluation phase of the neural network architecture, the standard training / testing split was used.

[0319] In the first stage of neural network architecture search, the optional operations O include: i skip connection operation; 3×3 maximum pooling; 3x3 separable convolution operation; 3x3 dilated separable convolution operation; zeroing operation.

[0320] A proxy parent network consisting of 14 units is trained for 1000 epochs using a one-layer optimization method. After the proxy parent network converges, the optimal set of operations is activated based on α.

[0321] In the second stage, we replace the mixing operation ob(i; j) with the weighted sum of all operations in the activation group in the first stage. Next, the surrogate mother architecture stacked with 20 building blocks is trained using a first-level optimization for 100 cycles.

[0322] Table 4

[0323] plan Testing Errors Parameter quantity (M) Search Cost Search Methods Existing solution 1 2.55 2.8 3150 Evolution-based search Existing solution 2 2.08 5.7 4 Gradient-based search Existing solution 3 2.89 4.6 0.5 Reinforcement Learning Existing solution 4 — 3.3 4 Gradient-based search This application plan 2.45 3.6 1 Gradient-based search

[0324] After stacking 20 cells to obtain the final optical network architecture, we can use the same traditional method as DARTS for training. The test results of the trained neural network architecture on the CIFAR-10 dataset are shown in Table 4.

[0325] Table 4 shows the test errors, parameter quantities, and search costs of the neural network architectures obtained by using different neural network architecture search schemes. The meanings of each scheme are as follows:

[0326] Existing solution 1: It can be expressed as AmoebaNet-B, which is specifically the regularized evolution of image classifier structure search (the source of the solution is Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926, 2018.);

[0327] Existing solution 2: It can be expressed as ProxylessNAS, which specifically searches for neural architectures directly on target tasks and hardware (the source of the solution is Han Cai, Ligeng Zhu, and Song Han.Proxylessnas:Direct neural architecture search on target task and hardware.arXiv preprintarXiv:1812.00332,2018.);

[0328] Existing solution 3: It can be expressed as ENAS, which is specifically an efficient neural architecture search via parameter sharing (the source of the solution is Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv:1802.03268, 2018.);

[0329] Existing solution 4: It can be expressed as DARTS, which is specifically a differentiable neural network architecture search (the source of the solution is Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055, 2018);

[0330] The present application scheme can be represented as iDARTS, where iDARTS can represent the above Figure 7 The search method for the neural network architecture shown in the figure is used, and the neural network architecture obtained by the search is trained using the traditional method that is exactly the same as the traditional DARTS solution.

[0331] As can be seen from Table 4, compared with the AmoebaNet-B solution and the ENAS solution, the present application solution has lower test errors and higher accuracy for the neural network. Although the ProxylessNAS solution has achieved higher accuracy than the present application solution, the ProxylessNAS solution requires more memory.

[0332] In the embodiment of the present application, when performing a neural network search, both Figure 7 The method shown in the figure performs a two-stage search, or Fig.14 The method shown performs single-layer optimization. The test results of the neural network architecture obtained by using two-stage search and single-layer / double-layer optimization when searching for the neural network architecture in the embodiment of the present application are described below.

[0333] Table 5

[0334]

[0335] As shown in Table 5, the original setting refers to the traditional DARTS scheme for searching the neural network architecture, and the two-stage search refers to the use of Figure 7 The method shown in the figure is used to search for the neural network architecture. Double-layer optimization means that different training data are used to optimize the network structure parameters and network model parameters in the construction unit of the search network, and single-layer optimization means that the same training data is used to optimize the network structure parameters and network model parameters in the construction unit of the search network (for details, please refer to this Fig.14 The method shown in the table shows that CIFAR10 and CIFAR100 are different test sets. The numbers in the table show the test error rate and the variance of the error rate.

[0336] As can be seen from Table 5, no matter which search method is used, the error rate after single-layer optimization is lower than that after double-layer optimization, and the variance is also reduced. Therefore, the single-layer optimization method can reduce the error rate of the final constructed neural network architecture, improve the accuracy of the final neural network architecture during testing, and improve the stability of the final neural network architecture.

[0337] For another example, it can be seen from Table 5 that the use of a two-stage search scheme can also improve the accuracy of the final neural network architecture during testing, as well as the stability of the final neural network architecture.

[0338] Fig.15 It should be understood that the above definition, explanation and extension of the acquisition process of the target neural network also apply to Fig.15The target neural network in the method shown below is introduced Fig.15 The repeated descriptions of the methods shown are appropriately omitted. Fig.15 The methods shown include:

[0339] 4010. Obtain an image to be processed;

[0340] 4020. Process the image to be processed according to the target neural network to obtain a processing result of the image to be processed.

[0341] The target neural network in step 4020 may be a neural network searched (constructed) according to the search method of the neural network architecture of the embodiment of the present application. Figure 7 , Fig.13 as well as Fig.14 The neural network architecture obtained by the method shown.

[0342] Since the search method of the neural network architecture of the embodiment of the present application can construct a target neural network with better performance, a more accurate image processing result can be obtained when the target neural network is used to process the image to be processed.

[0343] The above-mentioned processing of the image to be processed may refer to identifying, classifying, detecting, etc. the image to be processed.

[0344] Fig.15 The image processing method shown can be applied to specific scenarios such as image classification, semantic segmentation, and face recognition. These specific applications are introduced below.

[0345] Image Classification:

[0346] when Fig.15 When the method shown is applied in an image classification scenario, the image to be processed is first obtained, and then feature extraction is performed on the image to be processed according to the target neural network to obtain the features of the image to be processed, and then the image to be processed is classified according to the features of the image to be processed to obtain the classification result of the image to be processed.

[0347] Since the search method using the neural network architecture of the embodiment of the present application can construct a target neural network with better performance, using the target neural network to classify the image to be processed can obtain better and more accurate image classification results.

[0348] Semantic segmentation in autonomous driving scenarios:

[0349] when Fig.15When the method shown is applied in the semantic segmentation scenario of an autonomous driving system, a road image is first acquired, and then convolution processing is performed on the road image according to a target neural network to obtain multiple convolution feature maps of the road image; finally, deconvolution processing is performed on the multiple convolution feature maps of the road image according to the target neural network to obtain a semantic segmentation result of the road image.

[0350] Since the search method using the neural network architecture of the embodiment of the present application can construct a target neural network with better performance, using the target neural network to perform semantic segmentation on road images can achieve better semantic segmentation results.

[0351] Face Recognition:

[0352] when Fig.15 When the method shown is applied in the semantic segmentation scenario of an autonomous driving system, the road image is first acquired, and then the face image is convolved according to the target neural network to obtain a convolution feature map of the face image. Finally, the convolution feature map of the face image is compared with the convolution feature map of the identity document image to obtain a verification result of the face image.

[0353] Since the search method using the neural network architecture of the embodiment of the present application can construct a target neural network with better performance, using the target neural network to recognize facial images can achieve better recognition effects.

[0354] Fig.16 It is a schematic diagram of the hardware structure of the neural network architecture search device provided in an embodiment of the present application. Fig.16 The neural network architecture search device 3000 shown in the figure can execute each step of the neural network architecture search method of the embodiment of the present application. Specifically, the neural network architecture search device 3000 can execute the above Figure 7 , Fig.13 as well as Fig.14 The various steps in the method shown.

[0355] Fig.16 The neural network architecture search device 3000 shown (the device 3000 may be a computer device) includes a memory 3001, a processor 3002, a communication interface 3003 and a bus 3004. The memory 3001, the processor 3002 and the communication interface 3003 are connected to each other through the bus 3004.

[0356] The memory 3001 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 3001 may store a program, and when the program stored in the memory 3001 is executed by the processor 3002, the processor 3002 is used to execute each step of the search method for the neural network architecture of the embodiment of the present application.

[0357] Processor 3002 can adopt a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the search method of the neural network architecture of the method embodiment of the present application.

[0358] The processor 3002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the search method of the neural network architecture of the present application may be completed by an integrated logic circuit of hardware in the processor 3002 or by instructions in the form of software.

[0359] The processor 3002 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 3001, and the processor 3002 reads the information in the memory 3001, and completes the functions required to be performed by the units included in the neural network architecture search device 3000 in combination with its hardware, or executes the search method for the neural network architecture of the method embodiment of the present application.

[0360] The communication interface 3003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the device 3000 and other devices or communication networks. For example, the information of the neural network to be constructed and the training data required in the process of constructing the neural network can be obtained through the communication interface 3003.

[0361] The bus 3004 may include a path for transmitting information between various components of the device 3000 (eg, the memory 3001 , the processor 3002 , and the communication interface 3003 ).

[0362] Fig.17 It is a schematic diagram of the hardware structure of the image processing device according to an embodiment of the present application.

[0363] Fig.17 The image processing device 4000 shown in the figure can execute each step of the image processing method of the embodiment of the present application. Specifically, the image processing device 4000 can execute the above Fig.15 The various steps in the method shown.

[0364] Fig.17 The image processing device 4000 shown includes a memory 4001 , a processor 4002 , a communication interface 4003 and a bus 4004 . The memory 4001 , the processor 4002 , and the communication interface 4003 are connected to each other through the bus 4004 .

[0365] The memory 4001 may be a ROM, a static storage device, or a RAM. The memory 4001 may store a program. When the program stored in the memory 4001 is executed by the processor 4002, the processor 4002 and the communication interface 4003 are used to execute each step of the image processing method of the embodiment of the present application.

[0366] Processor 4002 can be a general-purpose CPU, microprocessor, ASIC, GPU or one or more integrated circuits, which are used to execute relevant programs to implement the functions required to be performed by the units in the image processing device of the embodiment of the present application, or to execute the image processing method of the method embodiment of the present application.

[0367] The processor 4002 may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the image processing method of the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 4002 or by instructions in the form of software.

[0368] The processor 4002 may also be a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 4001, and the processor 4002 reads the information in the memory 4001, and completes the functions required to be performed by the units included in the image processing device of the embodiment of the present application in combination with its hardware, or executes the image processing method of the method embodiment of the present application.

[0369] The communication interface 4003 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 4000 and other devices or a communication network. For example, the image to be processed can be obtained through the communication interface 4003.

[0370] The bus 4004 may include a path for transmitting information between various components of the device 4000 (eg, the memory 4001 , the processor 4002 , and the communication interface 4003 ).

[0371] Fig.18 Schematic diagram of the hardware structure of the neural network training device of the embodiment of the present application. Similar to the above-mentioned device 3000 and device 4000, Fig.18 The neural network training device 5000 shown includes a memory 5001, a processor 5002, a communication interface 5003 and a bus 5004. The memory 5001, the processor 5002 and the communication interface 5003 are connected to each other through the bus 5004.

[0372] In passing Fig.16 After the neural network is constructed by the neural network architecture search device shown in the figure, it can be Fig.18 The neural network training device 5000 shown trains the neural network, and the trained neural network can be used to execute the image processing method of the embodiment of the present application.

[0373] Specifically, Fig.18 The device shown can obtain training data and a neural network to be trained from the outside world through the communication interface 5003, and then the processor trains the neural network to be trained according to the training data.

[0374] It should be noted that although the above-mentioned devices 3000, 4000 and 5000 only show a memory, a processor and a communication interface, in the specific implementation process, those skilled in the art should understand that the devices 3000, 4000 and 5000 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the devices 3000, 4000 and 5000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the devices 3000, 4000 and 5000 may also only include the devices necessary for implementing the embodiments of the present application, and do not necessarily include Fig.16 , Fig.17 and Fig.18 All devices shown in .

[0375] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0376] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0377] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0378] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0379] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0380] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0381] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for searching a neural network architecture, characterized in that: include: Determine a search space and a plurality of construction units, the search space and the plurality of construction units are used to process image data, the search space includes a plurality of groups of candidate operators, each group of candidate operators includes operators of the same type, each of the plurality of construction units is a network structure obtained by connecting a plurality of nodes through basic operators of a neural network, and connections between nodes of each of the plurality of construction units form edges; The plurality of building blocks are stacked to obtain an initial neural network architecture of the first stage, wherein each edge of each building block in the initial neural network architecture of the first stage corresponds to a plurality of candidate operators, and each of the plurality of candidate operators corresponds to a group from the plurality of groups of candidate operators; Optimizing the initial neural network architecture of the first stage until convergence, so as to obtain the optimized initial neural network architecture of the first stage; Obtaining an initial neural network architecture of the second stage, wherein the mixed operator corresponding to the j-th edge in the i-th building unit in the initial neural network architecture of the second stage is composed of all operators in the k-th group of candidate operators in the initial neural network architecture optimized in the first stage, wherein the k-th group of candidate operators is a group of candidate operators including the operator with the largest weight among multiple candidate operators corresponding to the j-th edge in the i-th building unit in the initial neural network architecture optimized in the first stage, and i, j and k are all positive integers; Optimize the initial neural network architecture of the second stage until convergence to obtain the optimized building blocks; Building a target neural network based on the optimized building blocks; The image data is processed using the target neural network.

2. The search method according to claim 1, characterized in that: The search method further comprises: Clustering is performed on the multiple candidate operators in the search space to obtain the multiple groups of candidate operators.

3. The search method according to claim 1 or 2, characterized in that: The search method further comprises: An operator is selected from each group of candidate operators of the multiple groups of candidate operators to obtain the multiple candidate operators corresponding to each edge of each building unit in the initial neural network architecture of the first stage.

4. The search method according to claim 3, characterized in that: The search method further comprises: Determine the operator with the largest weight in each edge of each building block in the initial neural network architecture of the first stage; A mixed operator consisting of all alternative operators in a group of alternative operators where the operator with the largest weight in the j-th edge in the ith construction unit in the initial neural network architecture of the first stage is located is determined as the alternative operator corresponding to the j-th edge in the ith construction unit in the initial neural network architecture of the second stage.

5. The search method according to claim 1 or 2, characterized in that: The multiple sets of candidate operators include: The first group of candidate operators: 3x3 maximum pooling operation, 3x3 average pooling operation; The second group of alternative operators: skip operations; The third group of alternative operators: 3x3 separation convolution operation, 5x5 separation convolution operation; The fourth group of alternative operators: 3x3 dilated separable convolution operation and 5x5 dilated separable convolution operation.

6. The search method according to claim 1 or 2, characterized in that: The optimizing the initial neural network architecture of the first stage until convergence to obtain an optimized building block includes: Using the same training data, respectively optimize the network architecture parameters and network model parameters of the building blocks in the initial neural network architecture of the first stage until convergence, so as to obtain the optimized initial neural network architecture of the first stage; and / or The initial neural network architecture of the second stage is optimized until convergence to obtain an optimized building block, including: The same training data is used to optimize the network architecture parameters and network model parameters of the building units in the initial neural network architecture of the second stage respectively until convergence to obtain the optimized building units.

7. A neural network architecture search device, characterized in that: include: Memory, used to store programs; A processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to perform the following process: Determine a search space and a plurality of construction units, the search space and the plurality of construction units are used to process image data, the search space includes a plurality of groups of candidate operators, each group of candidate operators includes operators of the same type, each of the plurality of construction units is a network structure obtained by connecting a plurality of nodes through basic operators of a neural network, and connections between nodes of each of the plurality of construction units form edges; The plurality of building blocks are stacked to obtain an initial neural network architecture of the first stage, wherein each edge of each building block in the initial neural network architecture of the first stage corresponds to a plurality of candidate operators, and each of the plurality of candidate operators corresponds to a group from the plurality of groups of candidate operators; Optimizing the initial neural network architecture of the first stage until convergence, so as to obtain the optimized initial neural network architecture of the first stage; Obtaining an initial neural network architecture of the second stage, wherein the mixed operator corresponding to the j-th edge in the i-th building unit in the initial neural network architecture of the second stage is composed of all operators in the k-th group of candidate operators in the initial neural network architecture optimized in the first stage, wherein the k-th group of candidate operators is a group of candidate operators including the operator with the largest weight among multiple candidate operators corresponding to the j-th edge in the i-th building unit in the initial neural network architecture optimized in the first stage, and i, j and k are all positive integers; Optimize the initial neural network architecture of the second stage until convergence to obtain the optimized building blocks; Building a target neural network based on the optimized building blocks; The image data is processed using the target neural network.

8. The neural network architecture search device according to claim 7, characterized in that: The processor is further configured to: Clustering is performed on the multiple candidate operators in the search space to obtain the multiple groups of candidate operators.

9. The neural network architecture search device according to claim 7 or 8, characterized in that: The processor is further configured to: An operator is selected from each group of candidate operators of the multiple groups of candidate operators to obtain the multiple candidate operators corresponding to each edge of each building unit in the initial neural network architecture of the first stage.

10. The neural network architecture search device according to claim 9, characterized in that: The processor is further configured to: Determine the operator with the largest weight in each edge of each building block in the initial neural network architecture of the first stage; A mixed operator consisting of all alternative operators in a group of alternative operators where the operator with the largest weight in the j-th edge in the ith construction unit in the initial neural network architecture of the first stage is located is determined as the alternative operator corresponding to the j-th edge in the ith construction unit in the initial neural network architecture of the second stage.

11. The neural network architecture search device according to claim 7 or 8, characterized in that: The multiple sets of candidate operators include: The first group of candidate operators: 3x3 maximum pooling operation, 3x3 average pooling operation; The second group of alternative operators: skip operations; The third group of alternative operators: 3x3 separation convolution operation, 5x5 separation convolution operation; The fourth group of alternative operators: 3x3 dilated separable convolution operation and 5x5 dilated separable convolution operation.

12. The neural network architecture search device according to claim 7 or 8, characterized in that: The processor is further configured to: Using the same training data, the network architecture parameters and network model parameters of the building blocks in the initial neural network architecture of the first stage are optimized respectively until convergence, so as to obtain the optimized initial neural network architecture of the first stage; and / or The step of optimizing the initial neural network architecture of the second stage until convergence to obtain an optimized building block comprises: The same training data is used to optimize the network architecture parameters and network model parameters of the building units in the initial neural network architecture of the second stage respectively until convergence to obtain the optimized building units.

13. A computer-readable storage medium, characterized in that: The computer-readable medium stores a program code for execution by a device, wherein the program code includes a program for executing the search method according to any one of claims 1 to 6.

14. A chip, characterized in that: The chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the search method according to any one of claims 1-6.

Images