Image classification network architecture design method, device, equipment and medium

Abstract

The present invention provides a method, device, equipment and medium for designing an image classification network architecture, which uses a candidate sub-search space to organize the search for the architecture, finds a small and effective sub-search space, and then iteratively modifies the architecture and updates the sub-search space, and can adaptively adjust the neural network architecture in the sub-search space, greatly improving the effectiveness and computational efficiency of the neural architecture search, and breaking through the problem of low search efficiency in the predefined ultra-large search space in the prior art. In addition, the present invention uses a graph convolutional neural network method, with candidate sub-search spaces as nodes, which can efficiently encode the relationship information between neural network architectures, and the discriminant features learned are conducive to using recurrent neural networks to determine how to modify the network architecture.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, device, equipment and medium for designing an image classification network architecture. Background Art

[0002] Neural network architecture design refers to designing the structure of a neural network, including convolutional layers, pooling layers, fully connected layers, etc. For image classification network architecture design, it usually requires expert experience and multiple attempts, which is time-consuming and inefficient.

[0003] At present, there are also neural architecture search methods (NAS) that automatically complete the design of image classification network architecture. One method is mainly based on reinforcement learning (RL) to implement neural architecture search. This method uses reinforcement learning algorithm to train recurrent neural network (RNN) and uses recurrent neural network as controller to generate architecture code. Each architecture code corresponds to a neural network architecture, and the optimal architecture code corresponds to the final output neural network architecture. In the framework of neural architecture search method based on reinforcement learning algorithm, the neural network architecture is described as a variable length string, and recurrent neural network is used to generate such a string; then, reinforcement learning is used to optimize the controller and finally obtain the optimal neural network architecture. Subsequently, Markov decision process, reward reinforcement recording mechanism, sampling strategy and other methods were introduced into the framework based on reinforcement learning to reduce the computational complexity. Another method uses evolutionary computing to automatically learn the optimal neural network architecture. This method first uses the simplest network structure to initialize a large population, and then obtains the optimal network structure by copying, mutating and selecting the population. Subsequently, some improvements have been made to neural architecture search based on evolutionary algorithms, such as the introduction of fixed-length binary encoding schemes for neural network architectures and predefined genetic operations.

[0004] However, both of the above methods are sampling methods, which require maintaining a large population of candidate network structures. Since each sub-network structure of the candidate population needs to be trained and verified, a large amount of computing resources are consumed. At the same time, these methods are sensitive to the initial value of the network structure and their performance is extremely unstable. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for designing an image classification network architecture, so as to solve the defects of low efficiency and poor reliability in the prior art of image classification network architecture design.

[0006] The present invention provides a method for designing an image classification network architecture, comprising:

[0007] Taking each candidate sub-search space as a node, sorting each candidate sub-search space in order from low to high according to the performance of each candidate sub-search space, and connecting every two adjacent nodes from low to high to construct a directed graph of the candidate sub-search space; each candidate sub-search space includes multiple neural network architectures, and the architecture similarity between every two neural network architectures in each candidate sub-search space is greater than a preset similarity, and the performance of each candidate sub-search space is determined based on a sample image, a classification label of the sample image, and a predicted classification result of the sample image;

[0008] Based on the candidate sub-search space directed graph, a multi-layer graph convolutional neural network is constructed, and a feature vector of each node in the last layer of the graph convolutional neural network is extracted;

[0009] Determine the input vector of the first layer in the multi-layer recurrent neural network based on the feature vector of each node in the last layer of the graph convolutional neural network, and obtain the output vector of each layer in the recurrent neural network based on the input vector of the first layer;

[0010] Based on the output vectors of each layer in the recurrent neural network, determine a target sub-search space from each candidate sub-search space, update the target sub-search space, and update the graph convolutional neural network and the recurrent neural network based on the updated target sub-search space;

[0011] Determine a reward value based on the performance of the target sub-search space and the performance of the updated target sub-search space, and perform parameter iteration on the updated graph convolutional neural network and the updated recurrent neural network based on the reward value;

[0012] When the iteration stopping condition is met, the image classification network architecture is determined based on the updated target sub-search space.

[0013] According to an image classification network architecture design method provided by the present invention, each candidate sub-search space is determined based on the following steps:

[0014] Randomly sample multiple candidate neural network architectures from the global search space;

[0015] Based on each candidate neural network architecture, and a neural network architecture whose similarity to each candidate neural network architecture is greater than the preset similarity, a corresponding candidate sub-search space is constructed; each candidate neural network architecture is the central architecture of the corresponding candidate sub-search space.

[0016] According to a method for designing an image classification network architecture provided by the present invention, extracting the feature vector of each node in the last layer of the graph convolutional neural network includes:

[0017] Determine a contribution coefficient of node j to node i based on a feature vector of node i, a feature vector of node j, and a feature vector of an edge from node i to node j in the graph convolutional neural network;

[0018] Based on the contribution coefficient of the node j to the node i and the feature vector of the node j, the feature vector of the node i is updated.

[0019] According to a method for designing an image classification network architecture provided by the present invention, the feature vector of the edge is determined based on the following steps:

[0020] Determine a contribution coefficient of edge q to edge p based on a feature vector of edge p in the graph convolutional neural network, a feature vector of an adjacent edge q of edge p, and a feature vector of a junction node between edge p and edge q;

[0021] Based on the contribution coefficient of the edge q to the edge p, the feature vector of the edge p is updated.

[0022] According to a method for designing an image classification network architecture provided by the present invention, the output vectors of each layer in the recurrent neural network are used to determine a target sub-search space from each candidate sub-search space, and the target sub-search space is updated, including:

[0023] Determine a target sub-search space from each candidate sub-search space based on an output vector of the first layer in the recurrent neural network;

[0024] Determine a layer to be updated of a central architecture in the target sub-search space based on an output vector of the n-1th layer in the recurrent neural network;

[0025] Determine the update operation corresponding to the layer to be updated based on the output vector of the nth layer in the recurrent neural network; wherein the nth layer is the last layer in the recurrent neural network;

[0026] Based on the update operation corresponding to the layer to be updated, the layer to be updated is updated to obtain an updated target sub-search space.

[0027] According to a method for designing an image classification network architecture provided by the present invention, the method of determining a target sub-search space from each candidate sub-search space based on the output vector of the first layer in the recurrent neural network comprises:

[0028] Determine, based on the output vector of the first layer in the recurrent neural network, a calculated score for each element in the output vector of the first layer;

[0029] Determine a target sub-search space from each candidate sub-search space based on the calculated score of each element in the output vector of the first layer;

[0030] The step of determining the to-be-updated layer of the central architecture in the target sub-search space based on the output vector of the n-1th layer in the recurrent neural network comprises:

[0031] Determine the calculated score of each element in the output vector of the n-1th layer based on the output vector of the n-1th layer in the recurrent neural network;

[0032] Determining the layer to be updated from the central architecture based on the calculated scores of each element in the output vector of the n-1th layer;

[0033] The step of determining the update operation corresponding to the layer to be updated based on the output vector of the nth layer in the recurrent neural network includes:

[0034] Determine, based on the output vector of the nth layer in the recurrent neural network, a calculated score for each element in the output vector of the nth layer;

[0035] Based on the calculated score of each element in the output vector of the nth layer, an update operation corresponding to the layer to be updated is determined.

[0036] According to a method for designing an image classification network architecture provided by the present invention, the method further includes:

[0037] When the iteration stop condition is not met, after replacing the corresponding candidate sub-search space with the updated target sub-search space, the execution returns to the node of each candidate sub-search space and sorts the candidate sub-search spaces in descending order according to their performance.

[0038] The present invention also provides an image classification network architecture design device, comprising:

[0039] A construction unit is used to use each candidate sub-search space as a node, sort each candidate sub-search space in order from low to high according to the performance of each candidate sub-search space, and connect every two adjacent nodes from low to high to construct a directed graph of the candidate sub-search space; each candidate sub-search space includes a plurality of neural network architectures, and the architecture similarity between every two neural network architectures in each candidate sub-search space is greater than a preset similarity, and the performance of each candidate sub-search space is determined based on a sample image, a classification label of the sample image, and a predicted classification result of the sample image;

[0040] An extraction unit, configured to construct a multi-layer graph convolutional neural network based on the candidate sub-search space directed graph, and extract a feature vector of each node in the last layer of the graph convolutional neural network;

[0041] An input unit, used to determine the input vector of the first layer in the multi-layer recurrent neural network based on the feature vector of each node in the last layer of the graph convolutional neural network, and obtain the output vector of each layer in the recurrent neural network based on the input vector of the first layer;

[0042] An updating unit, configured to determine a target sub-search space from each candidate sub-search space based on the output vectors of each layer in the recurrent neural network, update the target sub-search space, and update the graph convolutional neural network and the recurrent neural network based on the updated target sub-search space;

[0043] an iteration unit, configured to determine a reward value based on the performance of the target sub-search space and the performance of the updated target sub-search space, and perform parameter iteration on the updated graph convolutional neural network and the updated recurrent neural network based on the reward value;

[0044] The determination unit is used to determine the image classification network architecture based on the updated target sub-search space under the condition that the iteration stops.

[0045] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image classification network architecture design method described in any one of the above is implemented.

[0046] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described image classification network architecture design methods.

[0047] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned image classification network architecture design methods.

[0048] The image classification network architecture design method, device, equipment and medium provided by the present invention use candidate sub-search spaces to organize the search for the architecture, find a small and effective sub-search space from it, and then iteratively modify the architecture and update the sub-search space, which can adaptively adjust the neural network architecture in the sub-search space, greatly improving the effectiveness and computational efficiency of the neural architecture search, and breaking through the problem of low search efficiency in the predefined ultra-large search space in the prior art. In addition, the embodiment of the present invention uses a graph convolutional neural network method, with candidate sub-search spaces as nodes, which can efficiently encode the relationship information between neural network architectures, and the discriminant features learned are conducive to using recurrent neural networks to determine how to modify the network architecture. Furthermore, the present invention constructs a cluster of sub-search spaces from the entire architecture search space, iteratively modifies the architecture and updates the sub-search space, and combines the global architecture search with the local architecture search, which can improve the performance and efficiency of the neural architecture search. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 It is a flowchart of the image classification network architecture design method provided by the present invention;

[0051] Figure 2 is a schematic diagram of distance calculation between the neural network architecture α and the neural network architecture β provided by the present invention;

[0052] Figure 3 It is a schematic diagram of the structure of a four-layer convolutional neural network provided by the present invention;

[0053] Figure 4 This is a schematic diagram of the architecture searched on ImageNet-16-120 using the MobileNet-like search space provided by the present invention;

[0054] Figure 5 This is a schematic diagram of the architecture searched on CIFAR-10 using the MobileNet-like search space of the present invention;

[0055] Figure 6 It is a schematic diagram of the architecture searched on CIFAR-100 using the MobileNet-like search space provided by the present invention;

[0056] Figure 7It is a structural schematic diagram of the image classification network architecture design device provided by the present invention;

[0057] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] There are also gradient-based neural architecture search methods in the prior art to design image classification network architectures. This method parameterizes the neural network architecture using learnable structural parameters and updates the structural parameters using the back-propagation algorithm commonly used in artificial neural networks. A common strategy is to relax the discrete search space, that is, each candidate operation related to the neural network architecture is characterized by its importance. Therefore, the gradient update strategy can be used to effectively optimize the search path of the neural network architecture. In this process, it is necessary to optimize the neural network architecture parameters and network weight parameters at the same time. In addition, there are also heuristic criteria used to select and prune candidate operations, and finally obtain an optimal neural network architecture. The prior art also divides the search space into multiple subspaces, and uses Monte Carlo tree search, architecture topology node growth, and network operation step-by-step expansion strategies to further reduce the search space.

[0060] However, in order to cover as many excellent architectures as possible, the search space of the gradient-based neural architecture search method needs to be designed to be large enough, resulting in high computational complexity. Secondly, although the search space division can reduce the neural architecture search time, the space division is preset and static and cannot be dynamically adjusted, which affects the flexibility of the architecture design.

[0061] To this end, the present invention provides a method for designing an image classification network architecture. Figure 1 It is a flow chart of the image classification network architecture design method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0062] Step 110, taking each candidate sub-search space as a node, sorting each candidate sub-search space in order from low to high according to the performance of each candidate sub-search space, and connecting every two adjacent nodes from low to high to construct a directed graph of the candidate sub-search space; each candidate sub-search space includes multiple neural network architectures, and the architecture similarity between every two neural network architectures in each candidate sub-search space is greater than a preset similarity, and the performance of each candidate sub-search space is determined based on the sample image, the classification label of the sample image, and the predicted classification result of the sample image.

[0063] Here, each candidate sub-search space can be a set of neural network architectures randomly initialized from the global search space, and a set of corresponding candidate sub-search spaces are constructed with these neural network architectures as the center point. Among them, the global search space defines the entire set of all possible neural network architectures, and the global search space can be described by a set consisting of all predefined components, component connections, pooling and other operations. A component includes convolution operations and the connection relationship between convolution operations. Convolution operations can include "1×1 convolution", "3×3 convolution", "5×5 convolution", "7×7 convolution", and so on. The connection relationship between convolution operations is divided into "connection" and "disconnection"; in the case of "connection", it can be further divided into different types such as "concatenation" and "addition". Pooling operations can be "added" or "not added" between different components.

[0064] In addition, in order to intuitively explain the global search space, all possible operations can be organized into a directed graph. The graph has all possible candidate components as nodes. The directed edges between nodes represent the connection relationship between two components. Among them, a directed graph can be formed in a layer-by-layer stacking manner, and each layer can contain multiple components. The directed graph constitutes a global search space. Using the neural architecture search algorithm, the neural network architecture searched from the global search space corresponds to a subgraph of the directed graph. The user can determine the size of the global search space according to application needs. Optionally, an embodiment of the present invention can organize the candidate components into a directed graph in a layered manner.

[0065] For example, let the global search space be Ω, and let α(∈Ω) and β(∈Ω) be two neural network architectures in the global search space Ω. Let D be the total number of components contained in a single neural network architecture. Formally, the neural network architectures α and β can be represented as an ordered sequence of components, that is, α = (α (1) ,α (2) ,…α (D) ) and β=(β (1) ,β (2) ,…β (D) ), where α (k) and β (k)They represent the kth component of the neural network architecture α and the neural network architecture β respectively, k = 1, 2,…, D.

[0066] Optionally, the architecture similarity between every two neural network architectures can be represented by the distance d(α, β) between the two neural network architectures, and d(α, β) can be calculated using the following formula:

[0067]

[0068] Among them, d(α,β) represents the distance between the neural network architecture α and the neural network architecture β; δ(·) is the counting function, that is, if the component α (k) and component β (k) If they are not the same, then its value is 1, otherwise its value is , that is, the similarity between neural network architectures can be measured by the number of different components.

[0069] Figure 2 is a schematic diagram of distance calculation between the neural network architecture α and the neural network architecture β provided by the present invention, such as Figure 2 As shown in the figure, "Conv3×3" and "Conv5×5" represent the "3×3 convolution" operation and the "5×5 convolution" operation respectively. Therefore, Figure 2 The distance between the two neural network architectures in is 3.

[0070] Second, randomly sample M candidate neural network architectures {α1,α2,…,α M}, where α i represents the i-th candidate neural network architecture, i=1,2,…,M.

[0071] A candidate sub-search space is constructed based on the distance between every two neural network architectures. Specifically, M neural network architectures {α1, α2, … α M}, respectively, with each neural network architecture α i (i=1,2,…M) is used as the center to construct the corresponding candidate sub-search space, each candidate sub-search space includes the neural network architecture α i All neural network architectures whose distance between them does not exceed a given threshold S can be constructed as follows:

[0072] Ω αi ={α j |d(α i ,S j )≤S,α j ∈Ω}

[0073] Among them, the candidate sub-search space Ω αi Represents the neural network architecture α i The sub-search space centered onj is a neural network architecture in the global search space Ω, d(α i ,α j ) represents the neural network architecture α i And the neural network architecture α j The architectural distance between them is , and S is the distance threshold.

[0074] According to the above method, M neural network architectures {α1, α2, …α M Each neural network architecture in} is centered, and a candidate sub-search space can be defined accordingly to obtain M candidate sub-search spaces. Here, represents the candidate sub-search space centered on the neural network architecture α1, represents the candidate sub-search space centered on the neural network architecture α2, and so on. Represents the neural network architecture α M The candidate sub-search space centered on .

[0075] It should be noted that the embodiment of the present invention uses the number of different components between two neural network architectures to measure the similarity between the two to ensure the diversity of neural network architecture selection and bring great computational efficiency. In the same candidate sub-search space, the architecture distance between each neural network architecture and its corresponding central architecture does not exceed the given threshold S. Therefore, the neural network architectures in each candidate sub-search space constitute an architecture cluster, which are all neighboring structures of the central architecture. In an embodiment of the present invention, the threshold S can be set to 8. In an embodiment of the present invention, selecting a relatively small threshold S can reduce the number of subsequent iterations and save computational overhead.

[0076] In order to utilize the relationship between different sub-search spaces, the M candidate sub-search spaces constructed above are Organized into a directed graph (i.e., a directed graph of candidate sub-search space) to guide the search to find a better neural network architecture. represents a directed graph of candidate sub-search spaces, where ν represents a set of nodes, each node corresponds to a candidate sub-search space, and ε represents a set of directed edges, each edge connecting two candidate sub-search spaces. The following is a directed graph of candidate sub-search spaces: (hereinafter referred to as directed graph ) The construction method of the edge set ε in :

[0077] First, from each candidate sub-search space Extract the corresponding central architecture The given training set (sample images and their classification labels) is used to train the model. After the training is completed, the validation set is used to evaluate its performance (such as the accuracy index in the image classification task). Then, the candidate sub-search space is The average performance of all the architectures it contains (such as the accuracy index in the classification task) is used as its corresponding candidate sub-search space Next, sort the M performances. Finally, connect the low-performance candidate sub-search space (node) to all the candidate sub-search spaces (nodes) with higher performance. For example, for the node corresponding to the candidate sub-search space with the lowest performance, starting from this node, it will be connected to all other nodes. And so on, the directed graph can be completed in the end. Construction of the middle edge set ε.

[0078] Among them, a directed graph is used to organize M candidate sub-search spaces The advantage of this approach is that a directed edge depicts the direction of performance growth from one architecture to another, thereby guiding the neural architecture search algorithm to search for the optimal architecture along the direction of better performance, reducing the search path and improving search efficiency.

[0079] Step 120: Based on the candidate sub-search space directed graph, a multi-layer graph convolutional neural network is constructed, and the feature vector of each node in the last layer of the graph convolutional neural network is extracted.

[0080] Specifically, the purpose of constructing a graph convolutional neural network is to obtain the discriminant feature vector of the candidate sub-search space, so as to determine M candidate sub-search spaces based on the discriminant feature vector. The candidate sub-search space with the most potential for further search is selected. The graph structure in the graph convolutional neural network here is based on the directed graph Constructed.

[0081] In order to use graph convolutional neural networks to learn directed graphs To identify the features of each node in the graph, we first need to define the directed graph The initial features of each node in a directed graph There are M nodes in total, each node corresponds to a candidate sub-search space. The corresponding node is simply referred to as node i. Specifically, for each candidate sub-search space Using its central architecture α i The network topology feature vector is used as the initial feature of node i.

[0082] Figure 3 Schematic diagram of the structure of a four-layer convolutional neural network provided by the present invention, such as Figure 3As shown, the four-layer convolutional neural network structure includes input layer L1, middle layer L2, middle layer L3, and output layer L4, which correspond to the solid circles in the figure respectively. represents information aggregation, which can be understood as a “feature addition” operation. Figure 3 As shown, “1” represents the “direct connection” operation, “2” represents the “Conv5×5” operation, “3” represents the “Conv3×3” operation, and “4” represents the “Conv1×1” operation. Figure 3 The network structure shown can be described by the following matrix H:

[0083]

[0084] Among them, "∞" means no connection. For example, the "∞" in the first row of the first column of the matrix H means that the L1 layer is not connected to the L1 layer itself, the "∞" in the first row of the second column of the matrix H means that the L2 layer is not connected to the L1 layer, the "1" in the fourth row of the first column means that there is a "direct connection" operation between the L1 layer and the L4 layer, and the "4" in the fourth row of the third column means that the L3 layer and the L4 layer are connected through the "Conv1×1" operation. It should be pointed out that in the actual implementation process, these operations can also be numbered starting from 0. Further, the lower left corner of the matrix H is organized into a vector h=(2,1,1,3,1,4) by column scanning. T , the vector h is called the network topology feature vector.

[0085] It should be pointed out that for a structure consisting of multiple stacked components, when the components further contain sub-operations, they can be first described by tensors and then converted into vector representations.

[0086] Refer to the above Figure 3 The characterization process of the network structure shown in the figure is for the directed graph The feature vector of each node is initialized to the network topology feature vector of its corresponding central architecture. In this way, the feature vectors of M nodes can be obtained, denoted as v1, v2, …, v M , where v1 is the feature vector of node 1, v2 is the feature vector of node 2, and so on. M is the feature vector of node M.

[0087] Step 130: determine the input vector of the first layer in the multi-layer recurrent neural network based on the feature vector of each node in the last layer of the graph convolutional neural network, and obtain the output vector of each layer in the recurrent neural network based on the input vector of the first layer.

[0088] Considering that when some components of any neural network architecture change, its corresponding performance will also change, the embodiment of the present invention uses a graph convolutional neural network to learn the characteristics of each candidate sub-search space to determine the performance of each neural network architecture.

[0089] First, a recurrent neural network (RNN) is constructed, which will be used as a predictor. Structurally, the RNN is composed of L conventional recurrent neural units connected in series. According to the conventional terminology of recurrent neural networks, each unit of the RNN corresponds to a layer of the RNN. Therefore, the RNN has a total of L layers of units. For the first layer of the RNN, the feature vectors of all nodes in the last layer of the graph convolutional neural network constructed above are used as input.

[0090] Optionally, the feature vectors of all nodes in the last layer of the graph convolutional neural network are concatenated in node order to form a long vector as the input vector of the first layer of the recurrent neural network. The output vector of the first layer is used as the input vector of the second layer of the recurrent neural network, and so on, the output vector of the L-1th layer is used as the input vector of the Lth layer. Among them, the embodiment of the present invention takes into account that the more layers of the recurrent neural network, the more likely it is that the gradient will disappear, thereby affecting the network training. Therefore, the embodiment of the present invention can set L to 7.

[0091] Step 140: Based on the output vectors of each layer in the recurrent neural network, determine the target sub-search space from each candidate sub-search space, update the target sub-search space, and update the graph convolutional neural network and the recurrent neural network based on the updated target sub-search space.

[0092] In this step, we first use a recurrent neural network to select the candidate sub-search space with the greatest search potential (i.e., the target sub-search space), and then based on the central architecture α corresponding to the target sub-search space s , obtain the architecture modification strategy, and complete the current architecture modification according to this strategy. In this process, the recurrent neural network will be used as a predictor, that is, predicting the most potential candidate sub-search space, the network layer to be updated in the candidate sub-search space, and the update operation corresponding to the network layer to be updated.

[0093] After determining the update operation corresponding to the network layer to be updated, the target sub-search space is updated, and the graph convolutional neural network and the recurrent neural network are updated based on the updated target sub-search space.

[0094] For example, the target sub-search space is The target sub-search space is is the M candidate sub-search space First, take out the candidate sub-search space The central architecture of α s , determine the central architecture α s The corresponding network layer to be updated is completed, and the update operation of the network layer to be updated is completed, and the new neural network architecture is obtained as β s At this point, the neural network architecture β s Will replace the neural network architecture α s As a candidate sub-search space Further, according to the above method, the neural network architecture β s Generate new candidate sub-search space for the center Thus, the candidate sub-search space Replace the original candidate sub-search space

[0095] Next, a new candidate sub-search space directed graph is constructed according to the method described in step 110. It should be noted that although only one candidate sub-search space is corrected, the directed graph with M candidate sub-search space nodes The edges will be modified, especially the connection to the original candidate sub-search space will no longer exist and will instead be connected to the candidate sub-search space The edge of the candidate sub-search space Start connecting to the edges of other candidate sub-search spaces.

[0096] It should be noted that the reason why the embodiment of the present invention does not use the newly generated candidate sub-search space to replace the candidate sub-search space with the lowest performance is that the diversity of the candidate sub-search space needs to be maintained during the iteration process to avoid falling into a local minimum.

[0097] Step 150: determining a reward value based on the performance of the target sub-search space and the performance of the updated target sub-search space, and performing parameter iteration on the updated graph convolutional neural network and the updated recurrent neural network based on the reward value;

[0098] Step 160: When the iteration stopping condition is met, determine the image classification network architecture based on the updated target sub-search space.

[0099] Specifically, since the feature vectors of all nodes in the last layer of the graph convolutional neural network are concatenated as the input vectors of the first layer of the recurrent neural network, after the target sub-search space is updated, the feature vectors of each node in the last layer of the corresponding graph convolutional neural network will also change, and then the input vector of the first layer of the recurrent neural network will also change. Therefore, it is necessary to update the candidate sub-search space directed graph based on the updated target sub-search space, and then update the graph convolutional neural network and the recurrent neural network.

[0100] In addition, the feature vectors of all nodes in the last layer of the graph convolutional neural network are concatenated as the input vectors of the first layer of the recurrent neural network, indicating that the graph convolutional neural network and the recurrent neural network are connected together, and their parameters need to be iteratively updated in order to finally find the optimal network structure.

[0101] In the standard policy gradient-based reinforcement learning framework, the agent is a combination of a graph convolutional neural network and a recurrent neural network, the state is the neural network architecture obtained based on the agent, and the action is a modification of the central architecture.

[0102] Assume that the target sub-search space is The updated target sub-search space is Each sub-search space contains S neural network architectures. First, the target sub-search space is calculated as The average performance of the S neural network architectures (such as the accuracy index in image classification tasks), denoted as acc α Then, the updated target sub-search space is calculated as The average performance of the S neural network architectures is denoted as acc β Optionally, for image classification tasks, compute the neural network architecture β s The accuracy of the neural network architecture α s accuracy, and use the difference as a reward. Therefore, for the action "modify the neural network architecture α s Get the neural network architecture β s The reward value of " is calculated according to the following formula:

[0103] reward=acc α -acc β

[0104] Among them, reward is the reward value, acc α The target sub-search space is Average performance of neural network architecture, acc β The updated target sub-search space is Average performance of neural network architectures.

[0105] It should be noted that, unlike the commonly used reward definition method, the embodiment of the present invention does not directly use accuracy as a reward, but uses accuracy difference as a reward. If the accuracy is improved (the reward is greater than zero), such a correction is worthy of encouragement, so the reward is positive; if the accuracy is reduced, such a correction needs to be punished, so the reward is less than zero. After obtaining the reward for the current action, the conventional reinforcement learning based reward policy gradient is further used to guide the training of the network.

[0106] It should be further pointed out that the reward-based policy gradient method will update the parameters of the graph convolutional neural network and the recurrent neural network in an iterative manner. In this process, the architecture and its performance data accumulated in previous iterations can be sampled to select a part to participate in the current update of the graph convolutional neural network parameters and the recurrent neural network parameters.

[0107] Finally, check whether the iteration stop condition is met. If so, modify the optimal neural network architecture based on the central architecture of the target sub-search space and its corresponding neural network architecture. Optionally, the iteration stop condition can be set to reach a specified maximum number of iterations or the performance of the searched neural network architecture reaches a preset value.

[0108] The image classification network architecture design method provided by the embodiment of the present invention adopts a candidate sub-search space to organize the search for the architecture, finds a small and effective sub-search space, and then iteratively modifies the architecture and updates the sub-search space, which can adaptively adjust the neural network architecture in the sub-search space, greatly improving the effectiveness and computational efficiency of the neural architecture search, and breaking through the problem of low search efficiency in the predefined ultra-large search space in the prior art. In addition, the embodiment of the present invention uses a graph convolutional neural network method, with candidate sub-search spaces as nodes, which can efficiently encode the relationship information between neural network architectures, and the discriminant features learned are conducive to using recurrent neural networks to determine how to modify the network architecture. Furthermore, the present invention constructs a cluster of sub-search spaces from the entire architecture search space, iteratively modifies the architecture and updates the sub-search space, and combines the global architecture search with the local architecture search, which can improve the performance and efficiency of the neural architecture search.

[0109] Furthermore, although the embodiment of the present invention uses the image classification task as the training target for technical description, the network search framework of the embodiment of the present invention can be applied to other visual tasks, and only the network structure needs to be partially modified. For example, for the target detection task, it is only necessary to replace the classifier for the image classification task with the target detection network structure at the top of the network (commonly known as the head network), which greatly increases the flexibility of the embodiment of the present invention.

[0110] Based on the above embodiment, each candidate sub-search space is determined based on the following steps:

[0111] Randomly sample multiple candidate neural network architectures from the global search space;

[0112] Based on each candidate neural network architecture, and a neural network architecture whose similarity to each candidate neural network architecture is greater than a preset similarity, a corresponding candidate sub-search space is constructed; each candidate neural network architecture is a central architecture of the corresponding candidate sub-search space.

[0113] Specifically, the global search space can be described by a set of all predefined components, component connections, pooling operations, etc. A component contains convolution operations and the connection relationship between convolution operations.

[0114] For example, randomly sample M candidate neural network architectures {α1,α2,…,α M}, where α i represents the i-th candidate neural network architecture, i=1,2,…,M.

[0115] A candidate sub-search space is constructed based on the distance between every two neural network architectures. Specifically, M neural network architectures {α1, α2, … α M}, respectively, with each neural network architecture α i (i=1,2,…M) is used as the center to construct the corresponding candidate sub-search space, each candidate sub-search space includes the neural network architecture α i All neural network architectures whose distance between them does not exceed a given threshold S can be constructed as follows:

[0116]

[0117] Among them, the candidate sub-search space Represents the neural network architecture α i The sub-search space centered on j is a neural network architecture in the global search space Ω, d(α i ,α j ) represents the neural network architecture α i And the neural network architecture α j The architectural distance between them is , and S is the distance threshold.

[0118] According to the above method, M neural network architectures {α1, α2, …α M Each neural network architecture in} is centered, and a candidate sub-search space can be defined accordingly to obtain M candidate sub-search spaces. Here, represents the candidate sub-search space centered on the neural network architecture α1, represents the candidate sub-search space centered on the neural network architecture α2, and so on. Represents the neural network architecture α M The candidate sub-search space centered on .

[0119] Based on any of the above embodiments, extracting the feature vector of each node in the last layer of the graph convolutional neural network includes:

[0120] Based on the feature vector of node i, the feature vector of node j, and the feature vector of the edge from node i to node j in the graph convolutional neural network, determine the contribution coefficient of node j to node i;

[0121] Based on the contribution coefficient of node j to node i and the feature vector of node j, the feature vector of node i is updated.

[0122] Specifically, since the directed graph The edges of are from low-performance nodes to high-performance nodes. Therefore, the directed graph It contains important discriminative information. In order to make full use of the directed graph The edge information of , further defines the edge features. Each edge in connects two nodes, and the difference between the feature vectors of the two nodes is used to represent the edge feature.

[0123] Assume a directed graph There is an edge from node i to node j in the network, and the eigenvector of this edge is e i→j , then the eigenvector of the edge is calculated as follows:

[0124] e i→j =v j -v i

[0125] Among them, e i→j is the feature vector of the edge from node i to node j, v i is the feature vector of node i, v j is the feature vector of node j.

[0126] In obtaining a directed graph After obtaining the node feature vector and edge feature vector of the candidate sub-search space, the discriminant feature learning of the node is further realized according to the conventional graph convolutional neural network. Stacking three layers constitutes a three-layer graph convolutional neural network. The feature vector output of each node in the last layer is used as the final discriminant feature vector. The feature vector of each node in the last layer of the graph convolutional neural network is determined based on the following steps:

[0127] First, a node contribution weighting mechanism is introduced to implement the convolution operation on the node feature vector. The convolution operation on the node feature vector is to converge the feature vectors of all neighboring nodes of the node to the node, thereby obtaining an updated node feature vector. In an embodiment of the present invention, feature convergence will be performed in a weighted manner, and the weight coefficient will be calculated by the node contribution weighting mechanism.

[0128] Specifically, for each node i, the node contribution weighting mechanism calculates the contribution coefficient θ with its neighbor node jij . In this process, the feature vector of node i, the feature vector of node j, and the corresponding edge feature vector will be concatenated. The node contribution weighting mechanism is parameterized by the weight vector a to be learned, and the exponential linear unit (ELU) is used as the activation function. Among them, the exponential linear unit is a correction when the convolution result is a negative number. That is to say, let the input of a neuron be x. If x is a positive number, the exponential linear unit outputs x; otherwise, it outputs α(exp(x)-1), where exp(·) is a commonly used exponential function with natural numbers as the base, and α is a scaling factor. In an embodiment of the present invention, α is 0.5. Compared with the original Rectified Linear Unit (ReLU), this function has a smaller slope for inputs less than zero. Since the derivative is always non-zero, the appearance of silent neurons can be reduced, which solves the problem that the zeroing operation causes the neurons to not learn after the linear rectifier unit enters the negative interval. The exponential linear unit has the characteristics of non-linearity and non-saturation, and can converge quickly when the conventional stochastic conjugate gradient descent method is used to optimize the model. Specifically, for node i, let node j be its neighbor node, and the node contribution coefficient θ of node j to node i is ij Calculated as follows:

[0129]

[0130] Among them, θ ij is the node contribution coefficient of node j to node i, ELU(·) is the exponential linear unit, exp(·) is a commonly used exponential function with natural numbers as the base, v i is the feature vector of node i, v j is the eigenvector of node j, e i→j is the eigenvector of the edge connecting node i and node j, is a node set that includes all first-order neighbor nodes of node i and node i itself, and k is a node that belongs to A node, v k is the eigenvector of node k, e i→k is the feature vector of the edge connecting node i and node k, || represents the concatenation operation along the feature dimension, vector a is the parameter to be learned, [] represents the concatenated feature, and superscript T represents the vector transposition operation. In the denominator of the above formula, the weights of all adjacent nodes are normalized to ensure that the weight coefficients of all neighbor nodes to the central node are positive numbers less than 1, and the sum of the weight coefficients of all neighbor nodes is equal to 1, thereby playing the role of weighted average.

[0131] For each node, the above feature aggregation process includes not only the feature vectors of neighboring nodes, but also the feature vector of the node itself. After obtaining the normalized contribution coefficient of each neighborhood, these neighboring node feature vectors are weighted summed. In addition, the commonly used linear rectifier unit is applied to the summation result as an activation function. Specifically, the feature vector of each node is updated as:

[0132]

[0133] Among them, v′ i is the updated feature vector of node i, RELU(·) is a commonly used linear rectifier unit, v j is the feature vector of node j, θ ij is the contribution coefficient of node j to node i, is a node set that includes all first-order neighbor nodes of node i and node i itself, and j is a node that belongs to A node of .

[0134] Based on any of the above embodiments, the feature vector of the edge is determined based on the following steps:

[0135] Determine the contribution coefficient of edge q to edge p based on the feature vector of edge p, the feature vector of edge q adjacent to edge p, and the feature vector of the intersection node between edge p and edge q in the graph convolutional neural network;

[0136] Based on the contribution coefficient of edge q to edge p, update the eigenvector of edge p.

[0137] Here, the edge contribution weighting mechanism is introduced to implement the convolution operation on the edge feature vector. In order to balance the importance between nodes and edges, the edge feature vector is also updated. The update of the edge feature vector is similar to the update of the node feature vector.

[0138] Specifically, the edge contribution weighting mechanism is parameterized by the weight vector b to be learned, and ELU is used as the activation function. Any edge p in p , the eigenvector of the adjacent edge q is denoted by e q , edge p and its adjacent edge q in a directed graph The feature vector of the intersection node in is expressed as The side contribution coefficient of side q to side p Calculated as follows:

[0139]

[0140] in, is the edge contribution coefficient of the adjacent edge q to the edge p, ELU(·) is the exponential linear unit, exp(·) is a commonly used exponential function with natural numbers as the base, ep is the eigenvector of edge p, e q is the eigenvector of edge q, is the characteristic vector of the intersection node between edge p and its adjacent edge q, is the set of all first-order neighbor edges of edge p and edge p itself, and k is the edge of An edge of k is the eigenvector of edge k, is the feature vector of the intersection node of edge p and its adjacent edge q, || represents the concatenation operation along the feature dimension, vector b is the parameter to be learned, [] represents the concatenated feature, and the superscript T represents the vector transpose operation.

[0141] For each edge p, the edge feature vector is updated as:

[0142]

[0143] Among them, e' p is the updated feature of edge p, RELU(·) is a commonly used linear rectification function, e q is the feature vector of node q, is the edge contribution coefficient of the adjacent edge q to the edge p, is the edge set that includes all first-order neighbor edges of edge p and edge p itself, and k is the edge set that belongs to An edge of .

[0144] The node contribution weighting mechanism and edge contribution weighting mechanism designed above can enable the node feature vector and edge feature vector to periodically update themselves to obtain high-level features.

[0145] Finally, using the above node feature vector and edge feature vector update strategy, multiple such node feature vector and edge feature vector update layers are stacked. Structurally, the embodiment of the present invention can stack three layers in total. In the above manner, the construction of the graph convolutional neural network is completed. The output vectors of each node of the last layer of the graph convolutional neural network will be used as a directed graph. The discriminant feature vector of the corresponding node in .

[0146] It can be seen that the embodiment of the present invention adopts the strategy of simultaneously updating the node feature vector and the edge feature vector to update the directed graph In this way, the graph convolutional neural network model can accept node feature vectors and edge feature vectors as input, integrate point and edge information into the feature representation at the same time, and iteratively update node features and edge features in parallel, which can fully extract the directed graph. The information contained in it.

[0147] In addition, in the embodiment of the present invention, there is no need to perform the pooling operation usually included in the conventional graph convolutional neural network, that is, there is no need to perform a downsampling operation on the nodes of the graph. This is because, in the embodiment of the present invention, the directed graph Each first edge in is from a low-performance node to all high-performance nodes. Therefore, the graph already reflects the correlation between the local area and the global area, and there is no need to expand the receptive field of feature learning by downsampling.

[0148] Based on any of the above embodiments, based on the output vectors of each layer in the recurrent neural network, determining a target sub-search space from each candidate sub-search space, and updating the target sub-search space, including:

[0149] Based on the output vector of the first layer in the recurrent neural network, a target sub-search space is determined from each candidate sub-search space;

[0150] Based on the output vector of the n-1th layer in the recurrent neural network, determine the layer to be updated of the central architecture in the target sub-search space;

[0151] Based on the output vector of the nth layer in the recurrent neural network, determining the update operation corresponding to the layer to be updated; wherein the nth layer is the last layer in the recurrent neural network;

[0152] Based on the update operation corresponding to the layer to be updated, the layer to be updated is updated to obtain an updated target sub-search space.

[0153] Wherein, based on the output vector of the first layer in the recurrent neural network, the target sub-search space is determined from each candidate sub-search space, including:

[0154] Based on the output vector of the first layer in the recurrent neural network, determining the calculation score of each element in the output vector of the first layer;

[0155] Determine a target sub-search space from each candidate sub-search space based on the calculated score of each element in the output vector of the first layer;

[0156] Based on the output vector of the n-1th layer in the recurrent neural network, determine the layer to be updated of the central architecture in the target sub-search space, including:

[0157] Based on the output vector of the n-1th layer in the recurrent neural network, determine the calculation score of each element in the output vector of the n-1th layer;

[0158] Based on the calculated scores of each element in the output vector of the n-1th layer, the layer to be updated is determined from the central architecture;

[0159] Based on the output vector of the nth layer in the recurrent neural network, the update operation corresponding to the layer to be updated is determined, including:

[0160] Determine the calculation score of each element in the output vector of the nth layer based on the output vector of the nth layer in the recurrent neural network;

[0161] Based on the calculated scores of each element in the output vector of the nth layer, the update operation corresponding to the layer to be updated is determined.

[0162] It should be noted that all the unit structures (ie, layer structures) of the recurrent neural network are the same. In the above processing, all calculations are performed according to the calculation process of the standard recurrent neural network unit.

[0163] Secondly, according to the output vector of the first layer of the recurrent neural network, determine the target sub-search space. Let the output vector of the first layer of the recurrent neural network be o. Since m candidate sub-search spaces are constructed in the above steps Therefore, the dimension of the output vector of the first layer of the recurrent neural network is M. Therefore, the output vector o is an M-dimensional vector. Now, calculate a score for each element of the output vector o. Let the score of the i-th element be The target sub-search space is It is M points The candidate sub-search space corresponding to the largest one in Calculated as follows:

[0164]

[0165] in, is the i-th score, o i is the i-th element of the output vector o of the first layer of the recurrent neural network, o k is the kth element of the output vector o of the first layer of the recurrent neural network, exp(·) is a commonly used exponential function with natural numbers as the base, and M is the number of candidate subspaces.

[0166] By finding the one with the largest score, a target sub-search space is obtained, rather than a specific neural network architecture. In addition, if the performance of each architecture is directly evaluated in the predefined global search space, it will inevitably lead to problems such as low search efficiency and poor effect. The embodiment of the present invention first finds a small and effective target sub-search space, and then iteratively modifies the neural network architecture and updates the target sub-search space. In this way, a better neural network architecture can be searched more efficiently.

[0167] From the target sub-search space Extract its central architecture α s Next, determine the central architecture α s The layer to be updated.

[0168] Specifically, let the output vector of the sixth layer of the recurrent neural network be r. The output vector r is a vector of length H, where H is the number of layers of the preset neural network architecture. Here, a score is calculated for each element of the output vector r. Let the score of the i-th element be Central framework to be modified α s The number of layers is H scores The index value corresponding to the largest one among them, where each score Calculated as follows:

[0169]

[0170] in, is the i-th score, r i is the i-th element of the output vector r of the sixth layer of the recurrent neural network, r k is the kth element of the output vector r of the sixth layer of the recurrent neural network, exp(·) is a commonly used exponential function with natural numbers as the base, and H is the number of layers of the preset neural network architecture.

[0171] Finally, modify the central architecture α s According to the seventh layer output of the recurrent neural network, the central architecture α is determined s The specific operation of . Let the output of the seventh layer of the recurrent neural network be vector s. The output vector s is a vector of length C, where C is the number of candidate operations inside the preset component. Now, calculate a score for each element of the output vector s. Let the score of the i-th element be For the central architecture α s , given a layer to be modified, its operation will be scored C The operation corresponding to the largest one is replaced by the operation corresponding to the largest one, where each score Calculated as follows:

[0172]

[0173] in, is the i-th score, s i is the i-th element of the output vector s of the seventh layer of the recurrent neural network, s k is the kth element of the output vector s of the seventh layer of the recurrent neural network, exp(·) is a commonly used exponential function with natural numbers as the base, and C is the preset single-layer candidate operand.

[0174] The above steps determine the target sub-search space, the layer to be updated of the central architecture of the target sub-search space, and the update operation corresponding to the layer to be updated. Therefore, after performing the corresponding update operation on the layer to be updated corresponding to the central architecture of the target sub-search space, a new neural network architecture is obtained.

[0175] Based on any of the above embodiments, the method further includes:

[0176] If the condition for stopping the iteration is not met, after replacing the corresponding candidate sub-search space with the updated target sub-search space, the execution returns to the step of taking each candidate sub-search space as a node and sorting each candidate sub-search space in order from low to high according to the performance of each candidate sub-search space, that is, looping through steps 110 to 160.

[0177] In order to verify the effectiveness of the present invention, two search spaces known in the art (NAS-Bench-201 and MobileNet-like) and image classification experiments based on image datasets known in the art (CIFAR-10, CIFAR-100, ImageNet) were used. Specifically, architecture search was performed on the CIFAR-10 dataset based on the NAS-Bench-201 search space, and architecture performance evaluation was performed on three datasets: the CIFAR-10 dataset, the CIFAR-100 dataset, and a subset of ImageNet; architecture search and architecture performance evaluation were performed on the ImageNet dataset based on the MobileNet-like search space.

[0178] Specifically, the NAS-Bench-201 search space is based on component architecture, and the global search space contains 15,625 candidate neural network architectures. Each component is a directed acyclic graph with 4 nodes and 6 edges. Each edge is associated with an operation. There are 5 different candidate operations, including zeroize, skip connection, 1×1 convolution layer (1-by-1convolution), 3×3 convolution layer (3-by-3convolution) and 3×3 average pooling layer (3-by-3average pooling). In addition, based on the NAS-Bench-201 search space, 1,000 candidate neural network architectures are randomly sampled from the global search space, that is, the above M takes the value of 1,000.

[0179] The MobileNet-like search space consists of 5 different units, each with consecutive layers. In each layer, the kernel sizes (kernel sizes from {3,5,7}), expansion rates (expansion rates from {1,3,6}) (where expansion rate 1 is only used in the first layer), and the number of layers in each unit (the number of layers from {2,3,4}) of the MBConv Block are searched. The global search space contains 3.65×1015 candidate neural network architectures. Based on the MobileNet-like search space, 1000 candidate neural network architectures are randomly sampled from the global search space, that is, M is 1000. The CIFAR-10 dataset contains 50,000 training images and 10,000 test images consisting of 10 classes, with a pixel resolution of 32×32. The images of the CIFAR-100 dataset are the same as those of CIFAR-10, but their categories are fine-grained and divided into 100 classes. The ImageNet dataset contains 1,281,167 training images and 50,000 validation images consisting of 1,000 classes. The ImageNet-16-120 dataset is a downsampled subset of ImageNet, containing 151,700 training images, 3,000 validation images, and 3,000 test images. ImageNet-16-120 downsamples the image pixels of ImageNet to 16×16 and only retains images of the first 120 categories. In this embodiment, the cross entropy loss (CE) is used as the loss function of the neural network architecture, and the optimization algorithm adopts the classic error back propagation algorithm in this field.

[0180] By executing the method of the embodiment of the present invention, the optimal neural network architecture can be searched from the image data set. It should be noted that the method provided by the embodiment of the present invention can be naturally applied to classification scenarios with various numbers of categories, that is, only the number of predicted categories of the classifier at the top of the network needs to be modified.

[0181] The experimental results show that, using the method provided by the embodiment of the present invention, neural network architectures with excellent performance can be searched on these data sets. Table 1 is a performance list of the NAS-Bench-201 search space on three data sets, and Table 2 is a performance list based on the MobileNet-like search space on ImageNet. In Table 1, the three columns of numbers on the left represent the image recognition accuracy obtained by the optimal structure searched on the corresponding database. In Table 2, "Top-1 Acc" represents the accuracy index ranked first, "Top-5 Acc" represents the top 5 accuracy indexes, "#MADDs (M)" represents the number of multiplication and addition cumulative operations (how many megabytes), and "Search cost (GPU days)" represents the number of days consumed by the algorithm. It can be found from Tables 1 and 2 that the method provided by the embodiment of the present invention can greatly improve the performance of neural network search, while greatly improving the efficiency of neural architecture search.

[0182] Table 1

[0183]

[0184] Table 2

[0185]

[0186] In order to further verify the method of the present invention, Figure 4 This is a schematic diagram of the architecture searched on ImageNet-16-120 using the MobileNet-like search space provided by the present invention. Figure 5 This is a schematic diagram of the architecture that the present invention searches for on CIFAR-10 using the MobileNet-like search space. Figure 6 : is a schematic diagram of the architecture searched on CIFAR-100 using the MobileNet-like search space provided by the present invention. Figure 4 , Figure 5 , Figure 6 As shown in the figure, “Conv-3×3” represents a 3×3 convolution operation, “MB1-3×3” represents a mobile inverted bottleneck convolution with a convolution kernel of 3x3 and a dilation rate of 1, “MB6-3×3” represents a mobile inverted bottleneck convolution with a convolution kernel of 3×3 and a dilation rate of 6, “MB6-7×7” represents a mobile inverted bottleneck convolution with a convolution kernel of 7×7 and a dilation rate of 6, and “Pooling+FC” represents pooling and full connection.

[0187] It can be seen that on different data sets, the embodiments of the present invention can search for the optimal neural network architecture for each data set.

[0188] The method provided by the embodiment of the present invention can greatly improve the effectiveness and efficiency of neural network search, aiming at the practical application demand that existing methods are difficult to perform efficient architecture search in a predefined ultra-large search space. Among them, the method provided by the embodiment of the present invention does not limit the number of candidate architectures in the global search space, and can meet various application needs; the method provided by the embodiment of the present invention models the neural architecture search task as a problem that can adaptively adjust the neural network architecture in the sub-search space, which has the characteristics of intuitive thinking, easy programming, and high computational efficiency; the method provided by the embodiment of the present invention uses images as input, and no additional visual feature extraction is required, so efficient neural architecture search can be achieved. The method provided by the embodiment of the present invention breaks through the problem of low efficiency and poor effect of the prior art in searching in a predefined ultra-large search space, and has broad application prospects in the fields of digital entertainment and art, consumer electronics, autonomous driving, medical image processing, edge computing, etc.

[0189] The image classification network architecture design device provided by the present invention is described below. The image classification network architecture design device described below and the image classification network architecture design method described above can be referenced to each other.

[0190] Based on any of the above embodiments, Figure 7 is a schematic diagram of the structure of the image classification network architecture design device provided by the present invention, such as Figure 7 As shown, the device comprises:

[0191] A construction unit 710 is used to use each candidate sub-search space as a node, sort each candidate sub-search space in order from low to high according to the performance of each candidate sub-search space, and connect every two adjacent nodes from low to high to construct a directed graph of the candidate sub-search space; each candidate sub-search space includes a plurality of neural network architectures, and the architecture similarity between every two neural network architectures in each candidate sub-search space is greater than a preset similarity, and the performance of each candidate sub-search space is determined based on a sample image, a classification label of the sample image, and a predicted classification result of the sample image;

[0192] An extraction unit 720 is used to construct a multi-layer graph convolutional neural network based on the candidate sub-search space directed graph, and extract a feature vector of each node in the last layer of the graph convolutional neural network;

[0193] An input unit 730 is used to determine an input vector of a first layer in a multi-layer recurrent neural network based on a feature vector of each node in the last layer of the graph convolutional neural network, and obtain output vectors of each layer in the recurrent neural network based on the input vector of the first layer;

[0194] An updating unit 740, configured to determine a target sub-search space from each candidate sub-search space based on the output vectors of each layer in the recurrent neural network, update the target sub-search space, and update the graph convolutional neural network and the recurrent neural network based on the updated target sub-search space;

[0195] An iteration unit 750, configured to determine a reward value based on the performance of the target sub-search space and the performance of the updated target sub-search space, and perform parameter iteration on the updated graph convolutional neural network and the updated recurrent neural network based on the reward value;

[0196] The determination unit 760 is used to determine the image classification network architecture based on the updated target sub-search space when the condition for stopping the iteration is met.

[0197] Figure 8 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 8As shown, the electronic device may include: a processor 810, a memory 820, a communication interface 830 and a communication bus 840, wherein the processor 810, the memory 820 and the communication interface 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 820 to execute the image classification network architecture design method, which includes: taking each candidate sub-search space as a node, sorting each candidate sub-search space in order from low to high according to the performance of each candidate sub-search space, and connecting every two adjacent nodes from low to high to construct a candidate sub-search space directed graph; each candidate sub-search space includes multiple neural network architectures, and the architecture similarity between each two neural network architectures in each candidate sub-search space is greater than the preset similarity, and the performance of each candidate sub-search space is determined based on the sample image, the classification label of the sample image and the predicted classification result of the sample image; based on the candidate sub-search space directed graph, a multi-layer graph convolutional neural network is constructed, and the feature vector of each node in the last layer of the graph convolutional neural network is extracted; based on the The feature vectors of each node in the last layer of the graph convolutional neural network determine the input vector of the first layer in the multi-layer recurrent neural network, and the output vectors of each layer in the recurrent neural network are obtained based on the input vectors of the first layer; based on the output vectors of each layer in the recurrent neural network, a target sub-search space is determined from each candidate sub-search space, and the target sub-search space is updated, and the graph convolutional neural network and the recurrent neural network are updated based on the updated target sub-search space; based on the performance of the target sub-search space and the performance of the updated target sub-search space, a reward value is determined, and parameters of the updated graph convolutional neural network and the updated recurrent neural network are iterated based on the reward value; when the iteration stop condition is met, the image classification network architecture is determined based on the updated target sub-search space.

[0198] In addition, the logic instructions in the above-mentioned memory 820 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0199] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the image classification network architecture design method provided by the above methods, the method including: taking each candidate sub-search space as a node, sorting each candidate sub-search space in order from low to high according to the performance of each candidate sub-search space, and connecting every two adjacent nodes from low to high to construct a directed graph of the candidate sub-search space; each candidate sub-search space includes multiple neural network architectures, and the architecture similarity between every two neural network architectures in each candidate sub-search space is greater than a preset similarity, and the performance of each candidate sub-search space is determined based on the sample image, the classification label of the sample image, and the predicted classification result of the sample image; based on the candidate sub-search space directed graph, a multi-layer structure graph is constructed. Convolutional neural network, and extract the feature vector of each node in the last layer of the graph convolutional neural network; determine the input vector of the first layer in the multi-layer structure of the recurrent neural network based on the feature vector of each node in the last layer of the graph convolutional neural network, and obtain the output vector of each layer in the recurrent neural network based on the input vector of the first layer; determine the target sub-search space from each candidate sub-search space based on the output vector of each layer in the recurrent neural network, and update the target sub-search space, and update the graph convolutional neural network and the recurrent neural network based on the updated target sub-search space; determine the reward value based on the performance of the target sub-search space and the performance of the updated target sub-search space, and iterate the parameters of the updated graph convolutional neural network and the updated recurrent neural network based on the reward value; when the iteration stop condition is met, determine the image classification network architecture based on the updated target sub-search space.

[0200] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned image classification network architecture design methods, the method comprising: taking each candidate sub-search space as a node, sorting each candidate sub-search space in order from low to high according to the performance of each candidate sub-search space, and connecting every two adjacent nodes from low to high to construct a directed graph of the candidate sub-search space; each candidate sub-search space includes multiple neural network architectures, and the architecture similarity between every two neural network architectures in each candidate sub-search space is greater than a preset similarity, and the performance of each candidate sub-search space is determined based on a sample image, a classification label of the sample image, and a predicted classification result of the sample image; based on the candidate sub-search space directed graph, constructing a multi-layer graph convolutional neural network, and extracting the graph convolutional neural network. The invention relates to a method for obtaining a plurality of image convolutional neural networks (the image convolutional neural networks) and a plurality of image convolutional neural networks (the image convolutional neural networks) according to the embodiment of the present invention. The method comprises the following steps: obtaining a feature vector of each node in the last layer of the graph convolutional neural network; determining an input vector of the first layer in the recurrent neural network of the multi-layer structure based on the feature vector of each node in the last layer of the graph convolutional neural network, and obtaining an output vector of each layer in the recurrent neural network based on the input vector of the first layer; determining a target sub-search space from each candidate sub-search space based on the output vector of each layer in the recurrent neural network, updating the target sub-search space, and updating the graph convolutional neural network and the recurrent neural network based on the updated target sub-search space; determining a reward value based on the performance of the target sub-search space and the performance of the updated target sub-search space, and performing parameter iteration on the updated graph convolutional neural network and the updated recurrent neural network based on the reward value; and determining an image classification network architecture based on the updated target sub-search space when an iteration stopping condition is met.

[0201] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor. Through the description of the above implementation methods, a person of ordinary skill in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for designing an image classification network architecture, characterized in that: include: Taking each candidate sub-search space as a node, sorting each candidate sub-search space in order of performance from low to high, and connecting low-performance nodes to all nodes with higher performance than them, to construct a directed graph of candidate sub-search spaces; each candidate sub-search space includes multiple neural network architectures, and the architecture similarity between every two neural network architectures in each candidate sub-search space is greater than a preset similarity, and the performance of each candidate sub-search space is determined based on a sample image, a classification label of the sample image, and a predicted classification result of the sample image; Based on the candidate sub-search space directed graph, a multi-layer graph convolutional neural network is constructed, and a feature vector of each node in the last layer of the graph convolutional neural network is extracted; Determine the input vector of the first layer in the multi-layer recurrent neural network based on the feature vector of each node in the last layer of the graph convolutional neural network, and obtain the output vector of each layer in the recurrent neural network based on the input vector of the first layer; Based on the output vectors of each layer in the recurrent neural network, determine a target sub-search space from each candidate sub-search space, update the target sub-search space, and update the graph convolutional neural network and the recurrent neural network based on the updated target sub-search space; Determine a reward value based on the performance of the target sub-search space and the performance of the updated target sub-search space, and perform parameter iteration on the updated graph convolutional neural network and the updated recurrent neural network based on the reward value; When the iteration stopping condition is met, the image classification network architecture is determined based on the updated target sub-search space; The step of determining a target sub-search space from each candidate sub-search space based on the output vector of each layer in the recurrent neural network, and updating the target sub-search space, comprises: Determine a target sub-search space from each candidate sub-search space based on an output vector of the first layer in the recurrent neural network; Determine a layer to be updated of a central architecture in the target sub-search space based on an output vector of the n-1th layer in the recurrent neural network; Determine the update operation corresponding to the layer to be updated based on the output vector of the nth layer in the recurrent neural network; wherein the nth layer is the last layer in the recurrent neural network; Based on the update operation corresponding to the to-be-updated layer, the to-be-updated layer is updated to obtain an updated target sub-search space; The step of determining a target sub-search space from each candidate sub-search space based on the output vector of the first layer in the recurrent neural network comprises: Determine, based on the output vector of the first layer in the recurrent neural network, a calculated score for each element in the output vector of the first layer; Determine a target sub-search space from each candidate sub-search space based on the calculated score of each element in the output vector of the first layer; The step of determining the to-be-updated layer of the central architecture in the target sub-search space based on the output vector of the n-1th layer in the recurrent neural network comprises: Determine the calculated score of each element in the output vector of the n-1th layer based on the output vector of the n-1th layer in the recurrent neural network; Determining the layer to be updated from the central architecture based on the calculated scores of each element in the output vector of the n-1th layer; The step of determining the update operation corresponding to the layer to be updated based on the output vector of the nth layer in the recurrent neural network includes: Determine, based on the output vector of the nth layer in the recurrent neural network, a calculated score for each element in the output vector of the nth layer; Based on the calculated score of each element in the output vector of the nth layer, an update operation corresponding to the layer to be updated is determined.

2. The image classification network architecture design method according to claim 1, characterized in that: Each candidate sub-search space is determined based on the following steps: Randomly sample multiple candidate neural network architectures from the global search space; Based on each candidate neural network architecture, and a neural network architecture whose similarity to each candidate neural network architecture is greater than the preset similarity, a corresponding candidate sub-search space is constructed; each candidate neural network architecture is the central architecture of the corresponding candidate sub-search space.

3. The image classification network architecture design method according to claim 1, characterized in that: The step of extracting the feature vector of each node in the last layer of the graph convolutional neural network includes: Determine a contribution coefficient of node j to node i based on a feature vector of node i, a feature vector of node j, and a feature vector of an edge from node i to node j in the graph convolutional neural network; Based on the contribution coefficient of the node j to the node i and the feature vector of the node j, the feature vector of the node i is updated.

4. The image classification network architecture design method according to claim 3, characterized in that: The eigenvector of the edge is determined based on the following steps: Determine a contribution coefficient of edge q to edge p based on a feature vector of edge p in the graph convolutional neural network, a feature vector of an adjacent edge q of edge p, and a feature vector of a junction node between edge p and edge q; Based on the contribution coefficient of the edge q to the edge p, the feature vector of the edge p is updated.

5. The image classification network architecture design method according to any one of claims 1 to 4, characterized in that: The method further comprises: When the iteration stop condition is not met, after replacing the corresponding candidate sub-search space with the updated target sub-search space, the execution returns to the node of each candidate sub-search space and sorts the candidate sub-search spaces in descending order according to their performance.

6. A device for designing an image classification network architecture, characterized in that: include: A construction unit is used to take each candidate sub-search space as a node, sort each candidate sub-search space in order from low to high according to the performance of each candidate sub-search space, and connect the low-performance nodes to all nodes with higher performance than them, so as to construct a directed graph of the candidate sub-search space; each candidate sub-search space includes a plurality of neural network architectures, and the architecture similarity between every two neural network architectures in each candidate sub-search space is greater than a preset similarity, and the performance of each candidate sub-search space is determined based on a sample image, a classification label of the sample image, and a predicted classification result of the sample image; An extraction unit, configured to construct a multi-layer graph convolutional neural network based on the candidate sub-search space directed graph, and extract a feature vector of each node in the last layer of the graph convolutional neural network; An input unit, used to determine the input vector of the first layer in the multi-layer recurrent neural network based on the feature vector of each node in the last layer of the graph convolutional neural network, and obtain the output vector of each layer in the recurrent neural network based on the input vector of the first layer; An updating unit, configured to determine a target sub-search space from each candidate sub-search space based on the output vectors of each layer in the recurrent neural network, update the target sub-search space, and update the graph convolutional neural network and the recurrent neural network based on the updated target sub-search space; An iteration unit, configured to determine a reward value based on the performance of the target sub-search space and the performance of the updated target sub-search space, and perform parameter iteration on the updated graph convolutional neural network and the updated recurrent neural network based on the reward value; A determination unit, used for determining the image classification network architecture based on the updated target sub-search space when the iteration stop condition is met; The step of determining a target sub-search space from each candidate sub-search space based on the output vector of each layer in the recurrent neural network, and updating the target sub-search space, comprises: Determine a target sub-search space from each candidate sub-search space based on an output vector of the first layer in the recurrent neural network; Determine a layer to be updated of a central architecture in the target sub-search space based on an output vector of the n-1th layer in the recurrent neural network; Determine the update operation corresponding to the layer to be updated based on the output vector of the nth layer in the recurrent neural network; wherein the nth layer is the last layer in the recurrent neural network; Based on the update operation corresponding to the to-be-updated layer, the to-be-updated layer is updated to obtain an updated target sub-search space; The step of determining a target sub-search space from each candidate sub-search space based on the output vector of the first layer in the recurrent neural network comprises: Determine, based on the output vector of the first layer in the recurrent neural network, a calculated score for each element in the output vector of the first layer; Determine a target sub-search space from each candidate sub-search space based on the calculated score of each element in the output vector of the first layer; The step of determining the to-be-updated layer of the central architecture in the target sub-search space based on the output vector of the n-1th layer in the recurrent neural network comprises: Determine the calculated score of each element in the output vector of the n-1th layer based on the output vector of the n-1th layer in the recurrent neural network; Determining the layer to be updated from the central architecture based on the calculated scores of each element in the output vector of the n-1th layer; The step of determining the update operation corresponding to the layer to be updated based on the output vector of the nth layer in the recurrent neural network includes: Determine, based on the output vector of the nth layer in the recurrent neural network, a calculated score for each element in the output vector of the nth layer; Based on the calculated score of each element in the output vector of the nth layer, an update operation corresponding to the layer to be updated is determined.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image classification network architecture design method as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image classification network architecture design method as described in any one of claims 1 to 5 is implemented.

Images