Method and device for optimizing neural network architecture, computer device and storage medium

Abstract

The embodiment of the application discloses a kind of optimization methods, device, computer equipment and storage medium of neural network architecture, belong to machine learning field.The method comprises: determining subnetwork, subnetwork is the differentiable network in neural network architecture, subnetwork is constituted by at least two nodes and the edge of connecting node, the edge of connecting node is used to characterize the basic operation in neural network;Based on first training data set, determine the network loss of subnetwork in network parameter range, network parameter range is the parameter range with network parameter as center, network parameter includes architecture parameter and model parameter, architecture parameter is used to characterize the weight of basic operation, model parameter is used to characterize the operation mode of basic operation;By optimizing network parameter to minimize network loss, obtain the optimized subnetwork.The scheme of the embodiment of the application can reduce the instability of parameter optimization in the optimization process of neural network architecture, improve the precision and generalization performance of neural network model.

Description

Technical Field

[0001] This application relates to the field of machine learning, and in particular to a method, apparatus, computer device, and storage medium for optimizing a neural network architecture. Background Technology

[0002] Neural Architecture Search (NAS) is a technique for automatically designing neural networks. It uses algorithms to automatically design high-performance network structures based on a sample set. Differentiable Architecture Search (DARTS), as a method of NAS, improves the efficiency of neural network search by constructing a continuously relaxed search space and is widely used in the field of machine learning.

[0003] However, DARTS suffers from skip connections during the search process. As the search progresses, these skip connections gradually dominate the network structure, leading to instability in parameter optimization and ultimately affecting the accuracy of the final network model. Related techniques utilize the eigenvalues ​​of the Hessian matrix to determine the stability of parameter optimization during DARTS. Large eigenvalues ​​of the Hessian matrix indicate unstable parameter optimization during the network structure search. By stopping parameter optimization or using regularization techniques to prevent excessive fluctuations in the Hessian matrix eigenvalues, the stability of parameter optimization during the DARTS network structure search can be ensured.

[0004] The problem with the above technical solutions is that using direct stopping optimization or regularization methods still fails to effectively solve the problem of unstable parameter optimization during the search network structure process. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, and storage medium for optimizing neural network architectures. These methods reduce the instability of parameter optimization during the optimization process, simplify the optimization process, and improve the accuracy and generalization performance of neural network models. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a method for optimizing a neural network architecture, the method comprising:

[0007] Determine a subnetwork, which is a differentiable network in a neural network architecture. The subnetwork consists of at least two nodes and edges connecting the nodes. The edges connecting the nodes are used to represent the basic operations in the neural network.

[0008] Based on the first training dataset, the network loss of the sub-network within the range of network parameters is determined. The range of network parameters is a parameter range centered on the network parameters. The network parameters include architecture parameters and model parameters. The architecture parameters are used to characterize the weights of the basic operations, and the model parameters are used to characterize the operation mode of the basic operations.

[0009] The optimized subnetwork is obtained by optimizing the network parameters to minimize the network loss.

[0010] On the other hand, embodiments of this application provide an optimization apparatus for a neural network architecture, the apparatus comprising:

[0011] The first construction module is used to determine the sub-network, which is a differentiable network in the neural network architecture. The sub-network consists of at least two nodes and edges connecting the nodes. The edges connecting the nodes are used to represent the basic operations in the neural network.

[0012] The determination module is used to determine the network loss of the sub-network within the range of network parameters based on the first training dataset. The range of network parameters is a parameter range centered on the network parameters. The network parameters include architecture parameters and model parameters. The architecture parameters are used to characterize the weights of the basic operations, and the model parameters are used to characterize the operation mode of the basic operations.

[0013] An optimization module is used to optimize the network parameters to minimize the network loss and obtain the optimized sub-network.

[0014] On the other hand, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the neural network architecture optimization method as described above.

[0015] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the neural network architecture optimization method as described above.

[0016] On the other hand, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the neural network architecture optimization method provided above.

[0017] The technical solution provided in this application may include the following beneficial effects:

[0018] In this embodiment, the network loss is determined by using all parameters within the network parameter range, and the network parameters are optimized so that the network loss converges to a flat loss region. In contrast, related technologies determine the network loss using a single network parameter, and during the optimization process, the network loss is prone to significant fluctuations, causing it to converge to a loss peak region, leading to a collapse in the performance of the neural network model. Compared to related technologies, the technical solution provided in this application determines the network loss within a specific network parameter range. During the optimization process, the network loss does not fluctuate significantly, allowing it to converge from the loss peak region to a flat loss region. This reduces the instability of the neural network model's performance and improves its accuracy and generalization performance. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 A schematic diagram illustrating an embodiment of the differentiable neural network search process provided in this application is shown.

[0021] Figure 2 This illustration shows a schematic diagram of the application process of a neural network model provided in an exemplary embodiment of this application;

[0022] Figure 3 A flowchart illustrating an exemplary embodiment of the method for optimizing a neural network architecture provided in this application is shown.

[0023] Figure 4 A schematic diagram of the sub-network structure shown in an exemplary embodiment of this application is illustrated;

[0024] Figure 5 This illustration shows a schematic diagram of how network loss varies with network parameters, provided in an exemplary embodiment of this application.

[0025] Figure 6 A schematic diagram illustrating a network loss comparison provided by another exemplary embodiment of this application is shown;

[0026] Figure 7 A flowchart illustrating an optimization method for a neural network architecture provided in another exemplary embodiment of this application is shown;

[0027] Figure 8 A schematic diagram illustrating the range of network parameters shown in an exemplary embodiment of this application is provided.

[0028] Figure 9 This illustration shows an implementation diagram of a target basic operation selection process provided in an exemplary embodiment of this application;

[0029] Figure 10 A flowchart illustrating the target network model generation and verification process provided in an exemplary embodiment of this application is shown.

[0030] Figure 11 A structural block diagram of an optimization apparatus for a neural network architecture provided in an exemplary embodiment of this application is shown.

[0031] Figure 12 A structural block diagram of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] This application relates to the application of neural networks. For ease of understanding, the relevant terms and concepts of neural networks that may be involved in this application are introduced below.

[0034] 1. Neural Networks

[0035] Neural networks are mathematical models that simulate biological neural networks and are widely used in the field of machine learning. A neural network consists of a large number of nodes and their interconnections. Nodes receive input signals and convert them into output values ​​through activation functions, which are then passed to the next node. This output value can then be used as the input for the next node. Activation functions introduce non-linear features into the neural network. Each connection between nodes represents a weighted value for the signal passing through that connection. By training with large amounts of data and continuously adjusting the number of nodes, weights, and other relevant parameters of the neural network structure, a neural network model is obtained and used to solve real-world problems such as image classification, object recognition, and data prediction.

[0036] 2. Neural Network Optimization (Search)

[0037] Neural network optimization (search) is a technique for automatically designing neural networks. It uses algorithms to automatically design high-performance network structures based on training datasets. The specific process of neural network optimization (search) is as follows: First, a search space is defined. Then, a neural network is selected from the predefined search space according to a structure search strategy. This neural network is then evaluated using a performance evaluation method. The evaluation results are returned to the structure search strategy to adjust the next neural network selection, until a neural network model that meets the performance requirements is found.

[0038] 3. Differentiable Neural Network Search

[0039] Differentiable neural network search is a method for optimizing (searching) neural networks. Instead of searching for discrete candidate network structures, this method relaxes the search space to be continuous and optimizes the neural network through gradient descent, ultimately finding a neural network model that meets performance requirements. Differentiable neural network search constructs a neural network model by searching a sub-network and then connecting multiple sub-networks. For example... Figure 1 This diagram illustrates the implementation of the search process for a differentiable neural network. The first step is to construct a search space, consisting of nodes and directed edges connecting them. The operations on the directed edges between nodes are unknown. The second step is to relax the search space by mixing candidate operations on each directed edge. The third step is to jointly optimize the architecture parameters and model parameters using a two-layer optimization method. The fourth step is to generate the final sub-network based on the architecture parameters.

[0040] The following section analyzes the problems existing in the search process of differentiable neural networks.

[0041] Differentiable neural network search methods can encounter skip connections during the search process, meaning connections appear across nodes. As the number of search rounds increases, skip connections gradually dominate the network architecture, leading to a reduction in the number of layers in the searched network structure, and consequently, poor performance of the final neural network model.

[0042] In addition, the differentiable neural network search method uses a two-layer optimization method to optimize the architecture parameters, which leads to low utilization of training data and poor performance of the final searched neural network model.

[0043] Specifically, the solutions provided in this application can be applied to fields such as image classification, image retrieval, image segmentation, object detection, and natural language processing.

[0044] In one possible application scenario, image classification will be used as an example. The optimization method of the neural network architecture provided in the embodiments of this application is used to obtain a sub-network suitable for image classification. This sub-network is then stacked to obtain a neural network, which is then trained using images from the training dataset to obtain a neural network model suitable for image classification. In one possible implementation, this neural network model suitable for image classification is applied to a computer device to classify images stored on the computer device.

[0045] Optionally, the computer device can be a mobile computer device such as a smartphone, tablet, or laptop, or a computer device such as a desktop computer or a projector computer. This application does not limit the specific type of computer device.

[0046] In another possible implementation, the neural network model suitable for image classification is applied to the server side to classify images stored on the server side.

[0047] Optionally, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. This application embodiment does not limit this.

[0048] Take, for example, the application of a neural network model suitable for image classification to a computer device. Figure 2As shown, during image capture, the raw image data (RAW format) from the camera sensor 21 is transmitted to the coprocessor chip 22 (CC). CC 22 integrates an image signal processor 221 (ISP) and an embedded neural network processing unit 222 (NPU). The ISP 221 performs denoising and sharpening on the raw image data, while the NPU 222 runs a neural network model and uses this model to further process the image processed by the ISP 221 using artificial intelligence (AI) to obtain image processing results. Examples include content recognition and AI denoising. CC 22 then transmits the image processing results to the application processor 23 (AP), which further processes the results, such as switching shooting modes based on content recognition results and further optimizing the AI-denoised image.

[0049] It should be noted that the embodiments of this application are only illustrated using the above application scenarios as examples. In practical applications, those skilled in the art can apply the neural network model obtained by the optimization method of the neural network architecture provided in the embodiments of this application to scenarios such as target detection tasks and image segmentation. The embodiments of this application do not limit this.

[0050] The optimization method for the neural network architecture provided in the embodiments of this application will be described in detail below. Please refer to... Figure 3 The diagram illustrates a flowchart of an optimization method for a neural network architecture provided in an exemplary embodiment of this application.

[0051] Step 310: Determine the subnetwork. The subnetwork is a differentiable network in the neural network architecture. The subnetwork consists of at least two nodes and edges connecting the nodes. The edges connecting the nodes are used to represent the basic operations in the neural network.

[0052] In this embodiment, the sub-network is a directed acyclic graph consisting of N ordered nodes (N being an integer greater than or equal to 2) and edges connecting the nodes. Each node represents a feature map, and each edge connecting a node represents a basic operation used to process the input feature map. The basic operation belongs to any one or more operations in the candidate operation set.

[0053] In addition, in the embodiments of this application, the sub-network can be constructed in real time or selected from several pre-constructed sub-networks. The embodiments of this application do not limit this.

[0054] Alternatively, the sub-network can be determined based on the application requirements of the target network model and the type of data being processed.

[0055] In one possible implementation, when the target network model is a neural network for processing image data, the constructed subnetworks must be adapted to the processing of image data.

[0056] In another possible implementation, when the target network model is a neural network used to process speech data, the subnetworks constructed should be adapted to the processing of speech data.

[0057] It should be noted that the requirements for sub-networks in this application embodiment are only illustrated by taking the target network model for processing image data or voice data as an example. In practical applications, those skilled in the art can determine the sub-networks according to the application requirements of the target network model and the type of data to be processed. This application embodiment does not limit this.

[0058] Optionally, the number of nodes in the subnetwork can be determined based on the application requirements of the target network model and the memory resources of the computer equipment. Specifically, the initial number of nodes can be determined first based on the application requirements of the target network model, and then the initial number of nodes can be further adjusted based on the memory resources of the computer equipment to determine the final number of nodes.

[0059] Optionally, the video memory resource requirement of a computer device can be the size of the video memory resources of the computer device.

[0060] Optionally, the candidate operation set includes input 0 operation, 3×3 depthwise separable convolution, 5×5 depthwise separable convolution, 3×3 dilated depthwise separable convolution, 5×5 dilated depthwise separable convolution, 3×3 maximum pooling, 3×3 mean pooling, etc., and this application embodiment does not limit this. Among them, input 0 operation means that there is no basic operation between nodes.

[0061] In one possible implementation, i and j represent the i-th node and the j-th node, respectively. Here, i and j are integers greater than or equal to 0, and i... <j。x i Let x represent the i-th feature map. j Let represent the j-th feature map. An edge (i, j) connecting nodes represents the connection relationship from node i to node j. Basic operations on edge (i, j) are... (i,j) The feature map x used to input node i (i) Transform into feature map x (j) And basic operations o(i,j) Belongs to the candidate operation set Any one or more operations in it.

[0062] For example, such as Figure 4 The subnetwork is a directed acyclic graph consisting of four nodes 41 and edges 42 connecting the nodes. The nodes 41 in the subnetwork are nodes 0, 1, 2, and 3. Where x... 0 x 1 x 2 and x 3 These represent the 0th, 1st, 2nd, and 3rd feature maps, respectively. The search space contains a total of 6 edges, 42 in total: edge (0, 1), edge (0, 2), edge (0, 3), edge (1, 2), edge (1, 3), and edge (2, 3).

[0063] Furthermore, the subnetwork is a differentiable network within the neural network architecture. During the optimization process of the neural network architecture, the optimized subnetwork is determined by selecting basic operations from the candidate operation set. In related technologies, methods such as reinforcement learning or evolutionary algorithms are used to select a basic operation from the candidate operation set in a discrete manner, meaning that the methods for selecting possible basic operations from the candidate operation set are unrelated. Discrete operations are difficult to differentiate mathematically, increasing the computational load and space required by computer equipment, resulting in low optimization efficiency. In this embodiment, the subnetwork is a differentiable network within the neural network architecture. Specifically, during the optimization process, each basic operation in the candidate operation set processes the feature map of each node in the subnetwork, and then the results obtained from all basic operations are weighted and summed. At this point, the relationship between any two nodes is not a single, definitive basic operation, but rather a hybrid operation composed of multiple basic operations. The basic operations selected from the candidate operation set are interconnected, making the subnetwork a differentiable network. This hybrid operation is defined as:

[0064]

[0065] in The mixed weights of operations between nodes (i, j); For the operation o between the i-th node and the j-th node (i,j) The weights, i.e., the architecture parameters in the network parameters; (i,j) This represents the possible basic operations between node i and node j; Represented as a set of candidate operations; This represents the probabilistic weighting of basic operations between node i and node j.

[0066] Therefore, based on hybrid operations, in this embodiment of the application, the process of selecting basic operations from the candidate operation set changes from discrete operations to continuous operations, and the optimization of the sub-network becomes the learning of the architecture parameter α, thereby reducing the amount of computation of the computer equipment and improving the optimization efficiency.

[0067] For example, the basic operation that might exist for the edge (0, 1) between node 0 and node 1 is a 3×3 depthwise separable convolution. 1 5×5 depthwise separable convolution 2 3×3 hole depth separable convolution 3 There are three basic operations. Each basic operation has a weight of α. 1 α 2 α 3 The mixed operation between node 0 and node 1 is defined as follows:

[0068]

[0069] Step 320: Based on the first training dataset, determine the network loss of the sub-network within the network parameter range. The network parameter range is the parameter range centered on the network parameters. The network parameters include architecture parameters and model parameters. Architecture parameters are used to characterize the weights of basic operations, and model parameters are used to characterize the operation mode of basic operations.

[0070] In optimizing a subnetwork, computer devices aim to make the predicted values ​​output by the subnetwork as close as possible to the ground truth values ​​corresponding to the training data. This is achieved by comparing the current predicted values ​​with the ground truth values, and then updating the subnetwork parameters based on the difference between the two. This process brings the predicted values ​​of the subnetwork output closer to the ground truth values, resulting in an optimized subnetwork. The network loss is the difference between the predicted and ground truth values. In related technologies, the network loss is determined using only one set of network parameters. When the network parameters are updated again, the network loss may fluctuate significantly, causing the network loss to converge in the loss peak region. For example, ... Figure 5 As shown in the graph of network loss variation with network parameters, the network loss fluctuates little during the optimization process from network parameter 1 to network parameter 2. However, during the optimization process from network parameter 3 to network parameter 4, the network loss fluctuates significantly, producing a spike region within the range of network parameter 4, leading to a collapse in network performance. In this embodiment, to reduce the large fluctuations in network loss during network parameter optimization, the network loss is determined by the range of network parameters, minimizing the network loss for all network parameters within that range. This results in the network loss converging within a flat loss region, improving the accuracy and generalization performance of the optimized sub-network. For example, as... Figure 6Two identical sub-networks are constructed. One sub-network employs an optimization method from related technologies, where the network loss is determined by a specific network parameter. The other sub-network employs the optimization method described in this embodiment, where the network loss is determined by all parameters within a given range. The network parameters are optimized based on the same first training dataset to minimize the network loss. As can be seen from the network loss topography map, in the related technologies, the network loss converges in the loss peak region, while in the embodiment of this application, the corresponding network loss converges in the loss flat region.

[0071] The true value is determined based on the target task to be processed by the target network model, and is the actual result required by the target task.

[0072] Optionally, when the target task to be processed by the target network model is image classification, the true value is the image classification label corresponding to the training data; when the target task to be processed by the target network model is object detection, the true value is the detection result label corresponding to the training data.

[0073] It should be noted that the embodiments of this application are only illustrated with the above-mentioned real values. In practical applications, those skilled in the art can determine the real values ​​based on the target task processed by the target network model. The embodiments of this application are not limited to this.

[0074] The first training dataset is used to optimize the sub-network, resulting in the optimized sub-network. The type of the first training dataset is related to the target task that the target network model will handle. The first training dataset includes a training dataset and a test dataset. The training dataset has a larger data volume than the test dataset.

[0075] Optionally, the first training dataset can be obtained by means of network download, manual collection, etc., and this application embodiment does not limit this.

[0076] Optionally, when the target network model is to process image classification, the first training dataset can be the CIFAR-10 dataset, CIFAR-100 dataset, ImageNet dataset, etc., and this embodiment does not limit it.

[0077] In this embodiment of the application, network parameters include architecture parameters and network parameters. The architecture parameters are the weights α of the basic operations between every two nodes, and the network parameters are the configuration parameters of the basic operations themselves.

[0078] Optionally, the basic operation is a convolution operation, and the model parameters can be the number of channels, the size of the convolution kernel, the number of convolution kernels, the stride of the convolution kernel, etc., which are not limited in this embodiment.

[0079] Optionally, the basic operation is a pooling operation, and the model parameters can be the pooling kernel size, pooling method, pooling length, etc., which are not limited in this embodiment.

[0080] It should be noted that the embodiments of this application only use convolution and pooling operations as examples of basic operations. In practical applications, those skilled in the art can set corresponding model parameters according to different basic operations, and the embodiments of this application do not limit this.

[0081] Step 330: Optimize network parameters to minimize network loss to obtain the optimized subnetwork.

[0082] The optimization method for neural network architectures involves a computer continuously adjusting network parameters to minimize network loss. Minimizing network loss means that the predicted value of the subnetwork's output is close to the true value; this subnetwork is considered optimized. The process of continuously adjusting network parameters is the process of optimizing network parameters.

[0083] In one possible implementation, when the network loss meets the convergence condition, or when the upper limit of the number of optimization attempts is reached, the computer device stops optimizing the network parameters and obtains the optimized subnetwork based on the network parameters obtained in the final optimization.

[0084] In summary, the embodiments of this application determine the corresponding network loss by using all parameters within the network parameter range, and optimize the network parameters so that the network loss converges to a flat loss region. In contrast, related technologies determine the network loss using a single network parameter, and during the optimization process, the network loss is prone to significant fluctuations, causing it to converge to a loss peak region, leading to a collapse in the performance of the neural network model. Compared to related technologies, the technical solution provided in this application, by determining the network loss within the network parameter range, ensures that the network loss does not fluctuate significantly during the optimization process, allowing the network loss to converge from the loss peak region to a flat loss region. This reduces the instability of the neural network model's performance and improves its accuracy and generalization performance.

[0085] In one possible implementation, the computer device determines the network loss of the sub-network within the range of network parameters using the Sharpness Aware Minimization (SAM) method based on a first training dataset. This method is described below. Please refer to... Figure 7 The diagram illustrates a flowchart of an optimization method for a neural network architecture provided in another exemplary embodiment of this application.

[0086] Step 701: Determine the subnetwork. The subnetwork is a differentiable network in the neural network architecture. The subnetwork consists of at least two nodes and edges connecting the nodes. The edges connecting the nodes are used to represent the basic operations in the neural network.

[0087] Step 701 is the same as step 310, and will not be described again in this embodiment.

[0088] Step 702: Determine the network parameter range based on the network parameters and range size.

[0089] In this embodiment, the network parameter range is a neighborhood range centered on the current network parameters. The size of the neighborhood range needs to be determined based on the range size. The range size needs to be determined based on the video memory resources of the computer device.

[0090] Optionally, the network parameter range can be a spherical range with the current network parameters as the center line, and the range size is the radius of the sphere. For example, such as... Figure 8 The network parameter range is a neighborhood range centered on the current network parameter, with a range size of sphere radius ρ.

[0091] Step 703: Select step size based on parameters and determine candidate network parameters within the range of network parameters.

[0092] Since there are countless parameters within the range of network parameters, computer equipment cannot determine the network loss corresponding to all parameters within this range. Therefore, it is necessary to segment the network parameter range and select candidate network parameters in step sizes. The network loss within the range of network parameters is determined by the candidate network loss corresponding to the candidate network parameters. This optimization method reduces the computational load on computer equipment and improves optimization efficiency.

[0093] The selection of the parameter step size needs to be determined based on the range of network parameters and the available memory resources of the computer. Furthermore, during the optimization of the neural network architecture, the parameter step size is not a fixed value and needs to be continuously adjusted based on the network loss.

[0094] Optionally, the parameter step size can be 0.5, 1, 1.5, 2, etc., and this application embodiment does not limit it.

[0095] Step 704: For candidate network parameters within the range of network parameters, input the training data from the first training dataset into the sub-network using the candidate network parameters to obtain the network output of the sub-network.

[0096] The computer device determines the parameters of each candidate network as the parameters of the sub-network, trains the sub-network using the training dataset in the first training dataset, and obtains the predicted value of the sub-network output corresponding to each candidate network parameter.

[0097] Step 705: Determine the candidate network loss corresponding to the candidate network parameters based on the network output.

[0098] The difference between the predicted value and the true value of the subnetwork output corresponding to each candidate network parameter is the candidate network loss for that parameter. The network loss is determined based on the target task to be processed by the target network model.

[0099] Optionally, when the target task to be processed by the target network model is image classification, the network loss is the result of image classification; when the target task to be processed by the target network model is target detection, the network loss is the result of target detection.

[0100] Step 706: The sum of the candidate network losses corresponding to each candidate network parameter is determined as the network loss of the subnetwork within the range of network parameters.

[0101] When the network loss converges to a flat region, the network loss is the sum of the network losses corresponding to all parameters within the network parameter range. In this embodiment, it is expressed as the sum of the candidate network losses corresponding to each candidate network parameter within the network parameter range, rather than the network loss corresponding to a single network parameter. Therefore, the network loss within the network parameter range is expressed as...

[0102] min w max ||∈||≤ρ L train (w+∈)

[0103] Where w represents the network parameter, L train Represented as network loss, ||∈||≤ρ represents the range of network parameters, specifically, the neighborhood range of size ρ centered on the current parameter, L train (w+∈) represents the sum of network losses of the subnetwork within the range of network parameters.

[0104] Step 707: Optimize the network parameters using a first-order optimization algorithm to minimize the network loss, and obtain the optimized network parameters.

[0105] In related technologies, a two-layer optimization method is used to optimize both architecture parameters and model parameters. Specifically, the training dataset is divided into two parts: one part is used to optimize the neural network architecture parameters, and the other part is used to optimize the model parameters. Gradient descent is employed, i.e., first fixing the architecture parameters and optimizing the model parameters, then fixing the model parameters and optimizing the architecture parameters, alternating between the two until the network loss converges. In related technologies, the utilization rate of training data is low, leading to a decrease in neural network performance. In this embodiment, a first-order optimization algorithm is used to simultaneously optimize network parameters. Specifically, the same training dataset is used to simultaneously optimize both architecture parameters and model parameters, improving the utilization rate of training data.

[0106] Optionally, the first-order optimization algorithm can be gradient descent, Newton's algorithm, etc., and this embodiment does not limit it.

[0107] Step 708: Based on the optimized architecture parameters, determine the target basic operations between nodes, wherein the weight of the target basic operations between nodes is higher than the weight of other basic operations between nodes.

[0108] In this embodiment, the sub-network is a differentiable neural network. All operations in the candidate operation set are processed on each node. Based on the optimized architecture parameters, the basic operations between nodes are obtained as hybrid operations. This hybrid operation includes several possible basic operations, and the weights of these basic operations are not the same. The optimized architecture parameters corresponding to these basic operations are arranged in descending order, and the one with the largest weight is selected as the target basic operation between nodes, while other operations besides the target basic operation are discarded.

[0109] For example, such as Figure 9 Let's take determining the target basic operation between node 0 and node 1 as an example. There are three possible basic operations between node 0 and node 1, represented by different lines. The weight of basic operation 1 is 0.1, the weight of basic operation 2 is 0.2, and the weight of basic operation 3 is 0.7. We select basic operation 3, which has the highest weight, as the target basic operation and discard the other basic operations. We use the same method to determine the target basic operations between other nodes.

[0110] Step 709: Determine the target model parameters corresponding to the target basic operation from the optimized model parameters.

[0111] In this embodiment, each basic operation has its own corresponding weights and model parameters. After determining the target basic operation based on the weights, the model parameters corresponding to the target basic operation are retained, while other model parameters are discarded. For example, as shown... Figure 9As shown, basic operation 1 is a 3×3 depthwise separable convolution operation, with corresponding model parameters of 3×3 kernel size, stride of 1, edge padding of 0, and 4 channels; basic operation 2 is a 5×5 depthwise separable convolution operation, with corresponding model parameters of 5×5 kernel size, stride of 1, edge padding of 0, and 4 channels; basic operation 3 is a 3×3 maximum pooling operation, with corresponding model parameters of 3×3 pooling kernel size, stride of 3, and edge padding of 0. Basic operation 3 with the largest weight is selected as the target basic operation, and its corresponding model parameters are used as the target model parameters, while the model parameters corresponding to basic operations 1 and 2 are discarded.

[0112] Step 710: Generate an optimized subnetwork based on the target basic operations and target model parameters.

[0113] In this embodiment, since the subnetwork is a differentiable neural network, there are multiple possible basic operations between nodes. The basic operation with the largest weight among the nodes is selected as the target basic operation, and other basic operations are discarded. Based on the target basic operation, the corresponding target model parameters are determined, and the model parameters corresponding to other basic operations are discarded, thus finally determining the network structure between nodes. Furthermore, the target basic operation and target model parameters between nodes may be the same or different. For example, such as... Figure 9 As shown, there are three possible basic operations between the four nodes 0, 1, 2, and 3: basic operation 1, basic operation 2, and basic operation 3. Basic operation 3, with the highest weight among the nodes, is selected as the target basic operation, while basic operations 1 and 2 are discarded. Based on the target basic operation, the corresponding target model parameters are determined from the optimized model parameters. The network structure between the nodes is then determined based on the target basic operation and the target model parameters, thereby generating the optimized subnetwork.

[0114] In summary, in this embodiment, by selecting candidate network parameters within a range of network parameters and determining the network loss within that range based on the sum of the losses of the candidate network parameters, the optimization process is simplified and optimization efficiency is improved. Furthermore, optimizing network parameters using a first-order algorithm to minimize network loss improves the utilization of training data compared to second-order algorithm optimization methods in related technologies.

[0115] In this embodiment, the final target network model is obtained by stacking the optimized sub-networks. The method for generating the target network model is described below. Please refer to... Figure 10 The diagram illustrates a flowchart of the target network model generation and verification process provided in an exemplary embodiment of this application.

[0116] Step 340: Construct the target network model based on the optimized sub-networks. The target network model is obtained by stacking the optimized sub-networks.

[0117] In this embodiment, the target network model is obtained by stacking optimized subnetworks in a predetermined manner.

[0118] Optionally, the target network model can be a Convolutional Neural Network (CNN) structure, a Recurrent Neural Network (RNN) structure, a Long Short Term Memory (LSTM) structure, etc., and this embodiment does not limit it.

[0119] Optionally, the number of optimized sub-networks can be one or more, and this embodiment does not limit this. The number of optimized sub-networks corresponds to the number of sub-networks determined.

[0120] In one possible implementation, the optimized subnetwork consists of two subnetworks: an upsampled subnetwork and a downsampled subnetwork. The optimized upsampled and downsampled subnetworks are obtained by constructing both subnetworks and training them using the same initial training dataset. Stacking the upsampled and downsampled subnetworks yields a U-net neural network architecture.

[0121] It should be noted that the embodiments of this application only illustrate the U-shaped neural network architecture obtained by stacking optimized sub-networks. In practical applications, those skilled in the art can obtain optimized sub-networks and stack them to obtain the corresponding target network model according to actual needs. This embodiment does not limit this.

[0122] Step 350: Train the target network model based on the first training dataset to obtain the trained target network model.

[0123] The first training dataset includes a training dataset and a test dataset. In this embodiment, the target network model is trained using the test dataset in the first training dataset to further optimize the target network model.

[0124] Step 360: Based on the second training dataset, perform performance verification on the trained target network model to obtain performance verification results. The second training dataset is different from the first training dataset. The performance verification results include accuracy verification results and generalization verification results.

[0125] Using the second training dataset, the performance of the target network model obtained in step 350 is verified and evaluated. Performance includes the accuracy and generalization ability of the target network model. Accuracy evaluates how well the target network model matches the true values, while generalization evaluates the target network model's ability to adapt to new training data. The performance evaluation criteria need to be determined based on the target task to be processed by the target network model. The second training dataset and the first training dataset are different datasets, with the first training dataset having a larger data volume than the second training dataset. Furthermore, the data in the second training dataset does not require preprocessing. The type of the second training dataset is related to the target task to be processed by the target network model.

[0126] Optionally, the second training dataset can be obtained through network download, manual collection, or other means, and this application embodiment does not limit this.

[0127] Optionally, when the target task to be processed by the target network model is image classification, the second training dataset can be the CIFAR-10 dataset, CIFAR-100 dataset, ImageNet dataset, etc., and this embodiment does not limit it.

[0128] In one possible implementation, when the target task to be processed by the target network model is image classification, the performance evaluation criteria of the target network model are Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index Measurement (SSIM).

[0129] It should be noted that the embodiments of this application are only used to illustrate the performance evaluation criteria of the target network model described above. In practical applications, those skilled in the art can select the evaluation criteria of the model according to the target task to be processed by the target network model. The embodiments of this application do not limit this.

[0130] In summary, in this embodiment, the sub-network is optimized first, and the target network model is constructed through the optimized sub-network. Compared with the method in related technologies of first constructing the target network model and then optimizing the constructed target network model, the computational load of computer equipment for optimization is reduced and the optimization efficiency is improved. In addition, the sub-network structure optimized based on this embodiment is simple, and the accuracy and generalization performance of the final constructed target network model are higher than those of the target network models constructed in related technologies.

[0131] Please refer to Figure 11 This illustration shows a structural block diagram of an optimization device for a neural network architecture provided in an exemplary embodiment of this application. The device may include:

[0132] The first building module 1101 is used to determine the sub-network, which is a differentiable network in the neural network architecture. The sub-network consists of at least two nodes and edges connecting the nodes. The edges connecting the nodes are used to represent the basic operations in the neural network.

[0133] The determination module 1102 is used to determine the network loss of the sub-network within the network parameter range based on the first training dataset. The network parameter range is the parameter range centered on the network parameters. The network parameters include architecture parameters and model parameters. The architecture parameters are used to characterize the weights of basic operations, and the model parameters are used to characterize the operation mode of basic operations.

[0134] The optimization module 1103 is used to minimize network loss by optimizing network parameters to obtain an optimized subnetwork.

[0135] Optionally, module 1102 includes:

[0136] The determination unit is used to determine the network loss of the subnetwork within the range of network parameters based on the first training dataset using a sharpness-aware minimization method.

[0137] Optionally, a unit is defined for:

[0138] Determine the range of network parameters based on network parameters and range size;

[0139] For candidate network parameters within the range of network parameters, the training data in the first training dataset is input into the sub-network using the candidate network parameters to obtain the network output of the sub-network.

[0140] Determine the candidate network loss corresponding to the candidate network parameters based on the network output;

[0141] The sum of the candidate network losses corresponding to each candidate network parameter is determined as the network loss of the subnetwork within the range of network parameters.

[0142] Optionally, determining the unit is also used for:

[0143] Based on the parameter selection step size, candidate network parameters within the range of network parameters are determined.

[0144] Optionally, the optimization module 1103 includes:

[0145] The optimization unit is used to optimize the network parameters using a first-order optimization algorithm to minimize the network loss and obtain the optimized network parameters.

[0146] The generation unit is used to generate an optimized subnetwork based on the optimized network parameters.

[0147] Optionally, the generating unit is used for:

[0148] Based on the optimized architecture parameters, the target basic operations between nodes are determined, where the weight of the target basic operations between nodes is higher than the weight of other basic operations between nodes.

[0149] From the optimized model parameters, determine the target model parameters corresponding to the target basic operation;

[0150] An optimized subnetwork is generated based on the target's basic operations and target model parameters.

[0151] The device also includes:

[0152] The second building module is used to build the target network model based on the optimized sub-networks. The target network model is obtained by stacking the optimized sub-networks.

[0153] The training module is used to train the target network model based on the first training dataset to obtain the trained target network model.

[0154] Optionally, the target network model adopts a U-shaped neural network architecture, and the target network model is constructed based on the optimized upsampling sub-network and downsampling sub-network.

[0155] The device also includes:

[0156] The performance verification module is used to perform performance verification on the trained target network model based on the second training dataset and obtain the performance verification results. The second training dataset is different from the first training dataset. The performance verification results include accuracy verification results and generalization verification results.

[0157] In summary, the embodiments of this application determine the corresponding network loss by using all parameters within the network parameter range, and optimize the network parameters so that the network loss converges to a flat loss region. In contrast, related technologies determine the network loss using a single network parameter, and during the optimization process, the network loss is prone to significant fluctuations, causing it to converge to a loss peak region, leading to a collapse in the performance of the neural network model. Compared to related technologies, the technical solution provided in this application, by determining the network loss within the network parameter range, ensures that the network loss does not fluctuate significantly during the optimization process, allowing the network loss to converge from the loss peak region to a flat loss region. This reduces the instability of the neural network model's performance and improves its accuracy and generalization performance.

[0158] It should be noted that the apparatus provided in the above embodiments is only illustrative of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0159] Please refer to Figure 12 This illustration shows a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Specifically, the computer device 1200 includes a Central Processing Unit (CPU) 1201, a system memory 1204 including Random Access Memory (RAM) 1202 and Read-Only Memory (ROM) 1203, and a system bus 1205 connecting the system memory 1204 and the CPU 1201. The computer device 1200 also includes a basic input / output (I / O) system 1206 that facilitates the transfer of information between various devices within the computer, and a mass storage device 1207 for storing the operating system 1213, application programs 1214, and other program modules 1215.

[0160] The basic input / output system 1206 includes a display 1208 for displaying information and an input device 1209 for user input, such as a mouse or keyboard. Both the display 1208 and the input device 1209 are connected to the central processing unit 1201 via an input / output controller 1210 connected to the system bus 1205. The basic input / output system 1206 may also include the input / output controller 1210 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1210 also provides output to a display screen, printer, or other types of output devices.

[0161] The mass storage device 1207 is connected to the central processing unit 1201 via a mass storage controller (not shown) connected to the system bus 1205. The mass storage device 1207 and its associated computer-readable media provide non-volatile storage for the computer device 1200. That is, the mass storage device 1207 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0162] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 1204 and the mass storage device 1207 described above can be collectively referred to as memory.

[0163] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1201. The one or more programs contain instructions for implementing the above-described model training methods. The central processing unit 1201 executes the one or more programs to implement the methods provided in the above-described method embodiments.

[0164] According to various embodiments of this application, the computer device 1200 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 1200 can be connected to a network 1212 via a network interface unit 1211 connected to the system bus 1205, or the network interface unit 1211 can be used to connect to other types of networks or remote computer systems (not shown).

[0165] The memory further includes one or more programs stored in the memory, and the one or more programs include steps performed by a computer device in the methods provided in the embodiments of this application.

[0166] In this application embodiment, a computer-readable storage medium is also provided, which stores at least one instruction that is loaded and executed by a processor to implement the method for optimizing the neural network architecture as described above.

[0167] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform an optimized method for a neural network architecture provided in various alternative implementations of the above aspect.

[0168] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0169] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for optimizing a neural network architecture, characterized in that, The method includes: A subnetwork is defined, wherein the subnetwork is a differentiable network in a neural network architecture, the subnetwork consists of at least two nodes and edges connecting the nodes, the edges connecting the nodes are used to represent the basic operations in the neural network, and the subnetwork is suitable for image classification; Based on the first training dataset, the network loss of the sub-network within the network parameter range is determined. The network parameter range is a parameter range centered on the current network parameters. The network parameters include architecture parameters and model parameters. The architecture parameters are used to characterize the weights of the basic operations, and the model parameters are used to characterize the operation mode of the basic operations. The first training dataset is an image dataset, and the real values ​​corresponding to the training data in the first training dataset are image classification labels. The optimized subnetwork is obtained by optimizing the network parameters to minimize the network loss.

2. The method according to claim 1, characterized in that, The step of determining the network loss of the sub-network within the range of network parameters based on the first training dataset includes: Based on the first training dataset, the network loss of the sub-network within the network parameter range is determined by the sharpness-aware minimization method.

3. The method according to claim 2, characterized in that, The step of determining the network loss of the sub-network within the network parameter range using a sharpness-aware minimization method based on the first training dataset includes: The range of network parameters is determined based on the network parameters and the range size; For candidate network parameters within the range of network parameters, the training data in the first training dataset is input into the sub-network using the candidate network parameters to obtain the network output of the sub-network. Based on the network output, determine the candidate network loss corresponding to the candidate network parameters; The sum of the losses of the candidate networks corresponding to each of the candidate network parameters is determined as the network loss of the sub-network within the range of the network parameters.

4. The method according to claim 3, characterized in that, After determining the network parameter range based on the network parameters and range size, the process further includes: Based on the parameter selection step size, the candidate network parameters within the range of network parameters are determined.

5. The method according to any one of claims 1 to 4, characterized in that, The step of optimizing the network parameters to minimize the network loss to obtain the optimized sub-network includes: The network parameters are optimized by a first-order optimization algorithm to minimize the network loss, resulting in the optimized network parameters. The optimized sub-network is generated based on the optimized network parameters.

6. The method according to claim 5, characterized in that, The process of generating the optimized sub-network based on the optimized network parameters includes: Based on the optimized architecture parameters, the target basic operations between nodes are determined, wherein the weight of the target basic operations between nodes is higher than the weight of other basic operations between nodes. From the optimized model parameters, determine the target model parameters corresponding to the target basic operation; The optimized sub-network is generated based on the target basic operations and the target model parameters.

7. The method according to any one of claims 1 to 4, characterized in that, After obtaining the optimized subnetwork by optimizing the network parameters to minimize the network loss, the method further includes: A target network model is constructed based on the optimized sub-networks, and the target network model is obtained by stacking the optimized sub-networks. The target network model is trained based on the first training dataset to obtain the trained target network model.

8. The method according to claim 7, characterized in that, The target network model adopts a U-shaped neural network architecture and is constructed based on optimized upsampling subnetworks and downsampling subnetworks.

9. The method according to claim 7, characterized in that, After training the target network model based on the first training dataset to obtain the trained target network model, the method further includes: Based on the second training dataset, the performance of the trained target network model is verified to obtain performance verification results. The second training dataset is different from the first training dataset. The performance verification results include accuracy verification results and generalization verification results.

10. An optimization device for a neural network architecture, characterized in that, The device includes: The first construction module is used to determine a sub-network, which is a differentiable network in a neural network architecture. The sub-network consists of at least two nodes and edges connecting the nodes. The edges connecting the nodes are used to represent the basic operations in the neural network. The sub-network is suitable for image classification. The determination module is used to determine the network loss of the sub-network within the range of network parameters based on the first training dataset. The range of network parameters is a range of parameters centered on the current network parameters. The network parameters include architecture parameters and model parameters. The architecture parameters are used to characterize the weights of the basic operations, and the model parameters are used to characterize the operation mode of the basic operations. The first training dataset is an image dataset, and the real values ​​corresponding to the training data in the first training dataset are image classification labels. An optimization module is used to optimize the network parameters to minimize the network loss and obtain the optimized sub-network.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the optimization method of the neural network architecture as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the optimization method of the neural network architecture as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes computer instructions that are executed by a processor to implement the optimization method for the neural network architecture as described in any one of claims 1 to 9.

Images