Optimization method and device of neural network, computer device and storage medium
By sampling and testing the performance of the supernetwork, and iteratively optimizing the supernetwork structure, the problem of insufficient performance in neural network structure search was solved, and more efficient neural network optimization and performance improvement were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
- Filing Date
- 2023-02-07
- Publication Date
- 2026-06-09
AI Technical Summary
The performance of neural network structure search in existing technologies still needs to be improved, and how to further optimize neural network structures has become an urgent problem to be solved.
By obtaining the first supernetwork, multiple subnetworks are obtained through sampling. The performance parameters of each subnetwork are tested using the test sample set of the target prediction task. Based on these parameters, the supernetwork structure is iteratively optimized until the optimization conditions are met, and the optimized supernetwork is obtained as the target model.
It improves the effectiveness and performance of neural network structure search, and enhances the predictive performance of the target model for the prediction task.
Smart Images
Figure CN116090536B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, and storage medium for optimizing neural networks. Background Technology
[0002] Artificial intelligence (AI) is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. With the rapid development of AI technology, neural networks (e.g., deep neural networks) have achieved great success in recent years in the processing and analysis of various media signals such as images, videos, and audio.
[0003] However, a high-performance neural network often possesses an intricate network structure, requiring significant effort from highly skilled and experienced human experts. In related technologies, to better construct neural networks, Neural Architecture Search (NAS) is typically used to automatically search for neural network structures, thereby obtaining more efficient designs. However, the performance of neural network structures found through this method still needs improvement; therefore, how to further optimize neural network structures has become a pressing issue. Summary of the Invention
[0004] This application proposes a method, apparatus, computer device, and storage medium for optimizing neural networks, in order to further optimize the performance of neural networks.
[0005] In a first aspect, embodiments of this application provide a method for optimizing a neural network. The method includes: acquiring a first supernetwork; sampling the first supernetwork to obtain multiple subnetworks; testing the network performance of each subnetwork using a test sample set corresponding to a target prediction task to obtain a first network performance parameter for each subnetwork; iteratively optimizing the network structure parameters of the first supernetwork based on the first network performance parameter of each subnetwork until a first optimization condition is met to obtain an optimized first supernetwork as a second supernetwork. The second supernetwork is used to obtain a target model corresponding to the target prediction task after iterative training using a training sample set.
[0006] Secondly, embodiments of this application provide a neural network optimization device, the device comprising: a supernetwork acquisition module, a first sampling module, a performance testing module, and a supernetwork optimization module. The supernetwork acquisition module is used to acquire a first supernetwork; the first sampling module is used to sample the first supernetwork to obtain multiple subnetworks; the performance testing module is used to test the network performance of each subnetwork using a test sample set corresponding to the target prediction task, obtaining first network performance parameters for each subnetwork; the supernetwork optimization module is used to iteratively optimize the network structure parameters of the first supernetwork based on the first network performance parameters of each subnetwork until a first optimization condition is met, obtaining an optimized first supernetwork as a second supernetwork, which is used for iterative training with a training sample set to obtain a target model corresponding to the target prediction task.
[0007] Thirdly, embodiments of this application provide a computer device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the methods described above.
[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code that can be invoked by a processor to execute the methods described above.
[0009] The solution provided in this application involves obtaining a first supernetwork; sampling the first supernetwork to obtain multiple subnetworks; testing the network performance of each subnetwork using a test sample set corresponding to the target prediction task to obtain first network performance parameters for each subnetwork; and iteratively optimizing the network structure parameters of the first supernetwork based on the first network performance parameters of each subnetwork until a first optimization condition is met, resulting in an optimized first supernetwork, which serves as a second supernetwork. The second supernetwork is used to obtain the target model corresponding to the target prediction task after iterative training using a training sample set. Thus, before searching for the neural network architecture, the initial search space corresponding to the first supernetwork is iteratively optimized based on the network performance of the multiple subnetworks sampled from the first supernetwork to obtain the optimal search space, i.e., a second supernetwork with more optimized network structure parameters. This improves the effectiveness of the search space design and the performance of the searched network structure, thereby enhancing the prediction performance of the target model trained on the second supernetwork for the target prediction task. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 A flowchart illustrating a neural network optimization method provided in an embodiment of this application is shown.
[0012] Figure 2 A schematic diagram of the network architecture of a first supernetwork provided in an embodiment of this application is shown.
[0013] Figure 3 It shows Figure 1 A flowchart illustrating a sub-step of step S120 in one embodiment.
[0014] Figure 4 It shows Figure 1 A flowchart illustrating steps S150 to S170 following step S140 in one embodiment.
[0015] Figure 5 A schematic diagram of the progressive contraction training process provided in one embodiment of this application is shown.
[0016] Figure 6 A flowchart illustrating a neural network optimization method provided in an embodiment of this application is shown.
[0017] Figure 7 It shows Figure 6 A flowchart illustrating a sub-step of step S230 in one embodiment.
[0018] Figure 8 It shows Figure 6 A flowchart illustrating a sub-step of step S250 in one embodiment.
[0019] Figure 9 This is a block diagram of a neural network optimization device according to an embodiment of this application.
[0020] Figure 10 This is a block diagram of a computer device for performing a neural network optimization method according to an embodiment of this application.
[0021] Figure 11 This is a storage unit in this application embodiment for storing or carrying program code that implements the neural network optimization method according to this application embodiment. Detailed Implementation
[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.
[0023] It should be noted that some processes described in the specification, claims, and accompanying drawings of this application include multiple operations that appear in a specific order. These operations may not be performed in the order they appear herein, or they may be performed in parallel. Operation numbers such as S110, S120, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel. Also, the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or server that includes a series of steps or sub-modules is not necessarily limited to those steps or sub-modules that are explicitly listed, but may include other steps or sub-modules that are not explicitly listed or that are inherent to such process, method, product, or device.
[0024] The inventors propose a method, apparatus, computer device, and storage medium for optimizing a neural network. The method involves sampling a first supernetwork to obtain multiple subnetworks, and updating the parameter values of the network structure parameters of the first supernetwork based on the network performance of the multiple subnetworks, thereby optimizing the search space corresponding to the first supernetwork. The neural network optimization method provided in this application is described in detail below.
[0025] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a neural network optimization method provided in one embodiment of this application. The following will be combined with... Figure 1 The optimization method for the neural network provided in this application embodiment is described in detail. The optimization method for the neural network may include the following steps:
[0026] Step S110: Obtain the first supernetwork.
[0027] In this embodiment, a super network can be understood as a set of all possible subnetworks during the model search process. The super network is generated based on a predefined search space, and the weights of the super network model are the weights of all its subnetwork models. The weights of the subnetwork models can be obtained from the super network model. Each layer in the super network model includes multiple operators; that is, each node has multiple candidate operators. The operators between layers are connected in a fully connected manner, and a path in the fully connected layer constitutes a subnetwork model. For example, selecting an operator in each layer results in a neural network model composed of selected operators from multiple layers, which is a subnetwork model. Updating the weights in this path updates the weights of the subnetwork model and simultaneously updates the weights of the super network model, thus achieving the effect of training the super network model.
[0028] Optionally, the first supernetwork can be a neural network extended from the lightweight deep neural network MobileNetV2. For an example, please refer to... Figure 2 , Figure 2 This paper illustrates the supernetwork structure of a first supernetwork built upon an extension of the MobileNetV2 network architecture. This supernetwork structure includes a convolutional block, multiple convolutional units, and a final fully connected layer. Each convolutional unit contains three convolutional blocks, each consisting of one point-wise convolution, one batchNorm, and one ReLU; one depth-wise convolution, one batchNorm, and one ReLU; and one point-wise convolution, one batchNorm, and one ReLU. Of course, the first supernetwork can also be a neural network derived from extensions of network architectures such as EfficientNet-b0, ResNet, Inception, or DenseNet; this embodiment does not impose any limitations on this. In this embodiment, the kernel size of the depth-wise convolution is modified to 7×7 to support kernel size search.
[0029] The search space can be set from multiple dimensions of network structure parameters, including but not limited to kernel size, dilation rate of input channels in a convolutional block, depth of convolutional units, etc. The search space can also be characterized by at least one of the following: the number of layers in the neural network architecture, the number of data units in each layer, and the number of neurons in each data unit. In essence, the search space defines the scope of the search for neural network architectures. Based on this scope, a set of searchable neural network architectures can be provided, and different search spaces can be characterized by the range of values for each network structure parameter.
[0030] Optionally, in the following text, the search space will be set using three dimensions of network structure parameters: the depth of the convolutional unit, the expansion ratio of the number of input channels in the convolutional block, and the kernel size of the depth-wise-conv convolutional kernel in the convolutional block, as a non-limiting example. The search space set to generate the first supernetwork can be understood as the initial search space, for example, Representing the initial search space, ε represents the depth of the convolutional unit, and ε represents the expansion rate of the input channels in the convolutional block. Characterizes the size of the depth-wise-conv convolution kernel in the convolution block.
[0031] Step S120: Sample the first supernetwork to obtain multiple subnetworks.
[0032] Furthermore, after obtaining the first supernetwork, the initial search space corresponding to the first supernetwork can be iteratively optimized to obtain the optimal search space, thereby improving the effectiveness of the search space design and the performance of the network structure searched based on the optimal search space. Specifically, for each round of optimization, the current first supernetwork needs to be trained. After convergence, the first supernetwork trained in this round is sampled to obtain multiple subnetworks. Then, the network performance parameters of the multiple subnetworks sampled after each round of optimization can be used to gradually optimize the network structure parameters of the first supernetwork in this round, thereby gradually improving the performance of the optimized first supernetwork. In particular, when training the current first supernetwork in each round, the sandwich method can be used to train the first supernetwork. That is, in each round of training, the largest subnetwork, the smallest subnetwork, and two randomly sampled subnetworks are sampled from the first supernetwork, and all subnetworks are trained until convergence. The weight parameters of the subnetworks are shared with the first supernetwork to obtain the trained first supernetwork. Of course, in addition to the sandwich method mentioned above, the training strategy can also be uniform sampling or independent sampling, etc. This embodiment does not limit this.
[0033] Understandably, the first supernetwork in step S120 is the first supernetwork after each round of training; it can be regarded as pre-training of the first supernetwork, through which the network performance of the first supernetwork is initially improved.
[0034] In some implementations, please refer to Figure 3 Step S120 may include the contents of steps S121 to S122:
[0035] Step S121: Obtain the search space corresponding to the first supernetwork, wherein the search space includes the first value range of each of the various network structure parameters of the first supernetwork.
[0036] Step S122: Based on the search space, sample multiple sub-networks, where the parameter values of each network structure parameter in each sub-network are all within the first value range of each network structure parameter.
[0037] The search space corresponding to the first supernetwork is the pre-set and stored range for searching the neural network architecture, such as the one mentioned above. in, ε represents the depth of the convolutional unit, and ε represents the expansion rate of the input channels in the convolutional block. This characterizes the size of the depth-wise-conv convolutional kernel in the convolutional block. Specifically, based on this search space, the maximum depth of the convolutional units in each sampled sub-network does not exceed [3,4,5], the expansion rate of the number of input channels in the convolutional block does not exceed [2,4,6], and the size of the depth-wise-conv convolutional kernel in the convolutional block does not exceed [3,5,7]. The sampling of multiple sub-networks can be based on random sampling from the search space corresponding to the first supernetwork, and each sub-network must have at least one different network structure parameter.
[0038] Step S130: Test the network performance of each sub-network using the test sample set corresponding to the target prediction task to obtain the first network performance parameter of each sub-network.
[0039] In practical applications, supernetworks possess a consistent attribute characteristic. This consistency refers to the degree of consistency between the network performance of the candidate network set obtained from the supernetwork and the network performance obtained by training the candidate network set individually. These subnetworks can generally be used to characterize the network performance of the first supernetwork. Therefore, for the multiple subnetworks sampled from the first supernetwork after each round of pre-training, the network performance of each subnetwork can be tested using the test sample set corresponding to the target prediction task, obtaining the first network performance parameters of each subnetwork. In this way, through the first network performance parameters of multiple subnetworks, the excellence of the first supernetwork's network performance can be indirectly known, so as to iteratively optimize the network structure parameters of the first supernetwork, that is, optimize the search space corresponding to the first supernetwork, and gradually improve the network performance of the first supernetwork.
[0040] The first network performance parameter can be the task prediction error rate, and may also include at least one of the task prediction latency and task prediction, which is not limited in this embodiment. The target prediction task can be determined according to the actual application requirements. For example, if the actual application requirements are face alignment, face recognition, eye tracking, etc., that is, it is necessary to estimate the face pose and improve the performance of face-related tasks based on the estimation results, then the target prediction task can be a face pose prediction task; correspondingly, the test sample set corresponding to the target prediction task can be a set of sample images containing faces, wherein each sample image in the sample image set carries its face pose information, so as to determine the prediction error rate of the face pose prediction task based on the pose information carried by the sample images. Of course, other performance index parameter values can also be set. Correspondingly, based on the other performance index parameter values, the performance of other networks of each sub-network can be determined, thereby realizing the evaluation of the network performance of the first super-network from a multi-dimensional network performance evaluation perspective, laying the foundation for subsequent optimization of the network structure parameters of the first super-network, and realizing the maximum iterative optimization of the network structure parameters of the first super-network.
[0041] Step S140: Based on the first network performance parameters of each sub-network, iteratively optimize the network structure parameters of the first supernetwork until the first optimization condition is met, and obtain the optimized first supernetwork as the second supernetwork. The second supernetwork is used to obtain the target model corresponding to the target prediction task after iterative training through the training sample set.
[0042] Furthermore, after obtaining the first network performance parameters of each sub-network, the target network performance parameters of the first super-network in this round can be determined based on the first network performance parameters of each sub-network in this round. The target network performance parameter value of the first super-network can be determined by weighted averaging of the first network performance parameter values. Of course, the determination method can be adjusted according to actual needs, and this embodiment does not impose any restrictions on this. For example, taking the task prediction error rate as the first network performance parameter, the average prediction error rate of the task prediction errors of multiple sub-networks can be obtained as the target prediction error rate of the first super-network. Further, the structural parameters of the first super-network can be optimized based on the target network performance parameters of the first super-network determined in this round to obtain the optimized first super-network. It can be understood that each round will optimize the network structural parameters according to steps S120 to S140, that is, the initial first super-network will undergo multiple rounds of network structural parameter optimization until the first optimization condition is met, thus obtaining the optimized first super-network, which serves as the aforementioned second super-network. Understandably, the second hypernetwork can serve as the target model for a target prediction task. For example, if the target prediction task is face pose prediction, the second hypernetwork can be used as a face pose prediction model. Furthermore, in this case, the task prediction performance of the second hypernetwork is significantly improved compared to the performance of the unoptimized first hypernetwork.
[0043] In some implementations, please refer to Figure 4 After step S140, the steps S150 to S170 may also be included:
[0044] Step S150: Using the training sample set, iteratively train the second supernetwork until the first target condition is met, and obtain the trained second supernetwork.
[0045] Understandably, the second supernetwork is a supernetwork optimized by searching the search space of the pre-trained first supernetwork. To further improve the network performance of the second supernetwork, iterative training can be performed on the second supernetwork using the training sample set corresponding to the target prediction task until the first target condition is met, thus obtaining the trained second supernetwork. At this point, the performance of the trained second supernetwork is greatly improved compared to the second supernetwork before training.
[0046] In some implementations, the second supernetwork can be trained using a progressive shrinkage approach. This progressive shrinkage training includes three phases, such as... Figure 5 As shown, in the first stage, the search space corresponding to the second supernetwork only contains the sub-search space. ε and The value is fixed at the maximum value in the corresponding set. After training converges, the second stage begins, where the search space of the second supernetwork only contains the sub-search space. and ε, The value is fixed at the maximum value in the corresponding set. After training converges, the process proceeds to the third stage, in which the search space corresponding to the second supernetwork contains... ε and Each stage of the second supernetwork inherits the weights from the previous stage. This progressive shrinkage training method mitigates the problems of large training oscillations and slow convergence caused by the mutual coupling between subnetworks. Furthermore, the sandwich method is employed during training: in each training round, the largest and smallest subnetworks are sampled, along with two networks of intermediate size. Gradients are accumulated and updated during backpropagation. During retraining, the larger network supervises the smaller network, improving its performance and thus enhancing the performance of the trained second supernetwork. For example, this improves the prediction accuracy of the second supernetwork in face pose prediction. Moreover, the subnetworks and supernetworks share weights during training; this one-shot training method accelerates network convergence. Simultaneously, the subnetworks with inherited weights sampled during search achieve high accuracy without fine-tuning or retraining from scratch.
[0047] The first objective condition can be: the loss value corresponding to the training of the second hypernetwork is less than a preset value, the loss value no longer changes, or the number of training iterations reaches a preset number. It is understood that after iteratively training the second hypernetwork for multiple training cycles based on the training sample set—each training cycle including multiple iterations—the weight parameters in the second hypernetwork are continuously optimized, causing the aforementioned loss value to become smaller and smaller, eventually reaching a fixed value or less than the preset value. At this point, it indicates that the second hypernetwork has converged. Alternatively, convergence can be determined after the number of training iterations reaches a preset number. The preset value and the preset number of iterations are pre-set and can be adjusted according to different application scenarios; this embodiment does not impose such restrictions.
[0048] Step S160: Based on the target performance index value, sample the trained second supernetwork to obtain the second subnetwork corresponding to the second supernetwork.
[0049] Step S170: Using a pre-trained performance prediction model, predict the second network performance parameters of the second sub-network until the sampled second network performance parameters of the second sub-network meet the second target condition, and use the second sub-network that meets the second target condition as the target model.
[0050] Furthermore, after training the second supernetwork is completed, the trained second supernetwork can be sampled based on the target performance metric value to obtain the second subnetwork corresponding to the second supernetwork. The target performance metric value can be a hardware-related performance metric value, including but not limited to any one or more of model inference latency, activation count, throughput, power consumption, and memory utilization. This target performance metric value can be pre-set according to the hardware conditions of the model deployment platform; this embodiment does not impose any restrictions on this.
[0051] In some implementations, an evolutionary computation search algorithm, combined with a target performance index value, can be used to quickly search for subnetworks that meet the target performance index value from the second supernetwork, serving as the target model. Specifically, a pre-trained performance prediction model can be used to predict the performance of each second subnetwork in the subnetwork population composed of sampled second subnetworks. Evolutionary operations (crossover and mutation) are performed under constraints (such as the number of floating-point operations per second), evaluating and iterating the second subnetworks in the population. After satisfying the target number of iterations, the second subnetwork with the best performance in the population is output, and this best-performing second subnetwork can serve as the target model. The performance prediction model can be trained on a multilayer perceptron (MLP) based on a target training sample set. The target training sample set can be based on sampling multiple subnetworks from the trained second supernetwork and performing structural encoding, i.e., generating a sub-search space corresponding to each subnetwork. Simultaneously, the network performance parameter values corresponding to each subnetwork are tested, generating training sample pairs of {subnetwork structure encoding, subnetwork performance}. Thus, by using a pre-trained performance prediction model, test set inference on a complex convolutional neural network can be avoided, saving a lot of time and accelerating the search for network structures. This, in turn, improves the search speed for finding a second subnetwork that meets the target performance index as the target model. Furthermore, since the second supernetwork is a supernetwork that has been trained after search space optimization, the search efficiency is greatly improved when searching for the second subnetwork, and the performance of the searched second subnetwork is also improved.
[0052] Alternatively, network structures can be searched using search algorithms such as random and grid search, gradient-based strategies, or reinforcement learning strategies. This embodiment does not limit this approach.
[0053] In other embodiments, steps S160 to S170 can also be executed on other computer devices. That is, after the training of the second supernetwork is completed, the other computer device can obtain the trained second supernetwork from the computer device used to train the second supernetwork; and then the search for the neural network structure is performed through the implementation of steps S160 to S170.
[0054] In this embodiment, before performing the neural network architecture search, based on the network performance of multiple sub-networks sampled from the first supernetwork, the initial search space corresponding to the first supernetwork is iteratively optimized to obtain the optimal search space, i.e., the second supernetwork. This improves the effectiveness of the search space design and the performance of the searched network structure. Then, the second supernetwork is trained using a progressive shrinking training method. Since the sub-networks and supernetworks share weights during training, the one-shot training method can accelerate network convergence. At the same time, the sub-networks with inherited weights sampled during the search do not need fine-tuning or retraining, and can achieve high accuracy. After the second supernetwork is trained, an evolutionary computation-based search algorithm is used to quickly search for and deploy the most suitable sub-network for different hardware platforms and efficiency constraints.
[0055] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating a neural network optimization method according to another embodiment of this application. The following will be combined with... Figure 6 The optimization method for the neural network provided in this application embodiment is described in detail. The optimization method for the neural network may include the following steps:
[0056] Step S210: Obtain the first supernetwork.
[0057] Step S220: Sample the first supernetwork to obtain multiple subnetworks.
[0058] In this embodiment, the specific implementation of steps S210 to S220 can be found in the content of the foregoing embodiments, and will not be repeated here.
[0059] Step S230: Test the network performance of each sub-network using the test sample set corresponding to the target prediction task to obtain the first network performance parameter of each sub-network. The first network performance parameter includes at least the error rate of task prediction for the target prediction task. Each test sample in the test sample set carries target label information.
[0060] In some implementations, please refer to Figure 7 Step S230 may include the contents of steps S231 to S232:
[0061] Step S231: Input each test sample in the test sample set into each sub-network to obtain the predicted label information for each test sample output by each sub-network.
[0062] In this embodiment, each test sample in the test sample set used to test the sub-network carries target label information. Based on this, each test sample can be input into each sub-network. Each sub-network can extract features based on the input test sample and output the predicted label information corresponding to the test sample based on the extracted feature information.
[0063] Step S232: Based on the predicted label information for each test sample output by each sub-network and the target label information carried by each test sample, determine the label prediction error rate of each sub-network.
[0064] Furthermore, the predicted label information for each test sample can be compared with the target label information carried by each test sample to determine whether they match. If they do not match, it indicates that the label prediction is incorrect; if they match, it indicates that the label prediction is correct. Then, based on the prediction and matching results of each sub-network for each test sample, the label prediction error rate of each sub-network for the test samples in the test sample set is calculated. The label prediction error rate of each sub-network can be understood as the mean average error (MAE) of the label prediction for the test sample set by each sub-network.
[0065] For example, taking face pose prediction as an example, the test sample set is a set of sample images containing faces. Each sample image carries target label information including the target pose information of the face, which includes the target yaw angle, target pitch angle, and target roll angle. Each sample image is input into each sub-network, which can predict the predicted label information of the face in the sample image. This predicted label information includes the predicted pose information of the face, including the predicted yaw angle, predicted pitch angle, and predicted roll angle. Then, it can be determined whether the pose error value between the target pose information and the predicted pose information is within a preset error value. If yes, the label prediction is determined to be correct; otherwise, the label prediction is determined to be incorrect.
[0066] Step S240: Based on the error rate of each sub-network for the target prediction task and the parameter value of the network structure parameter of each sub-network, generate a target mapping relationship between the error rate and the parameter value, wherein the parameter value of the network structure parameter of each sub-network is the maximum value of the network structure parameter in the sub-search space corresponding to each sub-network.
[0067] In this embodiment, the parameter value of the network structure parameter of each sub-network is the maximum value of the network structure parameter in the sub-search space corresponding to each sub-network. Therefore, the target mapping relationship between the generated error rate and the parameter value is the mapping relationship between the generated error rate and the value of the sub-search space.
[0068] Specifically, based on the statistically obtained task prediction error rate of each sub-network for the target prediction task and the parameter values of the network structure parameters of each sub-network, the mapping relationship between the two can be fitted using the following linear function:
[0069] y = ωx + b
[0070] Where y represents the error rate, and x represents the parameter values of the network structure parameters of each subnetwork. ω is a positive number greater than 0, meaning the error rate is positively correlated with the size of the subnetwork.
[0071] Step S250: According to the target mapping relationship, iteratively optimize the parameter values of the network structure parameters of the first hypernetwork until a preset number of iterations is reached to obtain the optimized first hypernetwork, which serves as the second hypernetwork. The error rate of the second hypernetwork in predicting the target prediction task is less than the error rate of the first hypernetwork in predicting the target prediction task. The parameter values of the network structure parameters of the first hypernetwork are the maximum values of the network structure parameters in the search space corresponding to the first hypernetwork.
[0072] In some implementations, the error rate in the target mapping relationship is positively correlated with the parameter value; please refer to [link to relevant documentation]. Figure 8 Step S250 may include the contents of steps S251 to S252:
[0073] Step S251: Determine the first parameter adjustment value based on the target mapping relationship.
[0074] Specifically, based on the linear mapping relationship, the rate of change of the error rate relative to the parameter value is determined; this rate of change is ω in the aforementioned linear function. Based on the rate of change and the second parameter adjustment value, the first parameter adjustment value is determined. The second parameter adjustment value is the current step size for optimizing the search space, i.e., the product of the rate of change and the second parameter adjustment value is obtained as the step size for this iteration of the search space. Understandably, if the current iteration is the first, then the current step size is the pre-set initial step size, for example, γ. d =γ e =γ k =1,γ d γ is used to characterize the evolution progression for the depth of a convolutional unit. eγ is used to characterize the evolution progression with respect to the expansion rate of the number of input channels in a convolutional block. k This parameter is used to characterize the step size for the depth-wise-conv convolutional kernel size within a convolutional block. If the current iteration number is T, then the current step size is the same as the step size at iteration T-1. In other words, the adjustment value of the second parameter is the current actual step size.
[0075] Step S252: Based on the first parameter adjustment value, iteratively adjust the parameter values of the network structure parameters of the first supernetwork until the preset number of iterations is reached to obtain the adjusted first supernetwork, which serves as the second supernetwork. The parameter values of the network structure parameters of the second supernetwork are less than the parameter values of the network structure parameters of the first supernetwork.
[0076] Furthermore, in each round of parameter adjustment, the network structure parameter values of the first supernetwork are reduced based on the first parameter adjustment value. Specifically, the difference between the current network structure parameter value of the first supernetwork and the first parameter adjustment value is obtained as the parameter value of the network structure parameter of the first supernetwork after this round of adjustment. The search space of the first supernetwork is gradually reduced until the preset number of iterations is reached, at which point the iterative adjustment of the parameter values stops, resulting in the first supernetwork corresponding to the final optimized search space, which is used as the second supernetwork. Understandably, the value range of the search space corresponding to the second supernetwork is smaller than that of the search space corresponding to the first supernetwork; that is, the parameter values of the network structure parameters of the second supernetwork are smaller than those of the first supernetwork. Thus, by gradually reducing the value range of the search space, the search space is optimized, thereby gradually reducing the error rate of the task prediction of the first supernetwork, i.e., gradually improving the network performance of the first supernetwork.
[0077] Specifically, step S250 can be characterized by the iterative update of the parameter values of the network structure parameters of the first supernetwork using the following formula:
[0078]
[0079] in, The parameter values used to characterize the network structure parameters of the i-th dimension in the search space corresponding to the first supernetwork in round t+1. The parameter values are used to characterize the parameters of the i-th network structure in the search space corresponding to the first supernetwork in round t, where w is the rate of change of the error rate in the aforementioned linear function relative to the parameter values in the search space. This is used to characterize the actual evolution growth of the network structure parameters of the i-th type in the search space corresponding to the first supernetwork in round t.
[0080] Understandably, the search space has a minimum threshold size; for example, the size of the convolutional kernel cannot be less than 1*1*1. Therefore, when obtaining... Next, it needs to be compared with the minimum threshold to ensure that the optimized search space can achieve network structure search. Furthermore, since this embodiment does not use a fixed step size to update the parameter values of the first supernetwork's network structure parameters, the larger the error rate change rate, the larger the step size. This allows for the maximum reduction of the current parameter values of the first supernetwork's network structure parameters, quickly lowering the task prediction error rate of the first supernetwork. Thus, the iterative update of the first supernetwork's network structure parameters based on the dynamically determined first parameter adjustment value according to the task prediction error rate allows for faster and more accurate adjustment of the first supernetwork's network structure parameters to an appropriate position. This enables faster and more accurate optimization of the search space corresponding to the first supernetwork, ultimately obtaining an optimal search space. This effectively improves the efficiency of setting the search space and enhances the performance of the searched network.
[0081] In this embodiment, the method of iteratively updating the network structure parameters of the first supernetwork based on the dynamically determined first parameter adjustment value according to the task prediction error rate allows for faster adjustment of the network structure parameters to an appropriate position. This enables faster optimization of the search space corresponding to the first supernetwork, ultimately yielding an optimal search space. In other words, it effectively optimizes the initial search space without requiring prior human knowledge, improving the effectiveness of the search space design and thus enhancing the performance of the search architecture.
[0082] Please refer to Figure 9 The diagram illustrates a structural block diagram of a neural network optimization device 300 according to an embodiment of this application. The device 300 may include: a supernetwork acquisition module 310, a first sampling module 320, a performance testing module 330, and a supernetwork optimization module 340.
[0083] The supernetwork acquisition module 310 is used to acquire the first supernetwork.
[0084] The first sampling module 320 is used to sample the first supernetwork to obtain multiple subnetworks.
[0085] The performance testing module 330 is used to test the network performance of each sub-network using the test sample set corresponding to the target prediction task, and obtain the first network performance parameter of each sub-network.
[0086] The supernetwork optimization module 340 is used to iteratively optimize the parameters until the first optimization condition is met, and obtain the optimized first supernetwork as the second supernetwork. The second supernetwork is used to obtain the target model corresponding to the target prediction task after iterative training with the training sample set.
[0087] In some implementations, the first network performance parameter includes at least the task prediction error rate for the target prediction task. Each test sample in the test sample set carries target label information. The performance testing module 330 includes a label prediction unit and an error rate prediction unit. The label prediction unit can be used to input each test sample in the test sample set into each sub-network to obtain the predicted label information output by each sub-network for each test sample. The error rate prediction unit can be used to determine the label prediction error rate of each sub-network based on the predicted label information output by each sub-network for each test sample and the target label information carried by each test sample.
[0088] In some implementations, the first network performance parameters include at least the task prediction error rate for the target prediction task. The hypernetwork optimization module 340 may include a mapping relationship generation unit and an iterative optimization unit. The mapping relationship generation unit can be used to generate a target mapping relationship between the error rate and the parameter values based on the task prediction error rate of each subnetwork for the target prediction task and the parameter values of the network structure parameters of each subnetwork. The parameter value of the network structure parameter of each subnetwork is the maximum value of the network structure parameter in the sub-search space corresponding to each subnetwork. The iterative optimization unit can be used to iteratively optimize the parameter values of the network structure parameters of the first hypernetwork according to the target mapping relationship until a preset number of iterations is reached, obtaining an optimized first hypernetwork as the second hypernetwork. The task prediction error rate of the second hypernetwork for the target prediction task is less than the task prediction error rate of the first hypernetwork for the target prediction task, and the parameter values of the network structure parameters of the first hypernetwork are the maximum values of the network structure parameters in the search space corresponding to the first hypernetwork.
[0089] In this approach, the error rate in the target mapping relationship is positively correlated with the parameter value. The iterative optimization unit may include a parameter adjustment value determination subunit and an iterative optimization subunit. The parameter adjustment value determination subunit can be used to determine a first parameter adjustment value based on the target mapping relationship. The iterative optimization subunit can be used to iteratively adjust the parameter values of the network structure parameters of the first hypernetwork based on the first parameter adjustment value until the preset number of iterations is reached, obtaining the adjusted first hypernetwork as the second hypernetwork. The parameter values of the network structure parameters of the second hypernetwork are less than the parameter values of the network structure parameters of the first hypernetwork.
[0090] In some implementations, the target mapping relationship is a linear mapping relationship, and the parameter adjustment value determination subunit can be specifically used to: determine the rate of change of the error rate relative to the parameter value based on the linear mapping relationship; and determine the first parameter adjustment value based on the rate of change and the second parameter adjustment value.
[0091] In some implementations, the first sampling module 320 may include a search space acquisition unit and a sampling unit. The search space acquisition unit may be used to acquire the search space corresponding to the first supernetwork, where the search space includes a first value range for each of the various network structure parameters of the first supernetwork. The sampling unit may be used to sample multiple subnetworks based on the search space, where the parameter value of each network structure parameter in each subnetwork is located within the first value range of each network structure parameter.
[0092] In some embodiments, the neural network optimization device 300 may further include: a supernetwork training module, a second sampling module, and an iterative optimization module. The supernetwork training module can be used to iteratively optimize the network structure parameters of the first supernetwork based on the first network performance parameters of each of the subnetworks until a first optimization condition is met, obtaining an optimized first supernetwork as the second supernetwork. Then, using the training sample set, the second supernetwork is iteratively trained until a first target condition is met, obtaining a trained second supernetwork. The second sampling module can be used to sample the trained second supernetwork according to a target performance index value to obtain a second subnetwork corresponding to the second supernetwork. The iterative optimization module can be used to predict the second network performance parameters of the second subnetwork using a pre-trained performance prediction model until the second network performance parameters of the sampled second subnetwork meet the second target condition, and the second subnetwork meeting the second target condition is taken as the target model.
[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0094] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.
[0095] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0096] In summary, the process involves: obtaining a first supernetwork; sampling the first supernetwork to obtain multiple subnetworks; testing the network performance of each subnetwork using a test sample set corresponding to the target prediction task to obtain the first network performance parameters of each subnetwork; iteratively optimizing the network structure parameters of the first supernetwork based on the first network performance parameters of each subnetwork until a first optimization condition is met, resulting in an optimized first supernetwork, which serves as the second supernetwork. The second supernetwork is used for iterative training using the training sample set to obtain the target model corresponding to the target prediction task. Thus, before searching for the neural network architecture, the initial search space corresponding to the first supernetwork is iteratively optimized based on the network performance of the multiple subnetworks sampled from the first supernetwork to obtain the optimal search space, i.e., a second supernetwork with more optimized network structure parameters. This improves the effectiveness of the search space design and the performance of the searched network structure, thereby enhancing the prediction performance of the target model trained on the second supernetwork for the target prediction task.
[0097] The following will combine Figure 10 This application describes a computer device.
[0098] Reference Figure 10 , Figure 10This diagram illustrates a structural block diagram of a computer device 400 provided in an embodiment of this application. The method described above in this embodiment can be executed by this computer device 400. The computer device can be an electronic terminal with data processing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, smartwatches, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, and smart home devices. Alternatively, the computer device can be a server. A server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms.
[0099] The computer device 400 in this application embodiment may include one or more of the following components: processor 401, memory 402, and one or more application programs, wherein the one or more application programs may be stored in memory 402 and configured to be executed by one or more processors 401, and the one or more programs are configured to perform the methods as described in the foregoing method embodiments.
[0100] Processor 401 may include one or more processing cores. Processor 401 connects to various parts within the computer device 400 using various interfaces and lines, and performs various functions and processes data of the computer device 400 by running or executing instructions, programs, code sets, or instruction sets stored in memory 402, and by calling data stored in memory 402. Optionally, processor 401 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 401 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the aforementioned modem can also be integrated into processor 401 and implemented using a separate communication chip.
[0101] The memory 402 may include random access memory (RAM) or read-only memory (ROM). The memory 402 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 402 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the computer device 400 during use (such as the various correspondences described above).
[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0103] In the several embodiments provided in this application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interface, device or module, and may be electrical, mechanical or other forms.
[0104] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0105] Please refer to Figure 11 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 500 stores program code that can be called by a processor to execute the methods described in the above method embodiments.
[0106] The computer-readable storage medium 500 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 500 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 500 has storage space for program code 510 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 510 may be compressed, for example, in a suitable form.
[0107] In some embodiments, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the steps in the above-described method embodiments.
[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for optimizing a neural network, characterized in that, The method includes: Obtain the first supernetwork; The first supernetwork is sampled to obtain multiple subnetworks; The network performance of each sub-network is tested using the test sample set corresponding to the target prediction task to obtain the first network performance parameter of each sub-network. The target prediction task includes an image processing task. The first network performance parameter includes at least the task prediction error rate for the target prediction task. Based on the error rate of each sub-network in the task prediction of the target prediction task and the parameter value of the network structure parameter of each sub-network, a target mapping relationship between the error rate and the parameter value is generated. The parameter value of the network structure parameter of each sub-network is the maximum value of the network structure parameter in the sub-search space corresponding to each sub-network. The error rate and the parameter value in the target mapping relationship are positively correlated, and the target mapping relationship is a linear mapping relationship. Based on the linear mapping relationship, determine the rate of change of the error rate relative to the parameter value; The adjustment value of the first parameter is determined based on the rate of change and the adjustment value of the second parameter; Based on the first parameter adjustment value, the parameter values of the network structure parameters of the first supernetwork are iteratively adjusted until a preset number of iterations is reached to obtain the adjusted first supernetwork, which serves as the second supernetwork. The parameter values of the network structure parameters of the second supernetwork are less than those of the first supernetwork, and the error rate of the second supernetwork in predicting the target prediction task is less than that of the first supernetwork. The parameter values of the network structure parameters of the first supernetwork are the maximum values of the network structure parameters in the search space corresponding to the first supernetwork. The second supernetwork is used to obtain the target model corresponding to the target prediction task after iterative training with the training sample set.
2. The method according to claim 1, characterized in that, The first network performance parameter includes at least the task prediction error rate for the target prediction task. Each test sample in the test sample set carries target label information. The step of testing the network performance of each sub-network using the test sample set corresponding to the target prediction task to obtain the first network performance parameter of each sub-network includes: Each test sample in the test sample set is input into each sub-network to obtain the predicted label information for each test sample output by each sub-network; Based on the predicted label information for each test sample output by each sub-network and the target label information carried by each test sample, the label prediction error rate of each sub-network is determined.
3. The method according to claim 1, characterized in that, The sampling of the first supernetwork yields multiple subnetworks, including: Obtain the search space corresponding to the first hypernetwork, wherein the search space includes a first value range of each of the various network structure parameters of the first hypernetwork; Based on the search space, multiple sub-networks are sampled, and the parameter values of each network structure parameter in each sub-network are all within the first value range of each network structure parameter.
4. The method according to any one of claims 1-3, characterized in that, After iteratively adjusting the network structure parameters of the first hypernetwork based on the first parameter adjustment value until the preset number of iterations is reached to obtain the adjusted first hypernetwork as the second hypernetwork, the method further includes: Using the training sample set, the second hypernetwork is iteratively trained until the first target condition is met, thus obtaining the trained second hypernetwork; Based on the target performance index value, the trained second supernetwork is sampled to obtain the second subnetwork corresponding to the second supernetwork; Using a pre-trained performance prediction model, the second network performance parameters of the second sub-network are predicted until the sampled second network performance parameters of the second sub-network meet the second target condition. The second sub-network that meets the second target condition is then used as the target model.
5. An optimization device for a neural network, characterized in that, The device includes: The hypernetwork acquisition module is used to acquire the first hypernetwork; The first sampling module is used to sample the first supernetwork to obtain multiple subnetworks; The performance testing module is used to test the network performance of each sub-network using the test sample set corresponding to the target prediction task, and to obtain the first network performance parameter of each sub-network. The target prediction task includes an image processing task, and the first network performance parameter includes at least the task prediction error rate for the target prediction task. A hypernetwork optimization module is used to generate a target mapping relationship between the error rate and the parameter values based on the task prediction error rate of each subnetwork for the target prediction task and the parameter values of the network structure parameters of each subnetwork. The parameter values of the network structure parameters of each subnetwork are the maximum values of the network structure parameters in the sub-search space corresponding to each subnetwork. The error rate and the parameter values in the target mapping relationship are positively correlated, and the target mapping relationship is a linear mapping relationship. Based on the linear mapping relationship, the module determines the rate of change of the error rate relative to the parameter values. Based on the rate of change and a second parameter adjustment value, the module determines a first parameter adjustment value. Based on the first parameter adjustment value, the module further optimizes the network structure. The parameter values of the network structure parameters of the first supernetwork are iteratively adjusted until a preset number of iterations is reached to obtain the adjusted first supernetwork, which serves as the second supernetwork. The parameter values of the network structure parameters of the second supernetwork are smaller than those of the first supernetwork, and the error rate of the second supernetwork in predicting the target prediction task is smaller than that of the first supernetwork. The parameter values of the network structure parameters of the first supernetwork are the maximum values of the network structure parameters in the search space corresponding to the first supernetwork. The second supernetwork is used to obtain the target model corresponding to the target prediction task after iterative training with the training sample set.
6. A computer device, characterized in that, include: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1-4.