A neural network search method and system

Abstract

The present invention discloses a neural network search method and system, the method comprising: randomly sampling two non-overlapping models on a hypernetwork to be searched, the two models having one copy each in an encoder and a momentum encoder, training the encoder based on momentum contrast learning; fixing the encoder network weights to set its batch normalization layer to training mode, adding a randomly initialized fully connected classification head after the encoder and training it; calculating the classification prediction accuracy of the model in the validation set as the performance index of the sampling model, re-searching and repeating the above steps of sampling model training, classification head training and performance evaluation, and obtaining the model with the best performance index as the final model. The search method provided by the present invention can train the hypernetwork in a self-supervised manner, the process does not require data annotation, and uses cross-learning to allow different structures to learn similar expressions, thereby alleviating the problem of unfair training or even training failure of different structures on the hypernetwork, and making the training of the hypernetwork more efficient and more robust.

Description

Technical Field

[0001] The present invention relates to the field of neural network search technology, and in particular to a neural network search method and system. Background Art

[0002] Neural Network Architecture Search (NAS) aims to automatically design neural network structures, thus avoiding tedious manual design. Currently, neural networks obtained by NAS search have achieved remarkable results in tasks such as image classification, object detection, image segmentation, and natural language. Starting from manually designing discrete model space and operation space, NAS uses search technology to explore the preset search field and find the best structure for single or multiple goals (such as accuracy, memory consumption, etc.), which often exceeds the performance of manual design.

[0003] Early NAS methods used dense methods to search for neural networks. They needed to sample a large number of network structures and train them from scratch. The huge computational cost makes them unsuitable for large data sets. Some recent methods have adopted a weight sharing strategy to reduce the computational cost. The idea of ​​this strategy is to design a super network that contains all structures, and the sampled neural networks are all sub-networks of the super network. In this way, only one complete training of the super network is required, and the sampled sub-networks can inherit and fine-tune the weights of the super network, which greatly reduces the computational cost. In this process, most weight allocation methods use continuous relaxation to parameterize the search space, but this will lead to the coupling of network structure parameters and super network weights during joint optimization. The greediness of gradient-based methods will inevitably introduce bias in the optimization process and mislead the search of network structure. At present, the training of super networks in NAS research is still a traditional supervised method. The parameters of the trained super network may not have the best generalization ability when searching and fine-tuning, so the results may not be optimal. Summary of the invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the defects of the prior art neural network search method in that the generalization ability is poor and the optimal search results cannot be obtained. Therefore, a neural network search method and system are provided, and the network obtained by the system achieves good results in image classification, semantic segmentation and instance segmentation tasks.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides a neural network search method, comprising the following steps:

[0007] Randomly sample two non-overlapping models on the hypernetwork to be searched. The two sampled models have one copy each in the encoder and momentum encoder. The encoder is trained based on the momentum contrastive learning method.

[0008] Take out the trained encoder and fix its network weights, set all batch normalization layers in the encoder to training mode, add a randomly initialized fully connected classification head after the encoder structure, and train the classification head;

[0009] The classification prediction accuracy of the model is calculated on the validation set and used as the performance indicator of the hypernetwork sampling model;

[0010] The ActionSpace method is used to sample the model, and the above steps of sampling model training, classification head training and performance evaluation are re-searched and repeated in the hypernetwork, and the model with the best performance index is obtained as the final model.

[0011] In one embodiment, the process of training the encoder based on the momentum contrastive learning method includes:

[0012] The momentum encoder is initialized using the same network structure as the encoder. In each training iteration, four random data augmentations are performed on each image sample x in the batch to obtain As network input; Feed the randomly sampled model 1 and model 2 on the encoder, and then pass through the shared multi-layer perceptron to obtain the feature vector Then separately Feeding the momentum encoders to model 1 and model 2, we get Will As a positive sample pair, As another pair of positive samples, the contrast loss is calculated with the negative sample queue to obtain two losses Add the two losses as the loss of sample x, reverse the loss to update the encoder, and then calculate the sliding average of the network weights of the updated encoder to update the momentum encoder; then Average and update to the negative sample queue.

[0013] In one embodiment, the network weights of the classification head network after training are fixed, the sliding means and sliding variances of all batch normalization layers are set to zero and set to training mode, the training images are fed to the sampling model on the hypernetwork, the parameters of the batch normalization layer are updated, and the batch normalization layer is set to verification mode, the verification images are fed to the sampling model on the hypernetwork, and its classification prediction accuracy on the classification head is used as the performance indicator of the model on the hypernetwork.

[0014] In one embodiment, the classification head is shared by multiple classifiers.

[0015] In one embodiment, the process of training the classification head network includes: in each training iteration, feeding the image to a model randomly sampled on the hypernetwork, then obtaining a classification prediction through the shared classification head, and calculating the classification loss between the classification prediction result and the true label of the image to train the classification head.

[0016] In a second aspect, an embodiment of the present invention provides a neural network search system, including: a sampling model training module, for randomly sampling two non-overlapping models on a hypernetwork to be searched, wherein the two sampled models have one copy each in an encoder and a momentum encoder, and the encoder and the momentum encoder are trained based on a momentum contrastive learning method;

[0017] The classification head training module is used to take out the trained encoder and fix its network weights, set all batch normalization layers in the encoder to training mode, add a randomly initialized fully connected classification head after the encoder structure, and train the classification head;

[0018] The sampling model performance indicator acquisition module is used to calculate the classification prediction accuracy of the model in the validation set and use it as the performance indicator of the hypernetwork sampling model;

[0019] The final model determination module is used to sample the model using the Action Space method, re-search and repeat the above steps of sampling model training, classification head training and performance evaluation in the hypernetwork, and obtain the model with the best performance index as the final model.

[0020] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the neural network search method of the first aspect of the embodiment of the present invention.

[0021] In a fourth aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the neural network search method of the first aspect of the embodiment of the present invention by executing the computer instructions.

[0022] The technical solution of the present invention has the following advantages:

[0023] 1. The neural network search method provided by the present invention is a self-supervised neural network search method, which can train a super network in a self-supervised manner. The process does not require data annotation. First, two non-overlapping models are randomly sampled on the super network to be searched, and the two sampled models are each in the encoder and the momentum encoder. The encoder is trained based on the momentum contrast learning method; the trained encoder is taken out and its network weights are fixed, all batch normalization layers in the encoder are set to training mode, and a randomly initialized fully connected classification head is added after the encoder structure, and the classification head is trained; the classification prediction accuracy of the model is calculated in the validation set, and it is used as the performance indicator of the super network sampling model; the model is sampled using the ActionSpace method, and the above-mentioned sampling model training, classification head training and performance evaluation steps are re-searched and repeated in the super network, and the model with the best performance indicator is obtained as the final model.

[0024] 2. The present invention provides a neural network search method and system, which utilizes cross-learning to allow different structures to learn similar expressions, alleviates the problem of unfair training or even training failure of different structures on the hypernetwork, and can obtain enough negative samples that are common to all structures. During cross-learning, positive samples come from the output of different structures, which can be regarded as a new type of data enhancement, making the training of the hypernetwork more efficient and robust. The network obtained by the neural network search method and system provided by the present invention can achieve good classification results in image classification, semantic segmentation, and example segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A workflow diagram of a specific example of the neural network search method provided in an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of training an encoder using a momentum contrastive learning method provided in an embodiment of the present invention;

[0028] Figure 3 A module composition diagram of a specific example of a neural network search system provided in an embodiment of the present invention;

[0029] Figure 4 A composition diagram of a specific example of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0032] Example 1

[0033] The embodiment of the present invention provides a neural network search method, such as Figure 1 As shown, the method comprises the following steps:

[0034] Step S1: Randomly sample two non-overlapping models on the hypernetwork to be searched, and each of the two sampled models has a copy in the encoder and the momentum encoder. The encoder is trained based on the momentum contrastive learning method.

[0035] In the embodiment of the present invention, the momentum encoder has the same initial structure as the corresponding encoder. During the training process, the momentum encoder parameters are updated based only on the encoder parameters, and the encoder is trained unsupervisedly through momentum contrastive learning. The specific process of training the encoder based on the momentum contrastive learning method is as follows: Figure 2 As shown, the momentum encoder is initialized using the same network structure as the encoder. In each training iteration, four random data enhancements are performed on each image sample x in the batch to obtain As network input; Feed the randomly sampled model 1 and model 2 on the encoder, and then pass through the shared multi-layer perceptron to obtain the feature vector Then separately Feeding the momentum encoders to model 1 and model 2, we get Will As a positive sample pair, As another pair of positive samples, the contrast loss is calculated with the negative sample queue to obtain two losses Add the two losses as the loss of sample x, reverse the loss to update the encoder, and then calculate the sliding average of the network weights of the updated encoder to update the momentum encoder; then Calculate the average and update it to the negative sample queue. This cross-learning allows different structures to learn similar expressions, alleviating the problem of unfair training or even training failure of different structures on the hypernetwork, and can obtain enough negative samples that are common to all structures. During cross-learning, positive samples come from the output of different structures, which is a new type of data enhancement, making the training of the hypernetwork more efficient and robust.

[0036] Step S2: Take out the trained encoder and fix its network weights, set all batch normalization layers in the encoder to training mode, add a randomly initialized fully connected classification head after the encoder structure, and train the classification head.

[0037] In the embodiment of the present invention, the classification head is shared by multiple classifiers. Multiple classifiers sharing one classification head can not only reduce the training time, but also compare the evaluation indicators of the sampling models more fairly.

[0038] Step S3: Calculate the classification prediction accuracy of the model on the validation set and use it as the performance indicator of the hypernetwork sampling model.

[0039] In an embodiment of the present invention, the network weights of the classification head network after training are fixed, the sliding means and sliding variances of all batch normalization layers are set to zero and set to training mode, the training images are fed to the sampling model on the hypernetwork, the parameters of the batch normalization layer are updated, and then the batch normalization layer is set to verification mode, the verification images are fed to the sampling model on the hypernetwork, and its classification prediction accuracy on the classification head is used as the performance indicator of the model on the hypernetwork.

[0040] Step S4: Use the ActionSpace method to sample the model, re-search and repeat the above steps of sampling model training, classification head training and performance evaluation in the hypernetwork, and obtain the model with the best performance index as the final model.

[0041] The embodiment of the present invention adopts the Action Space method to sample models, so that each sampling model is not exactly the same. By comparing the classification accuracy of different sampling models, the model with the highest classification accuracy is used as the final model to be applied to specific classification or segmentation tasks.

[0042] The neural network search method provided in the embodiment of the present invention can train the hypernetwork in a self-supervisory manner without data annotation, and use cross-learning to allow different structures to learn similar expressions, thereby alleviating the problem of unfair training of different structures on the hypernetwork or even training failure, and can obtain enough negative samples that are common to all structures. During cross-learning, positive samples come from outputs of different structures, making the training of the hypernetwork more efficient and robust. The network obtained by the neural network search method provided by the present invention can achieve good classification results in image classification, semantic segmentation and example segmentation tasks.

[0043] Example 2

[0044] The embodiment of the present invention provides a neural network search system, such as Figure 3 As shown, including:

[0045] The sampling model training module 1 is used to randomly sample two non-overlapping models on the hypernetwork to be searched, and the two sampled models are each stored in the encoder and the momentum encoder. The encoder and the momentum encoder are trained based on the momentum contrastive learning method. This module executes the method described in step S1 in Example 1, which will not be repeated here.

[0046] The classification head training module 2 is used to take out the trained encoder and fix its network weights, set all batch normalization layers in the encoder to training mode, add a randomly initialized fully connected classification head after the encoder structure, and train the classification head; this module executes the method described in step S2 in Example 1, which will not be repeated here.

[0047] The sampling model performance indicator acquisition module 3 is used to calculate the classification prediction accuracy of the model in the verification set and use it as the performance indicator of the super network sampling model; this module executes the method described in step S3 of embodiment 1, which will not be repeated here.

[0048] The final model determination module 4 is used to sample the model using the Action Space method, re-search and repeat the above-mentioned sampling model training, classification head training and performance evaluation steps in the hypernetwork, and obtain the model with the best performance index as the final model. This module executes the method described in step S4 of embodiment 1, which will not be repeated here.

[0049] The neural network search system provided by the embodiment of the present invention can train the hypernetwork in a self-supervisory manner without data annotation, and use cross-learning to allow different structures to learn similar expressions, thereby alleviating the problem of unfair training or even training failure of different structures on the hypernetwork, and can obtain enough negative samples that are common to all structures. During cross-learning, positive samples come from the output of different structures, making the training of the hypernetwork more efficient and robust. The network obtained by the neural network search method provided by the present invention can achieve good classification results in image classification, semantic segmentation, and example segmentation tasks.

[0050] Example 3

[0051] An embodiment of the present invention provides a computer device, such as Figure 4 As shown, the device may include a processor 51 and a memory 52, wherein the processor 51 and the memory 52 may be connected via a bus or other means. Figure 4 Take the example of connecting via a bus.

[0052] The processor 51 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above chips.

[0053] The memory 52 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the corresponding program instructions / modules in the embodiments of the present invention. The processor 51 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 52, that is, the neural network search method in the above method embodiment 1 is implemented.

[0054] The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the processor 51, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the processor 51 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, an intranet, a mobile communication network, and combinations thereof.

[0055] One or more modules are stored in the memory 52, and when executed by the processor 51, the neural network search method in Example 1 is executed.

[0056] The specific details of the above-mentioned computer device can be understood by referring to the corresponding related descriptions and effects in Example 1, and will not be repeated here.

[0057] Those skilled in the art can understand that the implementation of all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0058] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the scope of protection of the invention.

Claims

1. A neural network search method, characterized in that: The steps include: Randomly sample two non-overlapping models on the hypernetwork to be searched. The two sampled models have one copy each in the encoder and momentum encoder. The encoder is trained based on the momentum contrastive learning method, including: The momentum encoder is initialized using the same network structure as the encoder. In each training iteration, for each image sample in the batch Perform four random data augmentations to obtain , , , As network input; , Feed the randomly sampled model 1 and model 2 on the encoder, and then pass through the shared multi-layer perceptron to obtain the feature vector , , and then respectively , Feeding the momentum encoders to model 1 and model 2, we get , ,Will , As a positive sample pair, , As another pair of positive samples, the contrast loss is calculated with the negative sample queue to obtain two losses , , add the two losses together as a sample The loss is reversely derived to update the encoder, and then the sliding average of the network weights of the updated encoder is calculated to update the momentum encoder; then , Calculate the average and update it to the negative sample queue; Take out the trained encoder and fix its network weights, set all batch normalization layers in the encoder to training mode, add a randomly initialized fully connected classification head after the encoder structure, and train the classification head; The classification prediction accuracy of the model is calculated on the validation set and used as the performance indicator of the hypernetwork sampling model; The Action Space method is used to sample the model, and the above steps of sampling model training, classification head training and performance evaluation are re-searched and repeated in the hypernetwork, and the model with the best performance index is obtained as the final model.

2. The neural network search method according to claim 1, characterized in that: The process of calculating the classification prediction accuracy of the model on the validation set and using it as the performance indicator of the hypernetwork sampling model includes: Fix the network weights of the classification head network after training, set the sliding mean and sliding variance of all batch normalization layers to zero and set them to training mode, feed the training images to the sampling model on the hypernetwork, update the parameters of the batch normalization layer, set the batch normalization layer to verification mode, feed the verification images to the sampling model on the hypernetwork, and use its classification prediction accuracy on the classification head as the performance indicator of the model on the hypernetwork.

3. The neural network search method according to claim 2, characterized in that: The classification head is shared by multiple classifiers.

4. The neural network search method according to claim 3, characterized in that: The process of training the classification head network includes: In each training iteration, the image is fed to a model randomly sampled on the hypernetwork, and then passed through the shared classification head to obtain the classification prediction. The classification loss between the classification prediction result and the true label of the image is calculated to train the classification head.

5. A neural network search system, characterized in that: include: The sampling model training module is used to randomly sample two non-overlapping models on the hypernetwork to be searched. The two sampled models have one copy each in the encoder and momentum encoder. The encoder and momentum encoder are trained based on the momentum contrastive learning method, including: The momentum encoder is initialized using the same network structure as the encoder. In each training iteration, for each image sample in the batch Perform four random data augmentations to obtain , , , As network input; , Feed the randomly sampled model 1 and model 2 on the encoder, and then pass through the shared multi-layer perceptron to obtain the feature vector , , and then respectively , Feeding the momentum encoders to model 1 and model 2, we get , ,Will , As a positive sample pair, , As another pair of positive samples, the contrast loss is calculated with the negative sample queue to obtain two losses , , add the two losses together as a sample The loss is reversely derived to update the encoder, and then the sliding average of the network weights of the updated encoder is calculated to update the momentum encoder; then , Calculate the average and update it to the negative sample queue; The classification head training module is used to take out the trained encoder and fix its network weights, set all batch normalization layers in the encoder to training mode, add a randomly initialized fully connected classification head after the encoder structure, and train the classification head; The sampling model performance indicator acquisition module is used to calculate the classification prediction accuracy of the model in the validation set and use it as the performance indicator of the hypernetwork sampling model; The final model determination module is used to sample the model using the Action Space method, re-search and repeat the above steps of sampling model training, classification head training and performance evaluation in the hypernetwork, and obtain the model with the best performance index as the final model.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the neural network search method according to any one of claims 1 to 4.

7. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the neural network search method according to any one of claims 1 to 4 by executing the computer instructions.

Images