A lightweight local neural network structure search method for self-supervised training
A lightweight local neural network architecture search method with self-supervised training solves the problems of long search time and data adaptation on edge devices, enabling rapid generation of personalized models, protecting data privacy and reducing costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2024-05-11
- Publication Date
- 2026-07-21
AI Technical Summary
Existing neural network architecture search methods are time-consuming on resource-constrained edge devices and fail to effectively adapt to the device's data environment, while also posing issues of data privacy and transmission costs.
A lightweight local neural network architecture search method with self-supervised training is adopted. By constructing a sequence search space, a memory-FLOPs mapping table, a training quality predictor, and multi-objective Bayesian optimization, personalized model structures can be quickly searched locally on edge devices, avoiding data upload and hardware feedback acquisition.
Rapidly generate personalized neural network models on edge devices, reducing search time, protecting data privacy, lowering transmission costs, and improving model adaptability.
Smart Images
Figure CN118350443B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of edge computing technology, specifically relating to a lightweight local neural network structure search method for self-supervised training. Background Technology
[0002] Deep learning possesses powerful capabilities for automatically extracting features, and the design of the network architecture plays a crucial role in the feature representation of data and the performance of the final model. Neural Architecture Search (NAS) is a method that automatically generates suitable network structures through optimization algorithms. Existing NAS methods are mainly divided into two categories: pre-deployment NAS and post-deployment NAS. Pre-deployment NAS is further divided into: (i) Hardware-in-the-loop (HIL) methods, such as MnasNet, which deploy candidate models to target devices to obtain real hardware feedback and achieve a high-fit model; and (ii) Hardware-not-in-the-loop (HNIL) methods, such as ProxylessNAS, FBNet, and OFA, which use model latency predictors or latency correspondence tables to estimate the latency of candidate models on the device. HIL methods have a long search time because they require obtaining real hardware feedback; HNIL methods reduce the search time by estimating the latency of the model on the device, but reduce the model's adaptability to the device.
[0003] To address the aforementioned issues, the post-deployment NAS method, AdaptiveNet, proposes performing lightweight NAS directly on the target device, generating a model structure adapted to the target device's computing resources. This approach avoids repeated deployment of candidate models and uploading hardware information while obtaining real hardware feedback, effectively reducing search time.
[0004] However, the methods described above only consider optimization during the model inference stage, without addressing optimization for subsequent data environment adaptation (model training) on the target device. To adapt to the target device's data environment, these methods require uploading the target device's data to a server for pre-training, neglecting local data privacy concerns and additional transmission costs. Summary of the Invention
[0005] To address the problems of existing NAS methods, this invention provides a method that simultaneously achieves low search time, high device adaptability, and high data adaptability—namely, a lightweight local neural network architecture search method for self-supervised training—and achieves zero interaction between edge devices and cloud servers.
[0006] The technical solution adopted by this invention to solve the technical problem is as follows:
[0007] The present invention provides a lightweight local neural network architecture search method for self-supervised training, which specifically includes the following steps:
[0008] (1) Construct the sequence search space;
[0009] Obtain a pre-trained model, which includes an input layer, ordinary convolutional blocks, sequence MBV2 blocks, and a classifier; construct a sequence search space, where each MBV2 block sequence can choose different widths and depths;
[0010] (2) Construct a Memory-FLOPs mapping table;
[0011] Given a set of input resolutions and a set of channel expansion factors, the entire sequence search space is represented as a series of search subspaces. Then, by calculating the average memory footprint of candidate models and the average FLOPs of each search subspace, key-value pairs are established and stored in a Memory-FLOPs mapping table. A memory analyzer g is then constructed. m Calculate the memory footprint of the candidate model during forward inference; select the optimal search subspace S with the largest average FLOPs within the available memory limit. * ;
[0012] (3) Construct a training quality predictor;
[0013] Candidate models are constructed by changing the number of MBV2 blocks and the channel width of the pre-trained model; candidate models are sampled in the sequence search space, and the sampled candidate models are tested on the candidate dataset. The candidate model structure and the corresponding test error rate are recorded; the recorded candidate model structure is used as input data and the test error rate is used as labels to construct a dataset and train it. The training quality predictor obtained after training can effectively predict the error rate of the candidate model.
[0014] (4) Multi-objective Bayesian optimization;
[0015] Determining the optimal search subspace S * Then, the pre-trained model and training quality predictor are distributed to the local device, and multi-objective Bayesian optimization is used to search for the optimal subspace S. * The optimal model structure, i.e., the customized model structure, is obtained through the search. * ;
[0016] (5) Weight loading;
[0017] After obtaining the optimal model structure for the target device through search, the optimal model is constructed and the weights from the pre-trained model are loaded into the optimal model.
[0018] Furthermore, in step (1), each MBV2 block sequence can select different widths and depths, where the width is the channel expansion factor of the MBV2 block and the depth is the number of MBV2 blocks.
[0019] Furthermore, in step (2), the function for calculating memory usage is:
[0020] Mem = g m (G,r,b min )
[0021] Where G represents the candidate model structure, r represents the input resolution, and b min This represents the minimum batch size.
[0022] Furthermore, in step (2), given a set of input resolutions R and a set of channel expansion factors Q, the entire sequence search space is represented as a series of search subspaces as follows:
[0023] S = {S1,S2,...,S} n}, n=|R|×|Q|.
[0024] Furthermore, the training quality predictor is a multilayer perceptron.
[0025] Furthermore, in step (4), given the maximum available memory M of the target device... max Candidate model structure G, minimum batch size b min Multi-objective Bayesian optimization seeks a custom model structure θ * To minimize the error rate h of the candidate model on the hardware error Training memory usage (h) m and inference delay h latency The specific optimization process is as follows:
[0026]
[0027] Among them, f opt Let λ1, λ2, and λ3 represent the objective function, where λ1, λ2, and λ3 are configurable hyperparameters used to adjust the importance of different metrics. Error rate h error The predictions are obtained by training a quality predictor based on the candidate model structure G; the inference latency h of the candidate model on the device is also included. latency The memory usage h generated by the candidate model during training is obtained by performing multiple inference processes and taking the average. m By memory analyzer g m The calculation yielded:
[0028]
[0029] Among them, M max This indicates the maximum available memory of the target device.
[0030] In the initial optimization phase, random sampling is used to select from the optimal search subspace S. * Multiple different candidate model structures are obtained, and a fixed search period is set; the goal of multi-objective Bayesian optimization is to minimize the objective function f. opt After performing the multi-objective Bayesian optimization process, select the objective function f. opt The smallest candidate model structure is taken as the optimal model structure on the target device, i.e., the customized model structure θ. * .
[0031] Furthermore, in step (4), to reduce memory usage during the training phase and accelerate the training process, the optimal batch size is calculated:
[0032]
[0033] Among them, M p This indicates the memory usage of the candidate model during training.
[0034] Furthermore, in step (4), a custom model structure θ is searched using multi-objective Bayesian optimization. * At that time, a Gaussian process was used as a surrogate model for modeling, while Monte Carlo sampling was used as the sampling function to sample the next most likely point.
[0035] Furthermore, in step (5), each MBV2 block consists of two PW convolutional layers and one DW convolutional layer. By comprehensively considering all the convolutional layers in each MBV2 block, the L1 norm of each input channel in the DW convolution is calculated and the channels are sorted according to the L1 norm. Based on the width w of the MBV2 block in the optimal model, the weights of the first w channels are copied to the MBV2 block of the optimal model according to the L1 norm sorting order, thereby completing the weight loading from the pre-trained model to the optimal model.
[0036] The beneficial effects of this invention are:
[0037] This invention provides a lightweight local neural network architecture search method for self-supervised training. It is a lightweight post-deployment NAS method that can quickly search for personalized neural network model structures on resource-constrained edge devices. Compared to front-deployment NAS technology, this invention, based on a post-deployment NAS approach, considers factors such as memory overhead and training performance during local training, enabling efficient optimization of the model training process on the edge device. This invention can generate a customized model in just 300 to 3000 seconds on an edge device, 18.8 times faster than front-deployment NAS technology. Compared to state-of-the-art post-deployment NAS technology, this invention offers comparable search time, and it eliminates the need to upload local data to the cloud during subsequent data environment adaptation, protecting data privacy and reducing data transmission costs.
[0038] Furthermore, this invention constructs a training quality predictor and distributes it and the pre-trained model locally to each device to obtain the device's actual hardware information, enabling rapid searching for personalized neural network model structures adapted to the device's capabilities. Simultaneously, unlike existing NAS methods that optimize the inference process, this invention optimizes the subsequent data adaptation (local training) process, improving the performance of the personalized model. Moreover, this invention reduces model search time while maintaining accurate device adaptability, thereby finding a matching personalized neural network model structure for the target device. Attached Figure Description
[0039] Figure 1 This invention provides an overall design architecture diagram for a lightweight local neural network architecture search method for self-supervised training.
[0040] Figure 2 This is a schematic diagram of the weight loading process. Detailed Implementation
[0041] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0042] This invention provides a lightweight local neural network architecture search method for self-supervised training. The specific algorithm is mainly implemented in two stages: a cloud server preparation stage and a device-side customization stage.
[0043] like Figure 1As shown, this invention provides a lightweight local neural network architecture search method for self-supervised training, the specific algorithm flow of which includes the following steps:
[0044] Step S1: Cloud server preparation phase;
[0045] Step S1.1: Sequence search space;
[0046] In a cloud server, a neural network is pre-trained by inputting a pre-training dataset to obtain a pre-trained model.
[0047] The pre-trained model mainly consists of an input layer, ordinary convolutional blocks, sequence MBV2 blocks, and a classifier. In order to balance the complexity of the search algorithm and the training quality of the generated candidate models, this invention can construct candidate models by changing the number of MBV2 blocks and the channel width of the pre-trained model.
[0048] This invention constructs a sequence search space, where each MBV2 block sequence (stage) can select different widths (channel expansion factors of the MBV2 block) and depths (number of MBV2 blocks).
[0049] Step S1.2: Memory-FLOPs mapping table;
[0050] Given a set of input resolutions R and a set of channel expansion factors Q, the entire sequence search space can be represented as a series of search subspaces, as shown below:
[0051] S = {S1,S2,...,S} n}, n=|R|×|Q|
[0052] By calculating the average memory usage of candidate models and the average model computational power (FLOPs) of each search subspace, key-value pairs are established and stored in the Memory-FLOPs mapping table.
[0053] By building a memory analyzer g m The memory usage of the candidate model during forward inference is calculated using the function shown below:
[0054] Mem = g m (G,r,b min )
[0055] Where G represents the candidate model structure, r represents the input resolution, and b min This represents the minimum batch size. For similar candidate model structures, model accuracy is usually positively correlated with FLOPs. Therefore, for an edge device, it is first necessary to determine its available memory size, and then select the optimal search subspace S with the largest average FLOPs within the available memory size constraint.* In this way, the candidate models are restricted to the optimal search subspace S. * This significantly reduces search complexity.
[0056] For example, the sequence search space based on MobileNetV2 contains 4,723,920 model structures. Let R = {96,128,160,192,224} and Q = {0.35,0.5,0.75,1.0,1.3,1.4}. For a given input resolution r ∈ R and a given channel expansion factor q ∈ Q, only 157,464 model structures need to be searched in the search subspace.
[0057] Step S1.3: Train the quality predictor;
[0058] When searching for customized models on the device, the training quality of candidate models is a crucial indicator of their subsequent local training performance. Therefore, this invention samples a sufficient number of candidate models from the sequence search space, then tests these sampled models on a candidate dataset, recording the candidate model structure and corresponding test error rate. A training quality predictor is then constructed. This predictor is a multilayer perceptron, using the recorded candidate model structure as input data and the test error rate as labels to build the dataset. The resulting training quality predictor effectively predicts the error rate of candidate models (mean squared error below 0.7%).
[0059] Step S2: Device-side customization stage;
[0060] Determine the optimal search subspace S based on the hardware information of the edge device. * Then, the pre-trained model and training quality predictor are sent to the device, and a search is performed locally on the device to complete the model customization.
[0061] Step S2.1: Multi-objective Bayesian optimization;
[0062] Determine the optimal search subspace S on the target device. * Then, multi-objective Bayesian optimization is used to search the optimal subspace S. * The search yields a customized model. Given the maximum available memory M of the target device. max Candidate model structure G, minimum batch size b min Multi-objective Bayesian optimization seeks a custom model structure θ * To minimize the error rate h of the candidate model on the hardware error Training memory usage (h) m and inference delay h latency The specific optimization process is as follows:
[0063]
[0064] Among them, f opt Let λ1, λ2, and λ3 represent the objective function, where λ1, λ2, and λ3 are configurable hyperparameters used to adjust the importance of different metrics. Error rate h error The predictions are obtained by training a quality predictor based on the candidate model structure G; the inference latency h of the candidate model on the device is also included. latency The memory usage h generated by the candidate model during training is obtained by performing multiple inference processes and taking the average. m Can be analyzed by memory analyzer g m The calculations are as follows:
[0065]
[0066] Among them, M max This indicates the maximum available memory on the target device.
[0067] On certain devices, memory resources may be scarce. Reducing the batch size of the input data for candidate models can effectively reduce the memory usage during the training process. To reduce memory usage and speed up the training process, the optimal batch size can be calculated using the following formula:
[0068]
[0069] Among them, M p This represents the memory usage of the candidate model during training. A custom model structure θ is searched using multi-objective Bayesian optimization. * At that time, a Gaussian process was used as a surrogate model for modeling, and Monte Carlo sampling was used as the sampling function to sample the next most likely point. In the initial stage, random sampling was used to sample from the optimal search subspace S. * Twenty different candidate model structures were obtained, and a fixed search period was set. The goal of multi-objective Bayesian optimization is to minimize the objective function f. opt After performing the multi-objective Bayesian optimization process, select the objective function f. opt The smallest candidate model structure is taken as the optimal model structure on the target device, i.e., the customized model for the target device.
[0070] Step S2.2: Weight loading;
[0071] After obtaining the optimal model structure for the target device through search, it is necessary to construct the optimal model and load the weights from the pre-trained model into the optimal model. The optimal model can be obtained by changing the width and depth based on the pre-trained model.
[0072] like Figure 2 As shown, each MBV2 block consists of two point-wise (PW) convolutional layers and one depth-wise (DW) convolutional layer. By comprehensively considering all convolutional layers in each MBV2 block, the L1 norm of each input channel in the DW convolution is calculated, and the channels are sorted according to the L1 norm. It is believed that the channel with the larger the L1 norm is, the more important it is to the optimal model because it will output larger intermediate activation values, affecting the prediction results of the optimal model. This invention copies the weights of the first w channels into the MBV2 block of the optimal model according to the L1 norm sorting order, based on the width w of the MBV2 block in the optimal model, thereby completing the weight loading from the pre-trained model to the optimal model.
[0073] Assuming the MBV2 block sequence has N MBV2 blocks and the optimal model has N' MBV2 blocks, the weight loading method used in this invention retains the first N' MBV2 blocks in the MBV2 block sequence, omitting the remaining N-N' MBV2 blocks. This approach relies on a fundamental principle: in a DNN model, layers closer to the input are more important.
[0074] The present invention provides a lightweight local neural network structure search method for self-supervised training. This method is an algorithm that uses multi-objective Bayesian optimization to quickly search for neural network model structures. By collecting real-time feedback data from edge devices, the search process is optimized and a personalized neural network model structure adapted to the computing resources of edge devices is obtained.
[0075] This invention obtains a highly customized personalized neural network model structure for the target device by completing model structure search and data environment adaptation locally on the device. With subsequent data environment adaptation as the optimization goal, it can better perform training tasks locally on the device.
[0076] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A lightweight local neural network architecture search method for self-supervised training, characterized in that, Includes the following steps: (1) Construct the sequence search space; Obtain a pre-trained model, which includes an input layer, ordinary convolutional blocks, sequence MBV2 blocks, and a classifier; construct a sequence search space, where each MBV2 block sequence can choose different widths and depths; (2) Construct a Memory-FLOPs mapping table; Given a set of input resolutions and a set of channel expansion factors, the entire sequence search space is represented as a series of search subspaces. Then, by calculating the average memory usage of the candidate models and the average FLOPs of each search subspace, key-value pairs are established and stored in the Memory-FLOPs correspondence table. By building a memory analyzer Calculate the memory usage of the candidate model during forward inference; Select the optimal search subspace with the largest average FLOPs within the available memory size limit. ; (3) Construct a training quality predictor; Candidate models are constructed by changing the number of MBV2 blocks and the channel width of the pre-trained model; candidate models are sampled in the sequence search space, and the sampled candidate models are tested on the candidate dataset, and the candidate model structure and the corresponding test error rate are recorded. The dataset is constructed by using the recorded candidate model structure as input data and the test error rate as label, and then trained. The training quality predictor obtained after training can effectively predict the error rate of the candidate model. (4) Multi-objective Bayesian optimization; Determining the optimal search subspace Then, the pre-trained model and training quality predictor are distributed to the local device, and multi-objective Bayesian optimization is used to search for the optimal subspace. The optimal model structure, i.e., the customized model structure, is obtained through search. ; In step (4), the maximum available memory of the target device is given. Candidate model structure Minimum batch size Multi-objective Bayesian optimization seeks a custom model structure To minimize the error rate of the candidate model on the hardware Training memory usage and reasoning delay The specific optimization process is as follows: ； in, Describe the objective function. , , All are configurable hyperparameters used to adjust the importance of different metrics; error rate Train the quality predictor based on the candidate model structure Predicted inference latency of candidate models on the device. The memory usage of the candidate model during training is obtained by performing multiple inference processes and taking the average. By memory analyzer The calculation yielded: ； in, Indicates the maximum available memory of the target device; In the initial optimization phase, random sampling is used to select from the optimal search subspace. Multiple different candidate model structures are obtained, and a fixed search period is set; the goal of multi-objective Bayesian optimization is to minimize the objective function. After performing the multi-objective Bayesian optimization process, select the objective function that makes the objective function... The smallest candidate model structure is taken as the optimal model structure on the target device, i.e., the customized model structure. ; (5) Weight loading; After obtaining the optimal model structure for the target device through search, the optimal model is constructed and the weights from the pre-trained model are loaded into the optimal model.
2. The lightweight local neural network architecture search method for self-supervised training according to claim 1, characterized in that, In step (1), each MBV2 block sequence can select different widths and depths, where the width is the channel expansion factor of the MBV2 block and the depth is the number of MBV2 blocks.
3. The lightweight local neural network architecture search method for self-supervised training according to claim 1, characterized in that, In step (2), the function for calculating memory usage is: ； in, Indicates the candidate model structure, Indicates the input resolution. This represents the minimum batch size.
4. The lightweight local neural network architecture search method for self-supervised training according to claim 1, characterized in that, In step (2), a set of input resolutions is given. and a set of channel expansion factors The entire sequence search space is represented as a series of search subspaces as follows: 。 5. The lightweight local neural network architecture search method for self-supervised training according to claim 1, characterized in that, The training quality predictor is a multilayer perceptron.
6. The lightweight local neural network architecture search method for self-supervised training according to claim 1, characterized in that, In step (4), to reduce memory usage during the training phase and accelerate the training process, the optimal batch size is calculated: ； in, This indicates the memory usage of the candidate model during training.
7. The lightweight local neural network architecture search method for self-supervised training according to claim 1, characterized in that, In step (4), a custom model structure is searched using multi-objective Bayesian optimization. At that time, a Gaussian process was used as a surrogate model for modeling, while Monte Carlo sampling was used as the sampling function to sample the next most likely point.
8. The lightweight local neural network architecture search method for self-supervised training according to claim 1, characterized in that, In step (5), each MBV2 block consists of two PW convolutional layers and one DW convolutional layer. By comprehensively considering all convolutional layers in each MBV2 block, the L1 norm of each input channel in the DW convolution is calculated, and the channels are sorted according to the L1 norm. The width of the MBV2 block in the optimal model is used as the basis for the calculation. Sort the first few according to their L1 norm. The weights of each channel are copied into the MBV2 block of the optimal model, thus completing the weight loading from the pre-trained model to the optimal model.