Lightweight neural network design and updating method and system based on dynamic combination of network units
Through the method based on the dynamic combination of network elements, lightweight neural networks are designed and updated, which solves the problems of large parameters, large calculations and difficult to update dynamically when deploying deep neural networks on resource-constrained devices, and realizes efficient, fast response and adaptive neural network deployment in dynamic environments.
Patent Information
- Application Number
- CN202510046395.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-13
- Publication Date
- 2025-05-09
AI Technical Summary
When deploying existing deep neural networks on devices with limited resources, they face the problems of large amount of parameters, large amount of calculations, and difficulty in dynamic updates and adjustments. Especially in a dynamic environment, they cannot quickly respond to and adapt to changes in data flow and resource conditions.
We adopt a lightweight neural network design and update method based on dynamic combination of network units, create multiple network units through modular design, combine them to form a complete neural network, and dynamically add, delete or reconfigure network units, adjust network depth and width, use reinforcement learning to explore the optimal structure, and update network configuration in real time.
It realizes the possibility of deploying deep neural networks in resource-constrained environments, improves the performance and adaptability of the network, and can quickly respond and update, and adapt to changes in data flow and resource conditions in dynamic environments.
Smart Images
Figure CN119961675A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision, machine learning and artificial intelligence technology, and in particular to a lightweight neural network design and update method and system based on dynamic combination of network units. Background Art
[0002] In recent years, deep neural networks have been widely used in many technical fields due to their excellent performance, such as pattern recognition, natural language processing, recommendation systems, and unmanned driving. There are also many applications in industries such as medicine, finance, and agriculture. However, a neural network with good performance usually contains a large number of parameters, which places high demands on storage space and computing power. Therefore, it is currently very challenging to deploy deep neural networks on resource-constrained devices in practical applications. Considering the limited storage and computing power of some mobile and terminal devices, the concept of network compression and acceleration has been proposed.
[0003] As an effective way to compress networks, the design of lightweight networks has received a lot of attention. This type of method aims to build neural networks with fewer parameters and lower resource consumption to improve efficiency. However, most lightweight network structures are currently fixed and can usually only run in a static environment. In actual applications, network deployment platforms are often in a dynamic environment, where the input data flow or resource conditions may change at any time. In order to maintain good performance, the deployed neural network must have the ability to update or replace the structure regularly. On the one hand, the network needs to respond quickly to real-time changing input data and update parameters quickly and efficiently. On the other hand, in addition to changes in data flow, the available resources (storage space, computing power) of the deployment platform may also change, which requires the network to dynamically adjust its own structure to adapt to the changing resource conditions. Therefore, how to design the dynamic update and adjustment of the network to adapt to the ever-changing real environment is a technical problem that needs to be solved urgently. Summary of the invention
[0004] The technical task of the present invention is to provide a lightweight neural network design and update method and system based on dynamic combination of network units to solve the problem of how to design the dynamic update and adjustment of the network to adapt to the ever-changing real environment.
[0005] The technical task of the present invention is achieved in the following way: a lightweight neural network design and update method based on dynamic combination of network units, the method is as follows:
[0006] Network unit design: Collect modular design methods to create multiple network units with diverse structures or functions;
[0007] Build a complete neural network based on network units: Build a neural network by combining different network units;
[0008] Dynamic adjustment of network structure: Dynamically add, delete or reconfigure network units, and adjust the depth and width of the network. At the same time, use reinforcement learning to explore the optimal network structure and update the network configuration in real time to improve the performance and adaptability of the network.
[0009] Frozen parameter update strategy: Intelligently select frozen parameters and fine-tune parameters after unfreezing based on network performance and task requirements.
[0010] Preferably, the network unit includes multiple network layers and neurons; wherein the network layer includes an input layer, a hidden layer and an output layer; the number of neurons in different network units is flexibly designed according to demand;
[0011] Among them, a network unit containing three fully connected layers includes two weight parameters W1 and W2; for a given input tensor X, X outputs a new tensor Y after two layers of fully connected operations, and the transformation is expressed as:
[0012] Y=W2(W1X+b1)+b2;
[0013] Where b1 and b2 represent bias;
[0014] For a network unit containing two convolutional layers, the computational relationship between the input tensor X and the output tensor Y is expressed as:
[0015] Y = Conv(Conv(X, W1), W2);
[0016] Among them, Conv represents the convolution and activation operation, and then the output of each network unit is obtained as the premise of subsequent overall network calculation.
[0017] Preferably, when building a neural network, select a network architecture and determine the number of layers, unit types, and connection methods of each layer.
[0018] Among them, building a multi-layer perceptron (MLP) is as follows:
[0019] Select multiple network units to stack and combine to form a larger network layer, called a combination layer; each network unit in the combination layer is independent of each other and has no weight connection; if a combination layer includes n network unit combinations, the total weight parameter of the combination layer is the sum of the W1 and W2 parameters in the n network units; compared with ordinary network layers, the combination layer contains fewer weight parameters, thereby achieving the purpose of reducing the number of parameters;
[0020] In the forward propagation process, a slicing operation is performed on each combination layer; if there are n network units, the input X is divided into n parts, that is, X = [X1, X2, ..., Xn]; accordingly, each input slice is calculated by the corresponding network unit to obtain n outputs, that is, Y1, Y2, ..., Yn; furthermore, the output of a complete combination layer is the connection result of the outputs of n network units, that is,
[0021] Y out =Conc(X)=[Y1,Y2,…,Yn];
[0022] Among them, Conc represents the connection operation;
[0023] A complete lightweight neural network is built using multiple combined layers and ordinary layers; among them, ordinary network layers are used to connect multiple combined layers to solve the dimension matching problem.
[0024] As a preferred method, the neural network is constructed as follows:
[0025] Define network units: determine the number of neurons and layers of each network unit;
[0026] Design the network architecture: determine the overall architecture of the network, including the type and number of network units and the connections between them; and determine the configuration of the network's input and output layers and any hidden or intermediate layers;
[0027] Build the network framework: Add each designed network unit as a building block to the network framework. According to the designed network architecture, combine the network units to form a composite layer. At the same time, introduce the common network layer and use the common layer to connect two adjacent composite layers to ensure that data flows correctly between the network units.
[0028] Parameter initialization: All parameters in the network are initialized randomly, Xavier or He;
[0029] Forward propagation: defines the forward propagation process of the network, ensuring that the input data can be processed by the network units and produce output;
[0030] Loss function and optimizer: select or define a loss function to evaluate the difference between the network output and the target output, and select an optimizer (such as SGD, Adam, etc.) to update the network parameters based on the results of the loss function;
[0031] Train the network: Use the training data set to train the network and adjust the learning rate, batch size, and training cycle hyperparameters to optimize network performance;
[0032] Validation and testing: Use the validation set to evaluate the performance of the network during training, make necessary adjustments, and use the test set to evaluate the generalization ability of the network.
[0033] As a preference, when the network's environmental resource conditions change, the network structure will be adjusted to accommodate the latest memory or computing requirements, as follows:
[0034] Introducing a switching mechanism into the network, allowing the network to swap the positions of different network units according to performance requirements during operation, and using an optimization algorithm to find the best order of network units within a given time;
[0035] A gating parameter is set for each network unit to measure the importance of the corresponding network unit, and whether to delete the corresponding network unit is decided based on the characteristics of the input data.
[0036] As a preferred embodiment, the network structure is dynamically adjusted as follows:
[0037] Dynamic routing: By setting multiple exit points in the network structure, samples can be routed to network paths of different depths according to sample complexity, and each exit point corresponds to a different network depth;
[0038] Unit exchange: Introducing exchangeable units into the network, which dynamically exchange positions according to resource and performance requirements; specifically: using a dynamic routing mechanism to select units to process input data, the output of the dynamic routing mechanism is a probability distribution, each probability value corresponds to a network unit, and multiple units are selected to process input data based on the output probability distribution;
[0039] Unit addition and removal: remove units from the network based on the importance of the unit measured by the gating mechanism; specifically, set a trainable parameter for each network unit, called a gate, and selectively use or delete any network unit during training based on the importance of the unit measured by the value of the gating parameter. For example, delete the unit when the gating value is less than 0.
[0040] Adaptive learning rate adjustment: During the training process, use adaptive learning rate adjustment methods, such as Adam or RMSprop, to automatically adjust the learning rate according to the training status of the model, thereby better optimizing the network structure;
[0041] Training and fine-tuning: When training the network, use reinforcement learning to train the network so that reinforcement learning can learn the time points and methods of adjusting the structure during the training process;
[0042] Performance monitoring: Implement real-time performance monitoring to evaluate the performance of the network under different structures and dynamically adjust strategies based on performance feedback to ensure that the network maintains accuracy while minimizing the consumption of computing resources.
[0043] Preferably, the freezing parameter update strategy is as follows:
[0044] Assume that the network N has been trained on the original dataset D;
[0045] When new data arrives, the network N needs to n Study;
[0046] Given a pre-trained network N with a parameter set Θ, the goal is to select a subset of parameters to update where the subset Θ s It is expected that for the new dataset D n The most important; and by measuring the weight shift to measure the importance of parameters, which refers to the phenomenon that the weight space of the original pre-trained network changes when the network is fine-tuned; by using the dataset D n The weight offset is approximated by the gradient on , which is calculated as:
[0047]
[0048] Among them, g i (x v ) is the objective function relative to the data point x v The gradient of
[0049] Based on the calculated Value, select important parameters to update according to the preset selection ratio;
[0050] After determining the selected important parameters, the remaining parameters are frozen or kept unchanged during the update process, which means that they do not participate in the update step, which is beneficial to preserve the learned representation of the network and save computing resources for fast updates; in addition, the frozen parameters are regularly fine-tuned after a certain number of iterations;
[0051] The performance of the network is monitored and whether further updates to the selected parameters are needed. The iterative process enables the network to adapt to new data and update quickly while maintaining the stability and generalization ability provided by the frozen parameters.
[0052] A lightweight neural network design and update system based on dynamic combination of network units, the system is used to implement the above-mentioned lightweight neural network design and update method based on dynamic combination of network units; the system includes:
[0053] The network unit design module is used to collect modular design methods to create multiple network units with diverse structures or functions;
[0054] A neural network building module is used to build a neural network by combining different network units on the basis of the network unit;
[0055] Dynamic adjustment module, which is used to dynamically add, delete or reconfigure network units and adjust the depth and width of the network. It also uses reinforcement learning to explore the optimal network structure and update the network configuration in real time to improve the performance and adaptability of the network.
[0056] The frozen parameter update module is used to intelligently select frozen parameters and fine-tune parameters after unfreezing according to network performance and task requirements.
[0057] An electronic device comprising: a memory and at least one processor;
[0058] Wherein, the memory stores a computer program;
[0059] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the lightweight neural network design and update method based on dynamic combination of network units as described above.
[0060] A computer-readable storage medium having a computer program stored therein, wherein the computer program can be executed by a processor to implement the lightweight neural network design and update method based on dynamic combination of network units as described above.
[0061] The lightweight neural network design and update method and system based on dynamic combination of network units of the present invention have the following advantages:
[0062] (i) The present invention solves two challenges in the actual deployment of deep neural networks: one is to reduce the parameters and computational complexity of the network so that the network can be deployed in an environment with resource constraints such as memory and computing; the other is to efficiently learn and update the network to cope with real-time data and changing resource constraints;
[0063] (ii) The present invention first designs and defines a series of network units, which constitute the basic components of lightweight networks; secondly, a complete neural network is built using the designed network units. This step includes determining the overall architecture of the network, initializing parameters, and designing the training process; thirdly, in order to enable the network to adapt to samples and resource conditions of different complexities, a dynamic structural adjustment strategy is introduced, including dynamically adding, deleting, or reconfiguring network units during training, and updating the network configuration in real time; finally, in order to speed up the network's response and update speed to real-time data, a frozen parameter update strategy is implemented during training and fine-tuning, setting the parameters of certain layers to non-trainable, and keeping these parameters unchanged in subsequent training, thereby improving memory efficiency and training efficiency;
[0064] (III) The present invention can build a lightweight neural network system that is highly flexible and adaptable to a variety of tasks and resource constraints. It can not only optimize the use of computing resources while maintaining high performance, but also has good generalization capabilities. The dynamic adjustment and freeze update strategies are particularly suitable for scenarios of transfer learning and fine-tuning pre-trained models, enabling the network to quickly adapt to new tasks while retaining the knowledge of the pre-trained model.
[0065] (iv) The lightweight neural network that can be dynamically updated and adjusted by the present invention provides a new idea for the deployment of neural networks in dynamic environments, and can effectively reduce the number of network parameters to achieve the purpose of compression, while realizing dynamic network updates and structural adjustments;
[0066] (V) The present invention aims to build a lightweight neural network system that is highly flexible and adaptable to a variety of tasks, can optimize the use of computing resources while maintaining high performance, and has good generalization capabilities, while enhancing the deployment and application of deep neural networks in real-world environments;
[0067] (vi) By combining different network units, the present invention significantly reduces the number of parameters and the amount of computation of the neural network, thereby accelerating the training and reasoning speeds;
[0068] (VII) The dynamic adjustment between different combinations of the present invention enables the network to adapt to different resource constraints while maintaining good performance;
[0069] (VIII) The strategy of freezing some parameters of the present invention also speeds up the updating speed of the network, making it easier for the network to quickly process real-time input. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] The present invention is further described below in conjunction with the accompanying drawings.
[0071] Attached Figure 1 Schematic diagram of the lightweight neural network design and update method based on dynamic combination of network units, taking a fully connected network containing two types of network units (green and orange) as an example: (a) the original dense network layer; (b) the combination layer of network units; (c) the exchange of different network units. DETAILED DESCRIPTION
[0072] The lightweight neural network design and update method and system based on dynamic combination of network units of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.
[0073] Embodiment 1:
[0074] As attached Figure 1 As shown, this embodiment provides a lightweight neural network design and update method based on dynamic combination of network units, and the method is specifically as follows:
[0075] S1. Network unit design: Collect modular design methods to create multiple network units with diverse structures or functions;
[0076] S2. Build a complete neural network based on the network units: Build a neural network by combining different network units;
[0077] S3. Dynamic adjustment of network structure: Dynamically add, delete or reconfigure network units, and adjust the depth and width of the network. At the same time, use reinforcement learning to explore the optimal network structure and update the network configuration in real time to improve the performance and adaptability of the network.
[0078] S4, frozen parameter update strategy: intelligently select frozen parameters and fine-tune parameters after unfreezing based on network performance and task requirements.
[0079] The network unit in step S1 of this embodiment includes multiple network layers and neurons; the network layer includes an input layer, a hidden layer and an output layer; the number of neurons in different network units is flexibly designed according to needs;
[0080] Among them, a network unit containing three fully connected layers includes two weight parameters W1 and W2; for a given input tensor X, X outputs a new tensor Y after two layers of fully connected operations, and the transformation is expressed as:
[0081] Y=W2(W1X+b1)+b2;
[0082] Where b1 and b2 represent bias;
[0083] For a network unit containing two convolutional layers, the computational relationship between the input tensor X and the output tensor Y is expressed as:
[0084] Y = Conv(Conv(X, W1), W2);
[0085] Among them, Conv represents the convolution and activation operation, and then the output of each network unit is obtained as the premise of subsequent overall network calculation.
[0086] In step S2 of this embodiment, when building a neural network, a network architecture is selected, and the number of layers of the network, the unit type of each layer, and the connection method are determined;
[0087] Among them, building a multi-layer perceptron (MLP) is as follows:
[0088] Select multiple network units to stack and combine to form a larger network layer, called a combination layer; each network unit in the combination layer is independent of each other and has no weight connection; if a combination layer includes n network unit combinations, the total weight parameter of the combination layer is the sum of the W1 and W2 parameters in the n network units; compared with ordinary network layers, the combination layer contains fewer weight parameters, thereby achieving the purpose of reducing the number of parameters;
[0089] In the forward propagation process, a slicing operation is performed on each combination layer; if there are n network units, the input X is divided into n parts, that is, X = [X1, X2, ..., Xn]; accordingly, each input slice is calculated by the corresponding network unit to obtain n outputs, that is, Y1, Y2, ..., Yn; furthermore, the output of a complete combination layer is the connection result of the outputs of n network units, that is,
[0090] Y out =Conc(X)=[Y1,Y2,…,Yn];
[0091] Among them, Conc represents the connection operation;
[0092] A complete lightweight neural network is built using multiple combined layers and ordinary layers; among them, ordinary network layers are used to connect multiple combined layers to solve the dimension matching problem.
[0093] The specific details of building a neural network in step S2 of this embodiment are as follows:
[0094] S201, define network units: determine the number of neurons and layers of each network unit;
[0095] S202. Design network architecture: Determine the overall architecture of the network, which includes the type and number of network units and the connection between each unit; and determine the configuration of the network's input and output layers and any hidden layers or intermediate layers;
[0096] S203, building a network framework: adding each designed network unit as a building block to the network framework, combining the network units to form a combination layer according to the designed network architecture; at the same time, introducing a common network layer, using a common layer connection between two adjacent combination layers to ensure that data flows correctly between the network units;
[0097] S204, parameter initialization: performing random initialization, Xavier initialization or He initialization on all parameters in the network;
[0098] S205, forward propagation: define the forward propagation process of the network to ensure that the input data can be processed by the network units and generate output;
[0099] S206, loss function and optimizer: select or define a loss function to evaluate the difference between the network output and the target output, and select an optimizer (such as SGD, Adam, etc.) to update the network parameters according to the result of the loss function;
[0100] S207, training network: using the training data set to train the network, and adjusting the learning rate, batch size and training cycle hyperparameters to optimize network performance;
[0101] S208, Validation and Testing: Use the validation set to evaluate the performance of the network during training, make necessary adjustments, and use the test set to evaluate the generalization ability of the network.
[0102] In this embodiment, when the environmental resource conditions of the network change, the network structure will be adjusted to adapt to the latest memory or computing requirements, as follows:
[0103] Introducing a switching mechanism into the network, allowing the network to swap the positions of different network units according to performance requirements during operation, and using an optimization algorithm to find the best order of network units within a given time;
[0104] A gating parameter is set for each network unit to measure the importance of the corresponding network unit, and whether to delete the corresponding network unit is decided based on the characteristics of the input data.
[0105] The dynamic adjustment of the network structure in step S3 of this embodiment is specifically as follows:
[0106] S301, dynamic routing: by setting multiple exit points in the network structure, the samples are routed to network paths of different depths according to the sample complexity, and each exit point corresponds to a different network depth;
[0107] S302, unit exchange: introduce exchangeable units into the network, and the exchangeable units dynamically exchange positions according to resource and performance requirements; specifically: use a dynamic routing mechanism to select a unit to process input data, the output of the dynamic routing mechanism is a probability distribution, each probability value corresponds to a network unit, and multiple units are selected to process the input data according to the output probability distribution;
[0108] For example, the Top-K selection method can be used to select the K units with the highest probability. This selection process redistributes computing tasks within the model and can be regarded as a dynamic exchange of unit positions. During the training process, the dynamic routing mechanism will learn how to dynamically adjust routing decisions based on the characteristics of the input data so that the exchange of units is more adaptable to performance requirements. When making adjustments, a genetic algorithm is additionally introduced to help determine the best order of units to adapt to the current input samples and resource conditions. The genetic algorithm searches through a set of candidate solutions through operations such as selection, crossover, and mutation to find the optimal or near-optimal routing strategy.
[0109] S303, unit addition and removal: remove units in the network according to the importance of the units measured by the gating mechanism; specifically, set a trainable parameter for each network unit, called gating, and measure the importance of the unit according to the value of the gating parameter during the training process, and selectively use or delete any network unit, for example, delete the unit when the gating value is less than 0;
[0110] S304, adaptive learning rate adjustment: During the training process, an adaptive learning rate adjustment method, such as Adam or RMSprop, is used to automatically adjust the learning rate according to the training status of the model, thereby better optimizing the network structure;
[0111] S305, training and fine-tuning: When training the network, use reinforcement learning to train the network so that the reinforcement learning can learn the time point for adjusting the structure and the method for adjusting the network structure during the training process;
[0112] S306, Performance Monitoring: Implement real-time performance monitoring, evaluate the performance of the network under different structures, and dynamically adjust the strategy based on performance feedback to ensure that the network maintains accuracy while minimizing the consumption of computing resources.
[0113] The frozen parameter update strategy in step S4 of this embodiment is as follows:
[0114] S401, assuming that the network N has been trained on the original data set D;
[0115] S402, when new data arrives, the network N needs to n Study;
[0116] S403. Given a pre-trained network N with a parameter set Θ, the goal is to select a subset of parameters to be updated. where the subset Θ s It is expected that for the new dataset D n The most important; and by measuring the weight shift to measure the importance of parameters, which refers to the phenomenon that the weight space of the original pre-trained network changes when the network is fine-tuned; by using the dataset Dn The weight offset is approximated by the gradient on , which is calculated as:
[0117]
[0118] Among them, g i (x v ) is the objective function relative to the data point x v The gradient of
[0119] S404, based on the calculated Value, select important parameters to update according to the preset selection ratio;
[0120] S405. After determining the selected important parameters, the remaining parameters will be frozen or kept unchanged during the update process, which means that they do not participate in the update step, which is beneficial to retain the learned representation of the network and save computing resources for fast update; in addition, the frozen parameters will be regularly fine-tuned after a certain number of iterations;
[0121] S406. Monitor the performance of the network and evaluate whether further updates to the selected parameters are needed. The iterative process enables the network to adapt to new data and update quickly while maintaining the stability and generalization ability provided by the frozen parameters.
[0122] Embodiment 2:
[0123] This embodiment provides a lightweight neural network design and update system based on dynamic combination of network units, which is used to implement the lightweight neural network design and update method based on dynamic combination of network units in Example 1; the system includes:
[0124] The network unit design module is used to collect modular design methods to create multiple network units with diverse structures or functions;
[0125] A neural network building module is used to build a neural network by combining different network units on the basis of the network unit;
[0126] Dynamic adjustment module, which is used to dynamically add, delete or reconfigure network units and adjust the depth and width of the network. It also uses reinforcement learning to explore the optimal network structure and update the network configuration in real time to improve the performance and adaptability of the network.
[0127] The frozen parameter update module is used to intelligently select frozen parameters and fine-tune parameters after unfreezing according to network performance and task requirements.
[0128] Embodiment 3:
[0129] This embodiment also provides an electronic device, including: a memory and a processor;
[0130] Wherein, the memory stores computer-executable instructions;
[0131] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the lightweight neural network design and update method based on dynamic combination of network units in any embodiment of the present invention.
[0132] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.
[0133] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.
[0134] Embodiment 4:
[0135] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor to enable the processor to execute the lightweight neural network design and update method based on dynamic combination of network units in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.
[0136] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
[0137] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.
[0138] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.
[0139] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.
[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A lightweight neural network design and update method based on dynamic combination of network units, characterized in that: The method is as follows: Network unit design: Collect modular design methods to create multiple network units with diverse structures or functions; Build a complete neural network based on network units: Build a neural network by combining different network units; Dynamic adjustment of network structure: Dynamically add, delete or reconfigure network units, and adjust the depth and width of the network. At the same time, use reinforcement learning to explore the optimal network structure and update the network configuration in real time to improve the performance and adaptability of the network. Frozen parameter update strategy: Intelligently select frozen parameters and fine-tune parameters after unfreezing based on network performance and task requirements.
2. The lightweight neural network design and update method based on dynamic combination of network units according to claim 1 is characterized in that: The network unit includes multiple network layers and neurons; the network layer includes input layer, hidden layer and output layer; the number of neurons in different network units is flexibly designed according to needs; Among them, a network unit containing three fully connected layers includes two weight parameters W1 and W2; for a given input tensor X, X outputs a new tensor Y after two layers of fully connected operations, and the transformation is expressed as: Y=W2(W1X+b1)+b2; Where b1 and b2 represent bias; For a network unit containing two convolutional layers, the computational relationship between the input tensor X and the output tensor Y is expressed as: Y = Conv(Conv(X, W1), W2); Among them, Conv represents convolution and activation operations.
3. The lightweight neural network design and update method based on dynamic combination of network units according to claim 1 is characterized in that: When building a neural network, choose the network architecture and determine the number of layers, unit type and connection method of each layer; Among them, building a multi-layer perceptron is as follows: Select multiple network units to stack and combine to form a network layer, which is called a combination layer; each network unit in the combination layer is independent of each other and has no weight connection; among them, if a combination layer includes n network unit combinations, the total weight parameter of the combination layer is the sum of the W1 and W2 parameters in the n network units; In the forward propagation process, a slicing operation is performed on each combination layer; if there are n network units, the input X is divided into n parts, that is, X = [X1, X2, ..., Xn]; accordingly, each input slice is calculated by the corresponding network unit to obtain n outputs, that is, Y1, Y2, ..., Yn; furthermore, the output of a complete combination layer is the connection result of the outputs of n network units, that is, Y out =Conc(X)=[Y1,Y2,…,Yn]; Among them, Conc represents the connection operation; A complete lightweight neural network is constructed using multiple combined layers and common layers, wherein common network layers are used to connect multiple combined layers.
4. The lightweight neural network design and update method based on dynamic combination of network units according to claim 1 is characterized in that: The details of building a neural network are as follows: Define network units: determine the number of neurons and layers of each network unit; Design the network architecture: determine the overall architecture of the network, including the type and number of network units and the connections between them; and determine the configuration of the network's input and output layers and any hidden or intermediate layers; Build the network framework: Add each designed network unit as a building block to the network framework. According to the designed network architecture, combine the network units to form a composite layer. At the same time, introduce the common network layer and use the common layer to connect two adjacent composite layers to ensure that data flows correctly between the network units. Parameter initialization: All parameters in the network are initialized randomly, Xavier or He; Forward propagation: defines the forward propagation process of the network, ensuring that the input data can be processed by the network units and produce output; Loss function and optimizer: Choose or define a loss function that evaluates the difference between the network output and the target output, and choose an optimizer that updates the network parameters based on the results of the loss function; Train the network: Use the training data set to train the network and adjust the learning rate, batch size, and training cycle hyperparameters to optimize network performance; Validation and testing: Use the validation set to evaluate the performance of the network during training, make necessary adjustments, and use the test set to evaluate the generalization ability of the network.
5. The lightweight neural network design and update method based on dynamic combination of network units according to claim 1, characterized in that: When the network's environmental resource conditions change, the network structure will be adjusted to adapt to the latest memory or computing requirements, as follows: Introducing a switching mechanism into the network, allowing the network to swap the positions of different network units according to performance requirements during operation, and using an optimization algorithm to find the best order of network units within a given time; A gating parameter is set for each network unit to measure the importance of the corresponding network unit, and whether to delete the corresponding network unit is decided based on the characteristics of the input data.
6. The lightweight neural network design and update method based on dynamic combination of network units according to claim 1, characterized in that: The dynamic adjustment of network structure is as follows: Dynamic routing: By setting multiple exit points in the network structure, samples can be routed to network paths of different depths according to sample complexity, and each exit point corresponds to a different network depth; Unit exchange: Introducing exchangeable units into the network, which dynamically exchange positions according to resource and performance requirements; specifically: using a dynamic routing mechanism to select units to process input data, the output of the dynamic routing mechanism is a probability distribution, each probability value corresponds to a network unit, and multiple units are selected to process input data based on the output probability distribution; Unit addition and removal: remove units from the network based on the importance of the units measured by the gating mechanism; specifically, set a trainable parameter for each network unit, called a gate, and selectively use or delete any network unit during training based on the importance of the unit measured by the value of the gating parameter; Adaptive learning rate adjustment: During the training process, the adaptive learning rate adjustment method is used to automatically adjust the learning rate according to the training status of the model, so as to better optimize the network structure; Training and fine-tuning: When training the network, use reinforcement learning to train the network so that reinforcement learning can learn the time points and methods of adjusting the structure during the training process; Performance monitoring: Implement real-time performance monitoring to evaluate the performance of the network under different structures.
7. The lightweight neural network design and update method based on dynamic combination of network units according to any one of claims 1 to 6, characterized in that: The frozen parameter update strategy is as follows: Assume that the network N has been trained on the original dataset D; When new data arrives, the network N needs to n Study; Given a pre-trained network N with a parameter set Θ, the goal is to select a subset of parameters to update where the subset Θ s It is expected that for the new dataset D n The most important; and by measuring the weight shift to measure the importance of parameters, which refers to the phenomenon that the weight space of the original pre-trained network changes when the network is fine-tuned; by using the dataset D n The weight offset is approximated by the gradient on , which is calculated as: Among them, g i (x v ) is the objective function relative to the data point x v The gradient of Based on the calculated Value, select important parameters to update according to the preset selection ratio; After the selected important parameters are determined, the remaining parameters will be frozen or kept unchanged during the update process; The performance of the network is monitored and whether further updates to the selected parameters are needed. The iterative process enables the network to adapt to new data and update quickly while maintaining the stability and generalization ability provided by the frozen parameters.
8. A lightweight neural network design and update system based on dynamic combination of network units, characterized in that: The system is used to implement the lightweight neural network design and update method based on dynamic combination of network units according to any one of claims 1 to 7; the system comprises: The network unit design module is used to collect modular design methods to create multiple network units with diverse structures or functions; A neural network building module is used to build a neural network by combining different network units on the basis of the network unit; Dynamic adjustment module, which is used to dynamically add, delete or reconfigure network units and adjust the depth and width of the network. It also uses reinforcement learning to explore the optimal network structure and update the network configuration in real time to improve the performance and adaptability of the network. The frozen parameter update module is used to intelligently select frozen parameters and fine-tune parameters after unfreezing according to network performance and task requirements.
9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the lightweight neural network design and update method based on dynamic combination of network units as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement a lightweight neural network design and update method based on dynamic combination of network units as described in any one of claims 1 to 7.