Model training methods, processing chips, and edge terminals

Abstract

The embodiments of the present invention provide a model training method, a processing chip, and an edge terminal, which relate to the field of computer technology. The model training method includes: using training data to train the main path of a deep neural network model to obtain network parameters of each layer of the main path of the deep neural network model, the network parameters including weights; compressing the main path of the deep neural network model, and saving the weights of each layer of the main path of the compressed deep neural network model; using the collected sample data to continue training the current deep neural network model, updating and saving the network parameters of each layer of each side path module; wherein the weights of each layer of the main path remain unchanged. In the present invention, the storage space required to store the weights of each layer of the deep neural network model is greatly reduced, and the memory consumption during the continued training of the deep neural network model is also reduced, so that it can be suitable for edge terminals with smaller memory.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training method, a processing chip and an edge terminal. Background Art

[0002] Deep learning is a very popular research direction in the field of artificial intelligence in recent years. Compared with traditional shallow machine learning, it can mine more implicit features. At present, deep neural networks (DNN) have been widely used in many artificial intelligence fields including computer vision, speech recognition, autonomous driving, cancer detection, and robotics.

[0003] However, with the rapid development of deep neural networks, the number of layers of deep neural networks is gradually increasing. The more layers of deep neural networks, the more network parameters (such as weights, activation values, etc.) involved. Taking the residual neural network (ResNet) as an example, when the number of layers increases from 18 to 50, when the edge width of the input color image is 224 pixels, the number of network parameters increases from 11.4M to 24.7M, and the storage space required to store individual neuron data during training will also increase greatly. Based on this, the current training of deep neural networks has high requirements for terminal storage space and relies on training chips in the cloud, which cannot be applied to edge devices with small memory. In addition, the amount of training data that can be used for cloud training is small and cannot cover all data in the actual application scenarios of deep neural networks. When the deep neural network is deployed on the edge, due to the small memory of the edge, the training of the deep neural network has high requirements for terminal storage space, which also makes it impossible for the edge to use the generated data to retrain the deep neural network. Summary of the invention

[0004] The purpose of the present invention is to provide a model training method, a processing chip and an edge terminal, which greatly reduce the storage space required to store the weights of each layer of the deep neural network model, and also reduce the memory consumption during the continued training of the deep neural network model, so that it can be suitable for edge terminals with smaller memory.

[0005] To achieve the above-mentioned purpose, the present invention provides a model training method for training a deep neural network model including a main path and a side path, wherein the main path includes multiple sub-modules, each sub-module includes multiple layers, and the side path includes multiple side path modules, and the sub-modules correspond to the side path modules one by one and are connected in parallel. The method comprises: using training data to train the main path of the deep neural network model to obtain network parameters of each layer of the main path of the deep neural network model, wherein the network parameters include weights; compressing the main path of the deep neural network model, and saving the weights of each layer of the main path of the compressed deep neural network model; using the collected sample data to continue training the current deep neural network model, updating and saving the network parameters of each layer of each side path module; wherein the weights of each layer of the main path remain unchanged.

[0006] The present invention also provides a processing chip for executing the above-mentioned model training method.

[0007] The present invention also provides an edge terminal, comprising: at least one processing chip; and a memory communicatively connected to the at least one processing chip; wherein the memory stores instructions executable by the at least one processing chip, and the instructions are executed by the at least one processing chip so that the at least one processing chip can execute the model training method as described above.

[0008] An embodiment of the present invention provides a training method for a deep neural network model. The deep neural network model includes a main path and a side path. The main path includes multiple submodules, each submodule includes multiple layers, and the side path includes multiple side path modules. The submodules correspond to the side path modules one by one and are connected in parallel. When training the deep neural network model, the main path of the deep neural network model is first preliminarily trained using training data. After obtaining network parameters of each layer of the main path of the deep neural network model, the main path of the deep neural network model is compressed, and only the weights of each layer of the main path of the compressed deep neural network model can be saved, thereby greatly reducing the storage space required for storing the weights of each layer of the main path of the deep neural network model. The saved weights remain unchanged in subsequent training processes, and the memory consumption of the deep neural network model during continued training is also reduced, so that it can be suitable for edge terminals with smaller memory. After the preliminary training is completed, the current deep neural network model is continued to be trained using the collected sample data, and the network parameters of each side path module can be obtained based on the latest sample data. That is, the deep neural network model finally trained can be adaptively learned and updated based on the latest sample data, thereby improving the accuracy of the deep neural network model.

[0009] In one embodiment, each of the side path modules includes a downsampling layer, a feature convolution group, an output convolution layer and an upsampling layer connected in sequence; during the training process, the downsampling layer downsamples the input data from the main path, converts the obtained low-resolution input data into the feature convolution group for feature extraction, and then uses the output convolution layer to convert the extracted feature data into the same dimension as the main path, and uses the upsampling layer to upsample the feature data with the same dimension as the main path and return it to the main path.

[0010] In one embodiment, the number of convolution layers of the feature convolution group is 1, and the number of convolution kernels of each convolution layer is 16.

[0011] In one embodiment, the current deep neural network model is continued to be trained using the collected sample data, and the network parameters of each layer of each side module are updated and saved, including: selecting multiple layers of the current deep neural network model as checkpoints; performing forward propagation on the current deep neural network model using the collected sample data to obtain the output data of the current deep neural network model, and saving the activation value of each checkpoint; performing back propagation training on the current deep neural network model using the output data obtained by forward propagation, the activation value of each checkpoint and the annotation information of each sample data, updating and saving the network parameters of each layer of each side module, and during the back propagation training process, using the activation value of each checkpoint to calculate the activation value of each layer in the deep neural network model that is not used as the checkpoint; wherein the weights of each layer of the main path remain unchanged.

[0012] In one embodiment, selecting multiple layers of the current deep neural network model as checkpoints includes: selecting a layer in each submodule of the main path of the deep neural network model as a checkpoint.

[0013] In one embodiment, each submodule of the main path is divided based on the function or structure implemented by each layer of the main path.

[0014] In one embodiment, the weights of each layer of the deep neural network model are stored using 8-bit integers, and the learnable parameters of each layer of the deep neural network model are stored using 16-bit floating point numbers.

[0015] In one embodiment, the main path of the deep neural network model is compressed in any one or any combination of the following ways: pruning, quantization, and weight sharing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a specific flow chart of the model training method according to the first embodiment of the present invention;

[0017] Figure 2 is a schematic diagram of the connection between the submodules of the main path and the side path modules of the deep neural network model according to the first embodiment of the present invention;

[0018] Figure 3 It is a specific flow chart of the model training method in the second embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be described in detail with reference to the accompanying drawings to provide a clearer understanding of the purpose, features and advantages of the present invention. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0020] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0021] Unless the context requires otherwise, throughout the specification and claims, the word "comprise" and variations such as "include" and "have" should be construed in an open, inclusive sense, ie, should be interpreted as "including, but not limited to."

[0022] References throughout the specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0023] As used in this specification and the appended claims, the singular forms "a", "an", and "" include plural referents unless the context clearly dictates otherwise. It should be noted that the term "or" is generally employed in its sense including "or / and" unless the context clearly dictates otherwise.

[0024] In the following description, in order to clearly show the structure and working mode of the present invention, many directional words will be used for description, but the words "front", "back", "left", "right", "outside", "inside", "outward", "inward", "up", "down", etc. should be understood as convenient terms and should not be understood as restrictive terms.

[0025] The first embodiment of the present invention relates to a model training method, which is applied to a processing chip in an edge terminal, such as a mobile phone, a tablet computer, a laptop computer, a desktop host, etc. The processing chip of the edge terminal can execute the model training method in this embodiment to train a deep neural network model that has been constructed; the deep neural network model includes a main path and a side path, the main path includes multiple submodules, the side path includes multiple side path modules, the submodules correspond to the side path modules one by one and are connected in parallel, wherein the deep neural network model includes: an input layer, a convolutional layer, a fully connected layer, a pooling layer, etc., which will not be pointed out one by one in subsequent embodiments.

[0026] The specific process of the model training method in this embodiment is as follows Figure 1 shown.

[0027] Step 101, using training data to train the main path of the deep neural network model, to obtain network parameters of each layer of the main path of the deep neural network model, the network parameters including weights.

[0028] Specifically, for a deep neural network model that has been constructed to be trained, after setting the network parameters (including weights and biases) of the deep neural network model, the main path of the neural network model is trained using pre-stored training data; wherein the pre-stored training data may be historical data stored by the edge terminal in the current application scenario.

[0029] The training includes two stages, namely forward propagation and back propagation. In the forward propagation process, the activation value of each neuron in each layer of the main path of the deep neural network model is calculated according to the training data of the input layer of the main path of the deep neural network model, and the output data of the output layer of the deep neural network model is obtained; in the back propagation training process, the cost is calculated based on the difference between the output data of the output layer obtained in the forward propagation process and the annotation information in the training data, and combined with the preset loss function, and then the network parameters of each layer of the main path of the deep neural network are adjusted according to the set optimization strategy (such as gradient descent method) until the main path of the deep neural network model that meets the preset conditions is obtained through training. The preset conditions may be that the accuracy of the model reaches a preset accuracy threshold, the number of training times reaches a preset number of training times, etc.

[0030] In one example, each submodule of the main path is divided based on the function or structure that each layer of the main path is used to implement, which helps to retrain the deep neural network model faster later. When each submodule of the main path is divided based on the function that each layer of the main path is used to implement, the main path can implement multiple functions. The main path is divided into multiple submodules according to the function, and each submodule is used to implement a function, that is, the multiple layers contained in each submodule are used to implement the same function; taking the YOLO (You Only Look Once) V3 algorithm network as the main path as an example, the multiple layers corresponding to each output head can be divided into a submodule. When each submodule of the main path is divided based on the structure of the main path, a submodule is re-divided each time the feature dimension reduction is performed on the data; taking the ResNet network as the main path as an example, each Bottleneck structure with the same feature resolution is used as a submodule.

[0031] In one example, each side path module includes a downsampling layer, a feature convolution group, an output convolution layer, and an upsampling layer connected in sequence; during the training process, the downsampling layer downsamples the input data from the main path, converts the obtained low-resolution input data into the feature convolution group for feature extraction, and then uses the output convolution layer to convert the extracted feature data to the same dimension as the main path, and uses the upsampling layer to upsample the feature data of the same dimension as the main path and return it to the main path. Please refer to Figure 2 , taking any submodule 1 in the main path (wherein the specific structure of the submodule is not shown) as an example, the side path module 2 connected in parallel therewith comprises a downsampling layer 21, a feature convolution group 22, an output convolution layer 23 and an upsampling layer 24, the submodule 1 and the side path module 2 use the same input image, in the side path module 2, the downsampling layer 21 downsamples the input image according to a preset multiple to obtain an image of a preset resolution, for example, if the size of the input image is X*Y, and it is downsampled by S times, then a resolution image of (X / S)*(Y / S) size can be obtained; then the preset resolution image is The image with the same resolution is input into the feature convolution group 22 for feature extraction, and the extracted feature image is input into the output convolution layer 23, and the dimension of the feature image is restored to an image with the same dimension as the main path, that is, an image with the same dimension as the main path output. Subsequently, the feature image is restored to its original resolution through the upsampling layer 24 (that is, the sampling multiples of the upsampling layer 24 and the downsampling layer 21 are the same), that is, the same as the resolution of the main path output image. Finally, the side path module 2 obtains an output image with the same dimension and resolution as the main path, which is added to the output image of the sub-module 1 and input into the next sub-module and the side path module.

[0032] Among them, the number of convolution layers of the feature convolution group and the number of convolution kernels of each convolution layer can be configured based on the accuracy requirements. The more convolution layers and convolution kernels there are, the higher the accuracy of the trained deep neural network model. That is, the number of convolution layers of the feature convolution group can be one or more layers, and the number of convolution kernels of each convolution layer can be any number, such as 8, 16, 32, etc.; in this embodiment, the number of convolution layers of the feature convolution group 21 is set to 1 layer, and the number of convolution kernels of the convolution layer is 16, which can ensure the accuracy of the deep neural network model while reducing the storage space required for storing the weights of the side module as much as possible; the output convolution layer is a 1*1 convolution layer.

[0033] It should be noted that this embodiment does not impose any restrictions on the number of layers of the main path of the deep neural network model, which can be set according to the actual application scenario; in addition, the parameters of the deep neural network model also include the learning rate of gradient descent, which can be set to 0.05.

[0034] Step 102, compress the main path of the deep neural network model, and save the weights of each layer of the compressed main path of the deep neural network model.

[0035] Specifically, the main path of the deep neural network model is compressed. After compression, the number of weights required to be saved can be greatly reduced, that is, the memory required to save the weights of the main path of the deep neural network model is reduced, and the memory consumption of the edge terminal is reduced. Among them, the weights in the network parameters of the main path of the deep neural network model no longer change in the subsequent process, and can be stored in 8-bit integers to further reduce the storage space consumed by weight storage; in addition, when storing the bias of each layer of the main path of the deep neural network model, if the bias of each layer of the main path does not need to be learned and updated in the subsequent training process, the bias of each layer of the main path can also be stored in 8-bit integers. If the bias of each layer of the main path still needs to be learned and updated in the subsequent training process, 16-bit floating point numbers are used to store the bias of each layer of the main path, allowing the bias of each layer of the main path to be modified during the retraining process, which can ensure the accuracy of the subsequent training process.

[0036] In one embodiment, the compression method includes any one of the following or any combination: pruning, quantization, and shared weights; wherein pruning can remove parameters of low importance in the main path of the deep neural network model, and can remove 80% to 90% of the parameters in the deep neural network model, retaining only 10% to 20% of the parameters; quantization is to reduce the bit width of the floating-point weight parameters in the deep neural network model, for example, 64-bit floating-point type uses 8-bit integer quantization, and the storage space is reduced to one-eighth of the original size; shared weights are to extract values ​​that can be shared in each layer of the main path of the deep neural network model, reduce the number of possible values ​​to be stored, and reduce storage consumption.

[0037] Step 103, continue to train the current deep neural network model using the collected sample data, update the network parameters of each layer of each side road module and save them; wherein the weights of each layer of the main road remain unchanged.

[0038] Specifically, the edge terminal will still collect sample data in real time during the operation process. For the deep neural network model trained in step 101, the collected sample data will be used to continue training the deep neural network model. The training process is roughly the same as step 101. The main difference is that during this training process, the weights of each layer of the main path of the deep neural network model are kept unchanged, that is, this training can obtain the network parameters of each layer of each side path module for storage; in addition, during this training process, it can be set to allow the bias of each layer of the main path to be learned and updated, and the bias of each layer of the main path will also be updated during the continued training process; thereby, the final deep neural network model can be obtained.

[0039] This embodiment provides a training method for a deep neural network model. The deep neural network model includes a main path and a side path. The main path includes multiple sub-modules, each sub-module includes multiple layers, and the side path includes multiple side path modules. The sub-modules correspond to the side path modules one by one and are connected in parallel. When training the deep neural network model, the main path of the deep neural network model is first preliminarily trained using training data. After obtaining the network parameters of each layer of the main path of the deep neural network model, the main path of the deep neural network model is compressed, and only the weights of each layer of the compressed main path of the deep neural network model can be saved, thereby greatly reducing the storage space required for storing the weights of each layer of the main path of the deep neural network model. The saved weights remain unchanged in subsequent training processes, and the memory consumption of the deep neural network model during continued training is also reduced, so that it can be suitable for edge terminals with smaller memory. After the preliminary training is completed, the current deep neural network model is continued to be trained using the collected sample data, and the network parameters of each side path module can be obtained based on the latest sample data. That is, the deep neural network model finally trained can be adaptively learned and updated based on the latest sample data, thereby improving the accuracy of the deep neural network model.

[0040] The second embodiment of the present invention relates to a model training method. Compared with the first embodiment, this embodiment provides a specific implementation method for continuing to train the current deep neural network model using the collected sample data, obtaining the network parameters of each layer of each side module and saving them.

[0041] The specific process of the model training method in this embodiment is as follows Figure 3 shown.

[0042] Step 201, using training data to train the main path of the deep neural network model, to obtain network parameters of each layer of the main path of the deep neural network model, the network parameters including weights.

[0043] Step 202, compress the main path of the deep neural network model, and save the weights of each layer of the compressed main path of the deep neural network model.

[0044] Step 203, using the collected sample data to continue training the current deep neural network model, obtains and saves the network parameters of each layer of each side module, including the following sub-steps:

[0045] Sub-step 2031, selecting multiple layers of the main path of the current deep neural network model as checkpoints.

[0046] Sub-step 2032, using the collected sample data to perform forward propagation on the current deep neural network model, obtain the output data of the current deep neural network model, and save the activation value of each checkpoint.

[0047] Sub-step 2033, using the output data obtained by forward propagation, the activation value of each checkpoint and the annotation information of each sample data to perform back-propagation training on the current deep neural network model, update the network parameters of each layer of each side module and save them. During the back-propagation training process, the activation value of each checkpoint is used to calculate the activation value of each layer in the deep neural network model that is not used as a checkpoint; wherein the weights of each layer of the main path remain unchanged.

[0048] Specifically, the edge terminal will still collect sample data in real time during the working process. For the deep neural network model trained in step 201, the collected sample data will be used to continue training the deep neural network model. The continued training also includes two stages, namely forward propagation training and back propagation training.

[0049] Before forward propagation, multiple layers of the current deep neural network model main path are selected as checkpoints. The number of checkpoints can be set based on the computing power of the edge terminal. The lower the computing power of the edge terminal, the more checkpoints need to be set to avoid excessive time consumption for back propagation. In one example, a layer can be selected as a checkpoint in each submodule of the main path of the deep neural network model; however, it is not limited to this, and multiple submodules can also be set to share a checkpoint.

[0050] During forward propagation, the collected sample data is input into the input layer of the current deep neural network model, the activation value of each neuron in each layer of the deep neural network model is calculated, the output data of the output layer of the deep neural network model is obtained, and then the activation value of each checkpoint is saved.

[0051] In the process of back propagation training, based on the difference between the output data of the output layer of the deep neural network model in the forward propagation process and the labeled information in the sample data, the cost is calculated in combination with the preset loss function, and then the network parameters of each layer of each side module of the deep neural network are adjusted according to the preset optimization strategy (such as the gradient descent method) until the deep neural network model that meets the preset conditions is obtained through training. The preset conditions can be that the accuracy of the model reaches the preset accuracy threshold, the number of training times reaches the preset number of training times, etc. In the process of back propagation training, for each layer in the deep neural network model that does not save the activation value, the nearest checkpoint is first selected, and the activation value of the layer is obtained by deduction based on the activation value and output of the checkpoint. The weights of each layer of the main path remain unchanged in the back propagation training; in addition, it can also be set to allow the bias of each layer of the main path to be updated through learning, and the bias of each layer of the main path will also be updated during the continued training process to improve the accuracy of the trained deep neural network model.

[0052] Taking a 50-layer deep residual network model as an example, the input is a 224-pixel image. When the input batch size is 32, the compression of the deep neural network model in step 202 includes 90% pruning and 8-bit quantization; the number of convolution layers of the feature convolution group of the side module is 1, the number of convolution kernels is 16, and the sampling rate multiplier of the downsampling layer is 4; the number of checkpoints in the main path is 4, and the bias of the main path and the network parameters of each layer of the side module are stored in 16-bit floating point numbers; the deep neural network model training method in this embodiment can reduce the storage space required for training from 4.3GB to 55.17MB, and the storage space is reduced by 98.72%.

[0053] In this embodiment, during the continued training of the deep neural network model, checkpoints are set in the main path, and only the activation value of each checkpoint can be saved during the forward propagation process, so that during the back propagation training process, the activation values ​​of other layers similar to it can be derived through each checkpoint, thereby further reducing the memory consumption during the training of the deep neural network model.

[0054] The third embodiment of the present invention relates to a processing chip, which is applied to an edge terminal, such as a mobile phone, a tablet computer, a laptop computer, a desktop host, etc. The processing chip is used to execute the model training method in the first or second embodiment.

[0055] A fourth embodiment of the present invention relates to an edge terminal, such as a mobile phone, a tablet computer, a laptop computer, a desktop host, etc. The edge terminal includes at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the model training method in the first or second embodiment.

[0056] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0057] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0058] A fifth embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0059] Preferred embodiments of the present invention have been described above in detail, but it should be understood that aspects of the embodiments can be modified, if necessary, to employ aspects, features and concepts of the various patents, applications and publications to provide further embodiments.

[0060] These and other changes can be made to the embodiments in light of the above detailed description.In general, in the claims, the terms used should not be considered limited to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which these claims are entitled.

Claims

1. A model training method, characterized in that: Used to train a deep neural network model including a main path and a side path, the main path includes multiple submodules, each submodule includes multiple layers, the side path includes multiple side path modules, the submodules correspond to the side path modules one by one and are connected in parallel, each submodule of the main path is divided based on the function or structure implemented by each layer of the main path, and the input data of the deep neural network model is an image; the method includes: Using the training data to train the main path of the deep neural network model, and obtain network parameters of each layer of the main path of the deep neural network model, wherein the network parameters include weights; Compressing the main path of the deep neural network model, and saving the weights of each layer of the compressed main path of the deep neural network model; The main path and side path modules of the current deep neural network model are continuously trained using the collected sample data, and the network parameters of each layer of each side path module are updated and saved; wherein the weights of each layer of the main path remain unchanged.

2. The model training method according to claim 1, characterized in that: Each of the side modules includes a downsampling layer, a feature convolution group, an output convolution layer and an upsampling layer connected in sequence; During the training process, the downsampling layer downsamples the input data from the main path, converts the obtained low-resolution input data into the feature convolution group for feature extraction, and then uses the output convolution layer to convert the extracted feature data into the same dimension as the main path, and uses the upsampling layer to upsample the feature data with the same dimension as the main path and return it to the main path.

3. The model training method according to claim 2, characterized in that: The number of convolution layers of the feature convolution group is 1, and the number of convolution kernels of each convolution layer is 16.

4. The model training method according to claim 1, characterized in that: The method further trains the current deep neural network model using the collected sample data, updates the network parameters of each layer of each side path module and saves the parameters, including: Selecting multiple layers of the current deep neural network model as checkpoints; Perform forward propagation on the current deep neural network model using the collected sample data to obtain output data of the current deep neural network model, and save the activation value of each checkpoint; The output data obtained by forward propagation, the activation value of each checkpoint and the annotation information of each sample data are used to perform back-propagation training on the current deep neural network model, and the network parameters of each layer of each side module are updated and saved. During the back-propagation training process, the activation value of each checkpoint is used to calculate the activation value of each layer in the deep neural network model that is not used as the checkpoint; wherein the weights of each layer of the main path remain unchanged.

5. The model training method according to claim 4, characterized in that: The selecting multiple layers of the current deep neural network model as checkpoints includes: A layer is selected as a checkpoint in each submodule of the main path of the deep neural network model.

6. The model training method according to claim 1, characterized in that: The weights of each layer of the main path of the deep neural network model are stored using 8-bit integers, and the learnable parameters of each layer of the deep neural network model are stored using 16-bit floating point numbers.

7. The model training method according to claim 1, characterized in that: The main path of the deep neural network model is compressed by any one or any combination of the following methods: pruning, quantization, and weight sharing.

8. A processing chip, characterized in that: Used to execute the model training method described in any one of claims 1 to 7.

9. An edge terminal, characterized in that: include: at least one processing chip; as well as, A memory in communication with the at least one processing chip; wherein, The memory stores instructions that can be executed by the at least one processing chip, and the instructions are executed by the at least one processing chip so that the at least one processing chip can execute the model training method as described in any one of claims 1 to 7.

Images