Continuous training method in limited mobile environment

Abstract

Provided is a continuous training method in a limited mobile environment. The method for continuously training a deep learning model according to an embodiment of the present invention involves: fine-tuning a pre-trained deep learning model with a first data set; fixing some weights in the fine-tuned deep learning model in the first fine-tuning step, and pruning the remaining weights; and loading pre-trained weights on the pruned weights. Accordingly, effective continuous training is possible in a fixed small model structure without a neural network addition process in a resource-limited mobile device environment.

Description

Continuous learning methods in limited mobile environments

[0001] The present invention relates to deep learning model learning, and more particularly, to a continuous learning method in a limited mobile environment for object classification and location information data in computer vision.

[0002] Conventional CNN continuous learning relies on large parameters, focusing primarily on software-driven performance to minimize loss of previously learned information. Furthermore, the image data used for training is also large, and this technique fails to consider hardware design constraints, such as those in mobile environments.

[0003] In addition, in order to be applied to hardware, the model size must be small and the overall structure must be fixed, whereas conventional continuous learning-related technologies are not easy to apply because certain neural networks are added as the model is learned or they are established when the number of parameters exceeds a certain level.

[0004] The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a method capable of continuous learning within a fixed model structure without an additional neural network process in a mobile device environment with limited resources.

[0005] According to one embodiment of the present invention for achieving the above object, a continuous learning method for a deep learning model includes a first fine-tuning step of fine-tuning a pre-trained deep learning model using a first dataset; a first pruning step of fixing some weights in the fine-tuned deep learning model in the first fine-tuning step and pruning the remaining weights; and a first loading step of loading pre-trained weights to the weights pruned in the first pruning step.

[0006] The first loading step may be to load pre-trained weights onto all of the pruned weights in the first pruning step.

[0007] The first loading step may load the pre-trained weights only to some of the weights pruned in the first pruning step.

[0008] Some of the pruned weights may be determined based on the importance of the pre-trained weights.

[0009] The number of weights into which pre-learned weights are loaded may be equal to the number of weights fixed in the first pruning step.

[0010] The deep learning model continuous learning method according to the present invention may further include a second fine-tuning step of fine-tuning the deep learning model for which the first loading step has been performed using a second dataset.

[0011] The continuous learning method of a deep learning model according to the present invention may further include a second pruning step of fixing some weights in a fine-tuned deep learning model in a second fine-tuning step and pruning the remaining weights; and a second loading step of loading pre-learned weights onto the weights pruned in the second pruning step.

[0012] The second loading step may be to load the pre-trained weights to all the pruned weights in the second pruning step.

[0013] The second loading step may load the pre-trained weights only to some of the weights pruned in the second pruning step.

[0014] According to another aspect of the present invention, a deep learning computing device is provided, characterized by including: a computing device that fine-tunes a pre-trained deep learning model using a first dataset, fixes some weights in the fine-tuned deep learning model and prunes the remaining weights, and loads the pre-trained weights into the pruned weights; and a memory that provides storage space required for the computing device.

[0015] According to another aspect of the present invention, a method for continuous learning of a deep learning model is provided, comprising: a step of receiving pre-learned weights for a deep learning model; a first fine-tuning step of fine-tuning the pre-learned deep learning model with a first dataset; a first pruning step of fixing some weights in the fine-tuned deep learning model in the first fine-tuning step and pruning the remaining weights; and a first loading step of loading pre-learned weights to the weights pruned in the first pruning step.

[0016] According to another aspect of the present invention, a deep learning computing device is provided, comprising: a communication interface for receiving pre-learned weights for a deep learning model; an computing device for fine-tuning the pre-learned deep learning model using a first dataset, fixing some weights in the fine-tuned deep learning model and pruning the remaining weights, and loading the pre-learned weights into the pruned weights;

[0017] As described above, according to embodiments of the present invention, effective continuous learning is possible within a fixed small model structure without an additional neural network process in a resource-constrained mobile device environment.

[0018] Figure 1-2. Example of the difference between transfer learning and continuous learning.

[0019] Figure 3. Learning structure of the continuous learning PackNet model

[0020] Figure 4. Example of a continuous learning process based on Scratch learning.

[0021] Figure 5-7. Information compression and continuous learning through PackNet-based pruning.

[0022] Figure 8-9. Continuous learning method according to one embodiment of the present invention.

[0023] Figure 10-11. Continuous learning method according to another embodiment of the present invention.

[0024] Figure 12-13. Performance comparison between the existing continuous learning method and the continuous learning method according to the embodiment of the present invention.

[0025] Fig. 14. Mobile deep learning computing device according to another embodiment of the present invention.

[0026] Hereinafter, the present invention will be described in more detail with reference to the drawings.

[0027] Recently, as the issue of designing neural networks capable of learning in limited environments such as edge computing and mobile devices has emerged, deep learning models are no longer simply required to achieve high performance, but also to develop technologies that can achieve performance with limited image data information in much smaller models compared to existing large-sized models.

[0028] In addition, as interest and research and development in deep learning models that can be practically used in actual industrial settings progress, the need for continuous learning technology that can continuously accumulate and learn new image data information to the same neural network model, rather than the existing method of learning only one simple dataset, is increasing.

[0029] However, despite the various research and development efforts being conducted on actual continuous learning, it is very difficult to develop a small-scale continuous learning model applicable to hardware structures without additional model expansion as new image data is added.

[0030] Figures 1 and 2 illustrate the structural differences between transfer learning (Figure 1) and continuous learning (Figure 2) in their learning processes. Transfer learning (Figure 1) is a technique that expects higher performance while rapidly learning other image datasets based on model parameters learned from a large-scale image dataset.

[0031] However, transfer learning suffers from the problem of forgetting previously learned information when additional data sets are learned. A learning structure that addresses this issue is continuous learning (Figure 2). Continuous learning offers a method for additionally learning new data sets while preserving existing learning information.

[0032] Figure 3 illustrates the learning structure of PackNet, a continuous learning method. This technology proposes a method that enables continuous learning while minimizing the expansion of neural network models. This model utilizes the pruning method, previously used to reduce model weight, as a form of data compression, creating a parameter space for learning the next new dataset.

[0033] However, since the method learns based on a very large dataset and adds an FC layer structure that performs object classification for new learning, it is difficult to apply it to hardware design in a mobile environment that requires a fixed model structure.

[0034] In addition, in an environment where image datasets are input through a real-time camera and learning is performed, it is not common to perform learning on multiple classes at once, such as in the experiment of the model. In reality, a technology is needed for performing learning on a single class at a time, but the technology is not suitable for such an environment.

[0035] Accordingly, an embodiment of the present invention proposes a real-time continuous learning method based on a fixed model structure suitable for mobile hardware. Instead of requiring an additional model structure, the method has a structure similar to that shown in Figure 4 above, which accumulates and stores previous image datasets in memory and performs scratch learning for each continuous learning cycle.

[0036] In embodiments of the present invention, we focus on improving learning speed by enabling scratch-based real-time continuous learning. Existing continuous learning models, like transfer learning, have a structure in which pre-learned parameters are used as initial values ​​for the first learning, and subsequent learning parameters are initialized to random values ​​to proceed with continuous learning. However, the continuous learning method according to embodiments of the present invention learns by reflecting previously pre-learned parameter values ​​in the parameters corresponding to the remaining space for subsequent continuous learning.

[0037] For the images presented in Fig. 5, information compression through PackNet-based pruning and the neural network paths that new image data input during continuous learning mainly pass through are shown proportionally in Figs. 6 and 7. The model that performed continuous learning every quarter based on pre-learned parameters (Fig. 6) shows a much higher usage ratio of the neural network path learned in the early stages compared to the model that learned based on random parameter values ​​from the quarters after the first learning (Fig. 7).

[0038] This demonstrates that, if cumulative continuous learning is performed without further model expansion, there is a limit to the inevitably reduced learning space as new learning accumulates. Furthermore, the pre-trained parameters are based on information from large image datasets, implying that the parameter information is highly generalizable.

[0039] Based on these two facts, the intention to completely compress and separate the learning information for each dataset is far from the learning tendency of the actual model, and it is concluded that the method of continuously utilizing pre-learned parameters to proceed with learning can achieve optimal performance in a limited space.

[0040] Accordingly, in an embodiment of the present invention, a continuous learning method is proposed that continuously utilizes pre-learned parameters in a mobile environment with limited resources.

[0041] Figures 8 and 9 are diagrams for explaining a continuous learning method according to an embodiment of the present invention. In Figure 8, the weights illustrated on the left are the weights of the first layer, and the weights illustrated on the right are the weights of the second layer. Accordingly, the deep learning model trained in the embodiment of the present invention is composed of two layers, but this is merely an example for the convenience of explanation. There is no limit to the number of layers of the deep learning model to which the embodiment of the present invention is applicable. Meanwhile, the deep learning model has a structure in which images are input in multiple channels, and the feature map of the image is also generated in multiple channels and is divided into weights for each channel.

[0042] For continuous learning, as illustrated in Figure 8, a deep learning model is first pre-trained, and then the pre-trained deep learning model is fine-tuned using Dataset #1. Next, the weights of some channels (blue) in the fine-tuned deep learning model are fixed, and the weights of the remaining channels (white) are pruned. Weight pruning is performed on a channel-by-channel basis. That is, the weights for some channels are pruned, and the weights for the remaining channels are left. Meanwhile, weight pruning can prune the weights of different channels for each layer. As illustrated, the channels for weight pruning in the first layer (shown on the left) are different from the channels for weight pruning in the second layer (shown on the right). Afterwards, the pre-trained weights are loaded into all the weights of the pruned channels.

[0043] As shown in Figure 9, a deep learning model loaded with pre-trained weights is fine-tuned using Dataset #2. In the fine-tuned deep learning model, weights of some channels (blue and yellow) are fixed, and weights of the remaining channels (white) are pruned. Meanwhile, channel pruning prunes weights for different channels layer by layer. Afterwards, the pre-trained weights are loaded onto all weights of the pruned channels.

[0044] Figures 10 and 11 are drawings for explaining a continuous learning method according to another embodiment of the present invention.

[0045] As illustrated in Figure 10, a deep learning model is first pretrained, then the pretrained deep learning model is fine-tuned using Dataset #1. Next, the weights of some channels (in blue) in the fine-tuned deep learning model are fixed, and the weights of the remaining channels (in white) are pruned. Then, the pretrained weights are loaded only to the weights of some of the pruned channels (in green).

[0046] The weights loaded with pre-trained weights are determined based on their importance. This can be implemented by loading weights among the pre-trained weights that are expected to have high importance for learning, specifically, weights with high values. The weights of the remaining channels (white) that are not loaded with pre-trained weights are initialized to 0. Meanwhile, the number of channels that are fixed and not pruned within a layer is matched to the number of channels loaded with pre-trained weights.

[0047] As shown in Figure 11 below, a deep learning model loaded with pre-trained weights is fine-tuned using Dataset #2. In the fine-tuned deep learning model, the weights of some channels (blue, yellow) are fixed, and the weights of the remaining channels (white) are pruned. Afterwards, the pre-trained weights are loaded only to the weights of some of the pruned channels (green). The weights of the remaining channels (white) that are not loaded with pre-trained weights are initialized to 0.

[0048] Figures 12 and 13 show a performance comparison between a conventional continuous learning method (left) and a continuous learning method according to an embodiment of the present invention (right). The continuous learning method according to an embodiment of the present invention (left of Figures 12 and 13) exhibits higher performance for additional cumulative learning than the conventional method (right of Figures 12 and 13), and shows a high degree of learning performance even in shorter learning epochs.

[0049] This shows that the continuous learning method according to the embodiment of the present invention is easy to design for hardware because there is no additional model expansion, shows high performance for small models and image datasets, is easy to apply in limited environments such as mobile devices, and has the advantage of real-time learning because it enables fast learning within a short learning time.

[0050] FIG. 14 is a diagram illustrating the configuration of a mobile deep learning computing device according to another embodiment of the present invention. As illustrated, the mobile deep learning computing device according to an embodiment of the present invention is configured to include a communication interface (110), a deep learning computing unit (120), and a memory (130).

[0051] The communication interface (110) communicates with an external host system and receives a dataset and parameters (weights) of a pre-trained deep learning model. The deep learning operator (120) continuously trains the loaded deep learning model according to the method presented in FIGS. 8 and 9 described above. The memory (130) provides the storage space necessary for the deep learning operator (120) to perform calculations.

[0052] So far, we have described in detail a preferred embodiment of a continuous learning method and a mobile deep learning computing device that applies the method in a limited mobile environment.

[0053] The above example presents a hardware-friendly model capable of continuous learning within a fixed model structure without additional neural network processing, enabling continuous learning in mobile device environments that support only a limited number of image data and model sizes. This method enables continuous learning within a fixed, small model structure without the need for additional neural network processing, even in resource-constrained mobile device environments.

[0054] In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and various modifications can be made by a person having ordinary skill in the art to which the present invention pertains without departing from the gist of the present invention as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present invention.

Claims

1. The first fine-tuning step of fine-tuning the pre-trained deep learning model using the first dataset; A first pruning step that fixes some weights in the fine-tuned deep learning model in the first fine-tuning step and prunes the remaining weights; A method for continuous learning of a deep learning model, characterized by comprising a first loading step of loading pre-learned weights onto pruned weights in a first pruning step.

2. In claim 1, The first loading stage is, A continuous learning method for a deep learning model, characterized in that pre-learned weights are loaded onto all pruned weights in the first pruning step.

3. In claim 1, The first loading stage is, A method for continuous learning of a deep learning model, characterized in that pre-learned weights are loaded only on some of the pruned weights in the first pruning step.

4. In claim 3, Some of the pruned weights are, A continuous learning method for a deep learning model characterized in that the importance of pre-learned weights is determined based on the weights.

5. In claim 3, The number of weights that are loaded with pre-learned weights is A continuous learning method for a deep learning model characterized by having the same number of fixed weights in the first pruning step.

6. In claim 1, A method for continuous learning of a deep learning model, characterized in that it further includes a second fine-tuning step of fine-tuning a deep learning model for which a first loading step has been performed using a second dataset.

7. In claim 6, A second pruning step that fixes some weights in the fine-tuned deep learning model and prunes the remaining weights in the second fine-tuning step; A method for continuous learning of a deep learning model, characterized in that it further includes a second loading step of loading pre-learned weights onto pruned weights in a second pruning step.

8. In claim 7, The second loading stage is, A deep learning model continuous learning method characterized by loading pre-learned weights to all pruned weights in the second pruning step.

9. In claim 7, The second loading stage is, A method for continuous learning of a deep learning model, characterized in that pre-learned weights are loaded only on some of the pruned weights in the second pruning step.

10. A computation unit that fine-tunes a pre-trained deep learning model using the first dataset, fixes some weights in the fine-tuned deep learning model, prunes the remaining weights, and loads the pre-trained weights into the pruned weights; and A deep learning computing device characterized by including a memory that provides storage space required for the computing device.

11. Step of receiving pre-learned weights for the deep learning model; A first fine-tuning step for fine-tuning a pre-trained deep learning model using the first dataset; A first pruning step that fixes some weights in the fine-tuned deep learning model in the first fine-tuning step and prunes the remaining weights; A method for continuous learning of a deep learning model, characterized by comprising a first loading step of loading pre-learned weights onto pruned weights in a first pruning step.

12. A communication interface for receiving pre-trained weights for a deep learning model; A deep learning computing device characterized by including an computing unit that fine-tunes a pre-trained deep learning model using a first dataset, fixes some weights in the fine-tuned deep learning model, prunes the remaining weights, and loads the pre-trained weights into the pruned weights.

Images