Computer implementation method and device for deep learning

Abstract

The present disclosure relates to a computer-implemented method for deep learning, comprising: obtaining a meta-network composed of a set of incubation modules wherein each incubation module of the set comprises at least one base unit in a deep learning network architecture, the element network is pre-trained on a data set; independently training a set of modules on the data set by training the meta-network on the data set, where each module in the set of modules corresponds to a respective incubation module in the set of incubation modules, wherein one incubation module of the set of incubation modules is replaced with a module of the set of modules corresponding to the one incubation module of the set of incubation modules for training the one module of the set of modules, wherein the modules in the set of modules comprise more than one basic unit in the deep learning network architecture; assembling the independently trained modules in the set of modules to form an assembled model; and obtaining the deep learning network optimized on the data set at least partially based on the assembly model.

Description

Technical Field

[0001] Aspects of the present disclosure relate generally to artificial intelligence, and more particularly, to methods and apparatus for deep learning. Background Art

[0002] Recent years have seen a rapid increase in the use of deep learning models, which researchers and practitioners apply to great impact on a wide range of applications such as image and video classification, image and speech recognition, and language translation, etc. As deep learning models have been more widely developed and used, model sizes have grown to new levels (e.g., tens to hundreds of layers, 10 to 20 million total parameters, or even tens of thousands of layers), for example, to increase effectiveness.

[0003] Training such large models is not an easy task and usually faces two major challenges: 1) On the infrastructure side, large models place higher demands on computing resources. Very large models can only be trained on highly optimized clusters with strong computing, memory, and communication capabilities. 2) On the optimization side, large models also require complex designs of optimization algorithms, weight initialization, and other techniques in order to avoid optimization problems.

[0004] Modular training, where the model is divided into several modules, each of which is trained separately, can be a good solution to these two challenges. However, training deep models in a modular manner also faces the problem of the contradiction between independence and compatibility: modules need to be trained independently, but they also need to be compatible with each other when used as the whole model.

[0005] Therefore, it may be desirable to provide improved techniques for modular training of large models, considering both the independence and compatibility of modules. Summary of the invention

[0006] A simplified overview of one or more aspects of the present disclosure is presented below to provide a basic understanding of these aspects. This overview is not an extensive overview of all anticipated aspects, and is intended to neither identify the key or essential elements of all aspects nor the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description presented later.

[0007] In one aspect of the present disclosure, a computer-implemented method for deep learning is provided, the method comprising: obtaining a meta-network consisting of a set of incubation modules, wherein each incubation module in the set of incubation modules includes at least one basic unit in a deep learning network architecture, and the meta-network is pre-trained on a data set; independently training a set of modules on the data set by training the meta-network on the data set, wherein each module in the set of modules corresponds to a corresponding incubation module in the set of incubation modules, wherein an incubation module in the set of incubation modules is replaced with a module in the set of modules corresponding to the one incubation module in the set of incubation modules for training the one module in the set of modules, wherein the modules in the set of modules include more than one basic unit in the deep learning network architecture; assembling the independently trained modules in the set of modules to form an assembled model; and obtaining the deep learning network optimized on the data set based at least in part on the assembled model.

[0008] In another aspect of the present disclosure, a computer-implemented method for deep learning of a task is provided, the method comprising: obtaining a meta-network composed of a set of incubation modules, wherein each incubation module in the set of incubation modules includes at least one basic unit in a deep learning network architecture for a task, the task including image or speech recognition, and the meta-network is pre-trained on a dataset including image or speech signals; independently training a set of modules on the dataset by training the meta-network on the dataset, each module in the set of modules corresponding to a corresponding incubation module in the set of incubation modules, wherein an incubation module in the set of incubation modules is replaced with a module in the set of modules corresponding to the one incubation module in the set of incubation modules for training the one module in the set of modules, wherein the modules in the set of modules include more than one basic unit in the deep learning network architecture; assembling independently trained modules in the set of modules to form an assembled model; and obtaining the deep learning network optimized on the dataset based at least in part on the assembled model.

[0009] In another aspect of the present disclosure, there is provided an apparatus for deep learning, the apparatus comprising a memory and at least one processor, the at least one processor being coupled to the memory and configured to obtain a meta-network consisting of a set of incubation modules, wherein each incubation module in the set of incubation modules comprises at least one basic unit in a deep learning network architecture, and the meta-network is pre-trained on a data set; independently training a set of modules on the data set by training the meta-network on the data set, each module in the set of modules corresponding to a corresponding incubation module in the set of incubation modules, wherein an incubation module in the set of incubation modules is replaced with a module in the set of modules corresponding to the one incubation module in the set of incubation modules for training the one module in the set of modules, wherein the modules in the set of modules comprise more than one basic unit in the deep learning network architecture; assembling independently trained modules in the set of modules to form an assembled model; and obtaining the deep learning network optimized on the data set based at least in part on the assembled model.

[0010] In another aspect of the present disclosure, a computer program product for deep learning is provided, the computer program product comprising a processor executable computer code for the following operations: obtaining a meta-network consisting of a set of incubation modules, wherein each incubation module in the set of incubation modules includes at least one basic unit in a deep learning network architecture, and the meta-network is pre-trained on a data set; independently training a set of modules on the data set by training the meta-network on the data set, wherein each module in the set of modules corresponds to a corresponding incubation module in the set of incubation modules, wherein an incubation module in the set of incubation modules is replaced with a module in the set of modules corresponding to the one incubation module in the set of incubation modules for training the one module in the set of modules, wherein the modules in the set of modules include more than one basic unit in a deep learning network architecture; assembling the independently trained modules in the set of modules to form an assembled model; and obtaining a deep learning network optimized on the data set based at least in part on the assembled model.

[0011] In another aspect of the present disclosure, a computer-readable medium storing computer code for deep learning is provided, wherein the computer code, when executed by a processor, causes the processor to perform operations, the operations comprising: obtaining a meta-network consisting of a set of incubation modules, wherein each incubation module in the set of incubation modules includes at least one basic unit in a deep learning network architecture, and the meta-network is pre-trained on a data set; independently training a set of modules on the data set by training the meta-network on the data set, wherein each module in the set of modules corresponds to a corresponding incubation module in the set of incubation modules, wherein an incubation module in the set of incubation modules is replaced with a module in the set of modules corresponding to the one incubation module in the set of incubation modules for training the one module in the set of modules, wherein the modules in the set of modules include more than one basic unit in the deep learning network architecture; assembling independently trained modules in the set of modules to form an assembled model; and obtaining the deep learning network optimized on the data set based at least in part on the assembled model.

[0012] By using a pre-trained lightweight meta-network to incubate modules partitioned from a deep network, a decoupled or independent training process can be achieved while ensuring compatibility.

[0013] Other aspects or variations of the present disclosure, as well as other advantages thereof, will become apparent by consideration of the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Aspects of the disclosure will hereinafter be described in conjunction with the accompanying drawings, which are provided to illustrate and not to limit aspects of the disclosure.

[0015] Figure 1 A schematic diagram showing an exemplary solution for the first training phase of a two-phase modular learning framework.

[0016] Figure 2 is a graph showing the performance of the assembled model after an exemplary solution of the first training phase with and without fine-tuning.

[0017] Figure 3 A comparison using centered kernel alignment (CKA) similarity is shown between the output features of each module in the assembled model with greedy implementation and the output features of each module in the E2E trained model.

[0018] Figure 4 The CKA similarity between the input of a module at the end of the modularization training phase and the input of the module at the beginning of the assembly fine-tuning phase is shown.

[0019] Figure 5 An exemplary schematic diagram of a process for modular training using a meta-network according to one or more aspects of the present disclosure is shown.

[0020] Figure 6 An example of module reuse according to one or more aspects of the present disclosure is shown.

[0021] Figure 7 An exemplary workflow of a method for performing modular training using a meta-network to obtain a deep learning network according to one or more aspects of the present disclosure is shown.

[0022] Figure 8 An exemplary workflow of a method for module reuse according to one or more aspects of the present disclosure is shown.

[0023] Fig.9A and Fig. 9B Experimental performance of the proposed method and / or process in which the meta-network is frozen during the decoupled training phase according to one or more aspects of the present disclosure is shown.

[0024] Fig.10 An example of a hardware implementation for an apparatus according to one or more aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The present disclosure will now be discussed with reference to several exemplary implementations. It should be understood that these implementations are only intended to enable those skilled in the art to better understand and thus implement embodiments of the present disclosure, rather than to impose any limitation on the scope of the present disclosure.

[0026] Supervised end-to-end (E2E) learning can be a standard approach to neural network optimization. However, when training large models, E2E learning methods may face challenges on both the infrastructure side and the optimization side. For example, on the infrastructure side, large models place higher demands on computing resources. Very large models can only be trained on highly optimized computing clusters with strong computing, memory, and communication capabilities. For another example, on the optimization side, large models require complex designs of optimization algorithms, weight initialization, and other techniques to avoid optimization problems.

[0027] As an example, a conventional way to train a large model can be to add more computing power (e.g., more GPU nodes) and train the network using data-parallel stochastic gradient descent, where each workstation receives a portion of a global (micro) batch, such as a block in a global (micro) batch. The size of the block should be large enough to fully use the computing resources of the workstation. Therefore, expanding the number of workstations will lead to an increase in the batch size. However, using large batches may negatively affect the accuracy of the model. In order to maintain network accuracy, it is necessary to carefully adjust the training hyperparameters (e.g., learning rate, momentum, etc.).

[0028] Modular training, where a model is divided into several modules, where each module is trained independently, can be a good solution for training large models. However, modular training requires both independence and compatibility of the divided modules to address the challenges of training large-scale models. That is, modules need to be trained independently, but they also need to be compatible with each other in order to perform correctly when used as the entire model. However, there is an obvious contradiction between the requirements of independence and compatibility.

[0029] Existing alternatives to E2E can be viewed as weak modular training methods, where these methods only achieve incomplete independence to maintain compatibility. For example, delayed gradient-based methods and synthetic gradient-based methods approximate E2E training in order to preserve a certain level of cross-module compatibility. Local learning-based methods achieve weak coupling between modules by introducing auxiliary networks. However, all of these methods still require cross-module communication, especially during forward propagation. Therefore, the requirement for independence is not fully achieved, which in turn may hinder modular training from achieving its full potential.

[0030] Typically, large models can be divided into several modules, and these modules can be spread across multiple devices or nodes for training. However, the communication between these modules on multiple devices or nodes generated by the sequential nature of the forward propagation and backpropagation algorithms may lead to low resource utilization, which can significantly slow down the training process. In particular, when more devices are used, greater communication overhead is incurred.

[0031] As an example, consider a model M that is partitioned into K modules: Module M i The input and output spaces are represented as and In E2E training, by first forwarding the input signal To produce the output h i =M i (h i-1 ), and then back propagate the error signal To update the model parameters To train module M i , where the input signal and error signal are given as:

[0032]

[0033] Therefore, E2E training ensures strong dependencies of both the input signal on the previous module and the error signal on the subsequent module.

[0034] However, this also makes it impossible to achieve independence during the training process of a given module (e.g., without any cross-module communication). In other words, the two requirements of independence and compatibility actually conflict with each other.

[0035] To this end, according to one or more aspects of the present disclosure, the present disclosure provides a method for using a meta-network for modular training to obtain a deep learning network, wherein the modules are trained in a completely decoupled manner without any communication between the modules, and a certain degree of compatibility is injected into the modules even when the modules are trained in a completely decoupled manner. The proposed method can avoid causing any communication overhead while ensuring compatibility. Therefore, it can reduce the burden on GPU memory and computing power, and can also open up new possibilities in highly heterogeneous scenarios, where different devices have highly different communication capabilities. In addition, since the proposed method removes cross-module dependencies on other modules, the efficiency of computing resources can be improved, and the effectiveness of parallel computing can be maximized accordingly. In addition, this divide-and-conquer strategy by a completely decoupled manner may be less likely to cause optimization problems. The proposed method can be applied to various deep neural networks or graph neural networks for various tasks, which may include but are not limited to image or speech recognition, or image classification, or recommendation, etc.

[0036] Typically, training each module in a completely independent manner may lead to compatibility issues. For example, in a two-stage modular learning framework, the modules (e.g., M i , i = 1…K), and then the trained modules (e.g., ) are assembled together to form the entire model Then fine-tune M assm To facilitate cross-module compatibility to obtain the final model M * In the second phase of assembly fine-tuning, module compatibility is promoted by enabling cross-module communication.

[0037] Figure 1 FIG. 1 is a schematic diagram showing an exemplary solution for the first training phase 100 of a two-stage modular learning framework. In the first training phase 100, for module M iThe previous module of M can be greedily replaced by a simple feature feeder 130, which transforms the input x into a feature vector for module M. i The correct feature space for module M i The subsequent modules of M can be replaced by auxiliary classifier 110, which passes the output to loss function 120 to compare with the correct result y. Arrow 140 can represent forward propagation, and arrow 150 can represent the forward propagation in module M. i By using the first training stage 100, each divided module can be trained independently.

[0038] exist Figure 2 The performance of the assembled model after the first training phase 100 with and without fine-tuning is shown in the graph 200 of FIG. The vertical axis of the graph 200 represents the test accuracy in percentage, and the dashed line 210 represents the test on E2E, which is also presented as an upper limit. The stripe 230 represents the test of the assembled model without fine-tuning, and as shown by the low accuracy of much less than 20%, it does not produce better results than random guessing because compatibility is not guaranteed at all during the first training phase 100. However, as shown by the stripe 220, fine-tuning the assembled model still does not provide a greater gain, and the fine-tuning model M * and its counterpart M for E2E training E2E There is a large gap between them, as shown by the dashed line 210. This may indicate that the greedy implementation of the two-stage modular learning framework imposes too much burden on the assembly fine-tuning stage, which makes it impractical to restore compatibility by using the assembly fine-tuning stage.

[0039] Therefore, the proposed method pre-injects a certain degree of compatibility to reduce the burden even when training modules completely independently. To better achieve compatibility, we can first analyze Figure 2 The incompatibility is shown in . The incompatibility may be caused by feature rank mismatch in early modules and input distribution shift in subsequent modules. Figure 3 shows the assembly model M with greedy implementation using the centralized kernel alignment (CKA) similarity assm The output features of each module and the E2E training model M E2E Comparison between the output features of each module in , where the comparison is performed on the CIFAR-10 dataset using ResNet-110 with K=8. Figure 3 In the example, the assembly model M with greedy implementation assm The modules in are represented continuously along the horizontal axis, and the E2E training model M E2E The modules in are represented continuously along the vertical axis. Figure 3 As shown inassm Early module generation in M E2E This can be caused by the short-sighted nature of the greedy approach, where modules are trained to produce features that are best suited to the classifier. However, in an assembled model, subsequent modules often expect early modules to capture low-level fine-grained features for further processing. Thus, incompatibility results.

[0040] Figure 3 It also shows that M assm Later modules in produce another pattern of features with decreasing similarity to their E2E counterparts at all feature levels. This fading pattern is another manifestation of module compatibility, which can be referred to as the input distribution shift problem. To further analyze this problem, Figure 4 shows the modules at the end of the modular training phase The input of (e.g., the first training phase 100 with greedy implementation) is the same as the module at the beginning of the assembly fine-tuning phase The CKA similarity between the inputs of . Figure 4 In , the module index is represented by the horizontal axis, and the input similarity is represented by the vertical axis. Figure 4 As shown, the results can clearly demonstrate the increased input distribution shift problem faced by later modules.

[0041] In a modular training phase (eg, a first training phase 100 with greedy implementation), the module M i receives its input from a feature feeder (e.g., feature feeder 130), and in the assembly model, the module From its previous module Receive its input. Since the output of the feature feeder is There are no constraints between the outputs of The input distribution of is shifted. Moreover, as more modules are stacked together, the subsequent modules are more affected by the shifted input distribution. That is, the stacked modules produce increasingly incompatible features for the subsequent modules.

[0042] In order to solve the compatibility problem, the proposed method introduces i Lightweight pre-trained meta-networks with the same input and output space in, To achieve a certain degree of module compatibility when the modules are trained in a completely decoupled manner. i , you can use Replace the other modules M in formula (1) j ,get:

[0043]

[0044] Figure 5 An exemplary schematic diagram of a process 520 for performing modular training using a meta-network 510 according to one or more aspects of the present disclosure is shown. The process 520 may be performed according to equation (2). Box 530 represents a pre-training module, and circle 560 represents a loss function. Arrow 540 may represent a loss function in module M. i The forward propagation during the training of the module M can be represented by arrow 550. i Similarly, process 520 can be considered a “proxy incubation” process, where meta-network 510 can be used as a substitute for the original model M to “incubate” module M i . Through the meta-network incubation module M i , even in the absence of any cross-module communication, in module M i Compatibility is also achieved during training.

[0045] In one aspect of the present disclosure, a pre-trained meta-network (e.g., meta-network 510) can naturally form a ladder of feature levels as it converges on a dataset. Replace with the module M to be trained i (For example, as shown in process 520), it is possible to implicitly specify that the inserted module M i Therefore, compatibility can be encouraged by training each module with a meta-network to produce features with a rank that matches its final position in the assembled model. That is, the problem of feature rank mismatch can be alleviated, and a certain degree of compatibility can be introduced in the decoupled or independent training process of the modules.

[0046] In another aspect of the present disclosure, the introduction of the meta-network can also enable the ability to reuse modules. A single meta-network can train different versions of modules with different sizes. Modules trained in this way can be freely reused to assemble with different versions of other modules to obtain different model pools. For example, assuming that m modules with different depths are trained for each stage, the size of the model pool that can be obtained through model assembly is m K At the same time, the total number of modules that need to be trained is only Km, and each module can be reused m times. K-1 Second-rate.

[0047] Figure 6 An example of module reuse according to one or more aspects of the present disclosure is shown. Figure 6 In the example of FIG. 6 , the meta-network 630 may include three modules 630-1, 630-2 and 630-3, that is, K=3, and the module to can all be first modules but with different depths, and the modules to can all be second modules but with different depths, and the modules to All modules may be third modules but with different depths. In the decoupled training phase 610, each module may be trained in a training process 611, where an input 660 may be passed along a forward propagation direction 650, and an error signal based on a loss function 670 may be passed along a backward propagation direction 640 to update the parameters of the module being trained. Figure 6 In the example of , the total number of modules that need to be trained in the decoupled training phase 610 is 3×m. In the model assembly phase 620, the trained modules can be assembled to form a diverse model pool. Specifically, the assembly can be performed by using the trained modules separately. to A trained module in to A trained module in and a trained module in to A trained module in , to stack the first module, the second module and the third module together to form the whole model. Figure 6 In the example of FIG. 6 , the size of the model pool that can be obtained in the model assembly stage 620 is m 3 (i.e., m 3 It can be seen that by leveraging the compatibility of modules incubated by the meta-network, a diverse pool of assembled models can be obtained in a low-cost manner.

[0048] Figure 7 An exemplary workflow of a method 700 for using a meta-network for modular training to obtain a deep learning network according to one or more aspects of the present disclosure is shown. The method 700 may be performed according to the process 520, or may be or include a part of the process 520, and the dashed block 735 may be an optional operation that may be omitted. At block 710, a meta-network consisting of a set of incubation modules (e.g., Figure 5 As shown in ). Each of the set of incubation modules (e.g., ) may include at least one basic unit in the architecture of a deep learning network, and the meta-network is pre-trained on a dataset. Typically, a deep learning network typically starts with an initial processing header, followed by a cascade of blocks, and then ends with a final task-related header. In one aspect of the present disclosure, the basic unit may be a block. For example, a basic unit may be a residual block in ResNet (residual network) or a transformer block in DeiT (data efficient image transformer). Each of the group of incubation modules may include as few basic units as possible to achieve a lightweight meta-network. For example, each of the group of incubation modules may include only one basic unit. As another example, in addition to only one basic unit, the first and last incubation modules (e.g., and ) can also include an initial processing header and a final task-related header respectively.

[0049] At box 720, a group of modules (wherein each module in the group of modules corresponds to a corresponding incubation module in the group of incubation modules) can be independently trained on the data set by training a meta-network on the data set, wherein an incubation module in the group of incubation modules is replaced with a module in the group of modules corresponding to the incubation module in the group of incubation modules for training the module in the group of modules. The number of modules in the group of modules can be equal to the number of incubation modules in the group of modules. The group of modules can be divided from a deep learning network, and the modules in the group of modules can include more than one basic unit in the deep learning network architecture. In one aspect of the present disclosure, when we divide the model M into K modules of this article, that is, The initial processing header and the final task-related header are always assigned to M1 and M2, respectively. K All modules M K ,M K-1 ,…M i , ...M2,M1 can contain the same number of blocks, i.e., evenly divide the cascade of blocks, or be substantially the same if the total number of blocks is not divisible by K. This is not only for simplicity, but also for efficiency, since blocks in mainstream architectures usually have the same computational overhead. Therefore, evenly dividing the model can maximize the parallelization of each decoupled training process. In another aspect of the present disclosure, the module M K ,M K-1 ,…M i ,…M2,M1 can contain different numbers of blocks.

[0050] At block 730 , the independently trained modules in the set of modules may be assembled to form an assembled model.

[0051] At box 740, a deep learning network optimized on the data set can be obtained based at least in part on the assembled model. In one aspect of the present disclosure, a lightweight meta-network can train each module in the group of modules using a process similar to proxy incubation, where the meta-network can be used as a substitute for the original deep learning network to incubate the module. Compatibility between the group of modules can be encouraged by sharing the meta-network, which can implicitly bind the incubated modules together. In this way, there is no need to fine-tune the assembled model to obtain the deep learning network. Method 700 can obtain the deep learning network directly from the assembled model.

[0052] In another aspect of the present disclosure, at block 735, the assembled model can be fine-tuned on the dataset to obtain a deep learning network optimized on the dataset. For example, the assembled model can be fine-tuned in a short time to improve compatibility.

[0053] In one or more aspects of the present disclosure, each module in the set of modules may include the same input and output spaces as a corresponding incubation module in the set of incubation modules. i Contains downsampling blocks, then these downsampling blocks must all be saved in the corresponding incubation module Otherwise, M i and will have different output spaces. The design principle can be expressed as:

[0054]

[0055] In other aspects of the present disclosure, independently training the set of modules may include freezing remaining incubated modules in the meta-network that were not replaced by a module in the set of modules during training of the module in the set of modules. Figure 5 In the process 520, the i The remaining incubation modules are not updated during training (Right now, ). By freezing the meta-network throughout the decoupled training process (e.g., process 520), all modules in the set of modules can be forced to adapt to the exact same meta-network. Thus, an implicit binding can be created between modules trained in this manner, which can alleviate the problem of input distribution shift and encourage module compatibility.

[0056] Figure 8 An exemplary workflow of a method 800 for module reuse according to one or more aspects of the present disclosure is shown. The method 800 may be based on Figure 6At block 810, a meta-network consisting of a set of incubation modules may be obtained, wherein each incubation module in the set of incubation modules includes at least one basic unit in a deep learning network architecture, and the meta-network is pre-trained on a data set. At blocks 820-1 to 820-Km, more than one set of modules may be independently trained on a data set. Each module in each set of modules corresponds to a corresponding one of the incubation modules in the set of incubation modules. For example, Figure 6 In the example of FIG. 6 , the same meta-network 630 may be used to train m groups of modules, where the first group of modules may include and in, Corresponding to the incubation module 630-1, Corresponding to the incubation module 630-2, and Corresponding to the incubation module 630-3, etc. For training modules The meta-network can be trained on the dataset, where the corresponding 630-1 module Replace, as shown in training process 611. For m groups of modules and K modules in each group, a total of Km independent training processes can be performed. In one aspect of the present disclosure, a module in one group of modules and a module in another group of modules corresponding to the same incubation module can include the same input and output spaces, but different numbers of basic units. Figure 6 In the example, and Both correspond to the incubation module 630-1, but Can have More layers.

[0057] At block 830, independently trained modules from more than one set of modules may be assembled to form different assembled models. In one aspect of the present disclosure, the trained modules may be assembled with each other as long as each module is arranged in the assembled model according to its corresponding position (e.g., in Figure 6 In the example, is the first module, which should be arranged at the first position in the assembly model). For example, Figure 6 In the example, from the mth th ) group module has a first position From the first set of modules with second and third positions and can be cascaded to form an assembly model, and the first set of modules having a first position and a second position and from the mth th ) group module has a third position can be cascaded to form another assembled model, as shown in the model assembly stage 620. The size of the model pool can be m K , and each trained module can be reused K-1 Second-rate.

[0058] At optional block 835 , the assembled model may be fine-tuned on the dataset to improve compatibility.

[0059] At block 840 , corresponding deep learning networks having different depths optimized on the dataset may be obtained based at least in part on the different assembled models.

[0060] It should be understood that one or more aspects of the present disclosure described with reference to methods and / or processes may be combined with other aspects described with reference to other methods and / or processes without causing a departure from the present disclosure.

[0061] Fig.9A and Fig. 9B The experimental performance of the methods 700 and 800 and / or the process 520 with freezing the meta-network during the decoupled training phase according to one or more aspects of the present disclosure is shown. The experiments were conducted on the CIFAR-10 dataset using ResNet-110 with K=8. Fig.9A In the graph of FIG. 1 , the vertical axis represents test accuracy in percentage, and dashed line 910 represents testing on E2E, which is also presented as an upper limit. Strip 930 represents testing on the assembled model without fine-tuning, and strip 920 represents testing on the assembled model with fine-tuning. Fig.9A It can be seen that, despite being simple and almost debugging-free, the proposed methods 700 and 800 and / or process 520 can achieve good performance compared to E2E training. In addition, the methods 700 and 800 and / or process 520 can successfully train deep transformer-based models with large batch sizes up to 8192, for example, without causing optimization problems. Fig. 9B The CKA similarity between the assembled model without fine-tuning and the E2E trained model is shown, with all pairs of module outputs compared. Fig. 9B , the modules in the assembly model of the methods 700 and 800 and / or the process 520 are sequentially represented along the horizontal axis, and the E2E training model M is sequentially represented along the vertical axis. E2E Modules in . Fig. 9BIt can be seen that the problem of feature level mismatch in early modules and input distribution shift in late modules can be well solved, and the problem of feature level mismatch in early modules and input distribution shift in late modules can be well solved. E2E CKA similarities between them can reveal healthy patterns.

[0062] Fig.10 1 shows an example of a hardware implementation for the apparatus 1000 according to one or more aspects of the present disclosure. The apparatus 1000 for deep learning may include a memory 1010 and at least one processor 1020. The processor 1020 may be coupled to the memory 1010 and configured to execute the above reference Figure 7 , Figure 8 and Figure 5 The method 700, method 800 and process 520 described herein. The processor 1020 may be a general purpose processor, or may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The memory 1010 may store input data, output data, data generated by the processor 1020 and / or instructions executed by the processor 1020.

[0063] The various operations, models, and networks described in conjunction with the disclosure herein may be implemented in hardware, software executed by a processor, firmware, a computer, or any combination thereof. According to one or more aspects of the present disclosure, a computer program product for deep learning may include a computer program for executing the above referenced Figure 7 , Figure 8 and Figure 5 The processor executable computer code of the method 700, method 800 and process 520 described above. According to another embodiment of the present disclosure, the computer readable medium may store computer code for deep learning, which, when executed by the processor, enables the processor to perform the above reference Figure 7 , Figure 8 and Figure 5 Method 700, method 800, and process 520 are described. Computer-readable media include both non-transitory computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. Any connection may be properly referred to as a computer-readable medium. Other embodiments and implementations are within the scope of the present disclosure.

[0064] In an embodiment of the present disclosure, each module in the group of modules may include the same input and output spaces as a corresponding incubation module in the group of incubation modules. A module in the group of modules and a module in another group of modules corresponding to the same incubation module may include the same input and output spaces, but with a different number of basic units, such as different layers. For example, in Figure 6 In the example, and Both correspond to the incubation module 630-1 and has 630-1 The same input and output space (e.g., input or output map size 32×32, 16×16, or 8×8), but Can have More layers.

[0065] In an embodiment of the present disclosure, during independent training of the set of modules, the remaining incubated modules in the meta-network that are not replaced by a module in the set of modules may be frozen. Figure 5 In the process 520, the i The remaining incubation modules are not updated during training ( (Right now, ))'s parameters.

[0066] In an embodiment of the present disclosure, another group of modules can be independently trained on a data set by using a meta-network, wherein each module in the other group of modules corresponds to a corresponding incubation module in the group of incubation modules. The modules in the group of modules corresponding to the same incubation module and the modules in the other group of modules can include the same input and output spaces, but the number of basic units is different. Independently trained modules from both the group of modules and the other group of modules can be assembled to form another assembly model. Another deep learning network with a depth different from the deep learning network optimized on the data set can be obtained directly from the other assembly model, or by fine-tuning the other assembly model.

[0067] In an embodiment of the present disclosure, the apparatus 1000 for deep learning including the memory 1010 and the at least one processor 1020 may further include at least one cache in each of the at least one processor 1020 for storing a meta-network. For example, each of the at least one processor 1020 may obtain the meta-network from the memory 1010 and write the meta-network into its cache. As another example, the at least one processor 1020 may be used to independently train the group of modules using the same meta-network stored in the cache, wherein different modules in the group of modules may be trained simultaneously on separate processors to achieve parallel computing while using the same meta-network to ensure compatibility between different modules. The components of the apparatus 1000 for deep learning may be located in one place, or may be distributed in different locations.

[0068] The foregoing description of the embodiments of the disclosed content is provided to enable those skilled in the art to make or use the various embodiments. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the various embodiments. Therefore, the claims are not intended to be limited to the embodiments shown herein, but rather to be accorded the widest scope consistent with the appended claims and the principles and novel features disclosed herein.

Claims

1. A computer-implemented method for deep learning, comprising: Obtaining a meta-network consisting of a set of incubation modules, wherein each incubation module in the set of incubation modules includes at least one basic unit in a deep learning network architecture, and the meta-network is pre-trained on a data set; independently training a set of modules on the dataset by training the meta-network on the dataset, each module in the set of modules corresponding to a respective one of the set of incubation modules, wherein one of the incubation modules in the set of incubation modules is replaced with one of the modules in the set of modules corresponding to the one of the incubation modules in the set of incubation modules for training the one of the modules in the set of modules, wherein the modules in the set of modules include more than one basic unit in the deep learning network architecture; Assembling independently trained modules in the set of modules to form an assembled model; and A deep learning network optimized on the dataset is obtained based at least in part on the assembled model.

2. The computer-implemented method of claim 1 , wherein: Each module of the set of modules comprises the same input and output spaces as a corresponding one of the set of incubation modules.

3. The computer-implemented method of claim 1 , further comprising: The assembled model is fine-tuned on the dataset to obtain a deep learning network optimized on the dataset.

4. The computer-implemented method of claim 1 , wherein: Independently training the set of modules further comprises: Remaining incubated modules in the meta-network that are not replaced by the one module in the set of modules are frozen during the training of the one module in the set of modules.

5. The computer-implemented method of claim 1 , further comprising: independently training another set of modules on the data set using the meta-network, each module in the another set of modules corresponding to a respective incubation module in the set of incubation modules, wherein the modules in the set of modules and the modules in the another set of modules corresponding to the same incubation module include the same input and output spaces but different numbers of elementary units; assembling independently trained modules from both the set of modules and the another set of modules to form another assembled model; and Based at least in part on the another assembled model, another deep learning network optimized on the dataset is obtained having a different depth than the deep learning network.

6. A computer-implemented method for deep learning of a task, comprising: Obtaining a meta-network consisting of a set of incubation modules, wherein each incubation module in the set of incubation modules includes at least one basic unit in a deep learning network architecture for a task, the task including image or speech recognition, and the meta-network is pre-trained on a dataset including image or speech signals; independently training a set of modules on the data set by training the meta-network on the data set, each module in the set of modules corresponding to a respective one of the set of incubation modules, wherein one of the incubation modules in the set of incubation modules is replaced with one of the modules in the set of modules corresponding to the one of the incubation modules in the set of incubation modules for training the one of the modules in the set of modules, wherein the modules in the set of modules include more than one basic unit in the deep learning network architecture; Assembling independently trained modules in the set of modules to form an assembled model; and A deep learning network optimized on the dataset is obtained based at least in part on the assembled model.

7. A device for deep learning, comprising: Memory; as well as At least one processor coupled to the memory and configured to perform the computer-implemented method of any one of claims 1 to 6.

8. A computer program product for deep learning, comprising: Processor executable computer code for performing the computer implemented method of any of claims 1-6.

9. A computer readable medium storing computer code for deep learning, which, when executed by a processor, causes the processor to perform the computer-implemented method of any one of claims 1-6.

Images