Layer freezing and data screening for sparse training

Abstract

A layer freezing and data screening technique for use in a sparse training domain for providing end-to-end target recognition for efficient training of a dataset. Layer freezing and data screening methods are seamlessly incorporated into sparse training algorithms to form a generic framework. The general framework is always superior to the existing method, and the number of floating point operations per second (FLOPs) and the memory cost of training are remarkably reduced while high precision is kept. The reduction of the trained FLOPs comes from three sources: weight sparseness, a frozen layer and a shrunk data set. Training acceleration depends on different factors, such as support of sparse calculations, layer type and size, and system cost. The reduction of FLOPs from the frozen layer and the shrunk data set results in a higher actual training acceleration than the weight sparseness.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 893,241, filed on August 23, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This topic deals with sparse training of deep learning on edge devices to perform object recognition. Background Art

[0004] Sparse training for object recognition has emerged as a promising paradigm for efficient deep learning on edge devices. Increasing sparsity is not always ideal as it can introduce severe accuracy degradation at high sparsity levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The accompanying drawings depict one or more embodiments by way of example only and not by way of limitation. In the drawings, like reference numerals represent the same or similar elements.

[0006] The features of the various embodiments disclosed will be readily understood from the following detailed description, with reference to the accompanying drawings. In the several views of the specification and drawings, each element uses a reference numeral. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements, with the added letter representing the specific element.

[0007] Unless otherwise noted, the various elements shown in the figures are not drawn to scale. For clarity, the sizes of the various elements may be enlarged or reduced. Several figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The following figures are included in the drawings:

[0008] Figure 1A is a schematic diagram depicting an overview of the SpFDE framework;

[0009] Figure 1B It is a flow chart of the method for training sparse networks using the SpFDE framework;

[0010] Figure 2A is an algorithm for the training process of the SpFDE framework including gradual layer freezing;

[0011] Figure 2B It is shown Figure 2A A method flow chart showing the steps of the algorithm shown;

[0012] Figure 3A is a schematic diagram describing different layer freezing schemes;

[0013] Figure 3Bis a graph showing the trend of layer gradient norms and the difference of layer gradient norms during dynamic sparse training;

[0014] Figure 4 is a flowchart of a data screening method for achieving truly efficient training of a dataset throughout the sparse training process;

[0015] Figure 5 is a diagram showing high-level memory savings of the SpFDE framework;

[0016] Figure 6 is a table showing the comparison of accuracy and FLOPs calculation results on the CIFAR-100 dataset using a residual network (ResNet)-32;

[0017] Figure 7 is a table describing the comparison results on the ImageNet dataset using ResNet-50; and

[0018] Figure 8 is a block diagram of a sample configuration of a computer system for implementing the SpFDE framework. DETAILED DESCRIPTION

[0019] A layer freezing and data filtering technique used in a sparse training domain is used to provide end-to-end dataset-efficient training for object recognition. The layer freezing and data filtering methods are seamlessly incorporated into the sparse training algorithm to form a general framework (referred to as the "SpFDE framework" in this paper). The SpFDE framework consistently outperforms existing methods and significantly reduces the floating point operations per second (FLOPs) and memory cost of training while maintaining high accuracy. The reduction in training FLOPs comes from three sources: weight sparsity, frozen layers, and shrunken datasets. The training speedup depends on different factors, such as the support of sparse computation, layer type and size, and system cost. The FLOPs reduction from frozen layers and shrunken datasets leads to higher actual training speedup than weight sparsity. This makes the layer freezing and data filtering methods more valuable in sparse training. The overall computation FLOPs are used to measure the training speedup, which can be considered as a theoretical upper limit.

[0020] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products, which illustrate examples described in the present disclosure. In order to provide a thorough understanding of the disclosed subject matter and its related teachings, many details and examples are included. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for describing specific aspects, not for limitation. In general, it is not necessary to show in detail the well-known instruction instances, protocols, structures, and techniques.

[0021] As used herein, the terms "coupled" or "connected" refer to any logical, optical, physical, or electrical connection, including links and the like, by which an electrical or magnetic signal generated or provided by one system element is transmitted to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other, and may be separated by intermediate components, elements, or communication media, one or more of which may modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element, or indirectly supported by another element by being integrated into or supported by the element.

[0022] The term "proximal" is used to describe an item or part of an item that is near, adjacent to, or beside an object or person; or closer relative to the rest of the item, which may be described as "distal." For example, the end of an item closest to an object may be called the proximal end, while the generally opposite end may be called the distal end.

[0023] The orientations of the device, other mobile devices, related components, and any other devices incorporating a camera, an inertial measurement unit, or both shown in any of the figures are given as examples only, for purposes of illustration and discussion. In operation, the device may be oriented in any other orientation suitable for the particular application of the device; for example, up, down, sideways, or any other orientation. Furthermore, for the purposes of use herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, above, below, top, bottom, side, horizontal, vertical, and diagonal are used only as examples and do not limit the orientation or orientation of any camera or inertial measurement unit constructed or described herein.

[0024] Other purposes, advantages and novel features of the examples will be partially set forth in the following description, and will become apparent to those skilled in the art by studying the following and the accompanying drawings, or may be learned through the production or operation of the examples. The purposes and advantages of the subject matter may be realized and obtained by the methods, means and combinations particularly pointed out in the appended claims.

[0025] Reference will now be made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0026] It should be understood that, unless otherwise specifically stated herein, the terms and expressions used herein have the general meaning corresponding to these terms and expressions in their respective respective fields of investigation and research. Relational terms such as first and second can be used only to distinguish one entity or action from another entity or action, without requiring or implying any actual such relationship or order between these entities or actions. The terms "comprises", "comprising", "includes", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment that includes or includes a series of elements or steps includes not only those elements or steps, but also other elements or steps that are not explicitly listed or inherent to such process, method, article or equipment. Without further limitation, an element preceded by "a" or "an" does not exclude the presence of additional identical elements in the process, method, article or equipment that includes the element.

[0027] Sparse training has attracted great attention from both industry and academia as a promising solution for efficient training of object recognition on edge devices. Recent studies have proposed various sparse training algorithms with computational and memory savings to achieve training acceleration. These sparse training methods can be divided into two major categories. The first category is the fixed mask sparse training method, which aims to find a better sparse structure in the initial stage and keep the sparse structure unchanged throughout the training process. These methods have a simple sparse training process but suffer from high accuracy degradation. The other category is dynamic sparse training (DST), which usually starts training from a randomly selected sparse structure. DST methods tend to continuously update the sparse structure during the sparse training process while maintaining the overall sparsity ratio of the model. Compared with fixed mask sparse training, the state-of-the-art DST method shows superiority in accuracy and has recently become a more widely adopted sparse training paradigm.

[0028] However, although existing sparse training methods can reduce training costs meaningfully, most of them focus on studying how to reduce training costs by further increasing sparsity while mitigating accuracy degradation. As a result, traditional methods for improvement tend to focus on sparse training performance at extremely high sparsity ratios (e.g., 95% and 98%). However, even recent sparse training methods still lead to severe performance degradation at these high sparsity ratios. For example, on the Canadian Institute for Advanced Research, 10-class (CIFAR-10) dataset, Memory-Economic Sparse Training (MEST) has 2.5% and 4% accuracy degradation at 95% and 98% sparsity, respectively. The CIFAR-10 dataset is a subset of the tiny images dataset, which consists of 60,000 32x32 color images. At extremely high sparsity, network performance usually starts to drop sharply, while the actual gains from weight sparsity (i.e., computational and memory savings) tend to saturate. This suggests that when a certain sparsity level is reached, reducing training cost by pushing sparsity to extreme ratios at the expense of network performance is no longer a desirable approach.

[0029] Figure 1A is a schematic diagram of a SpFDE framework 10 processed by an image processor 11. The SpFDE framework 10 uses Figure 1B The method 100 is shown to train a sparse network (not shown). The entire end-to-end training process is logically divided into three stages, including an initial stage 12, an active training stage 14, and a gradual layer freezing stage 16.

[0030] In the initial stage 12, the processor 11 initializes the sparse network and the partial training data set at block 102. The structure of the sparse network is randomly selected. The partial training data set is obtained by randomly removing a given proportion of training samples from the entire training data set, which is different from the existing work that starts from the entire training data set. During the entire training process, only a portion of the entire training data set is used.

[0031] The initialization phase 12 is followed by an active training phase 14, where the processor 11 actively trains (non-freezes) all layers using a sparse training algorithm in block 104. In the example, dynamic sparse training (DST) is applied from MEST as the sparse training method due to its superior performance, although other sparse training algorithms are compatible with the SpFDE framework 10.

[0032] At block 106, the processor 11 uses a data screening method to update the current partial training data set during training (see Figure 4 ). In addition to the computational and memory savings provided by the sparse training algorithm, the SpFDE framework 10 also benefits from the data screening method to further save computational and memory costs. Specifically, the computational cost is reduced by reducing the number of training iterations in each epoch, and the memory cost is reduced by loading a partial data set.

[0033] In the step-by-step layer freezing phase 16, the processor 11 freezes the layers step by step in a sequential manner at block 108 (see Figure 2A ). During sparse training, the sparse structure and weight values of the frozen layers remain unchanged. The computational and memory costs of calculating all weight gradients and activation gradients in the frozen layers can be eliminated, which is useful for edge devices with limited resources.

[0034] Step-by-step layer freezing

[0035] Since it is observed that the structure and representation similarity of the front layers converge faster than the back layers in sparse training, the step-by-step layer freezing method is used to sequentially and gradually freeze the layers. Specifically, a layer is frozen only after all the layers in front of this layer have been frozen. Since the entire frozen part of the model does not require the calculation of back-propagation (BP), the step-by-step approach brings the benefit of maximizing the savings in training costs.

[0036] Layer freezing algorithm

[0037] Figure 2A The algorithm 20 described in Figure 2B shows an example training process of the SpFDE framework 10 using the step-by-step layer freezing algorithm.

[0038] At block 24, train_flops is initialized to the total sparse training FLOPs of the unfrozen layers, and all blocks are put into active_layers.

[0039] At block 26, for a given deep neural network (DNN) model with L layers, the DNN model is divided into N blocks, each block N consisting of several consecutive DNN layers, such as the bottleneck blocks in a Residual Network (ResNet). The total number of training epochs is denoted as T, Δτ as the sparse structure change interval for dynamic sparse training, T frz (0 < T frz < T) is the epoch when the step-by-step layer freezing phase starts and the first block is frozen.

[0040] At block 28, for each Δτ epoch, the next block is sequentially frozen until the expected total training FLOPs meets target_flops. The frozen block is considered to still need forward propagation during training. The training FLOPs reduction of the frozen block is calculated as its sparse backpropagation calculation FLOPs (calculated by BpFlops(·) in Algorithm 20) multiplied by the total frozen epochs of the block.

[0041] In block 30, to better integrate with DST and ensure that layers / blocks are properly trained before being frozen, the step-by-step layer freezing interval is synchronized with the structure change interval (i.e., Δτ) of sparse training, and a layer-by-layer / block-by-block cosine learning rate schedule is adopted according to the total active training epochs of each layer / block.

[0042] Figure 3A Different layer freezing schemes are shown, Figure 3B The trend of layer gradient norms and the difference of layer gradient norms during dynamic sparsity training are shown.

[0043] Design principles for layer freezing

[0044] The derived layer freezing algorithm 20 has two key principles, the freezing scheme and the freezing criterion.

[0045] Freeze scheme

[0046] Since sparse training may target resource-limited edge devices, it is desirable to keep the training method as simple as possible to reduce system cost and strictly meet budget requirements. Therefore, the cost-saving oriented freezing principle is followed to guarantee the target training cost and derive a layer freezing scheme, which can include single-shot, single-shot and recovery, periodic freezing, and delayed periodic freezing, such as Figure 3A The one-shot scheme is adopted because it achieves the highest accuracy with the same training FLOPs savings, for example, since the one-shot freezing scheme has the longest active training epoch at the beginning of training, which helps the layers converge to a better sparsity structure before freezing.

[0047] Freeze Standard

[0048] Another question is how to derive the freezing criteria, i.e., which iterations or epochs to choose to freeze the layers. Traditional works have explored adaptive freezing methods by computing and collecting gradients during fine-tuning of dense networks. However, the unique properties of sparse training make these methods inapplicable. For example, Figure 3BAs shown, at the beginning of sparse training, the difference of gradient norms from different layers decreases, while after a period of time, the difference keeps fluctuating due to pruning and growing weights. Extracting the freezing criterion based on the gradient norm will inevitably introduce additional computational and system complexity because the changing pattern of the gradient norm difference is variable. Therefore, the SpFDE framework 10 combines the layer freezing interval with the DST interval, which is more advantageous.

[0049] Circular Data Filter

[0050] refer to Figure 1A and Figure 1B , Figure 4 The data screening method 40 shown is used to achieve truly dataset efficient training throughout the sparse training process.

[0051] At block 42 , at the beginning of sparse training, p% of the total training samples in the training data set are randomly removed to create a partial training data set and a removed data set.

[0052] At block 44, during sparse training, for each Δτ epoch, the current partial training dataset is updated by removing the simplest p% of training samples from the partial training dataset and adding them back to the removed dataset.

[0053] At block 46, the same number of samples are retrieved from the removed data set and added back to the partial training data set to keep the total number of training samples constant.

[0054] The number of forgetting times is used as a criterion to indicate the complexity of each training sample. Specifically, for each training sample, the number of forgetting times is collected by counting the number of transitions from correct classification to incorrect classification within each Δτ interval. The number is collected again at each time interval to ensure that newly added samples are treated equally. In addition, the structural change frequency Δτ is used as the dataset update frequency in sparse training to minimize the impact of structural changes on the number of forgetting times.

[0055] The removed dataset is viewed as a queue structure, samples are retrieved from its head, and newly removed samples are added to its tail. After all initially removed samples are retrieved, the removed dataset is shuffled after each update so that all training samples can be used at least once. Therefore, relatively simple samples are gradually screened out, and only important samples are used for efficient dataset training.

[0056] Experimental Results

[0057] The SpFDE framework10 is evaluated on benchmark datasets including CIFAR-100 and ImageNet for image classification tasks using ResNet-32 and ResNet-50. ImageNet is based on A database of images organized in a hierarchical structure (currently just a noun), where each node of the hierarchy is described by hundreds or thousands of images. It is a large-scale English vocabulary database. Nouns, verbs, adjectives, and adverbs are divided into cognitive synonym groups (synsets), each of which expresses different concepts. The accuracy, training FLOPs, and memory cost of the SpFDE framework 10 are compared with the most representative parsing training works at different sparsity ratios. The model is trained by using PyTorch on an 8×A100 graphics processing unit (GPU) server. PyTorch is an open source machine learning framework based on the Torch library for applications such as computer vision and natural language processing, mainly developed by MetaAI. Torch is a scientific computing framework that widely supports machine learning algorithms that put GPUs first. Standard data augmentation and momentum stochastic gradient descent (SGD) optimizer are used. Layer-by-layer cosine annealing learning rate scheduling is used according to the frozen period. In order to make a fair comparison with the reference work, 160 training epochs are used on the CIFAR-100 dataset and 150 training epochs are used on the ImageNet dataset. ImageNet is based on A hierarchical (currently only noun) organized image database where each node of the hierarchy is described by hundreds or thousands of images. MEST+EM&S was chosen as the training algorithm for weight sparsity because it does not involve any intensive computation, which makes it desirable for edge device scenarios. A uniform unstructured sparsity is applied to all convolutional layers while keeping only the first layer dense.

[0058] Figure 6 A comparison of the accuracy and computational FLOPs results on the CIFAR-100 dataset using ResNet-32 is shown. Each accuracy result is the average of 3 runs. The configuration of the SpFDE framework 10 is represented using x%+y%, where x indicates the target training FLOPs reduction during layer freezing and y is the percentage of training data removed. Compared to recent sparse training methods, the SpFDE framework 10 consistently achieves higher or similar accuracy while significantly reducing training FLOPs. Specifically, at a sparsity ratio of 90%, SpFDE 20％+20％Maintaining similar accuracy to MEST while achieving 27% reduction in training FLOPs. Compared to DeepR, SET, and DSR, SpFDE 25％+25％ A 27% reduction in FLOPs and +1.36% to +4.24% higher accuracy were achieved. More importantly, when SpFDE 25％+25％ At 90% sparsity, both methods have the same training FLOPs (i.e., 0.96) when compared with MEST at 95% sparsity, while SpFDE 25％+25％ with significantly higher accuracy (i.e., +0.66%). Reducing training cost by pushing sparsity to extreme ratios is no longer an ideal approach. The SpFDE framework10 provides a new dimension to reduce training cost while maintaining accuracy.

[0059] Figure 7 Comparison results on the ImageNet dataset using ResNet-50 are provided. At each training FLOPs level, the SpFDE framework10 consistently achieves higher accuracy than existing works. Notably, the SpFDE framework10 outperforms the original MEST in both accuracy and FLOPs savings. The FLOPs savings are attributed to layer freezing and data filtering for end-to-end dataset efficient training. In addition, compared to the one-time dataset shrinkage used in MEST, data filtering dynamically updates the training dataset, thereby alleviating overfitting, leading to higher accuracy.

[0060] Reduce memory costs

[0061] refer to Figure 5 , we can see the superior memory savings of the SpFDE framework 10. The memory cost indicates the memory footprint used during sparse training, including weights, activations, and gradients of weights and activations represented in 32-bit floating points with a batch size of 64 on ResNet-32 using CIFAR-100. "SpFDE Min." represents the training memory cost after all target layers are frozen, while "SpFDE Avg." is the average memory cost during the entire training process. The baseline result of "DST methods Min." only considers the minimum memory cost requirement of the DST method, which ignores memory overheads (such as periodic dense backpropagation in RigL, dense sparse structure search at initialization, and soft memory limits in MEST). Even in this case, "SpFDE Avg." still outperforms "DST methods Min." by a large margin (20% to 25.3%). The “SpFDE minimum” results show that under different sparsity ratios, the minimum memory cost can be reduced by 42.2% to 43.9% compared with the “DST method minimum”. The significant reduction in memory cost is especially important for edge training.

[0062] Figure 8 A sample configuration of a computer system 800 suitable for implementing the framework described herein is shown.

[0063] Specifically, Figure 8 A block diagram of an example of a machine 800 that can implement one or more configurations is shown. In an optional configuration, the machine 800 can be run as a standalone device, or can be connected (e.g., a network) to other machines. In the deployment of the network, the machine 800 can be run in the capacity of a server machine, a client machine, or both in a server-client network environment. In an example, the machine 800 can act as a peer-to-peer (P2P) (or other distributed) peer machine in a network environment. In a sample configuration, the machine 800 can be a personal computer (personal computer, PC), a tablet PC, a set-top box (set-top box, STB), a personal digital assistant (personal digital assistant, PDA), a mobile phone, a smart phone, a network application, a server, a network router, a switch or a bridge, or any machine capable of executing instructions (sequential or other) specifying the actions to be taken by the machine. For example, the machine 800 can be used as a workstation, a front-end server, or a back-end server of a communication system. The machine 800 can implement the method described herein by running software for implementing the features described herein. Further, while a single machine 800 is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0064] As described herein, examples may include or may run on a processor, logic or multiple components, modules or mechanisms (referred to herein as "modules"). A module is a tangible entity (e.g., hardware) that is capable of performing a specified operation and may be configured or arranged in a certain manner. In an example, a circuit may be arranged as a module in a specified manner (e.g., internally or relative to an external entity such as other circuits). In an example, all or part of one or more computer systems (e.g., independent, client or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application portions or applications) as a module for performing a specified operation. In an example, the software may reside on a machine-readable medium. When executed by the underlying hardware of the module, the software causes the hardware to perform the specified operation.

[0065] Thus, the term "module" is understood to include at least one of a tangible hardware or software entity, which may be physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., temporarily) configured (e.g., programmed) to operate in a particular manner or perform part or all of any operations described herein. Considering an example where a module is temporarily configured, each module need not be instantiated at any time. For example, where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor may be configured as corresponding different modules at different times. The software may configure the hardware processor accordingly, for example, to constitute a particular module at one time and to constitute a different module at a different time.

[0066] The machine (e.g., computer system) 800 may include a hardware processor 802 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 804, and a static memory 806, some or all of which may communicate with each other via an interconnect (e.g., a bus) 808. The machine 800 may also include a display unit 810 (shown as a video display), an alphanumeric input device 812 (e.g., a keyboard), and a user interface (UI) navigation device 814 (e.g., a mouse). In an example, the display unit 810, the input device 812, and the UI navigation device 814 may be a touch screen display. The machine 800 may additionally include a mass storage device (e.g., a drive unit) 816, a signal generating device 818 (e.g., a speaker), a network interface device 820, and one or more sensors 822. Example sensors 822 include one or more of a global positioning system (GPS) sensor, a compass, an accelerometer, temperature, light, a camera, a video camera, a physical state or position sensor, a pressure sensor, a fingerprint sensor, a retinal scanner, or other sensors. The machine 800 may include an output controller 824, such as a serial (e.g., universal serial bus (USB)), parallel or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate or control one or more peripheral devices (e.g., printer, card reader, etc.).

[0067] The mass storage device 816 may include a machine-readable medium 826 having stored thereon one or more sets of data structures or instructions 828 (e.g., software) that embody or are utilized by any one or more of the techniques or functionality described herein. The instructions 828 may also reside, in whole or in part, within the main memory 804, within the static memory 806, or within the hardware processor 802 during execution of the instructions 828 by the machine 800. In an example, one or any combination of the hardware processor 802, the main memory 804, the static memory 806, or the mass storage device 816 may constitute a machine-readable medium.

[0068] Although the machine-readable medium 826 is illustrated as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, or at least one of an associated cache and server) configured to store one or more instructions 828. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions executed by the machine 800 and causing the machine 800 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media may include non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; disks, such as internal hard disks and removable hard disks; magneto-optical disks; random access memory (RAM); solid state drives (SSD); CD-ROMs (compact disc read-only memory) and DVD-ROMs (Digital Video Disk-Read Only Memory). In some examples, the machine-readable medium may include non-temporary machine-readable media. In some examples, the machine-readable medium may include machine-readable media that is not a temporarily propagated signal.

[0069] The instructions 828 may also be transmitted or received over the communication network 832 using a transmission medium through the network interface device 820. The machine 800 may communicate with one or more other machines using any of a variety of transmission protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile phone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., a wireless network known as a cellular network). The network interface device 820 may be a wireless communication device that is used to communicate with the network 832. In an example, the network interface device 820 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas 830 to connect to the communication network 832. In an example, the network interface device 820 may include multiple antennas 830 to use at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology for wireless communication. In some examples, the network interface device 820 may use multi-user MIMO technology for wireless communication.

[0070] The features and flowcharts described herein may be embodied as method steps in one or more methods, or in one or more applications as described above. According to some configurations, one or more "applications" are programs that perform the functions defined in the program. Various programming languages ​​may be used to generate one or more applications constructed in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Applications developed with a software development kit (SDK) can be run on mobile operating systems such as IOS TM ANDROID TM , The present invention relates to mobile software on a mobile phone (such as a smartphone or other mobile operating system). In this example, third-party applications can call API calls provided by the operating system to facilitate the functions described herein. These applications can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can alternatively be implemented in dedicated computer hardware or an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0071] The program aspects of the technology can be considered as "products" or "articles of manufacture", which are usually carried on or embodied in a type of machine-readable medium in the form of at least one of executable code or related data. For example, the programming code may include code for touch sensors or other functions described herein. "Storage" type media include any or all tangible memories of computers, processors, etc. or their related modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programming at any time. All or part of the software can sometimes communicate through the Internet or various other telecommunications networks. Such communication, for example, can enable software to be loaded from one computer or processor to another, such as from a service provider's server system or host computer to a computer platform of a smart watch or other portable electronic device. Therefore, another type of medium that can carry programs, media content or metadata files includes optical, electrical and electromagnetic, such as through wired and optical landline networks and through different air links across the physical interface between local devices. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered as media that carry software. As used herein, unless restricted to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0072] Thus, machine-readable media may take the form of tangible storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer, etc., such as may be used to implement client devices, media gateways, transcoders, etc., as shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wire and optical fiber, including the wires that make up a bus within a computer system. Carrier transmission media may take the form of electrical or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs (Digital Video Disks) or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with a pattern of holes, RAMs, PROMs (Programmable Read-Only Memory) and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves that transmit data or instructions, cables or links that transmit such carrier waves, or any other media from which a computer can read at least one program code or data. Many of these forms of computer-readable media may involve transmitting one or more sequences of one or more instructions to a processor for execution.

[0073] Unless otherwise stated, all measurements, values, ratings, positions, quantities, sizes and other specifications listed in this specification (including the appended claims) are approximate values ​​and not exact values. These quantities are intended to have a reasonable range that is consistent with the functions to which they are related and the customary practices in the field to which they belong. For example, unless explicitly stated, parameter values, etc. may vary from the stated amount by as much as ±10%.

[0074] Furthermore, in the foregoing detailed description, various features are grouped together in various examples for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the claims reflect, the subject matter to be protected is less than all features of any single disclosed example. Accordingly, the claims are hereby incorporated into the detailed description, with each claim standing on its own as separately claimed subject matter.

[0075] While what is considered to be the best mode and other examples have been described above, it should be understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in a variety of forms and examples and that they may be applied in a number of applications, only some of which are described herein. The claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. A framework for training a sparse network, the framework comprising a processor, the processor being configured to: Initializing the sparse network, wherein the sparse network has a sparse structure; Actively training all layers of the sparse network using a portion of the training data set including training samples; Performing data screening on the training samples to update the partial training data set; as well as The layers of the sparse network are gradually frozen in a sequential manner to obtain a trained sparse network.

2. The frame according to claim 1, wherein: The processor is further configured to: The partial training data set is obtained by randomly removing a certain proportion of training samples from the entire training data set.

3. The frame according to claim 1, wherein: A layer is frozen only if all layers preceding it are frozen.

4. The frame according to claim 1, wherein: The sparse structure and weight values ​​of the frozen layer remain unchanged.

5. The frame according to claim 4, wherein: All gradients of weights and activations in the frozen layer are eliminated.

6. The frame according to claim 1, wherein: The data filtering reduces the number of training iterations in each epoch.

7. The frame according to claim 1, wherein: The data screening includes circular data screening.

8. The frame according to claim 7, wherein: The circular data screening includes: Randomly selecting a certain proportion of the total training samples of the training data set to create the partial training data set and remove the data set; By removing a plurality of the training samples from the partial training data set and adding the removed training samples to the removed data set, updating the partial training data set for each epoch; and The same number of removed training samples is retrieved from the removed data set, and the retrieved training samples are added back to the partial training data set to keep the total number of training samples in the partial training data set unchanged.

9. The frame according to claim 1, wherein: The processor is configured to actively train the layer by applying dynamic sparse training DST from memory-economical sparse training MEST.

10. The frame according to claim 1, wherein: The processor is configured to combine a layer freeze interval with a DST interval.

11. A method of using a framework having a processor configured to train a sparse network having a sparse structure, the method comprising: Initializing the sparse network; Actively training all layers of the sparse network using a portion of the training data set including training samples; Performing data screening on the training samples to update the partial training data set; as well as The layers of the sparse network are gradually frozen in a sequential manner to obtain a trained sparse network.

12. The method according to claim 11, further comprising: The partial training data set is obtained by randomly removing a certain proportion of training samples from the entire training data set.

13. The method according to claim 11, wherein: A layer is frozen only if all layers preceding it are frozen.

14. The method according to claim 11, wherein: The sparse structure and weight values ​​of the frozen layer remain unchanged.

15. The method according to claim 14, wherein: All gradients of weights and activations in the frozen layer are eliminated.

16. The method according to claim 11, wherein: The data filtering reduces the number of training iterations in each epoch.

17. The method according to claim 11, wherein: The data screening includes circular data screening.

18. A non-transitory computer readable medium storing program code, wherein the program code, when executed, is operable to: cause a processor of a framework to train a sparse network having a sparse structure to perform the following steps: Initializing the sparse network; Actively training all layers of the sparse network using a portion of the training data set including training samples; Performing data screening on the training samples to update the partial training data set; as well as The layers of the sparse network are gradually frozen in a sequential manner to obtain a trained sparse network.

19. The non-transitory computer readable medium of claim 18, wherein: The code can be operated to: cause the processor to obtain the partial training data set by randomly removing a certain proportion of training samples from the entire training data set.

20. The non-transitory computer readable medium of claim 18, wherein: The code is operable to cause the processor to freeze a layer only if all layers preceding the layer are frozen.

Images