Self-adaptive digital-analog hybrid learning method and device for analog in-memory computing, and recording medium thereof

Abstract

The present invention relates to a self-adaptive learning method and device for efficiently training a deep neural network on neuromorphic hardware by utilizing a hybrid architecture in which an analog backbone network and a digital attention block are combined, in order to overcome performance degradation caused by non-ideal characteristics of an analog memory device, and to a recording medium for implementing same.

Description

Self-adaptive digital-analog hybrid learning method and device for analog in-memory computing, and a recording medium thereof

[0001] The present invention relates to a self-adaptive learning method and device for efficiently training a deep neural network on neuromorphic hardware by utilizing a hybrid structure combining an analog backbone network and a digital attention block to overcome performance degradation caused by the non-ideal characteristics of an analog memory device, and a recording medium for implementing the same.

[0002] Analog In-Memory Computing (AIMC) is gaining attention as a promising technology for the energy-efficient acceleration of deep learning workloads. While notable advancements in Deep Neural Networks (DNNs) have recently been made in fields such as computer vision, speech recognition, and robotics, training increasingly large and complex neural networks requires significant computation time and energy costs, posing a significant challenge to sustainability. In particular, optimizing the performance and energy efficiency of AI computing hardware has emerged as a key challenge for AI applications in low-power systems, such as Internet of Things (IoT) devices and edge computing platforms.

[0003] To address these issues, highly optimized digital Application-Specific Integrated Circuits (ASICs) have been developed to accelerate deep learning workloads. While the focus has been on reducing the size and number of required computations through various optimization strategies such as quantization, binarization, and compression, digital implementations still result in significant energy consumption and processing time for large-scale networks.

[0004] As an alternative to existing digital computing technologies, analog in-memory computing (AIMC) systems offer significant advantages that enable low power consumption and parallel computation. The AIMC framework, based on Ohm's law and Kirchhoff's current law, facilitates large-scale parallel matrix-vector multiplications (MVM) by using resistive memory arrays to store weight matrices and applying voltages corresponding to input vector values. Additionally, parallel computation of rank-one cross products across the entire crossbar array is possible through pulse matching techniques and progressive conductance adjustment, achieving a time complexity of approximately O(1). Consequently, AIMC systems are expected to provide significantly faster performance and greater energy efficiency compared to digital alternatives, and are successfully accelerating on-chip inference using pre-trained models.

[0005] However, the practical implementation of on-chip training systems utilizing AIMC faces several challenges due to the non-ideal characteristics of memory devices. One significant challenge is conductance update asymmetry, which severely impacts system performance because the increments and decrements of conductance at a given conductance level are not identical. Furthermore, compared to the high-precision floating-point operations used in standard SGD algorithms, the limited precision of analog devices significantly degrades performance as the number of bits decreases. Implicit variation between devices further complicates the implementation of many algorithmic ideas that assume transformation invariance. While efforts are underway to develop memory devices with symmetric conductance updates, achieving ideal symmetry remains a challenge.

[0006] Several innovative solutions to address the non-ideal characteristics of analog memory devices include the development of specialized on-chip training algorithms. The Tiki-Taka algorithm is an approach that effectively handles asymmetry by utilizing two arrays: a secondary array that records gradient history and a main array that stores weight values. Backpropagation gradients are temporarily updated in the secondary array, while accumulated gradients are periodically updated in the main array. Simulations show that the Tiki-Taka algorithm can achieve performance comparable to ideal devices across various network types and scales, even when using asymmetric device models. However, the requirement to double the analog hardware can result in additional costs.

[0007] Furthermore, the Mixed Precision method is an alternative to address the limited conductance states of analog memory devices and inter-device variability. This approach utilizes analog devices for forward and backward computations while storing weight updates in high-precision FP32 digital memory. This method allows updates to be transmitted as a single-shot pulse when a specific threshold is exceeded, thereby maintaining accuracy while sacrificing some speed and efficiency. As such, on-chip training approaches have been proposed to compensate for non-ideality in AIMC systems, requiring innovations in algorithms, circuits, and especially network systems that are co-designed.

[0008] Furthermore, complementary machine learning techniques such as Knowledge Distillation (KD) have been explored to address hardware implementation challenges. KD can be utilized to mitigate device variability by transferring knowledge from a larger model (teacher) to a smaller model (student). Related research developed a joint solution combining KD and On-line Sparse Adaptation (OSA) to effectively restore inference accuracy in situations involving RRAM variability while guaranteeing minimal domain overhead. Following the advancement of KD, Self-Distillation (SD) has significantly improved the efficiency of deep neural networks by integrating attention-based shallow classifiers. SD improves performance, particularly in resource-constrained environments such as On-Device AI and the Internet of Things (IoT), by streamlining the training process into a single cohesive step where the teacher and student models can be trained together.

[0009] However, existing approaches still have limitations, such as failing to fully overcome the non-ideal nature of analog in-memory computing or requiring additional hardware resources. Therefore, there is a need for a new on-chip training methodology that can effectively overcome the non-ideal nature of analog devices while maintaining hardware efficiency.

[0010] The problem that the present invention aims to solve is to provide a self-adaptive learning method and device that improves the accuracy of on-chip learning through a hybrid structure efficiently combining an analog backbone network and a digital attention block, while optimizing energy efficiency by removing unnecessary digital components during the inference stage, in order to overcome performance degradation problems caused by non-ideal characteristics such as asymmetric conductance updates, limited precision, and inter-device variation of memory devices in existing analog in-memory computing systems.

[0011] An embodiment is a self-adaptive digital-analog hybrid learning method for analog in-memory computing (AIMC) based on a backbone network including an analog memory element and a self-adaptive network including a digital attention block, comprising the steps of: performing on-chip learning using training data with both the backbone network and the digital attention block activated; calculating a loss function based on the output from the backbone network and the output from the digital attention block; and updating parameters of the backbone network using the loss function, wherein the gradient from the digital attention block corrects errors caused by non-ideal hardware characteristics during the learning process of the backbone network.

[0012] In the above-described on-chip learning, when the digital attention block is activated, a self-digestion process is performed so that knowledge is directly transferred from the label (ground truth) to each digital attention block.

[0013] The above-described on-chip learning is performed by activating the digital attention block at predetermined intervals (Ts) according to a sparse strategy.

[0014] The above-described on-chip learning is performed by activating the digital attention block only during the initial learning period (Tw) according to the warm-up strategy, and then deactivating the digital attention block thereafter.

[0015] The above loss function is calculated as a weighted sum of the cross-entropy loss for the output of the backbone network and the cross-entropy loss for each digital attention block output, and the loss for the output of the digital attention block is calculated directly from the label without a self-distillation process to the backbone network.

[0016] The non-ideal characteristics occurring during parameter learning of the above backbone network include at least one of conductance update asymmetry of the analog memory device, a limited number of conductance states, or variation between devices, and the digital attention block guides the backbone network in a direction that adapts to the non-ideal characteristics.

[0017] In the embodiment, an intermediate feature map obtained during the learning process of the backbone network is input into the digital attention block, and the digital attention block performs classification based on the intermediate feature map. The gradient generated from the digital attention block is transmitted to the corresponding connection point of the backbone network to serve as a guide for the learning of the backbone network, and the gradient induces the backbone network to adapt itself to the physical constraints of the analog memory device during the self-digestion process.

[0018] The embodiment further includes the step of deactivating the digital attention block among the backbone network and the digital attention block and performing inference using only the backbone network.

[0019] The present invention achieves a performance improvement of up to 13.1% in CIFAR-10 image classification tasks by using a hybrid structure that combines an analog backbone network and an attachable digital attention block, which supports the gradient from the digital attention block to overcome the non-ideal characteristics of analog hardware during training, significantly reduces power consumption by using only the analog backbone network during inference, and further reduces relative power consumption by about 70% by minimizing the use of a digital processing unit (DPU) through a sparse strategy and a warm-up strategy.

[0020] Figure 1 schematically shows the on-chip training structure of the present invention.

[0021] Figure 2 schematically shows the on-chip inference structure of the present invention.

[0022] Figure 3 shows the communication interface in on-chip training.

[0023] Figure 4 shows the communication interface in on-chip inference.

[0024] FIG. 5 is a diagram illustrating the self-adaptation process according to an embodiment.

[0025] FIG. 6 is a flowchart of a self-adaptive digital-analog hybrid learning method for analog in-memory computing (AIMC) according to one embodiment.

[0026] Figure 7 shows the configuration of a self-adaptive network according to an embodiment.

[0027] FIG. 8 is a block diagram of a computing device implementing a self-adaptive digital-analog hybrid learning method for analog in-memory computing (AIMC) according to one embodiment.

[0028] Embodiments of the present invention will be described in detail below with reference to the drawings. However, detailed descriptions of known functions or configurations that may obscure the essence of the present invention in the following description and the attached drawings are omitted. Additionally, throughout the specification, the term 'comprising' a component means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0029] Additionally, terms such as first, second, etc. may be used to describe various components, but said components should not be limited by said terms. said terms may be used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0030] The terms used in this invention are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "comprising" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0031] Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0032] The present invention is intended to address problems that arise when implementing large-scale neural networks in analog in-memory computing (AIMC) systems. While AIMC offers significant advantages in energy efficiency and parallel computation for deep learning tasks, non-ideal hardware characteristics such as conductance update asymmetry, limited precision, and inter-device variability have hindered effective on-chip learning.

[0033] The present invention is advanced in that it is a self-adaptive network (SAnet) architecture integrating an analog backbone network and a detachable digital attention block inspired by self-distillation techniques. During on-chip learning, analog and digital components work together, and the digital attention block provides a more accurate convergence direction that mitigates non-ideal effects occurring in resistive processing units (RPUs). However, during the inference phase, only the analog backbone is used, which can significantly reduce power consumption.

[0034] The present invention effectively solves the scalability problem of neuromorphic computing systems by enabling neural networks to adapt to the non-ideality of hardware instead of requiring perfect hardware. This approach differs from previous methods such as the Tiki-Taka algorithm or Mixed Precision and provides a more flexible and energy-efficient solution for deploying deep neural networks in edge computing and IoT applications.

[0035] The present invention is a training method proposed to overcome the non-ideal characteristics of memory devices in an analog in-memory computing (AIMC) system. As illustrated in FIG. 1, the present invention utilizes a self-adaptive network (SAnet) combining an analog backbone network and a digital attention block as a core structure. FIGS. 1 to 5 schematically illustrate the system configuration of the present invention.

[0036] Looking at the on-chip training structure shown in Fig. 1, the present invention consists of a backbone network composed of an Analog Residual Block and several Digital Attention Blocks. During the training process, input images are processed through a forward computation path, and intermediate features are processed in the Digital Attention Blocks through a distillation path. This dual-path approach enables the network to effectively adapt to the non-ideal characteristics of analog devices.

[0037] On the other hand, the on-chip inference structure shown in Fig. 2 uses only an analog backbone network without a digital attention block. This significantly improves energy efficiency by removing unnecessary digital components during the inference phase.

[0038] From a hardware perspective, the present invention utilizes both RPUs (Resistive Processing Units) and DPUs (Digital Processing Units) during on-chip training, as illustrated in FIG. 3. The RPU consists of memory-based unit cells in the form of a crossbar array and includes peripheral components (bitline driver, ADC / DAC, digital post-processing unit, controller, communication interface). The DPU is responsible for the computation of digital attention blocks and provides an accurate computing environment without hardware constraints. On the other hand, during on-chip inference shown in FIG. 4, only the RPU is activated to minimize power consumption.

[0039] One of the key features of the present invention is the self-adaptation process illustrated in FIG. 5. The gradient provided by the digital attention block provides a more accurate convergence direction than the gradient of the analog backbone network, thereby mitigating the effect of the non-ideal characteristics of the RPU. This self-adaptation mechanism induces the model to adapt to hardware non-idealities on its own, thereby improving the performance of the deep neural network.

[0040] The present invention presents a new approach that maintains energy efficiency while overcoming the limitations of existing AIMC systems. In particular, it can achieve excellent performance even under various non-ideal characteristics of analog memory devices, such as conductance update asymmetry, limited precision, and inter-device variability. This has been demonstrated through a performance improvement of up to 13.1% and a reduction in power consumption of approximately 70% in CIFAR-10 image classification tasks.

[0041] Ultimately, the present invention is an innovative hybrid approach that combines the energy efficiency of analog in-memory computing with the accuracy of digital computing, and is expected to make a significant contribution to the development of next-generation low-power artificial intelligence hardware.

[0042] The operation process of Analog In-Memory Computing (AIMC) based on a backbone network including an analog memory element and a self-adaptive network including a digital attention block, in which the present invention is implemented, will be explained in more detail below.

[0043] In the present invention, analog in-memory computing (AIMC) is implemented through a self-adaptive network (SAnet) illustrated in FIGS. 1 to 5. The specific operation method of this system is as follows.

[0044] 1. Network structure (see FIGS. 1 to 5)

[0045] The present invention consists of two main components.

[0046] One is an analog backbone network, and the other is a digital attention block.

[0047] First, the analog backbone network consists of analog residual blocks and uses resistive memory devices. The digital attention block is a digital processing unit inspired by the magnetic distillation approach and is connected to various layers of the backbone network.

[0048] Meanwhile, traditional knowledge distillation is the process of transferring knowledge from a large teacher model to a small student model. The teacher model learns first and then transfers that knowledge to the student model. This process requires two separate models.

[0049] In contrast, the self-distillation in the present invention develops this concept by having the teacher and student exist within the same network, attaching shallow classifiers at various depths of the network so that each classifier can perform predictions independently, and configuring the final output and the intermediate classifier output to be learned simultaneously.

[0050] In this invention, this magnetic distillation concept is utilized as follows.

[0051] 1. Location of digital attention blocks

[0052] Digital attention blocks were placed at various depths of the backbone network. This is similar to the concept of placing intermediate classifiers at various depths in autodistillation.

[0053] 2. Direction of Knowledge Transfer

[0054] Unlike conventional magnetic distillation, the present invention "applies distillation loss only from the label to the attention block." This improves the backbone network's ability to adapt to neuromorphic hardware constraints.

[0055] 3. Loss Calculation Method

[0056] The present invention may also apply cross-entropy loss to the output of the backbone network and the outputs of the three digital attention blocks.

[0057] In conclusion, the digital attention block of the present invention borrows the conceptual structure of magnetic distillation (classifiers of varying depths) while being modified to suit the characteristics of analog in-memory computing. This compensates for the non-ideal characteristics of analog hardware and can provide more accurate gradient directions during the learning process.

[0058] 2. Learning and reasoning process (Figs. 1 and 2)

[0059] First, the on-chip learning process (Fig. 1) is as follows.

[0060] When an input image is entered into the system, the data propagates forward through the analog backbone network.

[0061] Intermediate feature maps are passed to digital attention blocks, and the backbone network and each digital attention block output a classification result.

[0062] The loss function calculates the difference between these outputs and the actual labels, and during the backpropagation process, the gradient generated in the digital attention block drives the learning of the backbone network.

[0063] Next, the on-chip inference process (Fig. 2) is as follows.

[0064] During the inference phase, the digital attention block is deactivated, and only the backbone network is activated to generate output from the input image. This significantly improves energy efficiency.

[0065] 3. Hardware Architecture (See Figs. 3 and 4)

[0066] Hardware for on-chip learning (Fig. 3)

[0067] The resistive processing unit (RPU) (10) is a memory-based unit cell configured in a crossbar array and performs analog calculations.

[0068] The digital processing unit (DPU) (20) is a processing element that implements the digital attention block.

[0069] The communication interface (30) supports data exchange between the RPU (10) and the DPU (20).

[0070] Hardware for on-chip inference (Fig. 4)

[0071] During inference, only the RPU (10) is enabled, and the DPU (20) is disabled to reduce power consumption.

[0072] 4. Self-adaptation mechanism (see Fig. 5)

[0073] The self-adaptation mechanism, which is a key feature of the present invention, operates as follows.

[0074] First, the flow of the existing method (existing method (w / o SALMON)) experiences errors during forward and reverse propagation due to the non-ideal characteristics of analog devices, and these errors are amplified during the learning process, inevitably degrading overall performance.

[0075] In contrast, the gradient generated in the digital attention block of the present invention (w / SALMON) provides a more accurate convergence direction to the backbone network. This mitigates the effects caused by the non-ideal characteristics of analog devices (conductance update asymmetry, limited precision, variation between devices), and consequently, the network can adapt to the non-ideal characteristics of the hardware.

[0076] An embodiment of the present invention implemented based on such theoretical foundations is a self-adaptive digital-analog hybrid learning method for analog in-memory computing (AIMC) based on a backbone network including an analog memory element and a self-adaptive network including a digital attention block, comprising the steps of: performing on-chip learning using learning data while both the backbone network and the digital attention block are activated as in the example of FIG. 6 (S10); calculating a loss function based on the output from the backbone network and the output from the digital attention block (S20); and updating the parameters of the backbone network using the loss function (S30), wherein the gradient from the digital attention block corrects errors caused by non-ideal hardware characteristics during the learning process of the backbone network.

[0077] The above-described on-chip learning is performed by activating the digital attention block at predetermined intervals (Ts) according to a sparse strategy.

[0078] In the above-described on-chip learning, when the digital attention block is activated, a self-digestion process is performed so that knowledge is directly transferred from the label to each digital attention block.

[0079] The above-described on-chip learning is performed by activating the digital attention block at predetermined intervals (Ts) according to a sparse strategy, or by activating the digital attention block only during an initial learning period (Tw) according to a warm-up strategy, and then deactivating the digital attention block thereafter.

[0080] The above loss function is calculated as a weighted sum of the cross-entropy loss for the output of the backbone network and the cross-entropy loss for each digital attention block output, and the loss for the output of the digital attention block is calculated directly from the label without a self-distillation process to the backbone network.

[0081] The non-ideal characteristics occurring during parameter learning of the above backbone network include at least one of conductance update asymmetry of the analog memory device, a limited number of conductance states, or variation between devices, and the digital attention block guides the backbone network in a direction that adapts to the non-ideal characteristics.

[0082] The intermediate feature map obtained during the learning process of the backbone network is input into the digital attention block, and the digital attention block performs classification based on the intermediate feature map. The gradient generated from the digital attention block is transmitted to the corresponding connection point of the backbone network to serve as a guide for the learning of the backbone network, and the gradient induces the backbone network to adapt itself to the physical constraints of the analog memory device during the self-digestion process.

[0083] Additionally, the embodiment further includes the step of deactivating the digital attention block among the backbone network and the digital attention block and performing inference using only the backbone network.

[0084] In the embodiment, the configuration of the self-adaptive network is exemplified in FIG. 7, and the operation process is described as follows.

[0085] The Self-Adaptive Network (SAnet) illustrated in Fig. 7 is a hybrid structure combining analog and digital components that enables efficient learning in analog in-memory computing systems. This network is composed of four main components.

[0086] 1. Digital Input Convolution Block (100)

[0087] The digital input convolution block (100) is responsible for the first processing step of the network. In an embodiment, this block performs a 3x3 convolution operation and preserves the spatial dimension (32x32) of the CIFAR-10 image using a stride of 1 and padding of 1. This block serves to extract initial features of the input image.

[0088] 2. Analog Residual Blocks (200)

[0089] Analog residual blocks (200) are a pivotal part of the network and perform matrix-vector multiplication operations using analog memory elements. These blocks, which are the main components of the backbone network, have the following characteristics.

[0090] Analog residual blocks (200) are composed of BasicBlock or Bottleneck blocks according to the Resnet architecture, and Resnet-10, Resnet-18, and Resnet-34 use BasicBlocks containing two 3x3 convolutions.

[0091] Resnet-50 uses a Bottleneck block consisting of a 1x1 convolution, a 3x3 convolution, and another 1x1 convolution.

[0092] In the embodiment, the basic channel starts at 128, and in subsequent layers, it expands to 128, 256, 512, and 1024 channels.

[0093] Analog residual blocks act as feature extractors and capture more abstract features as the network depth increases.

[0094] 3. Digital Classifier Module (300)

[0095] The digital classifier module (300) is the final output layer of the backbone network. This module receives features extracted from the analog residual block (200) and generates a final classification result. For Resnet-10, Resnet-18, and Resnet-34, it receives 512 input features and converts them into 10 output features (CIFAR-10 class), and for Resnet-50, it receives 2048 input features and converts them into 10 output features.

[0096] 4. Digital Attention Blocks (400)

[0097] The digital attention blocks (400) are a feature of the present invention and are inspired by the magnetic distillation architecture. These blocks are located at different depths of the backbone network and consist of an attention module and a shallow classifier module.

[0098] The attention module performs separable convolution (SepConv) on the intermediate feature maps of the backbone network, with a kernel size of 3x3, the first convolution with a stride of 2, the second convolution with a stride of 1 and padding of 1, and operates in the order of depth-wise convolution followed by point-wise convolution, and applies batch normalization and the ReLU activation function after each convolution.

[0099] The shallow classifier receives the output of the attention module and generates a separate classification result.

[0100] In Fig. 7, the filled arrows represent the forward computation path leading from the input image → digital input convolution block → analog residual blocks → digital classifier → final output.

[0101] Also, the hollow arrow indicates the distillation path leading from the intermediate output of the analog residual block → digital attention block → attention output.

[0102] In an embodiment, the digital attention block (400) includes a first attention block connected to the initial part of the network, a second attention block connected to the middle part of the network, and a third attention block connected to the later part of the network.

[0103] In the embodiment, the backbone network and all digital attention blocks are activated during the learning process, and a loss is calculated for the output of the backbone network and the output of each digital attention block. Then, during the backpropagation process, the gradient of the digital attention blocks is passed to the backbone network to induce learning.

[0104] During the inference process, the digital attention block is deactivated, and only the backbone network is activated to generate output from the input image.

[0105] In the embodiment, a hybrid approach is implemented that effectively overcomes the non-ideal characteristics of analog in-memory computing through this configuration and connection relationship, while utilizing accurate gradients provided by digital attention blocks during the learning process and maximizing energy-efficient analog computation during the inference process.

[0106] The following is a detailed explanation of each step.

[0107] Step 1 (S10)

[0108] At this stage, the backbone network and the digital attention block are simultaneously activated. The backbone network is a primary network structure that includes an analog residual block (ResNet-10, 18, 34, 50, etc.). In the embodiment, the digital attention block (400) consists of three (first to third digital attention blocks), and each block targets a different feature map of the backbone network. In this embodiment, the effect of the self-distillation architecture is maximized by simultaneously activating these two components.

[0109] On-chip learning is performed directly within Analog In-Memory Computing (AIMC) hardware, and training data, such as the CIFAR-10 dataset, can be used as input images of size 32×32×3. Additionally, data augmentation techniques (random cropping, horizontal flipping) can be applied to enhance the learning effect. In the embodiment, training can be performed using an SGD optimizer with a batch size of 128, with an initial learning rate set to 0.01 and momentum to 0.9, and the learning rate reduced to a minimum of 0.0001 according to a cosine annealing schedule.

[0110] Phase 2 (S20)

[0111] This process is the process of calculating self-adaptive loss.

[0112] The knowledge distillation of the present invention operates differently from the conventional method.

[0113] First, existing knowledge distillation methods use various types of loss functions, such as cross-entropy loss for hard labels, KL divergence for soft labels, and L2 regularization for feature maps. However, the present invention simplifies hardware implementation by using only a single cross-entropy loss.

[0114] In general knowledge distillation, the outputs of all blocks are ensemble to derive the final result, but SALMON uses only the logits of the backbone network.

[0115] In addition, the method by which the digital attention block in the present invention compensates for losses caused by non-ideal characteristics of an analog device is as follows.

[0116] First, each digital attention block takes an intermediate feature map from the backbone network as input and performs classification. The gradients generated during this process are backpropagated to the backbone network to guide learning. Grad-CAM analysis revealed that gradient clarity improved as more attention blocks were added. This indicates that the digital attention blocks compensate for gradient information that has been blurred by noise from analog devices.

[0117] In the example, the total loss function (L total ) is equal to mathematical formula 1.

[0118]

[0119] Here, y0 is the output of the backbone network, and y1, y2, and y3 are the outputs of the three digital attention blocks. is the cross-entropy loss of a backbone network including analog components, and represents the cross-entropy loss between the predicted yi and the target label y, and is a loss factor used to balance the contributions of different outputs.

[0120] Thus, the total loss is calculated by combining the classification loss for the backbone network output (yo) and the classification loss for the outputs of the three digital attention blocks.

[0121] The total loss calculated in this way (L total ) is used for backpropagation. The gradient of the loss with respect to the trainable parameter θ is calculated by differentiating the loss function with respect to the model parameter (θ), as shown in Equation 2.

[0122]

[0123] In this gradient calculation, the first term (∇θLCE(y0, y)) is the gradient of the loss with respect to the output of the backbone network, and the second term (Σ∇θ(α·LCE(yi, y))) is the gradient of the loss with respect to the outputs of the three digital attention blocks, meaning that the gradient obtained from the digital attention blocks directly contributes to the learning of the backbone network. This gradient serves to correct for errors caused by non-ideal characteristics of the analog components (noise, nonlinearity, restricted conductance state, etc.).

[0124] Stage 3 (S30)

[0125] In the example, at this step, model parameters are updated using stochastic gradient descent (SGD) with the calculated gradient.

[0126] In the example, the initial learning rate was set to 0.01 and the momentum was set to 0.9. The learning rate decreases to a minimum of 0.0001 according to the cosine annealing schedule.

[0127] Meanwhile, in the present invention, during on-chip learning, a self-digestion process is performed when a digital attention block is activated so that knowledge is directly transferred from the label to each digital attention block. This is explained in more detail as follows.

[0128] The self-distillation process in the present invention takes a unique approach different from general knowledge distillation. In traditional knowledge distillation, knowledge is transferred from a teacher model to a student model, whereas in the present invention, each digital attention block learns knowledge directly from the label (ground truth). This can be verified through Equation 1. According to Equation 1, the output (yi) of each digital attention block is directly compared with the actual label (y) to calculate the loss.

[0129] In addition, in the present invention, the backbone network and digital attention blocks are trained simultaneously and each receives a learning signal directly from the same label. This generates a kind of ensemble learning effect, but only the output of the backbone network is used for the final prediction.

[0130] In addition, digital attention blocks use intermediate feature maps extracted from different layers of the backbone network as input. This structure learns feature representations at various levels, thereby helping the backbone network become a more robust feature extractor.

[0131] Meanwhile, the process of directly transferring knowledge to a digital attention block is carried out as follows.

[0132] 1. Feature extraction from forward passes:

[0133] As input data x passes through the backbone network, multiple layers of intermediate feature maps are generated. These intermediate feature maps are each transmitted to the corresponding digital attention blocks (first to third digital attention blocks).

[0134] 2. Independent classification of attention blocks

[0135] Each digital attention block performs a SepConv operation on the input feature map and then generates class predictions (yi) through a digital shallow classifier.

[0136] 3. Directly supervised learning

[0137] The prediction (yi) of each attention block is directly compared with the actual label (y) to calculate the cross-entropy loss (LCE(yi, y)). In this process, knowledge of the label is directly transferred to the attention block. This contrasts with the indirect transfer of knowledge through the soft target of the teacher model in traditional knowledge distillation.

[0138] 4. Knowledge Propagation via Gradients

[0139] The gradient of the loss calculated in each attention block (∇θ(α·LCE(yi, y))) is also transmitted to the backbone network during the backpropagation process. Since this gradient is calculated digitally, it is accurate and is not affected by the non-ideal characteristics of analog devices. Consequently, this clean gradient signal corrects for noise and non-ideal characteristics occurring in the analog part of the backbone network.

[0140] Meanwhile, the embodiment includes two main strategies for the efficient utilization of digital attention blocks: a sparse strategy and a warm-up strategy. These two strategies provide a method to improve the learning performance of analog in-memory computing while optimizing computational efficiency and energy consumption.

[0141] First, the sparse strategy does not use digital attention blocks at every training step, but selectively activates them at predefined periods (Ts).

[0142] The period Ts defines the interval at which the digital attention block is activated. For example, if Ts=10, the digital attention block is activated only at every 10th epoch. Accordingly, when the sparse strategy is activated, the output (y1, y2, y3) of the digital attention block is calculated only when epoch e is a multiple of Ts, and the loss of the digital attention block is included in the loss function only at this time. In other epochs, the digital attention block is deactivated so y1, y2, and y3 are not calculated, and the loss function simply becomes Ltotal = LCE(y0, y).

[0143] This scarcity strategy can significantly reduce the computational burden and energy consumption of digital processing units (DPUs).

[0144] According to experimental results, selecting an appropriate Ts value (e.g., Ts=10 or Ts=25) results in minimal performance degradation compared to continuously using digital attention blocks. In particular, the performance loss of the sparse strategy is even smaller on devices with less severe non-idealistic characteristics.

[0145] Next, the warm-up strategy activates the digital attention block only during the initial stages of learning and then deactivates it.

[0146] The warm-up period Tw defines the number of initial epochs during which the digital attention block is activated. For example, if Tw=50, the digital attention block is activated only for the first 50 epochs.

[0147] When the warm-up strategy is enabled, the outputs (y1, y2, y3) of the digital attention block are computed only when epoch e is less than Tw, and the loss of the digital attention block is included in the loss function only during this period. In epochs after Tw, the digital attention block is disabled and only the backbone network continues to train.

[0148] This warm-up strategy helps the backbone network learn good initial representations through the guidance of digital attention blocks during the initial training phase. Subsequently, computational efficiency is improved as the backbone network continues training independently.

[0149] According to experimental results, selecting an appropriate Tw value (e.g., Tw=20 or Tw=50) results in minimal performance degradation compared to using digital attention blocks throughout the entire training period. In particular, the strong gradient signal provided initially by digital attention blocks helps overcome initial training instability caused by the non-ideal characteristics of analog devices.

[0150] Another embodiment of the present invention relates to a computing device that implements the self-adaptive digital-analog hybrid learning method described above.

[0151] FIG. 8 is a block diagram showing the schematic configuration of the arithmetic unit, which reconstructs the series of configurations described above from the perspective of hardware configuration. Therefore, to avoid duplication of explanation, only an overview focusing on the function and operation of each configuration will be briefly described here.

[0152] The computing device (800) comprises a memory (830) that stores a program (820) coded so that the self-adaptive digital-analog hybrid learning method described above can be read by a computer, and a processor (810) that executes the program.

[0153] Meanwhile, a computer-readable medium according to an embodiment of the present specification includes a computer-readable recording medium and can record instructions / commands / programs for executing a method according to an embodiment of the present specification on a computer. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored.

[0154] Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, computer-readable recording media may be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner. Furthermore, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.

[0155] A computer-readable medium stores one or more instructions, and said instructions, executable by one or more processors, enable the device to perform on-chip learning using training data with both the backbone network and the digital attention block activated, calculate a loss function based on the output from the backbone network and the output from the digital attention block, and update the parameters of the backbone network using said loss function. At this time, the gradient from the digital attention block corrects for errors caused by non-ideal hardware characteristics during the learning process of the backbone network.

[0156] The present invention has been described above with reference to various embodiments. Those skilled in the art will understand that the present invention may be implemented in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

Claims

1. A self-adaptive digital-analog hybrid learning method for analog in-memory computing (AIMC) based on a backbone network including analog memory elements and a self-adaptive network including digital attention blocks, A step of performing on-chip learning using training data with both the backbone network and the digital attention block activated; A step of calculating a loss function based on the output from the backbone network and the output from the digital attention block; and A step of updating the parameters of the backbone network using the above loss function; Includes, A self-adaptive digital-analog hybrid learning method for analog in-memory computing, wherein the gradient from the digital attention block corrects errors caused by non-ideal hardware characteristics during the learning process of the backbone network.

2. In Paragraph 1, The above-described on-chip learning is a self-adaptive digital-analog hybrid learning method for analog in-memory computing, wherein when the digital attention block is activated, a self-digestion process is performed so that knowledge is directly transferred from the label (ground truth) to each digital attention block.

3. In Paragraph 1, The above-described on-chip learning is a self-adaptive digital-analog hybrid learning method for analog in-memory computing, which learns by activating the digital attention block at predetermined intervals (Ts) according to a sparse strategy.

4. In Paragraph 1, The above-described on-chip learning is a self-adaptive digital-analog hybrid learning method for analog in-memory computing, wherein the digital attention block is activated only during an initial learning period (Tw) according to a warm-up strategy, and thereafter the digital attention block is deactivated.

5. In Paragraph 1, A self-adaptive digital-analog hybrid learning method for analog in-memory computing, wherein the above loss function is calculated as a weighted sum of the cross-entropy loss for the output of the backbone network and the cross-entropy loss for each digital attention block output, and the loss for the output of the digital attention block is calculated directly from the label without a self-distillation process to the backbone network.

6. In Paragraph 1, The non-ideal characteristics occurring during parameter learning of the above backbone network include at least one of conductance update asymmetry of analog memory devices, a limited number of conductance states, or variation between devices, and The above digital attention block is a self-adaptive digital-analog hybrid learning method for analog in-memory computing that guides the backbone network in a direction that adapts to the above non-ideal characteristics.

7. In Paragraph 1, The intermediate feature map obtained during the learning process of the above backbone network is input into the above digital attention block, and A self-adaptive digital-analog hybrid learning method for analog in-memory computing, wherein the digital attention block performs classification based on the intermediate feature map, the gradient generated from the digital attention block is transmitted to the corresponding connection point of the backbone network to serve as a guide for learning the backbone network, and the gradient induces the backbone network to adapt itself to the physical constraints of the analog memory device during the self-distillation process.

8. In Paragraph 1, A step of deactivating the digital attention block among the backbone network and the digital attention block and performing inference using only the backbone network; A self-adaptive digital-analog hybrid learning method that further includes 9. A recording medium storing a computer-readable program coded for the self-adaptive digital-analog hybrid learning method described in any one of claims 1 through 7.

10. A self-adaptive digital-analog hybrid learning method in memory that stores a program coded to be computer-readable; and A processor that executes the above program; Includes, The above self-adaptive digital-analog hybrid learning method is, It is implemented in a backbone network including analog memory elements and a self-adaptive network including digital attention blocks, and A step of performing on-chip learning using training data with both the backbone network and the digital attention block activated; A step of calculating a loss function based on the output from the backbone network and the output from the digital attention block; and A step of updating the parameters of the backbone network using the above loss function; Includes, A computing device in which the gradient from the digital attention block corrects errors caused by non-ideal hardware characteristics during the learning process of the backbone network.

11. In Paragraph 10, The above-described on-chip learning is a computing device in which, upon activation of the digital attention block, a self-digestion process is performed so that knowledge is directly transferred from the label (ground truth) to each digital attention block.

12. In Paragraph 10, The above-described on-chip learning is a computing device that learns by activating the digital attention block at predetermined intervals (Ts) according to a sparse strategy.

13. In Paragraph 10, The above-described on-chip learning is a computing device that activates the digital attention block only during an initial learning period (Tw) according to a warm-up strategy, and then deactivates the digital attention block thereafter.

14. In Paragraph 10, A computing device in which the above loss function is calculated as a weighted sum of the cross-entropy loss for the output of the backbone network and the cross-entropy loss for each digital attention block output, and the loss for the output of the digital attention block is calculated directly from the label without a self-distillation process to the backbone network.

15. In Paragraph 10, The non-ideal characteristics occurring during parameter learning of the above backbone network include at least one of conductance update asymmetry of analog memory devices, a limited number of conductance states, or variation between devices, and The above digital attention block is a computing device that guides a backbone network in a direction that adapts to the above non-ideal characteristics.

16. In Paragraph 10, The intermediate feature map obtained during the learning process of the above backbone network is input into the above digital attention block, and A computing device wherein the digital attention block performs classification based on the intermediate feature map, the gradient generated from the digital attention block is transmitted to the corresponding connection point of the backbone network to serve as a guide for learning the backbone network, and the gradient induces the backbone network to adapt itself to the physical constraints of the analog memory device during the self-digestion process.

17. In Paragraph 10, The above self-adaptive digital-analog hybrid learning method is, A step of deactivating the digital attention block among the backbone network and the digital attention block and performing inference using only the backbone network; An arithmetic unit that further includes.

Images