Apparatus and method for on-device artificial intelligence execution using low-power pim memory in mobile and edge environments

Abstract

A method and system for executing on-device artificial intelligence based on Processing-In-Memory (PIM) for mobile and edge environments are disclosed. An on-device artificial intelligence execution system according to one embodiment may include: an offline optimization device for receiving a trained artificial intelligence model and converting the model into a form aligned with the hardware structure of a low-power PIM memory; an on-device runtime engine for performing artificial intelligence inference within a mobile device using the optimized model transmitted from the offline optimization device; an intelligent heterogeneous scheduler included in the on-device runtime engine for analyzing a computation graph according to an artificial intelligence inference request and distributing computations to at least one of the low-power PIM memory or the main processor based on computation characteristics; a PIM native graph compiler for converting computations allocated to the low-power PIM memory by the intelligent heterogeneous scheduler into a dedicated instruction sequence so that they can be executed within the low-power PIM memory; a QoS-based bandwidth arbitrator for controlling memory access of the low-power PIM memory by considering priority memory access requests, such as display updates or communication processing, in relation to the execution of the instruction sequence generated by the PIM native graph compiler; and a low-power PIM memory module configured to perform computations within the memory according to the dedicated instruction sequence.

Description

Technology Field

[0001] The following description relates to a method and system for executing on-device artificial intelligence based on Processing-In-Memory (PIM) for mobile and edge environments. It concerns a method and system for efficiently and stably executing artificial intelligence models on mobile and edge devices by utilizing low-power PIM memory to minimize data movement and performing dynamic scheduling between heterogeneous computational resources and Quality of Service (QoS)-based resource control through an execution structure that combines offline optimization and on-device runtime. Background Technology

[0002] Currently, the operation of artificial intelligence models on mobile and edge devices is primarily carried out through computing systems based on the Von Neumann Architecture. In these conventional technologies, a method is generally used in which computing units, such as a Central Processing Unit (CPU), Graphics Processing Unit (GPU), or Neural Processing Unit (NPU), read weights and input data stored in Low Power Double Data Rate DRAM (LPDDR), which serves as the main memory, load them into registers or caches within the computing unit, and then perform calculations. In this structure, the computing unit and the memory unit are physically separated; while the processing speed of the computing unit has continuously increased to enhance computational performance, the increase in bandwidth of the memory interface has been relatively limited.

[0003] (1) Performance bottleneck due to physical limitations of memory bandwidth

[0004] Recently, massive AI models such as Large Language Models (LLMs) contain billions of parameters, requiring enormous memory bandwidth during the inference process. However, due to physical constraints, LPDDR interfaces used in mobile and edge devices cannot keep up with the rate of increase in processing performance of computing units, leading to frequent idle states where computing units wait for data transmission. This phenomenon is generally referred to as the "Memory Wall" problem. As a result, inference latency for the entire system increases, and the quality of on-device AI services requiring real-time responsiveness deteriorates.

[0005] (2) Excessive power consumption and heat generation problems due to data transfer

[0006] In conventional technologies based on the Von Neumann architecture, it is known that the energy consumed during the process of moving data from memory to the computing unit is tens to hundreds of times greater than the energy consumed during the computation itself. This data-movement-centric architecture causes particularly critical problems in mobile environments where battery capacity is limited. Running large-scale AI models leads to rapid battery drain, and repetitive memory access increases chipset heat, frequently resulting in thermal throttling for system protection. Consequently, stable, high-performance inference over extended periods becomes impossible, and serious limitations are imposed on the user experience, such as reduced usage time of mobile devices.

[0007] (3) Absence of a software stack that reflects PIM hardware characteristics

[0008] Recently, Processing-In-Memory (PIM) hardware capable of performing computations within memory has been proposed, but conventional artificial intelligence frameworks have limitations in that they perceive such PIM hardware merely as a simple storage device. Consequently, they fail to effectively utilize the channel and bank-level parallelism provided by PIM and continue to rely on methods that export data to external computing devices to perform computations. Furthermore, due to the lack of preprocessing tools and runtime control technologies to lighten AI models or rearrange memory layouts in advance to fit the physical structure of PIM hardware, there is a problem where even the installation of expensive PIM hardware does not lead to actual performance improvements and hardware resources are used inefficiently.

[0009] As such, conventional AI execution technologies in mobile and edge environments face complex limitations, including ① performance bottlenecks caused by insufficient memory bandwidth, ② excessive power consumption and heat generation issues due to data movement, and ③ the absence of a software stack that does not reflect the structural characteristics of PIM hardware. Therefore, a new type of on-device AI execution framework is required that can generate binary and tensor metadata aligned with the PIM hardware structure in the offline phase, verify and enforce zero-copy PIM operations based on this in the runtime phase, and guarantee the system's Quality of Service (QoS) even when priority requests such as display or communication occur.

[0010] [Prior Literature Number]

[0011] Korean Patent Publication No. 10-2024-0004361 The problem to be solved

[0012] We provide a method and system for executing on-device artificial intelligence based on Processing-In-Memory (PIM) for mobile and edge environments. means of solving the problem

[0013] A system for executing on-device artificial intelligence using low-power Processing-In-Memory (PIM) memory in mobile and edge environments, comprising: an offline optimization device for receiving a trained artificial intelligence model and converting the model into a form aligned with the hardware structure of the low-power PIM memory; an on-device runtime engine for performing artificial intelligence inference within a mobile device using the optimized model transmitted from the offline optimization device; an intelligent heterogeneous scheduler included in the on-device runtime engine for analyzing a computation graph according to an artificial intelligence inference request and distributing computations to at least one of the low-power PIM memory or a main processor based on computation characteristics; a PIM native graph compiler for converting computations allocated to the low-power PIM memory by the intelligent heterogeneous scheduler into a dedicated instruction sequence so that they can be executed within the low-power PIM memory; and a QoS-based bandwidth arbitrator for controlling memory access of the low-power PIM memory by considering priority memory access requests, such as display updates or communication processing, in relation to the execution of the instruction sequence generated by the PIM native graph compiler. The present invention provides an on-device artificial intelligence execution system comprising a low-power PIM memory module configured to perform operations within memory according to the above-described dedicated instruction sequence, wherein the on-device runtime engine links the intelligent heterogeneous scheduler, the PIM native graph compiler, and the QoS-based bandwidth arbitrator to control the main processor and the low-power PIM memory to perform artificial intelligence operations in parallel.

[0014] According to one aspect, the offline optimization device may be characterized by including: a sensitivity analysis unit for analyzing output sensitivity according to weight change; a precision determination unit for determining computational precision per layer based on the result of the sensitivity analysis; and a quantization execution unit for quantizing model parameters according to the determined precision.

[0015] According to another aspect, the precision determining unit may be characterized by assigning a first precision format to some layers of the artificial intelligence model and a second precision format to the remaining layers.

[0016] According to another aspect, the offline optimization device may be characterized by including a memory layout prepacker for rearranging logical tensor addresses to correspond to the physical address structure of the low-power PIM memory.

[0017] According to another aspect, the memory layout prepacker may be characterized by interleaving data so that consecutive operation data is cross-placed across multiple memory banks so that the data is not concentrated in a single memory bank.

[0018] According to another aspect, the memory layout prepacker may be characterized by generating a metadata header including a tensor identifier, a physical base address, an offset, a stride, bank mapping information, and a tile size.

[0019] According to another aspect, the PIM native graph compiler may be characterized by ensuring that zero-copy execution is performed by verifying that the operand address of the generated private instruction sequence falls within the internal address range of the low-power PIM memory.

[0020] According to another aspect, the PIM native graph compiler may be characterized by dividing computational data into tile units based on the internal buffer capacity of the low-power PIM memory.

[0021] According to another aspect, the PIM native graph compiler may be characterized by fusing matrix multiplication operations and subsequent activation operations or element-wise operations into a single kernel to omit the storage of intermediate results in external memory.

[0022] According to another aspect, the QoS-based bandwidth arbitrator may be characterized by determining priority based on at least one of the requesting party, the request type, or the time of the request of a memory access request.

[0023] According to another aspect, the QoS-based bandwidth arbitrator may be characterized by restricting or delaying memory access to the low-power PIM memory when a display update request is detected.

[0024] According to another aspect, the QoS-based bandwidth arbitrator may be characterized by adjusting the operating frequency or instruction issuance cycle of the low-power PIM memory to reduce memory bus occupancy.

[0025] According to another aspect, the QoS-based bandwidth arbitrator may be characterized by pausing operations being performed in the low-power PIM memory when bandwidth shortage exceeds a threshold, and resuming operations from the same point after the priority request ends.

[0026] According to another aspect, under the control of the QoS-based bandwidth arbitrator, the low-power PIM memory module may be characterized by storing an execution context, including a program counter, internal register status, tile index, or partial sum information, in the internal registers or reserved internal memory area of ​​the low-power PIM memory module when an operation is paused.

[0027] According to another aspect, the intelligent heterogeneous scheduler may be characterized by dynamically adjusting the operation distribution ratio between the low-power PIM memory and the main processor, the number of active banks, or the operating frequency based on the remaining battery level, temperature, or priority of the running application.

[0028] A system for executing on-device artificial intelligence using low-power Processing-In-Memory (PIM) memory in mobile and edge environments, comprising: an on-device runtime engine for performing artificial intelligence inference by receiving an optimized artificial intelligence model aligned with the hardware structure of the low-power PIM memory; an intelligent heterogeneous scheduler included in the on-device runtime engine for analyzing a computation graph in response to an artificial intelligence inference request and distributing computations to at least one of the low-power PIM memory or a main processor based on computation characteristics; a PIM native graph compiler for converting computations allocated to the low-power PIM memory by the intelligent heterogeneous scheduler into a dedicated instruction sequence so that they can be executed within the low-power PIM memory; and a QoS-based bandwidth arbitrator for controlling memory access of the low-power PIM memory by considering priority memory access requests, such as display updates or communication processing, in relation to the execution of the instruction sequence generated by the PIM native graph compiler. The present invention provides an on-device artificial intelligence execution system comprising a low-power PIM memory module configured to perform operations within memory according to the above-described dedicated instruction sequence, wherein the on-device runtime engine links the intelligent heterogeneous scheduler, the PIM native graph compiler, and the QoS-based bandwidth arbitrator to control the main processor and the low-power PIM memory to perform artificial intelligence operations in parallel.

[0029] A method for executing on-device artificial intelligence using low-power PIM memory in mobile and edge environments, comprising: receiving an optimized artificial intelligence model aligned with the hardware structure of the low-power PIM memory into an on-device runtime engine; when an artificial intelligence inference request is received, an intelligent heterogeneous scheduler analyzes a computation graph corresponding to the artificial intelligence model to determine computation characteristics; based on the determined computation characteristics, distributing at least a portion of the computations of the computation graph to at least one of the low-power PIM memory or the main processor; for the computations distributed to the low-power PIM memory, a PIM native graph compiler converts them into a dedicated instruction sequence that can be executed within the low-power PIM memory; prior to the execution of the dedicated instruction sequence, a QoS-based bandwidth arbitrator controls memory access of the low-power PIM memory by considering priority memory access requests such as display updates or communication processing; and, according to the dedicated instruction sequence, the low-power PIM memory module performs computations within the memory and generates a final inference result together with the computation result performed by the main processor.

[0030] According to one aspect, the step of converting to the dedicated instruction sequence may be characterized by supporting zero-copy execution so that operations are performed without moving data to external memory by verifying that the operand address of the dedicated instruction sequence falls within the internal address range of the low-power PIM memory.

[0031] According to another aspect, if the QoS-based bandwidth arbitrator determines that the memory bus bandwidth shortage exceeds a threshold, it may pause the operation being performed in the low-power PIM memory and, after the priority memory access request is terminated, resume the operation from the same point based on the stored execution context.

[0032] A computer program stored on a non-transient computer-readable recording medium is provided, which is combined with a computer device and executes the above method on the computer device.

[0033] A non-transient computer-readable recording medium is provided on which a computer program for executing the above method on a computer device is recorded. Effects of the invention

[0034] We can provide a PIM (Processing-In-Memory) based on-device artificial intelligence execution method and system for mobile and edge environments.

[0035] According to embodiments of the present invention, by combining low-power PIM memory internal computation with intelligent heterogeneous scheduling, memory bandwidth bottlenecks occurring in mobile environments can be alleviated and data movement minimized, thereby simultaneously improving the processing speed and energy efficiency of on-device artificial intelligence inference. Furthermore, through a zero-copy execution structure and PIM-aware offline optimization, various artificial intelligence models can be efficiently deployed without hardware modification, and a stable multitasking environment can be maintained while preventing conflicts with high-priority tasks such as display updates or communication through QoS-based bandwidth arbitration and context-aware resource management. Accordingly, the present invention can provide an optimal operating environment that balances performance, power efficiency, and user experience in commercial on-device artificial intelligence services requiring real-time responsiveness. Brief explanation of the drawing

[0036] FIG. 1 is a diagram illustrating an example of the configuration of a Processing-In-Memory (PIM) based on-device artificial intelligence execution system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a core technology state diagram for a PIM-based on-device artificial intelligence execution system according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating step-by-step examples of detailed processing procedures for a PIM cognitive quantizer according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating step-by-step examples of detailed processing procedures for a memory layout prepacker according to an embodiment of the present invention. FIG. 5 is a flowchart illustrating, step-by-step, an example of a detailed processing procedure of an intelligent heterogeneous scheduler according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating step-by-step examples of detailed processing procedures of a PIM native graph compiler according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating step-by-step examples of detailed processing procedures for a Quality of Service (QoS) based bandwidth arbitrator according to an embodiment of the present invention. FIG. 8 is a block diagram illustrating an example of the detailed configuration of an offline optimization device according to an embodiment of the present invention. FIG. 9 is a block diagram illustrating an example of the detailed configuration of a PIM native graph compiler according to an embodiment of the present invention. FIGS. 10 and FIGS. 11 are drawings illustrating examples of AI inference and PIM acceleration processing sequence diagrams according to an embodiment of the present invention. FIGS. 12 and FIGS. 13 are drawings illustrating examples of system resource management and QoS linkage sequence diagrams according to an embodiment of the present invention. FIG. 14 is a flowchart illustrating an example of a PIM-based on-device artificial intelligence execution method according to an embodiment of the present invention. FIG. 15 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Specific details for implementing the invention

[0037] Hereinafter, embodiments will be described in detail with reference to the attached drawings.

[0038] Embodiments of the present invention may provide an execution framework in which an offline optimization phase, an on-device runtime phase, and a system resource management phase are organically combined to enable low-power and high-performance artificial intelligence inference in mobile and edge environments. This phased configuration can be designed to maintain stable execution and Quality of Service (QoS) while minimizing data movement by considering both the physical and structural characteristics of Processing-In-Memory (PIM) hardware and the constraints of the mobile system environment.

[0039] The offline optimization step according to embodiments of the present invention is a preprocessing step for converting a general-purpose artificial intelligence model into a form suitable for the structure of low-power PIM memory hardware, and can be performed in a server or high-performance computing environment. In this step, a high-precision artificial intelligence model that has completed training may be input, and said model may be converted into an optimized binary form aligned with the data format and memory structure supported by the internal computation unit of the low-power PIM memory.

[0040] Specifically, PIM cognitive quantization can convert weights into low-precision data formats such as MXFP4 or MXFP8, which are supported by the multiplication-accumulation unit within the PIM. At this time, a mixed-precision technique can be applied by analyzing the sensitivity of each layer of the neural network and applying high precision to layers with relatively large accuracy loss and low precision to layers with low sensitivity. This allows for the reduction of memory usage and computational costs while maintaining the accuracy of the model.

[0041] In addition, memory layout prepacking can store weight data in advance by interleaving it according to the multi-bank structure of PIM memory at compile time to eliminate data rearrangement overhead that may occur at runtime. The optimized binary generated during this process may also include a metadata header containing a physical address map per tensor, stride information, bank placement information, and an internal PIM address range, and the metadata header can be used later at runtime for generating zero-copy commands and validating addresses.

[0042] The on-device runtime step according to embodiments of the present invention is a step of performing artificial intelligence inference on a mobile or edge device, and can provide an execution environment that minimizes the intervention of a central processing unit and utilizes low-power PIM memory as a primary computational resource. In this step, the optimization binary and metadata generated in the offline optimization step may be loaded, and the artificial intelligence computation graph may be dynamically distributed between the PIM, the central processing unit, or the neural network processing unit.

[0043] To this end, the runtime engine can divide the entire computation graph into subgraphs by distinguishing between computations with high memory bandwidth dependency and computationally intensive computations. For example, computations with high memory access frequency can be assigned to the PIM, while computationally intensive computations can be assigned to the central processing unit or neural network processing unit.

[0044] Additionally, subgraphs allocated to PIM can be converted into PIM native command sequences through a zero-copy command generation process. The commands generated at this stage can be verified to ensure that operand addresses cycle only within the PIM internal address range; this eliminates unnecessary data copying between the host and memory and reduces latency and energy consumption associated with data movement.

[0045] The system resource management step according to embodiments of the present invention can provide a control function to prevent overall system performance degradation caused by PIM operations in a mobile environment where multiple applications are executed simultaneously. To this end, an intelligent heterogeneous scheduler may be provided, and the scheduler can monitor the load status of the central processing unit, graphics processing unit, neural network processing unit, and PIM, respectively, and the idle status of the memory channel in real time.

[0046] The intelligent heterogeneous scheduler can calculate a cost function using computational characteristics and system state information as input, and can dynamically reallocate workloads to PIM or other computational resources based on the calculation result.

[0047] Additionally, through a QoS-based bandwidth arbitration function, throttling can be performed to adjust the command issue cycle or operating frequency of the PIM operation when high-priority memory access requests, such as display updates or communication, occur. If necessary, the PIM operation may be paused after saving the execution context, which includes the program counter, registers, tile index, and partial sum, and resumed from the same point after the priority request ends.

[0048] Embodiments of the present invention may operate by including a static optimization step, a dynamic execution step, and a QoS-based adaptive control step. In the static optimization step, an optimized binary aligned with the PIM structure may be generated through PIM-aware quantization and memory layout prepacking, and in the dynamic execution step, in-memory operations may be performed through heterogeneous resource scheduling and zero-copy commands. Subsequently, in the QoS-based adaptive control step, PIM operations may be throttled or paused and resumed to maintain system stability.

[0049] FIG. 1 is a diagram illustrating an example of the configuration of a PIM-based on-device artificial intelligence execution system according to an embodiment of the present invention. The system illustrated in FIG. 1 may be configured to include an offline optimization area that optimizes an artificial intelligence model in an offline environment to be suitable for a low-power PIM hardware structure, an on-device execution area that executes the optimized model on a mobile or IoT device, and a hardware layer that supports the same.

[0050] The PIM cognitive quantizer (110) according to the present embodiment can receive a large-capacity artificial intelligence model that has been trained in a server or high-performance computing environment and perform the function of converting it into a low-precision data format supported by low-power PIM hardware. The input artificial intelligence model may include parameters in a high-precision (FP32) format, and the PIM cognitive quantizer (110) can convert it into a low-precision format such as MXFP4 or MXFP8.

[0051] At this time, the PIM cognitive quantizer (110) can automatically determine the quantization level with the least loss of accuracy by analyzing the distribution characteristics and sensitivity of parameters for each neural network layer, rather than simply reducing the number of bits in a uniform manner. In this process, a mixed precision technique may be applied, and data formats of different precision may be selected for each layer.

[0052] In addition, to minimize quantization noise that may occur during the conversion process, a calibration operation can be performed by comparing the output values ​​of the original model and the quantized model. Furthermore, by applying a block-wise scaling technique optimized for the MAC structure within the PIM, numerical stability can be ensured while reducing hardware implementation complexity.

[0053] Finally, the PIM cognitive quantizer (110) can generate a lightweight model with mixed precision and block-unit scaling applied while complying with hardware constraints and transmit it to the memory layout prepacker (120). At this time, the layer-specific precision map (MXFP4 / MXFP8), block scaling factor, and correction result are recorded in a binary header or as separate metadata, which can then be used for decoding and command parameter setting in the runtime stage.

[0054] The memory layout prepacker (120) according to the present embodiment can pre-pack weight data to fit the physical structure of the low-power PIM memory module during the compilation stage in order to eliminate data rearrangement overhead that may occur during the runtime stage.

[0055] The memory layout prepacker (120) can generate a bank interleaving pattern that cross-stores consecutive data blocks in different banks to enable parallel processing by utilizing the multi-bank structure of the low-power PIM memory. In addition, memory addresses can be allocated by considering data locality so that input data and weights required for PIM operations are located within the same bank group.

[0056] This static optimization process can drastically reduce data loading time during on-device inference and ensure that the on-device PIM runtime engine can start operations immediately without separate data shuffling. The optimized binary generated by the memory layout prepacker (120) can be distributed to a mobile device and loaded by an intelligent heterogeneous scheduler (310).

[0057] Additionally, the memory layout prepacker (120) can generate a metadata header including a tensor ID, physical base address, stride, bank and channel mapping information, internal address range, and scale factor location along with a bank interleaving pattern, and the metadata header can be provided for zero-copy command generation and address validation at the runtime stage.

[0058] The intelligent heterogeneous scheduler (310) according to the present embodiment can analyze the computation graph of an artificial intelligence model running within a mobile device and dynamically allocate each computation node to an optimal processor by considering the resource status of the system. For each node of the computation graph, the arithmetic intensity can be calculated as the Ops / Byte value obtained by dividing the amount of computation (Ops) by the amount of memory access (Bytes).

[0059] The intelligent heterogeneous scheduler (310) can classify memory-bound layers with frequent memory access relative to the amount of computation and computationally intensive compute-bound layers based on the arithmetic operation intensity. For example, layers with a high proportion of memory access, such as the Fully Connected Layer, can be allocated to a low-power PIM memory module, and computationally intensive layers, such as large matrix multiplication (MatMul), attention, and some convolution (Convolution), can be allocated to a main processor (CPU or NPU).

[0060] In addition, the intelligent heterogeneous scheduler (310) can monitor the load and heat status of each processor in real time and perform a load balancing function to reallocate tasks to other idle resources when an overload occurs on a specific processor. This allows for maintaining a balance between power efficiency and computational performance even in a limited mobile resource environment.

[0061] The PIM native graph compiler (320) according to the present embodiment receives a subgraph assigned to the PIM from an intelligent heterogeneous scheduler (310) and converts it into a dedicated instruction sequence that the PIM hardware can understand.

[0062] The PIM native graph compiler (320) can apply a tiling algorithm that divides large matrix operations into tiles of a manageable size, taking into account the limited buffer size within the PIM. Additionally, to increase memory bandwidth efficiency, it can prevent the generation and storage of intermediate data by performing operator fusion, which combines multiple consecutive operations into a single kernel.

[0063] In particular, the PIM native graph compiler (320) may generate a zero-copy command only after verifying, by referencing the metadata header generated by the memory layout prepacker (120), that the operand address in the zero-copy enforcement module does not exceed the PIM internal address range. If the verification fails, a tile redistribution or a fallback to the CPU / NPU path may be selected.

[0064] The generated PIM native instruction sequence can be passed to the hardware layer and executed through the low-power PIM memory integration driver (340).

[0065] The QoS-based bandwidth mediator (330) according to the present embodiment can monitor memory access requests from other system modules, such as a display, modem, and camera, in real time while PIM operations are being performed. In order to prevent screen tearing or communication delays, if a memory access request from a high-priority master, such as a display controller, is detected, the priority of the PIM operation can be temporarily lowered or control can be made to selectively utilize only the idle period where memory access does not occur.

[0066] At this time, the selection of idle banks can be performed based on the active status bits per bank group or the access count during the last N cycles, thereby minimizing conflicts with priority traffic. Additionally, the QoS-based bandwidth arbitrator (330) can apply a throttling policy to ensure that the memory bandwidth occupancy used by the PIM does not exceed a threshold, and throttling can be performed by adjusting the PIM operating frequency or the command issuance cycle.

[0067] In the event of an emergency interrupt, the QoS-based bandwidth arbitrator (330) may perform preemption by saving the operation context, including the program counter, registers, tile index, and partial sum of the PIM operation currently being performed, then stopping the command queue and immediately returning control of the memory bus. After the priority request is terminated, the saved context may be restored to resume the operation from the same point. The arbitration policy may be set based on a bandwidth occupancy threshold and a maximum allowable delay time for priority requests, and hysteresis may be applied to the resumption condition to prevent frequent interruptions and resumptions.

[0068] The offline optimization area and the on-device execution area may be implemented on the same device or distributed across different devices, and may include a structure in which optimized binaries and metadata are transmitted over a network.

[0069] FIG. 2 is a diagram illustrating an example of a core technology state diagram for a PIM-based on-device artificial intelligence execution system according to an embodiment of the present invention. The state diagram according to the embodiment of FIG. 2 shows the entire lifecycle from when an artificial intelligence model is input until a result is output, and the changes in the system's state in stages.

[0070] As illustrated in FIG. 2, the system can be started in an initial standby state (S100). The standby state (S100) corresponds to an idle state before an AI model input request is received, and when an AI model input request is received from the outside, the state transition can be made to a preprocessing and optimization step (S200).

[0071] The preprocessing and optimization step (S200) is an offline preprocessing step and may include a step of analyzing a computation graph for an input artificial intelligence model (S210), a PIM quantization step (S220) that converts the model into a data format supported by PIM hardware, and a step of rearranging the memory layout to fit the PIM memory structure (S230). In this step, an optimized binary with MXFP series format conversion and bank interleaving applied may be generated.

[0072] When the preprocessing and optimization step (S200) is completed, the optimized binary is loaded and the system can transition to an execution-ready state (S300). The execution-ready state (S300) is a state in which initialization for runtime execution is completed, and can subsequently transition to a runtime execution state (S400) in which the runtime execution engine is activated.

[0073] In the runtime execution state (S400), a workload scheduling step (S410) may be performed, and in this step, each computation node of the computation graph may be allocated to heterogeneous resources according to the computation characteristics. Computations with frequent memory access may be branched to the PIM accelerated execution state (S500), and computationally intensive computations may be branched to the legacy execution state (S600).

[0074] In the PIM accelerated execution state (S500), an instruction generation step (S510) for performing internal PIM operations may be performed, and an internal operation execution step (S520) in which MAC operations are executed internally within the PIM according to the generated instructions may be performed. On the other hand, in the legacy execution state (S600), operations using a CPU or NPU may be performed.

[0075] During runtime execution, if an urgent memory access request is detected from a display, communication module, etc., or if the memory bandwidth occupancy of the PIM exceeds a preset threshold, the system may transition to a QoS arbitration state (S700). In the QoS arbitration state (S700), memory bandwidth monitoring (S710) may be performed, and if necessary, an operation pause step (S720) may be performed to temporarily suspend the currently running PIM operation.

[0076] In the event that preemption occurs, the system may save an execution context including a program counter, internal registers, tile coordinates, and a partial sum storage location to resume the operation later. When the priority memory request is processed and the condition is released, the system returns to the PIM accelerated execution state (S500) and can resume the operation based on the saved context.

[0077] When all operations are completed, the system transitions to a result output state (S800) to output the inference results of the artificial intelligence model externally, and then returns to a waiting state (S100).

[0078] Accordingly, the state diagram illustrated in FIG. 2 indicates that an artificial intelligence execution system according to embodiments of the present invention can perform an entire operation flow including preprocessing, runtime execution, heterogeneous resource utilization, and QoS-based arbitration in stages.

[0079] FIG. 3 is a flowchart illustrating, step-by-step, an example of a detailed processing procedure of a PIM cognitive quantizer according to an embodiment of the present invention. The process illustrated in FIG. 3 may represent a process of lightweighting a trained high-precision artificial intelligence model so that it can be efficiently executed on low-power PIM hardware.

[0080] According to embodiments of the present invention, the PIM cognitive quantizer (110) may receive as input an artificial intelligence model of trained FP32 precision, a calibration dataset, and hardware specifications including data format information supported by PIM hardware. As output, it may generate a lightweight model converted into MXFP format and a scaling factor used for quantization.

[0081] As illustrated in FIG. 3, the quantization process may begin with a layer-by-layer sensitivity analysis step (S111). In the layer-by-layer sensitivity analysis step (S111), artificial noise is injected into each layer of the input artificial intelligence model, and the quantization sensitivity can be calculated by analyzing the change in output values ​​before and after the noise injection. At this time, the layer-by-layer sensitivity can be quantified by calculating the difference in output for the correction dataset as the Mean Squared Error (MSE) or the Kullback-Leibler divergence increment.

[0082] Subsequently, a format allocation step (S112, S113) may be performed. If the sensitivity per layer is above a preset threshold, an MXFP8 format providing relatively high precision may be allocated (S112), and if the sensitivity is below the threshold, an MXFP4 format with high compression ratio may be allocated (S113). Through this, a mixed precision policy that applies different precisions depending on the layer characteristics may be established.

[0083] After the format allocation is completed, a block-unit quantization step (S114) may be performed. In the block-unit quantization step (S114), 32 or 64 weights may be grouped into a single block according to the MAC operator structure within the PIM, for example, and quantization may be performed by extracting a shared exponent for the block. This block-unit quantization can improve computational efficiency while reducing hardware implementation complexity.

[0084] Next, a correction data-based error measurement step (S115) may be performed. In this step, an error between the output of the quantized model and the output of the original FP32 model may be measured using a correction dataset, and the error may be used as an input for a subsequent fine-tuning step.

[0085] Subsequently, a fine-tuning step of the scaling factor (S116) may be performed. In the fine-tuning step (S116), the scaling factor can be optimized to minimize the Kullback-Leibler divergence between the output of the original FP32 model and the output of the quantized model. This minimizes the degradation of accuracy caused by quantization.

[0086] The mixed precision policy, layer-specific MXFP4 or MXFP8 allocation information, and block-unit scaling factor generated through the above process can be recorded as metadata, and said metadata can subsequently be used for model decoding and PIM command parameter setting in the on-device PIM runtime engine. Finally, through the process illustrated in FIG. 3, a quantization model can be generated that satisfies PIM hardware constraints while maintaining accuracy.

[0087] FIG. 4 is a flowchart illustrating stepwise an example of a detailed processing procedure of a memory layout prepacker according to an embodiment of the present invention. The process illustrated in FIG. 4 may represent an offline preprocessing process for converting logical tensor addresses into physical addresses of low-power PIM memory to maximize parallelism and eliminate data rearrangement overhead at runtime.

[0088] According to embodiments of the present invention, a memory layout prepacker (120) may receive, as input, a quantized model and an address map of a low-power PIM memory, such as rank, bank, row, and column information. As output, it may generate a PIM binary with bank interleaving applied and a metadata header that can be referenced at runtime.

[0089] As illustrated in FIG. 4, the processing procedure may begin with a tensor shape parsing step (S121). In the tensor shape parsing step (S121), height, width, and channel information for the model's weight matrix is ​​analyzed to determine whether the tensor matches the PIM internal page size and memory access alignment constraints. This allows for determining the physical address placement strategy in subsequent steps.

[0090] Subsequently, a bank / channel structure mapping step (S122) may be performed. In the bank / channel structure mapping step (S122), the number of bank groups that can be simultaneously activated and the channel parallelism are determined to calculate the maximum data unit capable of parallel processing. The result of this calculation can be used as a criterion for weight block partitioning and interleaving pattern generation.

[0091] Next, a weight block partitioning step (S123) may be performed. In the weight block partitioning step (S123), weight data may be partitioned into blocks of a size suitable for PIM operations based on the analyzed tensor shape and bank / channel structure mapping results. This block partitioning may be determined by considering the processing unit of the PIM internal arithmetic unit and memory access efficiency.

[0092] Next, a cross-bank storage step (S124) may be performed. In the cross-bank storage step (S124), an interleaving method may be applied to physically distribute weight blocks so that different banks are accessed in consecutive operations. This prevents bank conflicts that may occur from consecutive access to the same bank and maximizes multi-bank parallelism.

[0093] Next, a binary header generation step (S125) may be performed. In the binary header generation step (S125), a metadata header including a tensor ID, physical base address, offset and stride information, bank-channel-bank group mapping information, tile size, PIM internal address range, and scaling factor location may be generated so that the runtime engine can recognize the memory layout and verify zero-copy address constraints.

[0094] According to embodiments of the present invention, an on-device PIM runtime engine may verify whether the source (src) and destination (dst) addresses of a command are outside the PIM internal address range or violate page alignment or bank alignment constraints by referring to the metadata header. If a constraint violation is detected as a result of the verification, the runtime may perform tile re-partitioning or padding and, if necessary, select a fallback path using a CPU or NPU.

[0095] Through the process of FIG. 4 as described above, the memory layout prepacker (120) can generate an optimized PIM binary that does not require data rearrangement at runtime, and as a result, can minimize data movement overhead and improve PIM-based parallel computation efficiency.

[0096] FIG. 5 is a flowchart illustrating stepwise an example of a detailed processing procedure of an intelligent heterogeneous scheduler according to an embodiment of the present invention. The process illustrated in FIG. 5 may represent a process of determining which processor among the CPU, NPU, or PIM to allocate a task to, taking into account the characteristics of the computational task and the state of system resources, when an artificial intelligence inference request occurs in a runtime environment.

[0097] According to embodiments of the present invention, an intelligent heterogeneous scheduler (310) may receive an inference request, a subgraph node constituting a computation graph, and current system resource status information as inputs. The system resource status information may include the load, temperature, and available resource status of each processor. As outputs, it may generate information regarding a target device ID to which a task will be assigned and an execution queue to which the task will be inserted.

[0098] As illustrated in FIG. 5, the scheduling procedure may begin with an operation node characteristic analysis step (S311). In the operation node characteristic analysis step (S311), an arithmetic intensity can be calculated for each node of the operation graph. The arithmetic intensity can be defined as the value obtained by dividing the operation amount per node (Ops) by the memory access amount (Bytes) (Ops / Byte), and through this, it is possible to determine whether the operation is a memory-bound operation where a bottleneck occurs due to memory bandwidth, or a compute-bound operation where a bottleneck occurs due to computational performance.

[0099] According to embodiments of the present invention, the threshold of arithmetic operation intensity can be managed by a policy table according to the device-specific memory bandwidth and operation performance profile, and can be dynamically adjusted according to changes in system state, such as system load or temperature changes.

[0100] If the operation node is determined to be memory-intensive, the scheduler may enter the PIM resource check step (S313). In this step, it can be determined whether the current PIM is in an available state (Idle) or in a state where other tasks are being performed (Busy). If the PIM is in an available state, the step of inserting the corresponding operation into the PIM execution queue (S314) may be performed.

[0101] On the other hand, if the PIM is in a state where it is performing other tasks, a waiting time prediction may be performed, followed by a cost-based decision-making step. In the cost-based decision-making, the estimated cost of processing the task in the PIM can be compared with the estimated cost of processing it on a CPU or NPU. For example, the PIM processing cost can be defined as Cost_PIM = Wait_PIM + Exec_PIM, which is the sum of waiting time and execution time, and the CPU processing cost can be defined as Cost_CPU = (Transfer_Bytes / BW) + Exec_CPU, which includes data transfer cost and execution time.

[0102] As a result of the comparison, if the processing cost in PIM is lower, the corresponding task can be inserted into the PIM execution queue (S314); conversely, if the processing cost in CPU or NPU is lower, it can be assigned to a legacy execution path through the CPU or NPU allocation step (S312). Even if the computation node is determined to be computationally intensive, the CPU or NPU allocation step (S312) may be selected.

[0103] According to embodiments of the present invention, the intelligent heterogeneous scheduler (310) may apply a dynamic offloading policy. For example, when the remaining battery level is below a preset threshold or the system temperature is above a threshold, it may operate in a power priority mode that forcibly increases the usage proportion of a power-efficient PIM. In the power priority mode, heat generation and power consumption may be controlled by applying a limit on the number of active banks or a dynamic voltage and frequency regulation (DVFS) policy while simultaneously increasing the PIM allocation ratio.

[0104] Through the process of FIG. 5 as described above, the intelligent heterogeneous scheduler (310) can dynamically assign tasks to the optimal processor by comprehensively considering computational performance, power efficiency, and system stability in a runtime environment.

[0105] FIG. 6 is a flowchart illustrating, step-by-step, an example of a detailed processing procedure of a PIM native graph compiler according to an embodiment of the present invention. The process illustrated in FIG. 6 may represent a process of receiving a subgraph allocated to the PIM by an intelligent heterogeneous scheduler as input and converting it into an instruction sequence that can be directly executed on low-power PIM memory hardware.

[0106] According to embodiments of the present invention, a PIM native graph compiler (320) can receive a subgraph determined as a PIM acceleration target as input and can generate a zero-copy PIM instruction sequence that does not involve data movement as output.

[0107] As illustrated in FIG. 6, the compilation procedure may begin with an SRAM size-based tiling step (S321). In the SRAM size-based tiling step (S321), large-scale matrix operations exceeding the capacity of the Global Buffer or local SRAM within the PIM can be divided into tiles suitable for the buffer size. This allows operations to be performed with each tile loaded into the PIM's internal buffer, thereby minimizing access to external memory.

[0108] Next, an operation fusion step (S322) can be performed. In the operation fusion step (S322), activation functions (Rectified Linear Unit, ReLU, etc.) or element-wise operations (Add, Mul, etc.) that immediately follow the matrix multiplication (MatMul) operation can be combined into a single kernel so that they can be processed continuously at the register level without separate memory storage. This eliminates memory access for intermediate results and reduces operation latency and energy consumption.

[0109] Subsequently, a PIM command generation step (S323) extending the LPDDR family interface may be performed. In this step, instead of a general memory read or write command, a special command that directly directs an internal PIM operation may be generated. The special command may include, for example, an operation opcode, a source (src) address, a destination (dst) address, a tile size, a barrier identifier (Barrier ID) for synchronization, etc.

[0110] According to embodiments of the present invention, when generating a command, the source address and destination address may be restricted to necessarily fall within the PIM internal address range, and the command may be issued only if it passes zero-copy enforcement verification. If an external DRAM area reference, page alignment violation, or bank alignment constraint violation is detected as a result of address verification, the compiler may readjust the tiling size or address placement. Additionally, if the constraint is not satisfied even with such readjustment, a fallback path may be selected to instruct execution on the CPU or NPU.

[0111] Next, a synchronization barrier insertion step (S324) may be performed. In the synchronization barrier insertion step (S324), when multiple PIM commands are executed in parallel or in a pipelined manner, a barrier command may be inserted to ensure dependencies between operations. This allows for control so that subsequent operations are safely performed after the completion of an operation on a specific tile.

[0112] Next, a pipeline optimization step (S325) may be performed. In the pipeline optimization step (S325), a pipeline structure that overlaps memory access and operation steps can be configured by rearranging the command execution order or allowing overlapping execution. This can reduce the idle time of the PIM operator and improve overall throughput.

[0113] Through the process of Fig. 6 as described above, the PIM native graph compiler (320) can generate zero-copy instruction packets in which operations are performed without data movement within the PIM, and these can be transmitted to the hardware and executed through a low-power PIM memory integration driver.

[0114] FIG. 7 is a flowchart illustrating step-by-step examples of detailed processing procedures for a QoS-based bandwidth mediator according to an embodiment of the present invention. The process illustrated in FIG. 7 may represent a bandwidth control process for ensuring system stability and user experience by coordinating memory access conflicts between PIM operations and other system components, such as displays and communication modules, in an on-device environment.

[0115] According to embodiments of the present invention, a QoS-based bandwidth arbitrator (330) may receive, as input, a memory access request originating from a PIM command queue and another system, such as a display or communication module. As output, it may generate a Grant signal or a Wait signal for the request.

[0116] As illustrated in FIG. 7, the arbitration procedure may begin with a memory request detection step (S331). In the memory request detection step (S331), the entity generating the request can be identified by snooping on transactions flowing into the memory controller. The identification of the request entity can be performed using the memory controller's master identifier (Master ID) or transaction tag.

[0117] If the requesting entity is determined to be a high-priority request with high real-time characteristics, such as display updates or communication processing, the QoS-based bandwidth arbitrator (330) can check the operating status of the PIM. If the PIM is idle, immediate execution approval can be granted for the said priority request. On the other hand, if the PIM is active, the PIM clock throttling step (S332) can be performed.

[0118] In the PIM clock throttling step (S332), the memory bus occupancy can be reduced by lowering the operating frequency of the PIM or delaying the PIM command issuance cycle. This allows the bandwidth required by high-priority requests, such as display updates, to be secured preferentially.

[0119] If it is determined that a bandwidth shortage persists or that there is a high probability of screen tearing or communication delay occurring, an operation pause step (S333) may be performed. In the operation pause step (S333), the currently executing PIM operation may be preempted to immediately return control of the memory bus. At this time, the execution context of the PIM operation may be backed up, and the saved context may include a program counter, internal register status, tile coordinates, partial sum storage location, and a waiting command pointer. The context may be stored in the PIM internal registers or in a pre-reserved internal buffer area.

[0120] When priority request processing is completed and the occupancy of the memory request queue decreases below a preset threshold, the arbitrator can restore the saved context and resume the PIM operation. At this time, hysteresis is applied to the resumption condition to prevent frequent repetition of preemption and resumption due to minute fluctuations in bandwidth conditions.

[0121] Through the process of Fig. 7 as described above, the QoS-based bandwidth mediator (330) can stably maintain the quality of high-priority services such as display and communication while minimizing system performance interference caused by PIM calculations.

[0122] FIG. 8 is a block diagram illustrating an example of the detailed configuration of an offline optimization device according to an embodiment of the present invention. The offline optimization device (100) illustrated in FIG. 8 may be a preprocessing device that reconstructs model data to match the physical memory address system of PIM-based hardware, going beyond simple model compression.

[0123] According to embodiments of the present invention, an offline optimization device (100) may receive a learned FP32 model as input and include a PIM cognitive quantizer (110) and a memory layout prepacker (120). The offline optimization device (100) may finally package metadata including an address mapping table, an interleaving pattern, a layer-by-layer precision map, and scaling factor location information together with the PIM optimization binary and transmit it to a runtime engine (300).

[0124] As illustrated in FIG. 8, the PIM cognitive quantizer (110) can perform mixed-precision based quantization on an input learned FP32 model. To this end, the PIM cognitive quantizer (110) may include a sensitivity analysis unit (111), a precision determination unit (112), a quantization execution unit (113), and an error correction unit (114).

[0125] The sensitivity analysis unit (111) can numerically evaluate the importance of each layer of the input model by calculating the output error rate according to the change in weights for each layer. The results of this sensitivity analysis can be used as basic information to determine the quantization precision for each layer.

[0126] The precision determination unit (112) can establish a mixed precision policy by considering the sensitivity analysis results, target accuracy, and hardware constraints. According to one embodiment, the precision determination unit (112) can assign MXFP8 precision to some upper layers within the model (e.g., top 20% of importance) and assign MXFP4 precision to the remaining layers.

[0127] The quantization execution unit (113) can convert the weights of each layer into the actual MXFP format according to the above-mentioned mixed precision policy, and the error correction unit (114) can perform scaling factor adjustment or correction operations to minimize the accumulated error caused by quantization. Through this process, an MXFP lightweight model can be generated.

[0128] Meanwhile, the memory layout prepacker (120) can receive the MXFP lightweight model as input and perform physical data placement suitable for the PIM memory structure. To this end, the memory layout prepacker (120) may include a tensor shape analysis unit (121), an address mapping unit (122), a data interleaving unit (123), and a metadata generation unit (124).

[0129] The tensor shape analysis unit (121) can analyze the dimension information and size of each tensor in the model to determine the consistency with the internal memory page and bank structure of the PIM. The address mapping unit (122) can generate a mapping table that converts a logical tensor index into a physical address of the low-power PIM memory, i.e., an address in the Bank, Row, and Column units.

[0130] The data interleaving unit (123) can arrange physical storage locations across multiple banks so that data accessed in consecutive operations is not concentrated in a specific bank. Through this, bank collisions can be minimized and parallel memory access efficiency can be improved when performing PIM operations.

[0131] The metadata generation unit (124) can generate metadata including the address mapping result, interleaving pattern, layer-specific precision information, and location information of the scaling factor. The metadata is packaged together with the PIM optimized binary and can subsequently be used by the runtime engine to verify zero-copy address constraints and perform efficient command generation.

[0132] Through the configuration of Fig. 8 as described above, the offline optimization device (100) can minimize overhead at runtime and improve the performance and power efficiency of the entire system by generating a data layout and quantization results that reflect PIM hardware characteristics in advance.

[0133] FIG. 9 is a block diagram illustrating an example of the detailed configuration of a PIM native graph compiler according to an embodiment of the present invention. The PIM native graph compiler (320) shown in FIG. 9 may be a core engine that receives a subgraph operation assigned to the PIM by an intelligent heterogeneous scheduler as input and generates an instruction sequence that enables the operation to be performed completely within the PIM.

[0134] According to embodiments of the present invention, the PIM native graph compiler (320) may include a tiling and optimization layer and an instruction generation layer, and each layer may include a plurality of functional modules. The PIM native graph compiler (320) may finally output a PIM instruction packet.

[0135] As illustrated in FIG. 9, the tiling and optimization layer may include a tiling manager (321) and a kernel fusion optimizer (322).

[0136] The tiling manager (321) can divide large matrix operations that exceed the capacity of the global buffer within the PIM module into tile units that fit the global buffer capacity. According to one embodiment, if the capacity of the global buffer is, for example, 256KB, the tiling manager (321) can break down the operation into multiple tiles based on the row, column, or channel dimensions of the matrix, and adjust each tile so that it can be loaded into the global buffer. Through this, it is possible to perform operations using only the PIM internal buffer without accessing external memory.

[0137] The kernel fusion optimizer (322) can perform kernel-level fusion optimization by detecting frequently consecutive operation patterns targeting divided tile operations. According to one embodiment, the kernel fusion optimizer (322) can fuse a sequence of operations performed consecutively, such as 'Matrix Multiplication (MatMul) + Add Bias (Add Bias) + Activation Function (ReLU)', into a single kernel. In this case, intermediate operation results are not written to external memory or a global buffer, but are processed immediately in a register or PIM internal operation path, thereby reducing the number of memory accesses and minimizing operation delays.

[0138] Meanwhile, the command generation layer may include a PIM command generator (323), a synchronization barrier inserter (324), and a zero-copy enforcement module (325).

[0139] The PIM command generator (323) can convert the operation information generated in the tiling and optimization layer into a command format recognizable by the PIM hardware. The command is not a general memory read / write command, but a special command that directly directs a PIM operation, and may include fields such as an operation opcode, source (src) and destination (dst) addresses, tile size, number of iterations, and a synchronization identifier.

[0140] The synchronization barrier inserter (324) can automatically insert synchronization barrier commands at necessary locations to align the completion times of operations between multiple PIM banks or bank groups operating in parallel. This prevents data race conditions that may occur due to data dependencies between operations performed in different banks.

[0141] The zero-copy enforcement module (325) can verify and enforce that the operand address of the generated instruction does not refer to an external DRAM area and must cycle within the PIM internal address range. To this end, the zero-copy enforcement module (325) can refer to a metadata header generated by the memory layout prepacker (120), and said metadata header may include an internal address range, bank mapping information, and offset / stride information.

[0142] According to one embodiment, the zero-copy enforcement module (325) may include the following verification items: (i) whether the source (src) and destination (dst) addresses fall within the PIM internal address range, (ii) whether page alignment and bank alignment constraints are satisfied, and (iii) whether there are no addresses referencing an external DRAM area. If any of the above verifications fail, the zero-copy enforcement module (325) may request tile re-division or select a fallback path to execute the corresponding operation on the CPU or NPU.

[0143] Through the configuration of Fig. 9 as described above, the PIM native graph compiler (320) can convert operations assigned to the PIM into instruction sequences that can be fully executed internally without external memory access, thereby minimizing memory bandwidth usage and improving the performance and power efficiency of the entire system.

[0144] FIGS. 10 and 11 are drawings illustrating examples of AI inference and PIM acceleration processing sequence diagrams according to an embodiment of the present invention. The sequence diagram according to the present embodiment shows the main processor and the PIM acceleration path operating in parallel according to an AI inference request, and shows the entire processing flow step by step from the inference request, instruction conversion, execution, and result merging.

[0145] FIG. 10 illustrates the scheduling phase (Scheduling Phase, Phase 1) in which resources are allocated after an inference request occurs, and FIG. 11 illustrates the instruction translation and execution phase (Execution Phase, Phase 2) in the PIM acceleration path and the subsequent result merging phase (Aggregation Phase, Phase 3). FIG. 10 and FIG. 11 are drawings that were originally a single sequence diagram but were divided for readability and clarity of explanation.

[0146] The first step is a step in which an AI inference request from a user is input, and resources are allocated so that the request is branched into a main processor path and a PIM acceleration path and executed in parallel, and may include steps 1 to 5.

[0147] In process 1, the user app / service may transmit an inference request to the system for performing artificial intelligence inference. The inference request may include information about a computation graph, input data, or an inference model.

[0148] In process 2, the intelligent heterogeneous scheduler (310) can analyze the structure, computational complexity, and memory access characteristics of the input computation graph after receiving the inference request.

[0149] In process 3, the intelligent heterogeneous scheduler (310) can assign non-linear operations or control flow-centered operations to be performed on the main processor (410) based on the analysis results.

[0150] In process 4, the intelligent heterogeneous scheduler (310) can direct parallel processing by branching operations that are highly dependent on memory bandwidth, such as large-scale matrix multiplication, into PIM accelerated tasks.

[0151] In process 5, the intelligent heterogeneous scheduler (310) can complete scheduling so that the main processor path and the PIM acceleration path are executed simultaneously.

[0152] The second step is a step in which an operation assigned to the PIM is converted into a PIM-specific instruction and executed in an actual PIM memory module under QoS control, and may include steps 6 to 14.

[0153] In process 6, the main processor (410) can pass the computation graph or subgraph branched into the PIM acceleration path to the PIM graph compiler (320).

[0154] In process 7, the PIM graph compiler (320) can perform tiling and operation fusion on the transmitted operation graph to match the physical structure of the PIM memory.

[0155] In process 8, the PIM graph compiler (320) can generate PIM native command packets based on the optimized result.

[0156] In process 9, the PIM integration driver (340) may request memory bus access rights from the QoS-based bandwidth arbitrator (330) before issuing the generated command packet.

[0157] In process 10, the QoS-based bandwidth mediator (330) can determine whether there is a request from a display or other high-priority module, and may approve access to the PIM acceleration task or switch to a waiting state.

[0158] In process 11, the low-power PIM memory module (420) can perform in-memory operations according to the instruction packet when memory access is approved.

[0159] In process 12, the low-power PIM memory module (420) can store the intermediate or final result of the operation being performed in the internal memory area.

[0160] In process 13, the QoS-based bandwidth mediator (330) can determine whether the interrupted PIM operation can be resumed depending on the system load state.

[0161] In process 14, the low-power PIM memory module (420) can restore the saved execution context to resume the operation from the breakpoint.

[0162] The third step is a step of synchronizing the results of operations performed in the main processor and the PIM path, respectively, and generating a final inference result, and may include steps 15 to 17.

[0163] In process 15, the intelligent heterogeneous scheduler (310) can synchronize the operation results performed in the main processor (410) with the operation results stored in the low-power PIM memory module (420).

[0164] In process 16, the intelligent heterogeneous scheduler (310) can merge multiple synchronized results to generate a final inference result.

[0165] In process 17, the intelligent heterogeneous scheduler (310) can terminate one AI inference cycle by returning the final inference result to the user app / service.

[0166] As such, according to the sequence illustrated in FIGS. 10 and 11, embodiments of the present invention can simultaneously improve inference performance and power efficiency while minimizing data movement by utilizing the main processor and the PIM acceleration path in parallel. In addition, some steps of the stepwise processing flow may be omitted, added, or the order changed depending on system resource status, QoS requirements, or workload characteristics, and such variations may also be included within the technical spirit and scope of the present invention.

[0167] FIGS. 12 and 13 are drawings illustrating examples of system resource management and QoS linkage sequence diagrams according to an embodiment of the present invention. The sequence diagram according to the present embodiment shows, in steps, a processing flow that maintains the stability of the entire system and the user experience by detecting high-priority requests, such as system overheating or display updates, when they occur while PIM acceleration operations are being performed, and adaptively controlling resources. FIGS. 12 and 13 are drawings that originally constituted a single sequence diagram, but have been divided for clarity of explanation.

[0168] Scenario A represents a process of adaptively controlling system resources, including PIM acceleration operations, when the system temperature or battery state reaches a critical condition, and may include processes 1 through 5.

[0169] In process 1, the temperature sensor or battery monitoring module of the mobile device can detect that the current system state exceeds a set temperature threshold or that the remaining battery level is below the threshold. The detection information can be transmitted to an intelligent heterogeneous scheduler (310) through the power manager of the host operating system (OS).

[0170] In process 2, the intelligent heterogeneous scheduler (310) can switch the system to Emergency Mode after receiving the status information. Emergency Mode may be an operating mode activated for the purpose of heat relief or power saving.

[0171] In process 3, the intelligent heterogeneous scheduler (310) may decide to limit the number of active banks used for PIM acceleration computation or to execute some computations by bypassing the main processor (CPU or NPU) path, depending on the entry into emergency mode.

[0172] In process 4, upon a control request from the intelligent heterogeneous scheduler (310), the low-power PIM memory integration driver (340) can issue a Dynamic Voltage and Frequency Scaling (DVFS) command to lower the operating frequency and voltage for the low-power PIM memory module (420).

[0173] In process 5, the low-power PIM memory module (420) reduces physical power consumption and heat generation according to the DVFS command and can operate in a low-power state until the system temperature stabilizes.

[0174] Scenario B represents a QoS-based arbitration process to prevent conflicts with PIM acceleration operations when a high-priority memory request requiring real-time, such as a display update, occurs, and may include processes 6 to 15.

[0175] In process 6, peripheral devices such as a display controller or a communication modem may generate a memory access request for screen refresh or data transmission.

[0176] In process 7, the QoS-based bandwidth arbitrator (330) monitors transactions flowing into the memory controller and can identify that the request originated from a high-priority master.

[0177] In process 8, the QoS-based bandwidth mediator (330) can evaluate the memory bandwidth share of the operation currently being performed by the low-power PIM memory module (420).

[0178] If it is determined that memory bandwidth is insufficient due to PIM operation in process 9, the QoS-based bandwidth mediator (330) can control the operation of the low-power PIM memory module (420) to be suspended.

[0179] When the operation is paused in process 10, the low-power PIM memory module (420) may store the execution context of the operation currently being performed in an internal register or a reserved internal buffer area. The execution context may include a program counter, internal register status, tile coordinates, and tile-unit partial sums.

[0180] In process 11, the QoS-based bandwidth arbitrator (330) returns control of the memory bus to the display controller so that data transmission required for screen updates can be performed without delay.

[0181] In process 12, the display controller can complete the memory access required for screen refresh.

[0182] In process 13, the QoS-based bandwidth mediator (330) can determine whether high-priority requests have been terminated and whether the priority request queue has stabilized below a set threshold.

[0183] If the resumption condition is met in process 14, the QoS-based bandwidth arbitrator (330) may instruct the low-power PIM memory module (420) to resume operation. At this time, hysteresis may be applied to the resumption condition to prevent frequent interruptions and resumptions.

[0184] In process 15, the low-power PIM memory module (420) can restore the saved execution context and resume PIM accelerated computation from the point where it was interrupted.

[0185] As such, according to the embodiments illustrated in FIGS. 12 and 13, the embodiments of the present invention can detect high-priority memory requests, such as the system's heating state, battery state, or display update, in real time while PIM acceleration computation is being performed, and accordingly perform adaptive resource control including computation throttling, preemption, and resumption. This allows for the prevention of user experience degradation, such as screen freezing or system instability, while maintaining artificial intelligence inference performance. Furthermore, the processing flow for each scenario may omit or add some processes depending on the system environment, terminal characteristics, or service requirements, and the order of each process may also be changed; however, such variations may also be included within the technical spirit and scope of the present invention.

[0186] FIG. 14 is a flowchart illustrating an example of a PIM-based on-device artificial intelligence execution method according to an embodiment of the present invention. The method illustrated in FIG. 14 is a method for executing on-device artificial intelligence using low-power PIM memory in mobile and edge environments, and can be performed by an on-device artificial intelligence execution system and may include steps (1410) to (1460). Each step may be specified into a plurality of detailed operations as described below.

[0187] In step (1410), the on-device artificial intelligence execution system may receive an optimized artificial intelligence model aligned with the hardware structure of the low-power PIM memory as input to the on-device runtime engine. At this time, the optimized artificial intelligence model may be a model generated in advance by an offline optimization device and may include a metadata header including layer-by-layer computational precision information, mixed precision allocation information, memory layout information, bank and channel mapping information, tile size information, and internal address range. The metadata header may include a tensor identifier, physical base address, offset, stride, bank mapping information, and scaling factor location information, and may be transmitted over a network or loaded from the local storage of a mobile device.

[0188] In step (1420), when an AI inference request is input to the on-device AI execution system, the system can determine computational characteristics by analyzing a computation graph corresponding to the inference request. In this step, the on-device AI execution system can analyze at least one of the computational amount per computation node, memory access frequency, arithmetic intensity, data locality, or intermediate result size through an intelligent heterogeneous scheduler, and thereby determine whether each computation is limited by memory bandwidth or computational performance. Additionally, the battery level, system temperature, priority of currently running applications, or system load status can be considered together.

[0189] In step (1430), the on-device artificial intelligence execution system may distribute at least some of the operations of the operation graph to at least one of the low-power PIM memory or the main processor (CPU / NPU) based on the operation characteristics determined in step (1420). At this time, the on-device artificial intelligence execution system may allocate operations that involve frequent memory access or operations that are highly dependent on memory bandwidth to the low-power PIM memory through an intelligent heterogeneous scheduler, and allocate operations that are operation-intensive or control-flow oriented to the main processor. Accordingly, the main processor path and the PIM acceleration path may be executed in parallel, and the operation distribution ratio, the number of active banks, or the operating frequency may be dynamically adjusted according to the system state.

[0190] In step (1440), the on-device AI execution system can convert the operation distributed to the low-power PIM memory into a dedicated instruction sequence that can be executed within the low-power PIM memory. In this step, the on-device AI execution system can divide the operation data into tile units based on the internal buffer capacity of the low-power PIM memory through a PIM native graph compiler, and can fuse matrix multiplication operations and subsequent activation operations or element-wise operations into a single kernel. Additionally, by verifying whether the operand address of the generated dedicated instruction sequence falls within the internal address range of the low-power PIM memory defined in the metadata header, zero-copy execution can be enforced or supported so that the operation is performed without moving data to external memory. If the verification is not passed, tile redistribution or a fallback to the main processor execution path may be selected.

[0191] In step (1450), the on-device AI execution system may control memory access to the low-power PIM memory by considering priority memory access requests requiring real-time capabilities, such as display updates, wireless communication processing, or other requests prior to the execution of a dedicated instruction sequence. In this step, the on-device AI execution system may perform throttling, including acknowledging or delaying memory access, adjusting the operating frequency of the low-power PIM memory, adjusting the instruction issue cycle, or limiting the number of active banks, through a QoS-based bandwidth arbitrator. Additionally, if the memory bus bandwidth shortage exceeds a threshold, it may perform preemption control to temporarily suspend operations being performed in the low-power PIM memory.

[0192] In step (1460), the on-device artificial intelligence execution system may have the low-power PIM memory module perform operations within memory according to a dedicated instruction sequence and generate a final inference result together with the results of the operations performed in the main processor. If the preemption occurs, the on-device artificial intelligence execution system may cause the low-power PIM memory module to store an execution context, including a program counter, internal register status, tile index, and partial sum information, in the internal registers of the low-power PIM memory module or in a reserved internal memory area, under the control of a QoS-based bandwidth arbitrator. Subsequently, when the priority memory access request is terminated, operations may resume from the same point based on the stored execution context, and the results of the operations performed in the main processor path and the PIM acceleration path, respectively, may be synchronized and merged to provide the final inference result to the user.

[0193] As such, according to the method illustrated in FIG. 14, the on-device artificial intelligence execution system can provide high inference performance, excellent power efficiency, and a stable user experience in mobile and edge environments by organically performing computational characteristic analysis, heterogeneous computation distribution, zero-copy based PIM execution, QoS-based memory access control, computation preemption and resumption, and context-aware resource management based on a model generated by offline optimization.

[0194] FIG. 15 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. For example, each of at least one computer device implementing the PIM-based on-device artificial intelligence execution system described above may correspond to the computer device (1500) of FIG. 15. As illustrated in FIG. 15, the computer device (1500) may include memory (1510), a processor (1520), a communication interface (1530), and an input / output interface (1540). The memory (1510) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, non-perishable mass storage devices such as ROM and disk drives may be included in the computer device (1500) as separate permanent storage devices distinct from memory (1510). Additionally, an operating system and at least one program code may be stored in memory (1510). These software components may be loaded into memory (1510) from a computer-readable recording medium separate from memory (1510). This separate computer-readable recording medium may include computer-readable recording media such as floppy drives, disks, tapes, DVD / CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into memory (1510) via a communication interface (1530) rather than a computer-readable recording medium. For example, software components may be loaded into memory (1510) of the computer device (1500) based on computer programs installed by files received through a network (Network, 1560).

[0195] The processor (1520) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (1520) via memory (1510) or a communication interface (1530). For example, the processor (1520) may be configured to execute instructions received according to program code stored in a recording device such as memory (1510).

[0196] The communication interface (1530) may provide a function for the computer device (1500) to communicate with other devices through a network (1560). For example, requests, commands, data, files, etc. generated by the processor (1520) of the computer device (1500) according to program code stored in a recording device such as memory (1510) may be transmitted to other devices through the network (1560) under the control of the communication interface (1530). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (1500) through the communication interface (1530) of the computer device (1500) via the network (1560). Signals, commands, data, etc. received through the communication interface (1530) may be transmitted to the processor (1520) or memory (1510), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (1500) may further include.

[0197] The input / output interface (1540) may be a means for interfacing with an input / output device (I / O device, 1550). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (1540) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. The input / output device (1550) may be composed of a computer device (1500) and a single device.

[0198] Additionally, in other embodiments, the computer device (1500) may include fewer or more components than those of FIG. 15. However, it is not necessary to clearly illustrate most of the prior art components. For example, the computer device (1500) may be implemented to include at least some of the input / output devices (1550) described above, or may include other components such as a transceiver, a database, etc.

[0199] In one embodiment, the processor (1520) or memory (1510) may be configured to be coupled with a low-power PIM memory module, and the computer device (1500) may implement the previously described PIM-based on-device artificial intelligence execution system or a part thereof.

[0200] According to embodiments of the present invention, the following technical effects can be expected.

[0201] (1) Securing ultra-high-speed on-device inference performance by overcoming memory barriers

[0202] Embodiments of the present invention can alleviate memory bandwidth bottlenecks occurring in mobile LPDDR-based systems by combining computational execution within a low-power PIM memory module with intelligent heterogeneous scheduling. In particular, the intelligent heterogeneous scheduler can select memory-bound computational layers with frequent memory access and process them as PIM, thereby reducing the stall cycles caused by the main processor (CPU or NPU) waiting for data arrival.

[0203] As a result, the effective computational throughput of the entire system can be improved, and token generation latency can be significantly reduced even when running AI models with a very large number of parameters, such as Large Language Models (LLM). This enables the improvement of the quality of real-time conversational services or on-device AI services that require immediate responsiveness.

[0204] (2) Energy efficiency improvement based on zero-copy architecture

[0205] According to embodiments of the present invention, through a zero-copy instruction sequence generated by a PIM native graph compiler, the process of data moving to the processor via a bus outside of memory can be omitted, and computation and result storage can be completed within memory. Accordingly, round-trip data traffic between the host and memory can be reduced, and energy consumed for data movement can be saved.

[0206] Furthermore, as unnecessary data transmission is eliminated, heat generation in the memory interface and chipset can be reduced, lowering the frequency of thermal throttling. These effects can be evaluated based on metrics such as inference latency, memory traffic (bytes), and energy consumption, and can provide various improvements depending on the model size, sequence length, and comparison criteria configuration.

[0207] (3) Ensuring development convenience and versatility through hardware-independent model deployment

[0208] According to embodiments of the present invention, through a PIM cognitive quantizer and a memory layout prepacker included in an offline optimization device, a model trained with a general-purpose artificial intelligence framework (PyTorch, TensorFlow, etc.) can be automatically converted into a binary optimized for the PIM structure without separate hardware modification.

[0209] By automatically handling details such as physical address mapping or bank interleaving of complex PIM hardware through software, AI model developers can easily utilize PIM-based acceleration without having a detailed understanding of the hardware structure. As a result, various types of AI models (CNN, RNN, Transformer, etc.) can be flexibly applied, and technological scalability and commercialization speed can be improved.

[0210] (4) Ensuring system stability in multitasking environments through QoS-based bandwidth arbitration

[0211] According to embodiments of the present invention, a QoS-based bandwidth arbitrator can be controlled to prioritize memory access requests from other system modules requiring real-time capabilities, such as display updates or wireless communication, even while PIM operations are being performed. This allows for the application of PIM occupancy limits, throttling, and preemption policies to ensure that the maximum delay of priority master requests is maintained below a set threshold.

[0212] If necessary, the PIM operation can be preempted to save the execution context, including the program counter, registers, tile index, and partial sum, and then resume from the same point when the priority traffic ends. As a result, AI operations can be performed in parallel without screen tearing or system response delays, thereby providing the stable user experience required in commercial environments.

[0213] (5) Maintaining an optimal operating environment through context-aware dynamic resource management

[0214] According to embodiments of the present invention, an intelligent heterogeneous scheduler can dynamically adjust the work distribution ratio between the PIM and the main processor by considering the remaining battery level, system temperature, and the priority of running applications in real time. For example, when the remaining battery level is low or heat generation is high, tasks are allocated primarily to the PIM, which has superior power efficiency, and when fast responsiveness is required, the utilization of the CPU or NPU can be increased.

[0215] Task distribution ratios, the number of active banks, PIM clock limits, and fallback conditions can be managed in the form of policy tables, enabling fine-grained tuning based on device characteristics. Through this active resource management technology, hardware lifespan can be extended, and users can always be provided with optimized performance and battery life.

[0216] The system or device described above may be implemented as a hardware component, or a combination of a hardware component and a software component. For example, the device and component described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0217] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0218] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The medium may continuously store a computer-executable program, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components; it is not limited to a medium directly connected to a computer system, but may also exist distributed over a network. Examples of computer-readable storage media include ROM, PROM, EPROM, EEPROM, flash memory (e.g., NAND / NOR), SSD, HDD, magnetic tape, optical recording media (CD-ROM, DVD, BD), magneto-optical media, memory cards, USB memory, etc. Such storage media refer to non-transient media and do not include transmission signals themselves or purely volatile memory (e.g., RAM). In addition, other examples of media include recording or storage media managed by app stores that distribute applications, or sites and servers that supply or distribute various other software. Examples of program instructions include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0219] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0220] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A system for executing on-device artificial intelligence using low-power Processing-In-Memory (PIM) memory in mobile and edge environments, comprising: an offline optimization device for receiving a trained artificial intelligence model and converting the model into a form aligned with the hardware structure of the low-power PIM memory; an on-device runtime engine for performing artificial intelligence inference within a mobile device using the optimized model transmitted from the offline optimization device; an intelligent heterogeneous scheduler included in the on-device runtime engine for analyzing a computation graph according to an artificial intelligence inference request and distributing computations to at least one of the low-power PIM memory or a main processor based on computation characteristics; a PIM native graph compiler for converting computations allocated to the low-power PIM memory by the intelligent heterogeneous scheduler into a dedicated instruction sequence so that they can be executed within the low-power PIM memory; and a QoS-based bandwidth arbitrator for controlling memory access of the low-power PIM memory by considering priority memory access requests, such as display updates or communication processing, in relation to the execution of the instruction sequence generated by the PIM native graph compiler. An on-device artificial intelligence execution system comprising a low-power PIM memory module configured to perform operations within memory according to the above-described dedicated instruction sequence, wherein the on-device runtime engine links the intelligent heterogeneous scheduler, the PIM native graph compiler, and the QoS-based bandwidth arbitrator to control the main processor and the low-power PIM memory to perform artificial intelligence operations in parallel. Claim 2 An on-device artificial intelligence execution system according to claim 1, wherein the offline optimization device comprises: a sensitivity analysis unit for analyzing output sensitivity according to weight change; a precision determination unit for determining computational precision per layer based on the sensitivity analysis result; and a quantization execution unit for quantizing model parameters according to the determined precision. Claim 3 An on-device artificial intelligence execution system according to paragraph 2, wherein the precision determining unit assigns a first precision format to some layers of the artificial intelligence model and a second precision format to the remaining layers. Claim 4 An on-device artificial intelligence execution system according to claim 1, characterized in that the offline optimization device includes a memory layout prepacker for rearranging logical tensor addresses to correspond to the physical address structure of the low-power PIM memory. Claim 5 An on-device artificial intelligence execution system according to claim 4, wherein the memory layout prepacker interleaves data so that consecutive computational data is cross-placed across multiple memory banks so that consecutive computational data is not concentrated in a single memory bank. Claim 6 An on-device artificial intelligence execution system according to claim 4, wherein the memory layout prepacker generates a metadata header including a tensor identifier, a physical base address, an offset, a stride, bank mapping information, and a tile size. Claim 7 An on-device artificial intelligence execution system according to claim 1, characterized in that the PIM native graph compiler ensures that zero-copy execution is performed by verifying that the operand address of the generated dedicated instruction sequence falls within the internal address range of the low-power PIM memory. Claim 8 An on-device artificial intelligence execution system according to claim 7, wherein the PIM native graph compiler divides computational data into tile units based on the internal buffer capacity of the low-power PIM memory. Claim 9 In claim 8, the on-device artificial intelligence execution system is characterized by the PIM native graph compiler fusing matrix multiplication operations and subsequent activation operations or element-wise operations into a single kernel to omit the storage of intermediate results in external memory. Claim 10 An on-device artificial intelligence execution system according to claim 1, wherein the QoS-based bandwidth arbitrator determines priority based on at least one of the requesting entity, request type, or request time of a memory access request. Claim 11 An on-device artificial intelligence execution system according to claim 10, wherein the above QoS-based bandwidth arbitrator restricts or delays memory access of the low-power PIM memory when a display update request is detected. Claim 12 An on-device artificial intelligence execution system according to claim 11, characterized in that the QoS-based bandwidth arbitrator adjusts the operating frequency or instruction issuance cycle of the low-power PIM memory to reduce memory bus occupancy. Claim 13 An on-device artificial intelligence execution system according to claim 10, wherein the above QoS-based bandwidth arbitrator pauses the operation being performed in the low-power PIM memory when the bandwidth shortage exceeds a threshold, and resumes the operation from the same point after the priority request ends. Claim 14 An on-device artificial intelligence execution system according to claim 13, characterized in that, under the control of the QoS-based bandwidth arbitrator, the low-power PIM memory module stores an execution context including a program counter, internal register status, tile index, or partial sum information when an operation is paused, in the internal register or reserved internal memory area of ​​the low-power PIM memory module. Claim 15 An on-device artificial intelligence execution system according to claim 1, wherein the intelligent heterogeneous scheduler dynamically adjusts the operation distribution ratio between the low-power PIM memory and the main processor, the number of active banks, or the operating frequency based on the remaining battery level, temperature, or the priority of the application being executed. Claim 16 A system for executing on-device artificial intelligence using low-power Processing-In-Memory (PIM) memory in mobile and edge environments, comprising: an on-device runtime engine for performing artificial intelligence inference by receiving an optimized artificial intelligence model aligned with the hardware structure of the low-power PIM memory; an intelligent heterogeneous scheduler included in the on-device runtime engine for analyzing a computation graph in response to an artificial intelligence inference request and distributing computations to at least one of the low-power PIM memory or a main processor based on computation characteristics; a PIM native graph compiler for converting computations allocated to the low-power PIM memory by the intelligent heterogeneous scheduler into a dedicated instruction sequence so that they can be executed within the low-power PIM memory; and a QoS-based bandwidth arbitrator for controlling memory access of the low-power PIM memory by considering priority memory access requests, such as display updates or communication processing, in relation to the execution of the instruction sequence generated by the PIM native graph compiler. An on-device artificial intelligence execution system comprising a low-power PIM memory module configured to perform operations within memory according to the above-described dedicated instruction sequence, wherein the on-device runtime engine links the intelligent heterogeneous scheduler, the PIM native graph compiler, and the QoS-based bandwidth arbitrator to control the main processor and the low-power PIM memory to perform artificial intelligence operations in parallel. Claim 17 A method for executing on-device artificial intelligence using low-power Processing-In-Memory (PIM) memory in mobile and edge environments comprises: receiving an optimized artificial intelligence model aligned with the hardware structure of the low-power PIM memory into an on-device runtime engine; when an artificial intelligence inference request is received, an intelligent heterogeneous scheduler analyzes a computation graph corresponding to the artificial intelligence model to determine computation characteristics; based on the determined computation characteristics, distributing at least a portion of the computations of the computation graph to at least one of the low-power PIM memory or the main processor; for the computations distributed to the low-power PIM memory, a PIM native graph compiler converts them into a dedicated instruction sequence that can be executed within the low-power PIM memory; and prior to the execution of the dedicated instruction sequence, a QoS-based bandwidth arbitrator controls memory access to the low-power PIM memory by considering priority memory access requests such as display updates or communication processing. An on-device artificial intelligence execution method characterized by including the step of the low-power PIM memory module performing operations within the memory according to the above-described dedicated instruction sequence and generating a final inference result together with the operation result performed by the main processor. Claim 18 In claim 17, the step of converting to the dedicated instruction sequence is characterized by supporting zero-copy execution so that operations are performed without moving data to external memory by verifying that the operand address of the dedicated instruction sequence falls within the internal address range of the low-power PIM memory. Claim 19 An on-device artificial intelligence execution method according to claim 17, characterized in that when the QoS-based bandwidth arbitrator determines that the memory bus bandwidth shortage exceeds a threshold, the operation being performed in the low-power PIM memory is paused, and after the priority memory access request is terminated, the operation is resumed from the same point based on the stored execution context. Claim 20 A computer program stored on a non-transient computer-readable recording medium that is combined with a computer device and executes the method of any one of claims 17 to 19 on the computer device.

Images