Hierarchical cim architecture system, scheduling method and device, equipment and storage medium

Abstract

The present disclosure relates to a layered CiM architecture system, a scheduling method, an apparatus, a device and a storage medium. The method comprises: a control layer and a 3D memory-in-compute array connected vertically, the 3D memory-in-compute array is integrated with a plurality of memory-compute layers for different modalities; the control layer is used for modal perception on acquired modal data, and the modal data is scheduled to a memory-compute layer in the 3D memory-in-compute array according to the result of the modal perception; the 3D memory-in-compute array is used for selecting a memory-compute layer corresponding to the modal data to infer the modal data according to the processing requirement of the modal data after receiving the modal data. The unreasonable bottleneck of bandwidth allocation of data loading is broken, the efficiency, energy efficiency and long-time running stability of edge multi-modal inference are significantly improved, and the resource constraints of edge scenarios are fully adapted.

Description

Technical Field

[0001] This disclosure relates to the fields of computing acceleration and storage technology, and in particular to a hierarchical CiM architecture system, scheduling method, apparatus, device and storage medium. Background Technology

[0002] Multimodal learning has become a core research direction and an important application scenario, used to achieve the effective fusion and accurate alignment of information from various modalities such as vision, language, and audio. It heavily relies on the parallel inference capabilities and cross-modal interaction processing capabilities of deep neural network models. Through feature extraction and correlation analysis of data from different modalities, it supports the implementation of various complex AI application scenarios. From a technical perspective, the emergence and development of multimodal alignment technology stems from the core need to enable artificial intelligence systems to possess human-like "cross-sensory understanding of the world." At the hardware support level, the existing 3D Compute-in-Memory (CiM) architecture provides an effective solution for improving the inference speed and energy efficiency of AI models. This architecture, by tightly integrating computing and storage units, significantly reduces the latency of data transfer between storage and computing modules, significantly optimizes the energy efficiency ratio during model inference, and provides an important foundation for hardware acceleration of multimodal learning tasks.

[0003] However, traditional 3D CiM architectures generally adopt a "single-layer array" design pattern, integrating computation and storage functions only within a single storage medium such as SRAM or ReRAM. This design cannot effectively address the specific acceleration issues of cross-modal computing. Existing multimodal models also suffer from severe structural heterogeneity; different operators exhibit significant differences in computational logic, data processing formats, and resource requirements, resulting in extremely high overall model complexity and severely limiting the inference efficiency of multimodal models at the edge. Furthermore, the input volumes of different modal data vary greatly, leading to bandwidth limitations and preventing the complete loading of multimodal models at once. Simultaneously, the frequent data transfer between storage and computation units consumes power far exceeding the power consumption of the computation itself, exacerbating the energy burden on edge devices and further restricting the long-term stable operation of multimodal learning tasks in edge scenarios. Therefore, a method is needed to address the bottlenecks in cross-modal feature interaction and computation in multimodal neural network inference. Summary of the Invention

[0004] This disclosure provides a hierarchical CiM architecture system, scheduling method, device, equipment, and storage medium, which breaks through the bottleneck of unreasonable bandwidth allocation for data loading, significantly improves the efficiency, energy efficiency, and long-term operational stability of multimodal inference at the edge, and fully adapts to the resource constraints of edge scenarios.

[0005] Firstly, this disclosure provides a layered CiM architecture system, including: A vertically interconnected control layer and a 3D in-memory computing array, wherein the 3D in-memory computing array integrates multiple in-memory computing layers for different modes; The control layer is used to perform modal perception on the acquired modal data and schedule the modal data to the in-memory computing layer in the 3D in-memory computing array based on the result of the modal perception. The 3D in-memory computing array is used to select the in-memory computing layer corresponding to the modal data to perform inference on the modal data according to the processing requirements of the modal data after receiving the modal data.

[0006] In some embodiments, the 3D in-memory computing array includes a first in-memory layer; the first in-memory layer employs an SRAM-CiM array for processing text modal computation and / or storing the active feature values ​​of the modal data and the intermediate processing results of the modal data.

[0007] In some embodiments, the 3D in-memory computing array further includes a second in-memory layer; the second in-memory layer employs an RRAM-CiM array for storing features and / or sparse computations of the visual modality.

[0008] In some embodiments, the 3D in-memory computing array further includes a third in-memory computing layer; the third in-memory computing layer employs a PCM-CiM or a hybrid memristor array for processing speech and / or time-series continuous modal signals.

[0009] In some embodiments, the first in-memory computing layer, the second in-memory computing layer, and the third in-memory computing layer are vertically interconnected with the control layer in sequence to form the 3D in-memory computing array of a 3D stacked architecture.

[0010] In some embodiments, the control layer includes a calibration unit and a scheduler; The calibration unit is used to perform modal basic calibration on the modal data after acquiring the modal data to obtain the modal sensing result; The scheduler is used to dynamically schedule the modal data to the in-memory computing layer in the 3D in-memory computing array based on the modal data information and the device computing load.

[0011] Secondly, this disclosure provides a hierarchical CiM-based dynamic scheduling method, applied to the aforementioned hierarchical CiM architecture system, including: The control layer performs modal perception on the acquired modal data and schedules the modal data to the in-memory computing layer in the 3D in-memory computing array based on the results of the modal perception. After receiving the modal data, the 3D in-memory computing array selects the in-memory computing layer corresponding to the modal data to perform inference on the modal data according to the processing requirements of the modal data. The 3D in-memory computing array integrates multiple in-memory computing layers for different modalities.

[0012] Thirdly, this disclosure provides a computer device including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the foregoing aspects.

[0013] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the methods described in the above aspects.

[0014] Fifthly, this disclosure provides a computer program product, including computer program instructions, which, when executed by a processor, implement the steps of the methods described above.

[0015] This disclosure provides a hierarchical CiM architecture system, scheduling method, apparatus, device, and storage medium. By vertically interconnecting the control layer and the 3D in-memory computing array, and integrating multiple in-memory computing layers for different modalities into the 3D in-memory computing array, the system effectively decouples the structural heterogeneity conflicts of multimodal models and reduces architectural complexity. At the same time, by leveraging the structural characteristics of 3D vertical interconnection, it overcomes the bottleneck of unreasonable bandwidth allocation for data loading, significantly improving the efficiency, energy efficiency, and long-term operational stability of multimodal inference at the edge, and fully adapting to the resource constraints of edge scenarios. Attached Figure Description

[0016] The present disclosure will be described in more detail below based on embodiments and with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a layered CiM architecture system provided in an embodiment of the present disclosure.

[0017] Figure 2 This is a schematic diagram of another layered CiM architecture system provided in an embodiment of this disclosure.

[0018] Figure 3 This is a schematic diagram of another layered CiM architecture system provided in an embodiment of the present disclosure.

[0019] Figure 4 A flowchart of a hierarchical CiM dynamic scheduling method provided in this embodiment of the disclosure.

[0020] In the accompanying drawings, the same parts are referred to by the same reference numerals, and the drawings are not drawn to scale. Detailed Implementation

[0021] Multimodal learning has become a core research direction and an important application scenario, used to achieve the effective fusion and accurate alignment of information from various modalities such as vision, language, and audio. It heavily relies on the parallel inference capabilities and cross-modal interaction processing capabilities of deep neural network models. Through feature extraction and correlation analysis of data from different modalities, it supports the implementation of various complex AI application scenarios. From a technical perspective, the emergence and development of multimodal alignment technology stems from the core need to enable artificial intelligence systems to possess human-like "cross-sensory understanding of the world." At the hardware support level, the existing 3D Compute-in-Memory (CiM) architecture provides an effective solution for improving the inference speed and energy efficiency of AI models. This architecture, by tightly integrating computing and storage units, significantly reduces the latency of data transfer between storage and computing modules, significantly optimizes the energy efficiency ratio during model inference, and provides an important foundation for hardware acceleration of multimodal learning tasks.

[0022] However, traditional 3D CiM architectures generally adopt a "single-layer array" design pattern, integrating computation and storage functions only within a single storage medium such as SRAM or ReRAM. This design approach makes it difficult to adapt to the core requirements of multimodal alignment and cannot effectively solve the dedicated acceleration problem of cross-modal computation, resulting in significant shortcomings in hardware support for multimodal learning tasks. In addition to the hardware architecture adaptability issue, existing multimodal models suffer from severe structural heterogeneity. They typically integrate various types of operators such as convolution (Conv), visual Transformer (ViT), attention mechanisms, multilayer perceptron (MLP), and modality fusion layers. Different operators have significant differences in computational logic, data processing formats, and resource requirements, leading to extremely high overall structural complexity of the model. Current mainstream edge AI chips (such as NPUs and AI accelerators) are mostly optimized for single types of operators (such as convolution or matrix multiplication operators) in their hardware design, lacking comprehensive adaptability to heterogeneous operators. This leads to problems such as low utilization of computing units and pipeline stalls caused by frequent operator switching, severely restricting the inference efficiency of multimodal models at the edge. In addition, memory and bandwidth pressure in the multimodal inference process has become another key bottleneck. The input volume of different modal data varies greatly. The input data volume of visual and audio modalities far exceeds that of text modalities. For example, a 224×224 resolution image, after encoding, can generate thousands of token embeddings, which is much larger than the text encoding result with the same semantic information. Edge AI chips, constrained by hardware cost and size limitations, generally suffer from bandwidth limitations, preventing the complete one-time loading of multimodal models. Furthermore, the frequent data transfer between storage and computing units consumes power far exceeding the power of the computing itself, exacerbating the energy burden on edge devices and further limiting the long-term stable operation of multimodal learning tasks in edge scenarios. Therefore, a method is needed to achieve vertical integration and hierarchical scheduling of heterogeneous in-memory computing units, providing functional mapping and calibration capabilities for different modalities within a structurally identical multi-layered CiM template.

[0023] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0026] Example One Figure 1 This is a schematic diagram of a layered CiM architecture system provided in an embodiment of the present disclosure.

[0027] This disclosure proposes a hierarchical CiM (Compute-in-Memory) architecture for feature cache / KV-cache in multimodal inference processes, enabling energy efficiency optimization through cross-layer dynamic scheduling. The hierarchical CiM architecture system in this disclosure configures heterogeneous memory arrays in different tiers and dynamically allocates cache and computing power for each modality using a modality-aware scheduler, achieving intelligent resource management by allocating storage media according to modal characteristics and adjusting computing paths according to task load.

[0028] likeFigure 1 As shown, a layered CiM architecture system includes: A vertically interconnected control layer and a 3D in-memory computing array, wherein the 3D in-memory computing array integrates multiple in-memory computing layers for different modes.

[0029] Specifically, the CiM architecture, or in-memory computing architecture, is a hardware architecture that deeply integrates storage and computing functions. Its core logic is to allow computation to be completed directly within the storage medium, breaking with traditional von Neumann architectures. The limitation of physically separating computing units (such as CPU / GPU) and storage units (such as memory / hard disk) in the von Neumann architecture fundamentally reduces the frequent data transfer between storage and computing, thereby solving the memory wall and power wall problems with high data transfer latency and high power consumption. It is especially suitable for matrix multiplication or multiply-accumulate (MAC) intensive tasks in the field of AI, such as multimodal inference and deep learning acceleration. By storing the weights of the neural network in the storage array (such as SRAM, DRAM, MRAM, memristor array), the input data is directly loaded into the storage array. The multiply-accumulate operation is completed in parallel within the array by utilizing the physical characteristics of the storage medium (such as the conductivity characteristics of memristors and the current superposition characteristics of SRAM), without the need to transfer data to external computing units. Finally, only the operation result is output.

[0030] In embodiments of this disclosure, the control layer is used to perform modal perception on the acquired modal data and schedule the modal data to the in-memory computing layer in the 3D in-memory computing array based on the result of the modal perception.

[0031] In embodiments of this disclosure, the 3D in-memory computing array is used to, upon receiving the modal data, select an in-memory computing layer corresponding to the modal data to perform inference on the modal data according to the processing requirements of the modal data.

[0032] Specifically, the in-memory computing layer (Tier), or in-memory computing unit layer, consists of independent functional modules that combine storage and computing capabilities, specifically designed to process one type or mode of data (such as text, images, and audio). The core hardware foundation of the in-memory computing layer is the CiM array, which, in collaboration with auxiliary components such as local control units and cache interfaces, forms a complete unit capable of independently storing and computing data. Multiple in-memory computing layers are stacked vertically through interconnection to form the overall structure of a 3D CiM array.

[0033] This disclosure provides a hierarchical CiM architecture system, scheduling method, apparatus, device, and storage medium. By vertically interconnecting the control layer and the 3D in-memory computing array, and integrating multiple in-memory computing layers for different modalities into the 3D in-memory computing array, the system effectively decouples the structural heterogeneity conflicts of multimodal models and reduces architectural complexity. At the same time, by leveraging the structural characteristics of 3D vertical interconnection, it overcomes the bottleneck of unreasonable bandwidth allocation for data loading, significantly improving the efficiency, energy efficiency, and long-term operational stability of multimodal inference at the edge, and fully adapting to the resource constraints of edge scenarios.

[0034] Example Two Building upon the above embodiments, this disclosure provides a more detailed hierarchical CiM architecture system. Specifically, it proposes a hierarchical CiM (Compute-in-Memory) architecture for feature cache (KV-cache) in multimodal inference processes, achieving energy efficiency optimization through cross-layer dynamic scheduling. The hierarchical CiM architecture system in this disclosure configures heterogeneous memory arrays in different tiers and dynamically allocates cache and computing power for each modality via a scheduler, achieving intelligent resource management by allocating storage media according to modal characteristics and adjusting computing paths according to task load.

[0035] like Figure 2 As shown, the layered CiM architecture system includes: A vertically interconnected control layer and a 3D in-memory computing array, wherein the 3D in-memory computing array integrates multiple in-memory computing layers for different modes.

[0036] Specifically, the CiM architecture, or in-memory computing architecture, is a hardware architecture that deeply integrates storage and computing functions. Its core logic is to allow computation to be completed directly within the storage medium, breaking with traditional von Neumann architectures. The limitation of physically separating computing units (such as CPU / GPU) and storage units (such as memory / hard disk) in the von Neumann architecture fundamentally reduces the frequent data transfer between storage and computing, thereby solving the memory wall and power wall problems with high data transfer latency and high power consumption. It is especially suitable for matrix multiplication or multiply-accumulate (MAC) intensive tasks in the field of AI, such as multimodal inference and deep learning acceleration. By storing the weights of the neural network in the storage array (such as SRAM, DRAM, MRAM, memristor array), the input data is directly loaded into the storage array. The multiply-accumulate operation is completed in parallel within the array by utilizing the physical characteristics of the storage medium (such as the conductivity characteristics of memristors and the current superposition characteristics of SRAM), without the need to transfer data to external computing units. Finally, only the operation result is output.

[0037] In embodiments of this disclosure, the control layer is used to perform modal perception on the acquired modal data and schedule the modal data to the in-memory computing layer in the 3D in-memory computing array based on the result of the modal perception.

[0038] In some embodiments, such as Figure 2 As shown, the control layer includes a calibration unit and a scheduler; the calibration unit is used to perform modal basic calibration on the modal data after acquiring the modal data to obtain the modal perception result; the scheduler is used to dynamically schedule the modal data to the in-memory computing layer in the 3D in-memory computing array according to the data information of the modal data and the device computing load.

[0039] Specifically, due to the differences in feature distribution, accuracy requirements, dynamic range, and temporal characteristics among different modalities (images, speech, text, time-series signals, etc.), a modality-aware calibration unit is set up in the hierarchical CiM architecture system in this embodiment of the disclosure. This unit is used to perform basic calibration of the amplitude or feature range of data from different modalities during multimodal inference, thereby improving the accuracy of multimodal alignment. The calibration unit can select the corresponding calibration path or strategy based on the modality information provided by the scheduler to achieve the adaptation of different modal features in a unified feature space. It should be noted that the specific calibration mechanism of the calibration unit can be implemented by hardware or software, and its internal structure can be flexibly configured according to the application scenario. This disclosure does not impose any limitations on this.

[0040] Specifically, the modality-aware scheduler in the hierarchical CiM architecture system dynamically allocates and schedules computing units and in-memory array resources for each modality during multimodal inference based on the processing requirements, data characteristics, and computational load status of different modalities. The scheduler can identify the modality type of the input data and select appropriate computational paths or resource configurations based on the system's operating status, thereby improving overall computing power utilization efficiency while ensuring the synchronization of multimodal tasks. The scheduler can work in conjunction with the calibration unit to perform unified task scheduling and execution management for calibrated multimodal data. It should be noted that the scheduler's scheduling mechanism can be implemented by hardware logic circuits, programmable control units, or embedded software modules, and its specific algorithms and scheduling strategies can be flexibly configured according to the application scenario; this disclosure does not impose any limitations.

[0041] In embodiments of this disclosure, the 3D in-memory computing array is used to, upon receiving the modal data, select an in-memory computing layer corresponding to the modal data to perform inference on the modal data according to the processing requirements of the modal data.

[0042] Specifically, the in-memory computing layer (Tier), or in-memory computing unit layer, consists of independent functional modules that combine storage and computing capabilities, specifically designed to process one type or mode of data (such as text, images, and audio). The core hardware foundation of the in-memory computing layer is the CiM array, which, in collaboration with auxiliary components such as local control units and cache interfaces, forms a complete unit capable of independently storing and computing data. Multiple in-memory computing layers are stacked vertically through interconnection to form the overall structure of a 3D CiM array.

[0043] In the embodiments of this disclosure, the same multi-layer in-memory computing structure is used to achieve inter-layer alignment and scheduling of different modalities, solving the problem of cross-modal feature interaction and computational bottlenecks in multimodal neural network inference. Here, "same" means that each layer follows the same template in circuit design and interconnection topology (e.g., all include CiM arrays, control units, and cache interfaces); however, they differ in storage media, bit precision, task mapping, or modal calibration strategies. By using a multi-layer CiM template with the same structure, functional mapping and calibration functions for different modalities are achieved.

[0044] This disclosure provides a hierarchical CiM architecture system, scheduling method, apparatus, device, and storage medium. By vertically interconnecting the control layer and the 3D in-memory computing array, and integrating multiple in-memory computing layers for different modalities into the 3D in-memory computing array, the system effectively decouples the structural heterogeneity conflicts of multimodal models and reduces architectural complexity. At the same time, by leveraging the structural characteristics of 3D vertical interconnection, it overcomes the bottleneck of unreasonable bandwidth allocation for data loading, significantly improving the efficiency, energy efficiency, and long-term operational stability of multimodal inference at the edge, and fully adapting to the resource constraints of edge scenarios.

[0045] Example Three Building upon the above embodiments, this disclosure provides a more detailed hierarchical CiM architecture system. Specifically, it proposes a hierarchical CiM (Compute-in-Memory) architecture for feature cache (KV-cache) in multimodal inference processes, achieving energy efficiency optimization through cross-layer dynamic scheduling. The hierarchical CiM architecture system in this disclosure configures heterogeneous memory arrays in different tiers and dynamically allocates cache and computing power for each modality using a modality-aware scheduler, achieving intelligent resource management by allocating storage media according to modal characteristics and adjusting computing paths according to task load.

[0046] like Figure 3 As shown, the layered CiM architecture system includes: A vertically interconnected control layer and a 3D in-memory computing array, wherein the 3D in-memory computing array integrates multiple in-memory computing layers for different modes.

[0047] Specifically, the CiM architecture, or in-memory computing architecture, is a hardware architecture that deeply integrates storage and computing functions. Its core logic is to allow computation to be completed directly within the storage medium, breaking with traditional von Neumann architectures. The limitation of physically separating computing units (such as CPU / GPU) and storage units (such as memory / hard disk) in the von Neumann architecture fundamentally reduces the frequent data transfer between storage and computing, thereby solving the memory wall and power wall problems with high data transfer latency and high power consumption. It is especially suitable for matrix multiplication or multiply-accumulate (MAC) intensive tasks in the field of AI, such as multimodal inference and deep learning acceleration. By storing the weights of the neural network in the storage array (such as SRAM, DRAM, MRAM, memristor array), the input data is directly loaded into the storage array. The multiply-accumulate operation is completed in parallel within the array by utilizing the physical characteristics of the storage medium (such as the conductivity characteristics of memristors and the current superposition characteristics of SRAM), without the need to transfer data to external computing units. Finally, only the operation result is output.

[0048] In embodiments of this disclosure, the control layer is used to perform modal perception on the acquired modal data and schedule the modal data to the in-memory computing layer in the 3D in-memory computing array based on the result of the modal perception.

[0049] In some embodiments, the control layer includes a calibration unit and a scheduler; the calibration unit is used to perform modal basic calibration on the modal data after acquiring the modal data to obtain the modal perception result; the scheduler is used to dynamically schedule the modal data to the in-memory computing layer in the 3D in-memory computing array according to the data information of the modal data and the device computing load.

[0050] Specifically, due to the differences in feature distribution, accuracy requirements, dynamic range, and temporal characteristics among different modalities (images, speech, text, time-series signals, etc.), a modality-aware calibration unit is set up in the hierarchical CiM architecture system in this embodiment of the disclosure. This unit is used to perform basic calibration of the amplitude or feature range of data from different modalities during multimodal inference, thereby improving the accuracy of multimodal alignment. The calibration unit can select the corresponding calibration path or strategy based on the modality information provided by the scheduler to achieve the adaptation of different modal features in a unified feature space. It should be noted that the specific calibration mechanism of the calibration unit can be implemented by hardware or software, and its internal structure can be flexibly configured according to the application scenario. This disclosure does not impose any limitations on this.

[0051] In Compute-in-Memory (CiM) or Processing-in-Memory (PIM) architectures, computations are often performed directly within the storage array. For example, ReRAM / PCM implements analog multiplication and addition, while SRAM implements bitwise digital accumulation. In this structure, numerical accuracy, dynamic range, noise distribution, and nonlinear distortion characteristics all affect the final computational results. Therefore, different storage media are required for modal data of different modes. Furthermore, when the model is a multimodal generative model (Multimodal LLM / VLM), the eigenvectors of different modes are fundamentally different, causing the same hardware error to be amplified or masked in different modes.

[0052] As shown in Table 1 below, the same hardware noise will cause semantic misalignment in the text modality, but is almost invisible in the image modality.

[0053] Table 1. Correspondence between different modal calibrations, hardware, and error sensitivities.

[0054] Therefore, modal data calibration is required through a modal calibration unit, rather than performing uniform linear compensation on the modal data.

[0055] Specifically, the modality-aware scheduler in the hierarchical CiM architecture system dynamically allocates and schedules computing units and in-memory array resources for each modality during multimodal inference based on the processing requirements, data characteristics, and computational load status of different modalities. The scheduler maps data from different modalities to the corresponding in-memory layers based on modality characteristics and task priorities, achieving modality-adaptive computational path allocation. It can identify the modality type of input data and select appropriate computational paths or resource configurations based on the system's operating status, thereby improving overall computing power utilization efficiency while ensuring the synchronization of multimodal tasks. The scheduler can work in conjunction with the calibration unit to perform unified task scheduling and execution management for calibrated multimodal data. It should be noted that the scheduler's scheduling mechanism can be implemented by hardware logic circuits, programmable control units, or embedded software modules, and its specific algorithms and scheduling strategies can be flexibly configured according to the application scenario; this disclosure does not impose any limitations.

[0056] In addition, in some embodiments, the scheduler has a dynamic feature cache scheduling mechanism. The feature cache is used for both the KV-cache of the attention model and for storing intermediate modal features. The scheduler selects different cache layers for hierarchical cache mapping according to modality type, dynamically adjusts the cache retention period according to the access frequency and time window of the modality, and dynamically reduces the power supply voltage or accuracy of some layers according to the task load to optimize energy consumption through energy efficiency self-adjustment.

[0057] In some embodiments, the scheduler and calibration unit can operate in parallel or partially overlap in timing. Specifically, at the control logic level, the system first identifies the modality type of the input task by the scheduler and determines the mapped in-memory computing level and resource allocation scheme based on modal characteristics (such as bandwidth requirements, accuracy requirements, access timing, etc.). The scheduler generates corresponding inter-layer mapping tables and scheduling instructions, and sends them to the corresponding layer control units. At the data flow level, before entering the designated layer for computation, the input multimodal features are normalized and scaled by the calibration unit to ensure consistency between different modalities in feature space, precision bit width, timing step size, etc. The calibrated feature data flows to the corresponding in-memory computing layer according to the scheduling instructions, realizing layered in-memory computing execution and dynamic feature cache management. In the system control path, the scheduler first determines the resource allocation and execution strategy; in the data execution path, the calibration unit prioritizes the consistency processing of input features. The scheduler and calibration unit coordinate in parallel in time sequence to ensure that cross-modal features can be efficiently migrated between different levels and maintain semantic consistency. This avoids the serial bottleneck of scheduling and alignment in traditional schemes, and takes into account both system-level decision flexibility and data-level processing accuracy, providing high-energy-efficiency hardware support for the dynamic scheduling of multimodal feature cache.

[0058] In addition, in some embodiments, the control layer also includes an interconnection and task control module, which enables the transmission of data and control signals between the functional layers through a vertical interconnection structure.

[0059] In embodiments of this disclosure, the 3D in-memory computing array is used to, upon receiving the modal data, select an in-memory computing layer corresponding to the modal data to perform inference on the modal data according to the processing requirements of the modal data.

[0060] Specifically, the in-memory computing layer (Tier), or in-memory computing unit layer, consists of independent functional modules that combine storage and computing capabilities, specializing in processing a specific type or class of data (such as text, images, and audio). The core hardware foundation of the in-memory computing layer is the CiM array, which, in collaboration with auxiliary components such as local control units and cache interfaces, forms a complete unit capable of independently storing and computing data. Multiple in-memory computing layers are stacked vertically through interconnection to form the overall structure of a 3D CiM array.

[0061] The hardware architecture in this embodiment includes multi-layered in-memory computing units (CiM cores), each layer corresponding to a modality or modal processing stage (such as vision, speech, audio, etc.). These in-memory computing units are connected via vertical interconnects to form a 3D stacked architecture, improving energy efficiency and reducing access latency. Each tier can utilize heterogeneous storage media (such as DRAM, ReRAM, PCM, etc.) to optimize the storage and processing of modal features according to the needs of different modalities. Each layer can be configured with different computing units according to different modal requirements, supporting efficient cross-modal embedding alignment.

[0062] Specifically, in some embodiments, such as Figure 3 As shown, the 3D in-memory computing array comprises three in-memory layers. The first in-memory layer is a high-speed in-memory layer, employing an SRAM-CiM array, used for processing text modal computation and / or storing the active feature values ​​of the modal data and intermediate processing results of the modal data. Further, the 3D in-memory computing array also includes a second in-memory layer; this second in-memory layer is a high-density in-memory layer, employing an RRAM-CiM array, used for storing visual modal features and / or sparse computation. Further, the 3D in-memory computing array also includes a third in-memory layer; this third in-memory layer is an analog / temporal in-memory layer, employing a PCM-CiM or hybrid memristor array, used for processing continuous modal signals such as speech and / or temporal signals. In some embodiments, the first in-memory computing layer, the second in-memory computing layer, and the third in-memory computing layer are vertically interconnected with the control layer to form the 3D in-memory computing array of the 3D stacked architecture. This achieves vertical integration and hierarchical scheduling of heterogeneous in-memory computing units at the physical level, which can significantly reduce cross-modal data transmission bandwidth and improve energy efficiency and parallel inference performance.

[0063] In the embodiments of this disclosure, the same multi-layer in-memory computing structure is used to achieve inter-layer alignment and scheduling of different modalities, solving the problem of cross-modal feature interaction and computational bottlenecks in multimodal neural network inference. Here, "same" means that each layer follows the same template in circuit design and interconnection topology (e.g., all include CiM arrays, control units, and cache interfaces); however, they differ in storage media, bit precision, task mapping, or modal calibration strategies. By using a multi-layer CiM template with the same structure, functional mapping and calibration functions for different modalities are achieved.

[0064] This disclosure provides a hierarchical CiM architecture system, scheduling method, apparatus, device, and storage medium. By vertically interconnecting the control layer and the 3D in-memory computing array, and integrating multiple in-memory computing layers for different modalities into the 3D in-memory computing array, the system effectively decouples the structural heterogeneity conflicts of multimodal models and reduces architectural complexity. At the same time, by leveraging the structural characteristics of 3D vertical interconnection, it overcomes the bottleneck of unreasonable bandwidth allocation for data loading, significantly improving the efficiency, energy efficiency, and long-term operational stability of multimodal inference at the edge, and fully adapting to the resource constraints of edge scenarios.

[0065] Example Four Based on the above embodiments, this disclosure provides a hierarchical CiM-based dynamic scheduling method, including: Step 401: The acquired modal data is modally sensed by the control layer, and the modal data is scheduled to the in-memory computing layer in the 3D in-memory computing array according to the result of the modal sense.

[0066] Step 402: After receiving the modal data, the 3D in-memory computing array selects the corresponding in-memory computing layer to perform inference on the modal data according to the processing requirements of the modal data. The 3D in-memory computing array integrates multiple in-memory computing layers for different modalities.

[0067] This disclosure provides a hierarchical CiM architecture system, scheduling method, apparatus, device, and storage medium. By vertically interconnecting the control layer and the 3D in-memory computing array, and integrating multiple in-memory computing layers for different modalities into the 3D in-memory computing array, the system effectively decouples the structural heterogeneity conflicts of multimodal models and reduces architectural complexity. At the same time, by leveraging the structural characteristics of 3D vertical interconnection, it overcomes the bottleneck of unreasonable bandwidth allocation for data loading, significantly improving the efficiency, energy efficiency, and long-term operational stability of multimodal inference at the edge, and fully adapting to the resource constraints of edge scenarios.

[0068] Example Five Based on the above embodiments, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the above embodiments.

[0069] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in the above embodiments.

[0070] In some embodiments of this example, a computer program product is provided, including computer program instructions, which, when executed by a processor, implement the steps of the method described in the above embodiments.

[0071] The processor may include, but is not limited to, one or more processors or microprocessors. Each processor may be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic component, for executing the methods in the above embodiments.

[0072] Computer-readable storage media can be implemented by any type of volatile or non-volatile storage device or a combination thereof. Computer-readable storage media can include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, and computer storage media (e.g., hard disks, floppy disks, solid-state drives, removable disks, CDs). ROM, DVD ROM, Blu-ray discs, etc.

[0073] Computer-readable storage media may also store at least one computer-executable program instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0074] In addition, the computer device may include (but is not limited to) a data bus, an input / output (I / O) bus, a display, and input / output devices (e.g., keyboard, mouse, speakers, etc.).

[0075] The processor can communicate with external devices via the I / O bus through wired or wireless networks.

[0076] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0077] This disclosure provides a hierarchical CiM architecture system, scheduling method, apparatus, device, and storage medium. By vertically interconnecting the control layer and the 3D in-memory computing array, and integrating multiple in-memory computing layers for different modalities into the 3D in-memory computing array, the system effectively decouples the structural heterogeneity conflicts of multimodal models and reduces architectural complexity. At the same time, by leveraging the structural characteristics of 3D vertical interconnection, it overcomes the bottleneck of unreasonable bandwidth allocation for data loading, significantly improving the efficiency, energy efficiency, and long-term operational stability of multimodal inference at the edge, and fully adapting to the resource constraints of edge scenarios.

[0078] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0079] It should be noted that, in this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element limited by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0080] While the embodiments disclosed herein are as described above, the foregoing content is merely for the purpose of facilitating understanding of this disclosure and is not intended to limit this disclosure. Any person skilled in the art to which this disclosure pertains may make any modifications and changes in form and detail of the implementation without departing from the spirit and scope of this disclosure; however, the scope of patent protection of this disclosure shall still be determined by the scope defined in the appended claims.

Claims

1. A layered CiM architecture system, characterized in that, include: A vertically interconnected control layer and a 3D in-memory computing array, wherein the 3D in-memory computing array integrates multiple in-memory computing layers for different modes; The control layer is used to perform modal perception on the acquired modal data and schedule the modal data to the in-memory computing layer in the 3D in-memory computing array based on the result of the modal perception. The 3D in-memory computing array is used to select the in-memory computing layer corresponding to the modal data to perform inference on the modal data according to the processing requirements of the modal data after receiving the modal data.

2. The system according to claim 1, characterized in that, The 3D in-memory computing array includes a first in-memory computing layer; The first in-memory layer uses an SRAM-CiM array to process text modality calculations and / or store the active feature values ​​of the modality data and the intermediate processing results of the modality data.

3. The system according to claim 2, characterized in that, The 3D in-memory computing array also includes a second in-memory computing layer; The second in-memory layer uses an RRAM-CiM array to store visual modal features and / or sparse computations.

4. The system according to claim 3, characterized in that, The 3D in-memory computing array also includes a third in-memory computing layer; The third storage layer employs PCM-CiM or a hybrid memristor array to process speech and / or time-series continuous modal signals.

5. The system according to claim 4, characterized in that, The first in-memory computing layer, the second in-memory computing layer, and the third in-memory computing layer are vertically interconnected with the control layer in sequence to form the 3D stacked architecture of the 3D in-memory computing array.

6. The system according to any one of claims 1 to 5, characterized in that, The control layer includes a calibration unit and a scheduler; The calibration unit is used to perform modal basic calibration on the modal data after acquiring the modal data to obtain the modal sensing result; The scheduler is used to dynamically schedule the modal data to the in-memory computing layer in the 3D in-memory computing array based on the modal data information and the device computing load.

7. A hierarchical CiM-based dynamic scheduling method, characterized in that, Applied to a hierarchical CiM architecture system as described in any one of claims 1 to 6, the method comprises: The control layer performs modal perception on the acquired modal data and schedules the modal data to the in-memory computing layer in the 3D in-memory computing array based on the results of the modal perception. After receiving the modal data, the 3D in-memory computing array selects the in-memory computing layer corresponding to the modal data to perform inference on the modal data according to the processing requirements of the modal data. The 3D in-memory computing array integrates multiple in-memory computing layers for different modalities.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method of claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method of claim 7.

10. A computer program product comprising computer program instructions, characterized in that, When the computer program is executed by the processor, it implements the method of claim 7.

Images