Memory scheduling methods, related devices and media
By obtaining the available capacity information of the multi-level on-chip storage space of the computing unit and the storage occupancy of the target operator, candidate block sizes are generated and block strategies that meet the capacity constraints are selected. This solves the problem that static block strategies cannot adapt to runtime storage changes, and achieves efficient utilization of storage resources and stability of operator execution.
Patent Information
- Application Number
- CN202611604390.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-16
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies, when dealing with complex operators, suffer from problems such as memory overflow or low resource utilization because the static block partitioning strategy cannot adapt to the dynamic changes in multi-level on-chip storage space during runtime.
By obtaining the available capacity information of the multi-level on-chip storage space of the computing unit in the current running state, the storage occupancy of the target operator is determined, candidate block sizes are generated, block strategies that meet the capacity constraints are selected, and the blocking and layout of data in the multi-level storage space are dynamically adjusted.
This avoids the risk of on-chip memory overflow, reduces idle storage resources, and improves the executability and operational stability of complex operators.
Smart Images

Figure CN123114777A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence chip technology, and in particular to a memory scheduling method, related apparatus and medium. Background Technology
[0002] With the rapid development of artificial intelligence technology, in-memory computing (CIM) architecture has attracted widespread attention because it can effectively alleviate the memory wall problem in the traditional von Neumann architecture. In CIM computing units, a multi-level storage hierarchy is usually adopted to balance performance and cost: off-chip memory (e.g., dynamic random access memory, DRAM) is used to store large-scale input / output tensors, with large capacity but high memory access latency; on-chip scratchpad memory (e.g., Scratchpad Memory, SPM) is explicitly controlled by software, with low latency and high bandwidth, but limited capacity; on-chip shared memory (e.g., Shared Memory, SMEM) can be accessed by multiple computing units and is suitable for storing parameter data such as weights and biases.
[0003] Existing operator execution frameworks typically require pre-allocating a fixed-size on-chip memory space for data movement when processing input data from off-chip memory, and also need to know in advance which input data resides in off-chip memory. This static pre-allocation method lacks a fixed paradigm for the selection of on-chip temporary storage and on-chip shared memory, relying mainly on the developer's experience in estimating the current input, failing to maximize memory resource utilization, and not conforming to the generalization design principle. Existing block partitioning methods are usually planned for fixed network layers, fixed block structures, or during offline compilation, making it difficult to adaptively adjust the block size according to the actual available on-chip memory status of the runtime device. When on-chip temporary storage is already reserved by the runtime allocator, cached, or occupied by other operators, static block partitioning parameters cannot accurately reflect the actual available capacity, easily leading to on-chip memory overflow during operator execution, or reducing on-chip memory utilization due to overly conservative block partitioning strategies. Summary of the Invention
[0004] This disclosure provides a memory scheduling method, related apparatus, and medium, aiming to solve the problem of memory overflow or low resource utilization caused by the inability of static block partitioning strategies to adapt to the dynamic changes of multi-level on-chip storage space during runtime when processing complex operators.
[0005] According to one aspect of this disclosure, a memory scheduling method is provided, characterized by comprising:
[0006] Obtain information on the available capacity of multi-level on-chip storage space of the computing unit in the current running state;
[0007] The execution information of the target operator is obtained, and multiple storage occupancy items involved in the execution of the target operator are determined based on the execution information. Each storage occupancy item corresponds to at least one data object and a corresponding storage space, and is used to characterize the storage occupancy of the corresponding data object in the storage space.
[0008] One or more candidate block sizes are generated, and the capacity requirement in the multi-level on-chip storage space is determined based on each of the aforementioned storage occupancy items when using each of the aforementioned candidate block sizes for block partitioning.
[0009] Based on the available capacity information, the capacity constraints of the multi-level on-chip storage space are determined, and according to the capacity requirements of each candidate block size, candidate block sizes that meet the capacity constraints of the multi-level on-chip storage space are selected, and a target block strategy is determined from the selected candidate block sizes.
[0010] The target operator is executed according to the target block strategy.
[0011] Optionally, determining the multiple storage items involved in the execution of the target operator based on the execution information includes:
[0012] Obtain the data volume of each data object and its storage usage-related attribute information;
[0013] Based on the data volume of each data object and the attribute information, the storage occupancy of the corresponding storage item is determined;
[0014] The attribute information includes at least one of the following: data type, storage layout, alignment requirements, buffering method, and temporary storage requirements.
[0015] Optionally, determining the multiple storage items involved in the execution of the target operator based on the execution information includes:
[0016] Based on the data role of each data object in the target operator, each data object is determined as a storage occupancy item corresponding to a corresponding storage space, wherein data objects with different data roles correspond to different storage spaces.
[0017] Optionally, determining the capacity constraints of the multi-level on-chip storage space based on the available capacity information includes:
[0018] Based on the available capacity of each storage space in the current running state and the corresponding security factor, the capacity budget of each storage space is determined, and the capacity budget is determined as the capacity constraint of the corresponding storage space.
[0019] The step of filtering candidate block sizes that meet the capacity constraints of the multi-level on-chip storage space based on the capacity requirements of each candidate block size includes:
[0020] When dividing the storage space using each of the candidate block sizes, the capacity requirement in each of the storage spaces will be compared with the capacity constraint of the corresponding storage space.
[0021] Candidate block sizes that satisfy the capacity constraints of the multi-level on-chip storage space are determined if the capacity requirements in each of the aforementioned storage spaces do not exceed the corresponding capacity constraints.
[0022] Optionally, the multi-level on-chip storage space includes at least a first storage space and a second storage space;
[0023] The generation of one or more candidate blocks includes:
[0024] The generation of one or more candidate block sizes includes:
[0025] Determine the first candidate block size for the first block dimension;
[0026] Based on the storage occupancy item corresponding to the first storage space, determine the capacity requirement of the first storage space when using the first candidate block size;
[0027] If the capacity requirement of the first storage space does not exceed the capacity constraint of the first storage space, the second candidate block size of the second block dimension is determined based on the available capacity information of the second storage space and the storage occupancy item corresponding to the second storage space.
[0028] The corresponding candidate block size is obtained by combining the first candidate block size and the second candidate block size.
[0029] Optionally, the first storage space is a shared memory, the second storage space is a temporary storage memory, the first block dimension is the column direction, and the second block dimension is the row direction;
[0030] The step of determining the capacity requirement of the first storage space when using the first candidate block size based on the storage occupancy item corresponding to the first storage space includes:
[0031] Based on at least one of the corresponding storage occupancy items in the weight data and bias data, determine the shared memory capacity requirement corresponding to the block size of each candidate column;
[0032] The step of determining the second candidate block size for the second block dimension based on the available capacity information of the second storage space and the storage occupancy item corresponding to the second storage space includes:
[0033] If the shared memory capacity requirement does not exceed the shared memory capacity constraint, the row block size under the current candidate column block size is determined according to the temporary storage capacity constraint and at least one corresponding storage occupancy item among the input data, output data and intermediate data.
[0034] Optionally, determining the row block size under the current candidate column block size includes:
[0035] The initial value of the row block size is determined based on the capacity constraint of the temporary storage and the storage occupancy corresponding to a unit row block;
[0036] Based on the target storage layout and / or alignment requirements, determine the actual temporary storage capacity requirement corresponding to the initial value;
[0037] If the actual temporary storage capacity requirement exceeds the capacity constraint of the temporary storage, the row block size is reduced, and the actual temporary storage capacity requirement corresponding to the reduced row block size is re-determined until the actual temporary storage capacity requirement does not exceed the capacity constraint of the temporary storage.
[0038] Optionally, determining the target segmentation strategy from the selected candidate segmentation sizes includes:
[0039] When there are multiple candidate block sizes selected, the evaluation result of each candidate block size is determined according to the preset evaluation index;
[0040] The target segmentation strategy is determined based on the evaluation results of each candidate segmentation size;
[0041] The preset evaluation indicators include at least one of the following: the amount of data processed in a single transaction, the number of blocks, the storage space utilization rate, and the number of data moves between different storage spaces.
[0042] Optionally, executing the target operator according to the target block strategy includes:
[0043] Determine the current storage location of the data to be processed;
[0044] When the data to be processed needs to be migrated from the current storage location to the target storage location for executing the target operator, the data to be processed is divided into blocks according to the target block division strategy, and the blocks of data to be executed are migrated to the target storage location.
[0045] The target operator is executed based on the data located at the target storage location.
[0046] Optionally, before executing the target operator based on the data located at the target storage location, the method further includes:
[0047] Determine the current storage layout of the data blocks to be executed;
[0048] If the current storage layout does not meet the execution requirements of the target operator, the storage layout of the data blocks to be executed is transformed.
[0049] If the current storage layout meets the execution requirements, the target operator is executed directly based on the data block to be executed.
[0050] Optionally, the storage layout conversion includes:
[0051] Given that the current storage layout is global row-first and the target storage layout is intra-block row-first, the data blocks are rearranged so that the data within each data block is stored continuously in rows.
[0052] According to one aspect of this disclosure, a memory scheduling apparatus is provided, characterized in that it comprises:
[0053] The information acquisition unit is used to acquire information about the available capacity of the multi-level on-chip storage space of the computing unit in the current running state;
[0054] A storage occupancy determination unit is used to obtain the execution information of the target operator and determine multiple storage occupancy items involved in the execution of the target operator based on the execution information. Each storage occupancy item corresponds to at least one data object and a corresponding storage space, and is used to characterize the storage occupancy of the corresponding data object in the storage space.
[0055] A capacity requirement determination unit is used to generate one or more candidate block sizes and, based on each of the storage occupancy items, determine the capacity requirement in the multi-level on-chip storage space when using each candidate block size for block division.
[0056] The strategy determination unit is used to determine the capacity constraints of the multi-level on-chip storage space based on the available capacity information, and to filter candidate block sizes that meet the capacity constraints of the multi-level on-chip storage space according to the capacity requirements of each candidate block size, and to determine the target block strategy from the filtered candidate block sizes.
[0057] An execution control unit is used to execute the target operator according to the target block strategy.
[0058] According to one aspect of this disclosure, a computing unit is provided, characterized in that it comprises:
[0059] The execution unit is used to execute the target operator;
[0060] Multi-level on-chip storage space is used to store data involved in the execution of the target operator;
[0061] The memory scheduling module is used to implement the memory scheduling method described above, thereby dynamically determining the target block strategy to guide the execution unit to perform block movement and layout transformation of data under the premise of satisfying the capacity constraints of the multi-level on-chip storage space, and thus completing the execution of the target operator.
[0062] Optionally, the multi-level on-chip storage space includes at least a temporary storage memory and a shared memory; the temporary storage memory is used to store at least one of input data, output data, intermediate data, residual data, and gating data; the shared memory is used to store at least one of weight data and bias data; the execution unit is used to access the data in the temporary storage memory and the shared memory to execute the target operator.
[0063] According to one aspect of this disclosure, an electronic device is provided, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program is executed by the processor to implement the memory scheduling method as described in any of the preceding claims.
[0064] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores one or more programs, which can be executed by one or more processors to implement the memory scheduling method as described in any of the preceding claims.
[0065] In the memory scheduling method proposed in this disclosure, by obtaining the available capacity information of the multi-level on-chip storage space of the computing unit in the current runtime state, and determining the multiple storage occupancy items involved based on the execution information of the target operator, the actual storage occupancy of each data object in the corresponding storage space can be accurately characterized. Based on this, candidate block sizes are generated and their capacity requirements in the multi-level on-chip storage space are calculated. Then, candidate block sizes that meet the capacity constraints of the multi-level on-chip storage space are selected to determine the target block strategy, realizing dynamic block scheduling based on the actual available resources at runtime. Compared with the existing technology that relies on static pre-allocation or developer experience estimation, this solution can avoid the overflow risk caused by on-chip storage being occupied by other tasks, while reducing storage resource idleness caused by overly conservative estimation, and improving the executability and operational stability of large-size inputs or complex fusion operators in a multi-level storage architecture.
[0066] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description
[0067] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.
[0068] Figure 1 This is a schematic diagram of the structure of the computing chip provided in the embodiments of this disclosure;
[0069] Figure 2 This is a schematic diagram of the block execution data flow provided in an embodiment of this disclosure;
[0070] Figure 3 This is a flowchart illustrating the memory scheduling method provided in an embodiment of this disclosure;
[0071] Figure 4 This is a flowchart illustrating the candidate block size generation steps provided in an embodiment of this disclosure;
[0072] Figure 5 This is a flowchart illustrating the steps for determining the target segmentation strategy provided in an embodiment of this disclosure;
[0073] Figure 6 This is a structural block diagram of the memory scheduling device provided in the embodiments of this disclosure;
[0074] Figure 7 This is a system architecture diagram of the electronic device provided in the embodiments of this disclosure. Detailed Implementation
[0075] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.
[0076] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:
[0077] An operator is an independently identifiable computational function that performs a predetermined data processing operation on a set of input data and produces corresponding output data. The operator can be described by a software behavioral model and has a corresponding hardware implementation. Operators include, but are not limited to, artificial intelligence operators, matrix operation operators, vector operation operators, data transformation operators, data transfer operators, and fusion operators composed of multiple sub-operators. An operator can be implemented by a single hardware computing unit or by multiple hardware computing units working together; a single hardware computing unit can also support the execution of multiple different operators.
[0078] Off-chip memory: refers to large-capacity main memory located outside the computing unit, such as dynamic random access memory (DRAM), which is used to store large-scale input / output tensors and operator execution results. It has the characteristics of large capacity but high memory access latency.
[0079] Scratchpad Memory (SPM): refers to high-speed local memory located inside the computing unit and explicitly controlled by software. It features low latency and high bandwidth, but has limited capacity. It is usually used to temporarily store blocks of input data, blocks of output data, and intermediate buffer data currently involved in the computation.
[0080] Shared memory (SMEM) refers to the storage space located inside the computing unit that can be accessed by multiple execution units. It is suitable for storing parameter data such as weights and biases, as well as temporary data that needs to be frequently shared and accessed. It can be read repeatedly during the entire execution of the operator without having to be moved from off-chip memory repeatedly.
[0081] Global Row First (ROW): This refers to the physical organization of data in which rows are stored consecutively throughout the entire tensor dimension. That is, all elements in the same row of the tensor are arranged adjacent to each other in terms of physical address, and the first element of the next row is immediately followed by the last element of the previous row. This arrangement is suitable for large-scale data storage in off-chip memory, which is convenient for accessing data in row order and batch moving. However, in block computing scenarios, it may lead to problems such as poor locality of cross-row data and low cache hit rate.
[0082] Intra-block row-first layout (TROW): This refers to dividing a tensor into multiple independent data blocks, where the data within each data block is still stored contiguously in rows, but different data blocks are physically separated or aligned to boundaries, and the contiguous row data of adjacent blocks is not required. This layout is designed specifically for temporary storage, enabling execution units to achieve optimal memory locality and memory access efficiency when processing individual data blocks. It also supports efficient mapping from global row-first layout through layout transformation operations, and is a key data organization form connecting off-chip storage and on-chip computing.
[0083] System architecture description applied in the embodiments of this disclosure
[0084] Figure 1 This is a schematic diagram of the structure of a computing chip according to an embodiment of this disclosure. Figure 1 As shown in the figure, the hierarchical connection between multiple computing units 110 in the computing chip 100 and the global memory (also known as off-chip memory) 190, as well as the functional module division within each computing unit, are illustrated. The computing chip 100 is suitable for high-concurrency, high-throughput computing tasks, including graphics processing, scientific computing, artificial intelligence training and inference scenarios. The computing units 110 can be any computing device with multi-level on-chip memory managed by software, such as in-memory computing devices, neural network processors (NPUs), graphics processing units (GPUs), and digital signal processors (DSPs).
[0085] The computing chip 100 includes multiple computing units 110 and global memory 190. The computing units 110 can be included in a cluster according to design requirements. Computing units 0 to 3 are shown as examples in the figure, but it should be understood that the architecture is not limited to this; the computing chip 100 can contain more or fewer computing units 110 depending on actual design needs. The global memory 190, as a unified address space in the chip system, provides a unified data access interface for the multiple computing units 110, supporting data sharing and collaborative computing across computing units 110. The global memory 190 is connected to each computing unit 110 through a high-bandwidth interconnect network, supporting cache coherency protocols to ensure data read / write consistency across multiple computing units.
[0086] Each computing unit 110 further includes a first instruction processing unit 120, a second instruction processing unit 130, an instruction fetch request processing unit 140, an instruction cache unit 150, shared memory 160, temporary storage memory 170, and a memory scheduling module 180. The first instruction processing unit 120 adopts a SIMT (Single Instruction Multiple Thread) execution model, including multiple execution units 121 for executing thread bundle-level instructions and a first scheduler 122 for thread bundle scheduling. Execution units 0 to 3 are shown in the figure, but in actual implementation, any number of execution units 121 may be included. The instruction cache unit 150 is responsible for caching thread bundle-level instructions and workgroup-level instructions prefetched from global memory 190 to reduce instruction access latency and improve instruction fetch efficiency.
[0087] The second instruction processing unit 130 includes a heterogeneous execution unit for executing workgroup-level instructions, such as a tensor operation component, a vector operation component, or a dedicated accelerator core. This heterogeneous execution unit 131 shares data access permissions with the first instruction processing unit 120 to the shared memory 160 and the temporary memory 170, but has independent instruction fetch paths and control logic to support the parallel scheduling and execution of different types of computing tasks.
[0088] like Figure 1 As shown, the instruction fetch request processing unit 140 is coupled to the first instruction processing unit 120 and the second instruction processing unit 130, respectively, and is used to distribute the instruction stream from the global memory 190 or the instruction cache unit 150 to the corresponding processing unit according to the instruction type. Specifically, for thread bundle-level parallel computing tasks, the instruction fetch request processing unit 140 routes SIMT instructions to the first instruction processing unit 120; for workgroup-level control flow, data transfer, or special function computation tasks, they are routed to the heterogeneous execution unit 131 in the second instruction processing unit 130. This hierarchical instruction fetching mechanism effectively reduces the instruction decoding pressure and improves the overall instruction throughput efficiency.
[0089] Shared memory 160, as a high-speed, configurable on-chip storage resource, is accessed by execution units 121 and heterogeneous execution units 131 within multiple computing units 110. It is used to store frequently reused data such as weights, biases, and intermediate results. Its capacity and bandwidth can be dynamically allocated according to operator requirements and managed at runtime through the memory scheduling module 180. Temporary storage 170 is explicitly controlled by software and is dedicated to temporarily storing currently processed data tiles. It supports fast read / write and layout conversion operations and serves as a crucial buffer layer connecting global memory 190 and the computing core.
[0090] The memory scheduling module 180 is the core functional unit of this embodiment. Coupled between the temporary storage 170, the shared storage 160, and the global memory 190, it is used to obtain the available capacity information of each storage space during runtime, determine the storage occupancy based on the execution information of the target operator, generate candidate block sizes, and filter target block strategies that meet capacity constraints, thereby guiding the movement and organization of data between multi-level storage. Through the dynamic scheduling capability of this module, the computing chip 100 can adaptively adapt to the memory access modes and runtime resource states of different operators without relying on static compilation planning, thereby achieving stable and efficient operator execution in high-concurrency and high-throughput scenarios.
[0091] The actual computation of the target operator is completed in the execution unit 121 of the first instruction processing unit 120 or the heterogeneous execution unit 131 of the second instruction processing unit 130. Before execution, the memory scheduling module 180 moves the data to be processed from the global memory 190 to the temporary storage memory 170 according to the block size, based on the target block partitioning strategy, and completes the layout conversion from global row-first arrangement to intra-block row-first arrangement during the transfer process; at the same time, parameter data such as weights and biases are preloaded into the shared memory 160 for repeated reading by the computation unit. Subsequently, the execution unit 121 or the heterogeneous execution unit 131 directly reads the current data block to be computed from the temporary storage memory 170 and reads the corresponding parameters from the shared memory 160, and performs matrix multiplication, convolution, normalization or other fusion operations according to the operator instructions. The calculated output block is stored in the temporary storage memory 170 through the write-back path, and then refreshed to the output tensor region of the global memory 190 as needed. Throughout the execution process, the memory scheduling module 180 continuously monitors the on-chip storage occupancy status and dynamically adjusts the size and storage allocation of the next block after each block is executed, ensuring that the operator achieves optimal execution efficiency while meeting multi-level storage capacity constraints. It should be noted that the number of levels of the multi-level on-chip storage space described in this application embodiment is not limited to two levels, and may include more levels such as data level memory (DLM). This application does not impose specific limitations on this.
[0092] Figure 2 This is a schematic diagram of the block execution data flow provided in an embodiment of this disclosure. For example... Figure 2As shown, the block execution data flow of this embodiment involves data interaction between off-chip memory 190, temporary storage memory 170, shared memory 160, and execution units (which may be execution unit 121 or heterogeneous execution unit 131). Specifically, the input tensor is stored in off-chip memory 190 in row-first order (ROW). According to the memory scheduling method of this embodiment, the input tensor is divided into multiple data blocks, denoted as tile0, tile1, tile2, etc. The input tensor is moved from off-chip memory 190 to temporary storage memory 170 block by block according to the target block strategy. During the moving process, the data storage layout is changed from global row-first order (ROW) to intra-block row-first order (TROW), that is, the data within each data block is stored continuously in rows, but the blocks maintain independent physical boundaries. This layout conversion enables the execution unit to access data with more efficient locality and reduces memory access latency. After the moving and layout conversion, the current data block to be computed is loaded into temporary storage memory 170, waiting to be read by the execution unit. This data block represents the input data unit participating in the computation within the current iteration cycle. The execution unit reads the current data block to be computed from the temporary storage memory 170, reads parameters such as weight data and bias data from the shared memory 160, and executes the computation operation of the target operator. Weights and biases typically remain constant or change at a low frequency throughout the operator's execution; therefore, they are suitable for storage in the shared memory 160 for repeated access by the execution unit, avoiding the overhead of repeatedly moving data from the external memory 190. The output data block generated after the execution unit's computation is completed is written back to the external memory 190. Specifically, the output data block is written to the output tensor region in the external memory 190 in row-major (ROW) format, and is sequentially denoted as out0, out1, out2, etc.
[0093] Through the aforementioned data flow, this embodiment of the disclosure achieves efficient block execution under a multi-level storage architecture: input data enters temporary storage 170 after layout transformation from off-chip memory 190; the execution unit accesses the input data blocks in temporary storage 170 and the weight / bias parameters in shared memory 160 in parallel, completes the calculation, and writes the result back to off-chip memory 190. Throughout this process, the block size is dynamically determined by the runtime memory budget, which avoids on-chip memory overflow and maximizes on-chip memory utilization and execution unit efficiency. It should be noted that... Figure 2 The execution flow of only one data block is shown as an example; in actual execution, all data blocks will be processed sequentially according to the target block strategy until the computation of the entire input tensor is completed.
[0094] The memory scheduling method provided in this disclosure is as follows:
[0095] Figure 3This is a flowchart illustrating a memory scheduling method according to an embodiment of this disclosure. Figure 3 As shown, the memory scheduling method of this disclosure embodiment is composed of Figure 1 The memory scheduling module 180 in the middle executes the following steps:
[0096] Step S310: Obtain the available capacity information of the multi-level on-chip storage space of the computing unit in the current running state.
[0097] Step S320: Obtain the execution information of the target operator, and determine multiple storage occupancy items involved in the execution of the target operator based on the execution information, wherein each storage occupancy item corresponds to at least one data object and a corresponding storage space, and is used to characterize the storage occupancy of the corresponding data object in the storage space.
[0098] Step S330: Generate one or more candidate block sizes, and determine the capacity requirement in the multi-level on-chip storage space when using each candidate block size for block division based on each of the storage occupancy items.
[0099] Step S340: Determine the capacity constraints of the multi-level on-chip storage space based on the available capacity information, and filter candidate block sizes that meet the capacity constraints of the multi-level on-chip storage space according to the capacity requirements of each candidate block size, and determine the target block strategy from the filtered candidate block sizes.
[0100] Step S350: Execute the target operator according to the target block strategy.
[0101] Specifically, in step S310, the memory scheduling module 180 obtains the available capacity information of the multi-level on-chip storage space at the current moment by querying the runtime storage manager of the computing unit. The multi-level on-chip storage space includes at least a first storage space and a second storage space. The first storage space may be a temporary storage memory 170, and the second storage space may be a shared memory 160. In some embodiments, the memory scheduling module 180 also obtains the available capacity information of the off-chip memory 190 at the current moment. This available capacity information reflects the actual remaining space after deducting the space allocated to other concurrent tasks, cached data, and system reserved areas, providing accurate resource boundary constraints for the dynamic generation of subsequent block allocation strategies.
[0102] In step S320, the memory scheduling module 180 parses the execution descriptor of the target operator and extracts metadata information of data objects such as input tensors, output tensors, weights, biases, intermediate buffers, residuals, and gating. Specifically, the memory scheduling module 180 obtains the data volume of each data object and attribute information related to storage usage, and determines the storage usage of the corresponding storage item based on the data volume of each data object and the attribute information; wherein, the attribute information includes at least one of data type, storage layout, alignment requirements, buffering method, and temporary storage requirements. For example, considering alignment requirements can avoid the waste of extra padding space caused by address misalignment; when double buffering is enabled, the storage usage of the corresponding storage item is calculated as twice that of single buffering; considering the buffering method can accurately assess the additional storage overhead brought by the double buffering strategy; when temporary buffer space is needed, the data volume of the temporary buffer space is included in the corresponding storage item.
[0103] Meanwhile, the memory scheduling module 180 can also determine different data objects as storage occupancy items corresponding to different storage spaces according to their data roles in the target operator. In one example, the first type of data object corresponds to the storage occupancy item of the first storage space, and the second type of data object corresponds to the storage occupancy item of the second storage space. The first type of data object and the second type of data object have different data roles in the target operator. For example, the first type of data object may include large-capacity intermediate result data such as input tensors, output tensors, residuals, and gating. Its data role is the main read and write load in the operator execution process, and it has high requirements for storage bandwidth and capacity. Therefore, it is mapped to the storage occupancy item of the temporary storage memory 170. The second type of data object may include parameter data such as weights and biases. Its data role is a shared parameter that is read-only or written infrequently. Its access mode has high reusability but its individual size is small. Therefore, it is mapped to the storage occupancy item of the shared storage memory 160. Through the aforementioned data role-based differentiated mapping mechanism, the memory scheduling module 180 can construct a refined storage occupancy model that matches the actual hardware storage level in step S320. This not only accurately reflects the real resource consumption of each data object in the corresponding storage space, but also provides a structured modeling basis for the multi-level capacity requirement calculation of candidate block sizes in the subsequent step S330. This ensures that the generated target block strategy can simultaneously meet the collaborative constraints of the temporary storage 170 and the shared storage 160, avoiding performance bottlenecks or resource waste caused by storage allocation misalignment.
[0104] In step S330, the memory scheduling module 180 uses a two-level decoupling decision mechanism to generate candidate block sizes. Figure 4 This is a flowchart illustrating the candidate block size generation steps provided in an embodiment of this disclosure. Figure 4 As shown, the candidate block generation step in this embodiment includes:
[0105] Step S410: Determine the first candidate block size for the first block dimension.
[0106] Step S420: Determine the capacity requirement of the first storage space when using the first candidate block size based on the storage occupancy item corresponding to the first storage space.
[0107] Step S430: If the capacity requirement of the first storage space does not exceed the capacity constraint of the first storage space, determine the second candidate block size of the second block dimension based on the available capacity information of the second storage space and the storage occupancy item corresponding to the second storage space.
[0108] Step S440: Combine the first candidate block size and the second candidate block size to obtain the corresponding candidate block size.
[0109] Specifically, in step S410, the memory scheduling module 180 determines one or more first candidate block sizes (i.e., candidate column block sizes) for the first block dimension (i.e., column direction) based on the data type, tensor dimension, and preset block granularity search space of the target operator. The range of values for the first candidate block size can be set based on the physical parallelism of the hardware computing unit, the minimum addressing unit of the shared memory, or high-frequency block values in historical execution statistics to ensure that the generated candidate size meets both hardware constraints and is practically executable.
[0110] In step S420, the memory scheduling module 180 determines the capacity requirement of the shared memory when using each candidate column block size based on the storage occupancy item corresponding to the first storage space (i.e., the storage occupancy item corresponding to at least one parameter data among weight data and bias data). Specifically, for each candidate column block size, the total capacity requirement of the shared memory under that candidate size is obtained by combining the projection size of the weight and bias data in that column direction, the number of bytes of the data type, and the alignment padding overhead. If a candidate column block size causes the capacity requirement of the shared memory to exceed its capacity constraint, then the candidate size is directly excluded to avoid subsequent invalid calculations.
[0111] In step S430, the memory scheduling module 180 further determines the second candidate block size (i.e., candidate row block size) of the second block dimension (i.e., row direction) only when the capacity requirement of the first storage space does not exceed the capacity constraint of the first storage space. This determination is based on the available capacity information of the second storage space (i.e., temporary storage 170) and the storage occupancy items corresponding to the second storage space (e.g., at least one of the corresponding storage occupancy items of input data, output data, and intermediate data). Specifically, the memory scheduling module 180 first calculates the initial value of the row block size based on the capacity constraint of the temporary storage 170 and the storage occupancy corresponding to each row block. If the number of row blocks corresponding to the initial value exceeds the actual number of rows of input data in the row direction, the number of row blocks is taken as the actual number of rows. Subsequently, the actual temporary storage capacity requirement is verified when the initial value is used, in conjunction with the target storage layout (e.g., row-first arrangement within a block, TROW) and / or alignment requirements. If the actual capacity requirement exceeds the capacity constraint of the temporary storage, the row block size is gradually reduced and re-verified until the actual temporary storage capacity requirement does not exceed the capacity constraint of the temporary storage. This iterative adjustment mechanism ensures that, while maintaining layout compliance, the largest possible row block size is selected to reduce the total number of blocks.
[0112] In step S440, the memory scheduling module 180 combines the first candidate block size and the second candidate block size, which have undergone the above two-level verification, to form a complete corresponding candidate block size. Each candidate block size carries a clear multi-level storage allocation scheme and corresponding capacity requirement metadata, serving as the basic unit for screening and evaluation in step S340. Through this column-first, row-last two-dimensional decoupled decision-making mechanism, the embodiments of this disclosure effectively avoid the computational overhead caused by high-dimensional joint search, while prioritizing the storage feasibility of key parameter data such as weights and biases in the shared memory. On this basis, it maximizes the processing granularity of input and output data in the temporary storage memory, laying a solid foundation for the subsequent efficient determination of the target block strategy.
[0113] In step S340, the memory scheduling module 180 first determines the capacity budget of each storage space based on the available capacity of each storage space in the current running state and the corresponding safety factor, and determines the capacity budget as the capacity constraint of the corresponding storage space; then, it compares the capacity requirement in each storage space when using each candidate block size for block division with the capacity constraint of the corresponding storage space; and determines the candidate block size that does not exceed the corresponding capacity constraint in each storage space as the candidate block size that satisfies the capacity constraint of the multi-level on-chip storage space. Figure 5 This is a flowchart illustrating the target segmentation strategy determination steps provided in an embodiment of this disclosure. Figure 5 As shown, the target block strategy determination step in this embodiment includes:
[0114] Step S510: Based on the available capacity of each storage space in the current running state and the corresponding security factor, determine the capacity budget of each storage space, and determine the capacity budget as the capacity constraint of the corresponding storage space.
[0115] Step S520: Compare the capacity requirements in each storage space when dividing the blocks using each of the candidate block sizes with the capacity constraints of the corresponding storage space.
[0116] Step S530: Determine the candidate block size that satisfies the capacity constraints of the multi-level on-chip storage space, where the capacity requirements in each of the storage spaces do not exceed the corresponding capacity constraints.
[0117] Specifically, in step S510, the memory scheduling module 180 obtains the current available capacity of the shared memory 160 and the temporary memory 170 from the runtime memory manager, and multiplies them by a preset safety factor (e.g., 0.85 or 0.9) to obtain the capacity budget for each storage space, and determines the capacity budget as the capacity constraint for the corresponding storage space. This safety factor is used to reserve buffer space to cope with sudden memory allocation, interrupt handling, or system-level reserved areas during runtime, preventing operator execution failure or system crash caused by on-chip memory being completely filled due to capacity estimation deviation or dynamic interference.
[0118] In step S520, the memory scheduling module 180 traverses all candidate block sizes. For each candidate block size, it compares its capacity requirement in the shared memory 160 with the capacity constraint of the shared memory 160, and simultaneously compares its capacity requirement in the temporary storage 170 with the capacity constraint of the temporary storage 170. This comparison operation is performed independently on a per-memory-space basis, ensuring that any out-of-limit situation in any memory space can be accurately identified.
[0119] In step S530, a candidate block size is marked as a candidate block size that satisfies the capacity constraints of the multi-level on-chip storage space only if the capacity requirements of both the shared memory 160 and the temporary storage 170 do not exceed their respective capacity constraints; if the capacity requirement of any storage space exceeds its capacity constraint, the candidate block size is directly excluded. Through the above dual-constraint screening mechanism, this embodiment of the present disclosure eliminates operator execution anomalies caused by single-level storage exceeding limits from the source, ensuring the stable operation of various operators on the CIM computing unit under the multi-level storage architecture.
[0120] After screening candidate block sizes that meet capacity constraints, the memory scheduling module 180 further evaluates the selected candidate block sizes based on preset evaluation indicators when there are multiple selected candidate block sizes, in order to determine the final target block strategy. Specifically, the preset evaluation indicators include at least one of the following: data volume processed in a single run, number of blocks, storage space utilization, and number of data moves between different storage spaces. The memory scheduling module 180 calculates the quantitative value of each candidate block size under each evaluation indicator: the larger the data volume processed in a single run, the more computational tasks can be completed in each iteration, which helps to reduce the total number of blocks and scheduling overhead; the fewer the number of blocks, the lower the synchronization and switching frequency during operator execution, which is beneficial to improving end-to-end throughput; the higher the storage space utilization, the lower the idle degree of on-chip storage resources and the better the resource utilization efficiency; the fewer the number of data moves, the smaller the memory access latency and bandwidth occupation between off-chip storage and on-chip storage, and the better the overall execution performance. Subsequently, the memory scheduling module 180 sorts the evaluation indicators according to the pre-configured weight or priority rules, and selects the candidate block size with the highest comprehensive score as the target block strategy. This target block strategy not only explicitly specifies the specific block size in the column and row directions, but also binds it to the matching multi-level storage allocation scheme and layout transformation parameters, providing complete guidance for the efficient and stable execution of the operator in step S350.
[0121] In step S350, the memory scheduling module 180 first determines the current storage location of the data to be processed. If the data to be processed needs to be moved from the current storage location to the target storage location for executing the target operator (e.g., from off-chip memory 190 to temporary storage 170), the data to be processed is divided into blocks according to the target block strategy, and the blocks of data to be executed are moved to the target storage location; if the data to be processed is already located in the target storage location and meets the layout requirements, the moving step is skipped and the execution preparation stage is entered directly.
[0122] Furthermore, before executing the target operator based on the data located at the target storage location, the memory scheduling module 180 also determines the current storage layout of the data block to be executed. If the current storage layout does not meet the execution requirements of the target operator (e.g., the current layout is global row-first, while the operator requires intra-block row-first (TROW) layout), then the storage layout of the data block to be executed is transformed. Specifically, if the current storage layout is global row-first and the target storage layout is intra-block row-first, the data blocks are rearranged so that the data within each data block is stored contiguously in rows, while maintaining the independence of the physical boundaries between different data blocks. If the current storage layout already meets the execution requirements, then the target operator is executed directly based on the data block to be executed without additional transformation overhead.
[0123] Through the complete execution process described above, which includes data location determination, on-demand block relocation, and adaptive layout conversion, the embodiments of this disclosure ensure that input data is located in the correct storage space and has a physical organization form that meets the operator requirements when it enters the computing unit. This process tightly couples the three key steps of block relocation, relocation, and layout conversion under the target block relocation strategy, avoiding data redundancy relocation or format mismatch problems caused by independent decisions in each step; especially for specific layout requirements such as TROW, by completing the layout conversion immediately after relocation, the computing unit can access data with the most efficient locality, significantly reducing cache miss rate and memory access latency, and further improving the actual computing performance and end-to-end execution stability of operators under the CIM architecture.
[0124] The memory scheduling device provided in this disclosure
[0125] Figure 6 This is a block diagram of a memory scheduling apparatus according to an embodiment of this disclosure. Figure 6 As shown, the memory scheduling device 600 includes an information acquisition unit 610, a storage occupancy determination unit 620, a capacity requirement determination unit 630, a strategy determination unit 640, and an execution control unit 650.
[0126] The information acquisition unit 610 is used to acquire the available capacity information of the multi-level on-chip storage space of the computing unit in the current running state.
[0127] The storage occupancy determination unit 620 is used to obtain the execution information of the target operator and determine multiple storage occupancy items involved in the execution of the target operator based on the execution information. Each storage occupancy item corresponds to at least one data object and a corresponding storage space, and is used to characterize the storage occupancy of the corresponding data object in the storage space.
[0128] The capacity requirement determination unit 630 is used to generate one or more candidate block sizes and, based on each of the storage occupancy items, determine the capacity requirement in the multi-level on-chip storage space when using each candidate block size for block division.
[0129] The strategy determination unit 640 is used to determine the capacity constraints of the multi-level on-chip storage space based on the available capacity information, and to filter candidate block sizes that meet the capacity constraints of the multi-level on-chip storage space according to the capacity requirements of each candidate block size, and to determine the target block strategy from the filtered candidate block sizes.
[0130] The execution control unit 650 is used to execute the target operator according to the target block strategy.
[0131] Description of apparatus and devices according to embodiments of this disclosure
[0132] This disclosure also provides an electronic device 700, comprising:
[0133] At least one processor, and,
[0134] A memory that is communicatively connected to at least one processor; wherein,
[0135] The memory stores instructions that are executed by at least one processor to cause the at least one processor to perform the method as described in any of the above embodiments of this application when executing the instructions.
[0136] The following is combined Figure 7 The hardware structure of the electronic device is described in detail.
[0137] The electronic device includes a processor 710, a memory 720, a power supply component 730, a network interface 740, and an input / output interface 750. The processor 710 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 720 can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 720 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710 using the memory scheduling method of the embodiments of this disclosure. The power supply component 730 is used to provide stable operating voltage and current to the various components of the electronic device to ensure normal operation of the device. The network interface 740 is used to realize communication and interaction between this device and other devices. Communication can be realized through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.). The input / output interface 750 is used to realize information input and output, such as connecting peripherals such as keyboard, mouse, and monitor. The processor 710, memory 720, power supply component 730, network interface 740 and input / output interface 750 are connected to each other within the device through an internal bus.
[0138] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the memory scheduling method described in the above embodiments, which will not be repeated here.
[0139] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.
[0140] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0141] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.
[0142] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.
[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0144] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0145] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.
[0146] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.
Claims
1. A memory scheduling method, characterized in that, include: Obtain information on the available capacity of multi-level on-chip storage space of the computing unit in the current running state; The execution information of the target operator is obtained, and multiple storage occupancy items involved in the execution of the target operator are determined based on the execution information. Each storage occupancy item corresponds to at least one data object and a corresponding storage space, and is used to characterize the storage occupancy of the corresponding data object in the storage space. One or more candidate block sizes are generated, and the capacity requirement in the multi-level on-chip storage space is determined based on each of the aforementioned storage occupancy items when using each of the aforementioned candidate block sizes for block partitioning. Based on the available capacity information, the capacity constraints of the multi-level on-chip storage space are determined, and according to the capacity requirements of each candidate block size, candidate block sizes that meet the capacity constraints of the multi-level on-chip storage space are selected, and a target block strategy is determined from the selected candidate block sizes. The target operator is executed according to the target block strategy.
2. The memory scheduling method according to claim 1, characterized in that, The step of determining multiple storage occupancy items involved in the execution of the target operator based on the execution information includes: Obtain the data volume of each data object and its storage usage-related attribute information; Based on the data volume of each data object and the attribute information, the storage occupancy of the corresponding storage item is determined; The attribute information includes at least one of the following: data type, storage layout, alignment requirements, buffering method, and temporary storage requirements.
3. The memory scheduling method according to claim 1, characterized in that, The step of determining multiple storage occupancy items involved in the execution of the target operator based on the execution information includes: Based on the data role of each data object in the target operator, each data object is determined as a storage occupancy item corresponding to a corresponding storage space, wherein data objects with different data roles correspond to different storage spaces.
4. The memory scheduling method according to claim 1, characterized in that, Determining the capacity constraints of the multi-level on-chip storage space based on the available capacity information includes: Based on the available capacity of each storage space in the current running state and the corresponding security factor, the capacity budget of each storage space is determined, and the capacity budget is determined as the capacity constraint of the corresponding storage space. The step of filtering candidate block sizes that meet the capacity constraints of the multi-level on-chip storage space based on the capacity requirements of each candidate block size includes: When dividing the storage space using each of the candidate block sizes, the capacity requirement in each of the storage spaces will be compared with the capacity constraint of the corresponding storage space. Candidate block sizes that satisfy the capacity constraints of the multi-level on-chip storage space are determined if the capacity requirements in each of the aforementioned storage spaces do not exceed the corresponding capacity constraints.
5. The memory scheduling method according to claim 4, characterized in that, The multi-level on-chip storage space includes at least a first storage space and a second storage space; The generation of one or more candidate block sizes includes: Determine the first candidate block size for the first block dimension; Based on the storage occupancy item corresponding to the first storage space, determine the capacity requirement of the first storage space when using the first candidate block size; If the capacity requirement of the first storage space does not exceed the capacity constraint of the first storage space, the second candidate block size of the second block dimension is determined based on the available capacity information of the second storage space and the storage occupancy item corresponding to the second storage space. The corresponding candidate block size is obtained by combining the first candidate block size and the second candidate block size.
6. The memory scheduling method according to claim 5, characterized in that, The first storage space is a shared memory, the second storage space is a temporary storage memory, the first block dimension is the column direction, and the second block dimension is the row direction; The step of determining the capacity requirement of the first storage space when using the first candidate block size based on the storage occupancy item corresponding to the first storage space includes: Based on at least one of the corresponding storage occupancy items in the weight data and bias data, determine the shared memory capacity requirement corresponding to the block size of each candidate column; The step of determining the second candidate block size for the second block dimension based on the available capacity information of the second storage space and the storage occupancy item corresponding to the second storage space includes: If the shared memory capacity requirement does not exceed the shared memory capacity constraint, the row block size under the current candidate column block size is determined according to the temporary storage capacity constraint and at least one corresponding storage occupancy item among the input data, output data and intermediate data.
7. The memory scheduling method according to claim 6, characterized in that, Determining the row block size under the current candidate column block size includes: The initial value of the row block size is determined based on the capacity constraint of the temporary storage and the storage occupancy corresponding to a unit row block; Based on the target storage layout and / or alignment requirements, determine the actual temporary storage capacity requirement corresponding to the initial value; If the actual temporary storage capacity requirement exceeds the capacity constraint of the temporary storage, the row block size is reduced, and the actual temporary storage capacity requirement corresponding to the reduced row block size is re-determined until the actual temporary storage capacity requirement does not exceed the capacity constraint of the temporary storage.
8. The memory scheduling method according to claim 1, characterized in that, The step of determining the target block strategy from the selected candidate block sizes includes: When there are multiple candidate block sizes selected, the evaluation result of each candidate block size is determined according to the preset evaluation index; The target segmentation strategy is determined based on the evaluation results of each candidate segmentation size; The preset evaluation indicators include at least one of the following: the amount of data processed in a single transaction, the number of blocks, the storage space utilization rate, and the number of data moves between different storage spaces.
9. The memory scheduling method according to claim 1, characterized in that, The execution of the target operator according to the target block strategy includes: Determine the current storage location of the data to be processed; When the data to be processed needs to be migrated from the current storage location to the target storage location for executing the target operator, the data to be processed is divided into blocks according to the target block division strategy, and the blocks of data to be executed are migrated to the target storage location. The target operator is executed based on the data located at the target storage location.
10. The memory scheduling method according to claim 9, characterized in that, Before executing the target operator based on the data located at the target storage location, the method further includes: Determine the current storage layout of the data blocks to be executed; If the current storage layout does not meet the execution requirements of the target operator, the storage layout of the data blocks to be executed is transformed. If the current storage layout meets the execution requirements, the target operator is executed directly based on the data block to be executed.
11. The memory scheduling method according to claim 10, characterized in that, The storage layout conversion includes: Given that the current storage layout is global row-first and the target storage layout is intra-block row-first, the data blocks are rearranged so that the data within each data block is stored continuously in rows.
12. A memory scheduling device, characterized in that, include: The information acquisition unit is used to acquire information about the available capacity of the multi-level on-chip storage space of the computing unit in the current running state; A storage occupancy determination unit is used to obtain the execution information of the target operator and determine multiple storage occupancy items involved in the execution of the target operator based on the execution information. Each storage occupancy item corresponds to at least one data object and a corresponding storage space, and is used to characterize the storage occupancy of the corresponding data object in the storage space. A capacity requirement determination unit is used to generate one or more candidate block sizes and, based on each of the storage occupancy items, determine the capacity requirement in the multi-level on-chip storage space when using each candidate block size for block division. The strategy determination unit is used to determine the capacity constraints of the multi-level on-chip storage space based on the available capacity information, and to filter candidate block sizes that meet the capacity constraints of the multi-level on-chip storage space according to the capacity requirements of each candidate block size, and to determine the target block strategy from the filtered candidate block sizes. An execution control unit is used to execute the target operator according to the target block strategy.
13. A computing unit, characterized in that, include: The execution unit is used to execute the target operator; Multi-level on-chip storage space is used to store data involved in the execution of the target operator; The memory scheduling module is used to implement the memory scheduling method as described in any one of claims 1-11, thereby dynamically determining the target block strategy to guide the execution unit to perform block movement and layout transformation of data under the premise of satisfying the capacity constraints of the multi-level on-chip storage space, and thus completing the execution of the target operator.
14. The computing unit according to claim 13, characterized in that, The multi-level on-chip storage space includes at least a temporary storage memory and a shared memory; the temporary storage memory is used to store at least one of input data, output data, intermediate data, residual data, and gating data; the shared memory is used to store at least one of weight data and bias data; the execution unit is used to access the data in the temporary storage memory and the shared memory to execute the target operator.
15. An electronic device comprising a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory, wherein the program is executed by the processor to implement the memory scheduling method as described in any one of claims 1-11.
16. A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the memory scheduling method as described in any one of claims 1-11.
