Large model compilation optimization method and system based on in-memory computing device

Abstract

The embodiment of the invention provides a large model compiling optimization method and system based on in-memory computing equipment. According to the method, a high-dimensional tensor of a large model is split into sub-tensors adaptive to a storage array, waste of storage resources caused by mismatching of data sizes is avoided, weight data with the reuse rate larger than a preset threshold value is preloaded to an equipment storage area, and repeated carrying of the weight data can be avoided; on the basis of a preset operator instruction matching table, a mapping relation between a large model operator and an in-memory calculation instruction is constructed, and it is ensured that equipment instruction resources can be efficiently called by the large model operator; then, based on the sub-tensor data size and the mapping relation between operators and instructions, the target calculation task is split into sub-tasks and distributed to a target calculation unit, and accurate matching of calculation unit resources and task types is achieved; and finally, generating an equipment executable instruction stream according to an allocation result to ensure that storage, instructions and computing unit resources of the computing equipment in the optimized memory are all adaptive to large model computing requirements, so that the utilization rate of the equipment resources can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and system for large-scale model compilation optimization based on in-memory computing devices. Background Technology

[0002] With the rapid development and continuous breakthroughs in large-scale model technology, in-memory computing devices have emerged. By innovatively integrating computing units directly inside the storage array, they have realized the groundbreaking concept of data-local computing, effectively alleviating the performance and energy efficiency dilemmas faced by traditional architectures.

[0003] However, the hardware characteristics of in-memory computing in the existing technology are not compatible with the computational characteristics of large models. The storage array of in-memory computing devices has a fixed size limit, and the high-dimensional tensors of large models cannot be directly adapted to the storage array, which will lead to a waste of device storage resources. Summary of the Invention

[0004] This invention provides a method and system for optimizing large model compilation based on in-memory computing devices, which can improve the utilization rate of device storage resources.

[0005] In a first aspect, embodiments of the present invention provide a method for optimizing the compilation of large models based on in-memory computing devices, the method comprising:

[0006] The high-dimensional tensor of the large model is split into at least one sub-tensor that adapts to the storage array, and the weight data of the large model with a reuse rate greater than a preset threshold is preloaded into the storage area of ​​the in-memory computing device.

[0007] Based on a pre-defined operator instruction matching table, a mapping relationship between large model operators and in-memory computation instructions is constructed.

[0008] Based on the mapping relationship between the at least one sub-tensor, the large model operator, and the in-memory computation instructions, the target computation task is split into at least one sub-task and assigned to the corresponding target computation unit;

[0009] Based on the allocation result of the at least one subtask, generate an instruction stream executable by the in-memory computing device, and output the optimized instruction stream and the current deployment configuration file.

[0010] Preferably,

[0011] Before splitting the high-dimensional tensor of the large model into at least one sub-tensor adapted to the storage array, and preloading the weight data with a large model reuse rate greater than a preset threshold into the storage area of ​​the in-memory computing device, the method further includes:

[0012] Parse the large model configuration file and extract the operator type, tensor dimension, and data dependencies of the large model configuration file;

[0013] Based on the extracted operator type, tensor dimension, and data dependency, a list of operator features is generated.

[0014] The hardware parameters of the in-memory computing device are detected, and a hardware capability list is generated, wherein the hardware parameters include: storage array size, number of parallel paths, instruction set, and precision range.

[0015] Preferably,

[0016] The process of splitting the target computation task into at least one subtask and assigning it to the corresponding target computation unit based on the mapping relationship between the at least one sub-tensor, the large model operator, and the in-memory computation instructions includes:

[0017] Based on the operator feature list and the hardware capability list, a task granularity evaluation function is constructed using the task granularity evaluation model;

[0018] The task granularity evaluation function is: G = α × C + β × P + γ × D;

[0019] Where G is the optimal task granularity for each operator, C is the computational complexity weight coefficient, P is the number of parallel paths weight coefficient, and D is the data volume weight coefficient.

[0020] Calculate the optimal task granularity for each operator based on the task granularity evaluation function;

[0021] Based on the optimal task granularity of each operator, the mapping relationship between the at least one sub-tensor, the large model operator, and the in-memory computation instructions, the target computation task is split into at least one sub-task and assigned to the corresponding target computation unit;

[0022] The computational priority of the at least one subtask is set based on the degree of data localization.

[0023] Preferably,

[0024] The step of generating an executable instruction stream for the in-memory computing device based on the allocation result of the at least one subtask, and outputting the executed instruction stream and the current deployment configuration file, includes:

[0025] Based on the allocation result of the at least one subtask, generate and execute a binary instruction stream executable by the in-memory computing device;

[0026] The accuracy and latency data of the large model are collected in real time, and the accuracy data and latency data are determined to be necessary for optimization based on preset accuracy thresholds and preset latency thresholds, respectively.

[0027] When the accuracy data is lower than the preset accuracy threshold, the current hierarchical quantization parameter is adjusted;

[0028] When the latency data is higher than the preset latency threshold, the current task granularity is adjusted, and the high-dimensional tensor of the large model splitting in the execution step is returned to at least one sub-tensor adapted to the storage array. The weight data of the large model with a reuse rate greater than the preset threshold is preloaded into the storage area of ​​the in-memory computing device.

[0029] Output the optimized binary instruction stream and the current deployment configuration file, wherein the current deployment configuration file includes: the current hierarchical quantization parameters and the current task granularity.

[0030] Secondly, embodiments of the present invention provide a large model compilation optimization system based on in-memory computing devices, the system comprising:

[0031] Splitting module: used to split the high-dimensional tensor of a large model into at least one sub-tensor adapted to the storage array, and preload the weight data of the large model with a reuse rate greater than a preset threshold into the storage area of ​​the in-memory computing device;

[0032] Instruction mapping module: used to construct the mapping relationship between large model operators and in-memory computation instructions based on a preset operator instruction matching table;

[0033] Task allocation module: Based on the mapping relationship between the at least one sub-tensor split by the splitting module, the large model operator constructed by the instruction mapping module, and the in-memory computation instructions, the target computation task is split into at least one sub-task and allocated to the corresponding target computation unit.

[0034] Execution optimization module: used to generate an executable instruction stream for the in-memory computing device based on the allocation result of at least one subtask of the task allocation module, and output the optimized instruction stream and the current deployment configuration file.

[0035] Preferably,

[0036] Prior to the splitting module, a preprocessing module is further included;

[0037] The preprocessing module is used to perform:

[0038] Parse the large model configuration file and extract the operator type, tensor dimension, and data dependencies of the large model configuration file;

[0039] Based on the extracted operator type, tensor dimension, and data dependency, a list of operator features is generated.

[0040] The hardware parameters of the in-memory computing device are detected, and a hardware capability list is generated, wherein the hardware parameters include: storage array size, number of parallel paths, instruction set, and precision range.

[0041] Preferably,

[0042] The task allocation module is used to perform:

[0043] Based on the operator feature list and the hardware capability list, a task granularity evaluation function is constructed using the task granularity evaluation model;

[0044] The task granularity evaluation function is: G = α × C + β × P + γ × D;

[0045] Where G is the optimal task granularity for each operator, C is the computational complexity weight coefficient, P is the number of parallel paths weight coefficient, and D is the data volume weight coefficient.

[0046] Calculate the optimal task granularity for each operator based on the task granularity evaluation function;

[0047] Based on the optimal task granularity of each operator, the mapping relationship between the at least one sub-tensor, the large model operator, and the in-memory computation instructions, the target computation task is split into at least one sub-task and assigned to the corresponding target computation unit;

[0048] The computational priority of the at least one subtask is set based on the degree of data localization.

[0049] Preferably,

[0050] The execution optimization module is used to perform:

[0051] Based on the allocation result of the at least one subtask, generate and execute a binary instruction stream executable by the in-memory computing device;

[0052] The accuracy and latency data of the large model are collected in real time, and the accuracy data and latency data are determined to be necessary for optimization based on preset accuracy thresholds and preset latency thresholds, respectively.

[0053] When the accuracy data is lower than the preset accuracy threshold, the current hierarchical quantization parameter is adjusted;

[0054] When the latency data is higher than the preset latency threshold, the current task granularity is adjusted, and the high-dimensional tensor of the large model splitting in the execution step is returned to at least one sub-tensor adapted to the storage array. The weight data of the large model with a reuse rate greater than the preset threshold is preloaded into the storage area of ​​the in-memory computing device.

[0055] Output the optimized binary instruction stream and the current deployment configuration file, wherein the current deployment configuration file includes: the current hierarchical quantization parameters and the current task granularity.

[0056] Thirdly, embodiments of the present invention provide a large model compilation optimization system based on in-memory computing devices, comprising: at least one memory and at least one processor;

[0057] The at least one memory is used to store a machine-readable program;

[0058] The at least one processor is configured to invoke the machine-readable program to execute any of the methods described in the first aspect.

[0059] Fourthly, embodiments of the present invention provide a computer-readable medium storing computer instructions, which, when executed by a processor, cause the processor to perform any of the methods described in the first aspect.

[0060] This invention provides a method and system for optimizing the compilation of large models based on in-memory computing devices. The method first decomposes the high-dimensional tensors of the large model into sub-tensors adapted to the storage array, avoiding waste of storage resources due to data size mismatch. Simultaneously, weight data with a reuse rate greater than a preset threshold is preloaded into the device's storage area, reducing storage bandwidth consumption caused by repeated weight data transfer. Next, based on a preset operator-instruction matching table, a mapping relationship between large model operators and in-memory computing instructions is constructed, ensuring that device instruction resources can be efficiently invoked by large model operators, avoiding idle instruction resources. Then, based on the sub-tensor data volume and the mapping relationship between operators and instructions, the target computation task is decomposed into subtasks and allocated to target computing units, achieving precise matching between computing unit resources and task types, avoiding computational resource mismatch. Finally, an executable instruction stream is generated based on the allocation results, ensuring that the optimized in-memory computing device's storage resources, instruction resources, and computing unit resources are all adapted to the computational needs of the large model, thereby significantly improving device resource utilization. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart of a large model compilation optimization method based on in-memory computing devices provided by an embodiment of the present invention;

[0063] Figure 2This is a flowchart of another large model compilation optimization method based on in-memory computing devices provided by an embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of a large model compilation optimization system based on an in-memory computing device provided in an embodiment of the present invention;

[0065] Figure 4 This is a schematic diagram of another large model compilation optimization system based on in-memory computing devices provided in an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0067] like Figure 1 As shown, this embodiment of the invention provides a method for optimizing the compilation of large models based on in-memory computing devices. This method may include the following steps:

[0068] Step 101: Split the high-dimensional tensor of the large model into at least one sub-tensor that adapts to the storage array, and preload the weight data of the large model with a reuse rate greater than a preset threshold into the storage area of ​​the in-memory computing device.

[0069] Step 102: Based on the preset operator instruction matching table, construct the mapping relationship between large model operators and in-memory computation instructions;

[0070] Step 103: Based on the mapping relationship between at least one sub-tensor, large model operator and in-memory computation instructions, the target computation task is split into at least one sub-task and assigned to the corresponding target computation unit;

[0071] Step 104: Based on the allocation results of at least one subtask, generate an instruction stream executable by the in-memory computing device, and output the optimized instruction stream and the current deployment configuration file.

[0072] This invention provides a method for optimizing the compilation of large models based on in-memory computing devices. The method first decomposes the high-dimensional tensors of the large model into sub-tensors adapted to the storage array, avoiding waste of storage resources due to data size mismatch. Simultaneously, weight data with a reuse rate greater than a preset threshold is preloaded into the device's storage area, reducing storage bandwidth consumption caused by repeated weight data transfer. Next, based on a preset operator-instruction matching table, a mapping relationship between large model operators and in-memory computing instructions is constructed, ensuring that device instruction resources can be efficiently invoked by large model operators, avoiding idle instruction resources. Then, based on the sub-tensor data volume and the mapping relationship between operators and instructions, the target computation task is decomposed into sub-tasks and allocated to target computing units, achieving precise matching between computing unit resources and task types, avoiding computational resource mismatch. Finally, an executable instruction stream is generated based on the allocation results, ensuring that the optimized in-memory computing device's storage resources, instruction resources, and computing unit resources are all adapted to the computational needs of the large model, thereby significantly improving the utilization rate of device storage resources.

[0073] To improve the compatibility with large models and in-memory devices, in one embodiment of the present invention, the above embodiment further includes the following before step 101:

[0074] Parse the large model configuration file and extract the operator type, tensor dimension, and data dependencies of the large model configuration file;

[0075] Based on the extracted operator type, tensor dimension, and data dependency, a list of operator features is generated.

[0076] The hardware parameters of the in-memory computing device are detected, and a hardware capability list is generated, wherein the hardware parameters include: storage array size, number of parallel paths, instruction set, and precision range.

[0077] In this embodiment of the invention, by parsing the configuration file of a large model, the core computational features of the large model are systematically extracted. The operator types cover the types of operators used in the large model to implement key operations such as matrix multiplication, attention mechanisms, and vector addition (e.g., matrix multiplication operators and softmax operators in the Transformer layer). The tensor dimensions correspond to the specific size of high-dimensional data blocks in the large model (e.g., feature tensors and weight tensors with dimensions of [1, 1024, 1024]). The data dependencies clarify the calling order and data flow logic between each operator and tensor. The extracted operator types, tensor dimensions, and data dependencies are generated into an operator feature list, which clearly presents the computational requirements of the large model. The process involves identifying data characteristics to provide a model-side basis for subsequent adaptation and optimization. Then, hardware probing methods are used to obtain the hardware parameters of the in-memory computing device, including the storage array size (e.g., 512×512) which determines data storage adaptability, the number of parallel paths affecting parallel computing capabilities, the instruction set that restricts operator instruction mapping (e.g., support for analog multiply-accumulate instructions and digital accumulation instructions), and the precision range that ensures computational accuracy. Integrating these parameters into a hardware capability list clarifies the hardware-side adaptation boundaries and capability limits. Through the operator feature list and the hardware capability list, a high degree of adaptation between the computational needs of large models and the hardware capabilities of in-memory computing devices can be achieved, laying the foundation for subsequent data layout reconstruction, operator instruction mapping, and task granularity allocation.

[0078] To further improve resource utilization, in one embodiment of the present invention, step 103 in the above embodiment may specifically include the following steps:

[0079] Based on the operator feature list and the hardware capability list, a task granularity evaluation function is constructed using the task granularity evaluation model;

[0080] The task granularity evaluation function is: G = α × C + β × P + γ × D;

[0081] Where G is the optimal task granularity for each operator, α is the computational complexity weight coefficient, C is the computational complexity, β is the parallel path weight coefficient, P is the parallel path, γ is the data volume weight coefficient, and D is the data volume.

[0082] Calculate the optimal task granularity for each operator based on the task granularity evaluation function;

[0083] Based on the optimal task granularity of each operator, the mapping relationship between the at least one sub-tensor, the large model operator, and the in-memory computation instructions, the target computation task is broken down into the at least one sub-task and assigned to the corresponding target computation unit.

[0084] The computational priority of the at least one subtask is set based on the degree of data localization.

[0085] In this embodiment of the invention, a task granularity evaluation model is constructed based on the preprocessed operator feature list and hardware capability list, thereby establishing a task granularity evaluation function and calculating the optimal task granularity for each operator. This ensures that the task granularity matches both the operator's own computational requirements and the hardware's parallel processing capabilities. Furthermore, by combining the mapping relationship between at least one sub-tensor of the already split adaptive storage array, large model operators, and in-memory computation instructions, the target computation task is split into at least one subtask according to the optimal task granularity. Large-granularity subtasks (e.g., matrix multiplication subtasks with high computational complexity and large data volume) are allocated to array computing units, while small-granularity subtasks (e.g., vector operation subtasks with low computational complexity and small data volume) are allocated to digital computing units, avoiding resource waste. Simultaneously, computational priorities can be set for each subtask based on the degree of data localization. Subtasks with high data localization (e.g., matrix multiplication subtasks involving preloaded weights) are given high priority, allowing them to occupy computational resources preferentially and avoiding hardware idleness due to data waiting.

[0086] To improve the compatibility with large models and in-memory computing devices, in one embodiment of the present invention, step 104 in the above embodiment includes:

[0087] Based on the allocation result of the at least one subtask, generate and execute a binary instruction stream executable by the in-memory computing device;

[0088] The accuracy and latency data of the large model are collected in real time, and the accuracy data and latency data are determined to be necessary for optimization based on preset accuracy thresholds and preset latency thresholds, respectively.

[0089] When the accuracy data is lower than the preset accuracy threshold, the current hierarchical quantization parameter is adjusted;

[0090] When the latency data is higher than the preset latency threshold, the current task granularity is adjusted, and the high-dimensional tensor of the large model splitting in the execution step is returned to at least one sub-tensor adapted to the storage array. The weight data of the large model with a reuse rate greater than the preset threshold is preloaded into the storage area of ​​the in-memory computing device.

[0091] Output the optimized binary instruction stream and the current deployment configuration file, wherein the current deployment configuration file includes: the current hierarchical quantization parameters and the current task granularity.

[0092] In this embodiment of the invention, a device-executable binary instruction stream is generated and executed based on the subtask allocation results. Simultaneously, during instruction stream execution, the accuracy and latency data of the large model are collected in real time, and the collected data are compared with preset accuracy and latency thresholds to determine whether the current optimization effect meets deployment requirements. If the accuracy data is lower than the preset accuracy threshold (e.g., accuracy loss exceeds 3%), the current hierarchical quantization parameters are adjusted to ensure the computational accuracy of the large model. If the latency data is higher than the preset latency threshold, the current task granularity is adjusted to reduce inference latency. After parameter adjustment, the data layout reconstruction, operator instruction mapping, and task allocation processes need to be restarted, and the optimized binary instruction stream and the current deployment configuration file are output to ensure efficient adaptation of the large model to in-memory computing hardware.

[0093] like Figure 2 As shown, to more clearly illustrate the technical solution and advantages of the present invention, the following provides a detailed description of the large model compilation optimization method based on in-memory computing devices provided by the embodiments of the present invention, which may specifically include the following steps:

[0094] Step 201: Parse the large model configuration file and extract the operator type, tensor dimension, and data dependencies of the large model configuration file;

[0095] Step 202: Based on the extracted operator type, tensor dimension, and data dependency, generate an operator feature list;

[0096] Step 203: Detect the hardware parameters of the in-memory computing device and generate a hardware capability list, wherein the hardware parameters include: storage array size, number of parallel paths, instruction set, and precision range;

[0097] Step 204: Split the high-dimensional tensor of the large model into at least one sub-tensor that adapts to the storage array, and preload the weight data of the large model with a reuse rate greater than a preset threshold into the storage area of ​​the in-memory computing device.

[0098] Step 205: Based on the preset operator instruction matching table, construct the mapping relationship between large model operators and in-memory computation instructions;

[0099] Step 206: Based on the operator feature list and the hardware capability list, construct a task granularity evaluation function using the task granularity evaluation model;

[0100] Specifically, the task granularity evaluation function is: G = α × C + β × P + γ × D;

[0101] Where G is the optimal task granularity for each operator, α is the computational complexity weight coefficient, C is the computational complexity, β is the parallel path weight coefficient, P is the parallel path, γ is the data volume weight coefficient, and D is the data volume.

[0102] Step 207: Calculate the optimal task granularity for each operator based on the task granularity evaluation function;

[0103] Step 208: Based on the optimal task granularity of each operator, the at least one sub-tensor, the mapping relationship between the large model operator and the in-memory computation instructions, the target computation task is split into at least one sub-task and assigned to the corresponding target computation unit;

[0104] Step 209: Set corresponding computation priorities for the at least one subtask based on the degree of data localization;

[0105] Step 210: Based on the allocation result of the at least one subtask, generate and execute the binary instruction stream executable by the in-memory computing device;

[0106] Step 211: Collect the accuracy data and latency data of the large model in real time, and determine whether the accuracy data and latency data need to be optimized based on the preset accuracy threshold and preset latency threshold, respectively;

[0107] Step 212: When the accuracy data is lower than the preset accuracy threshold, adjust the current hierarchical quantization parameter and return to step 204;

[0108] Step 213: When the delay data is higher than the preset delay threshold, adjust the current task granularity and return to step 204;

[0109] Step 214: Output the optimized binary instruction stream and the current deployment configuration file, wherein the current deployment configuration file includes: the current hierarchical quantization parameters and the current task granularity.

[0110] like Figure 3 As shown, this embodiment of the invention provides a large model compilation optimization system based on in-memory computing devices, the system comprising:

[0111] Splitting module 301: used to split the high-dimensional tensor of a large model into at least one sub-tensor adapted to the storage array, and preload the weight data of the large model with a reuse rate greater than a preset threshold into the storage area of ​​the in-memory computing device.

[0112] Instruction mapping module 302: used to construct the mapping relationship between large model operators and in-memory computation instructions based on a preset operator instruction matching table;

[0113] Task allocation module 303: Based on the mapping relationship between the at least one sub-tensor split by the splitting module 301, the large model operator constructed by the instruction mapping module 302, and the in-memory computation instructions, the target computation task is split into at least one sub-task and allocated to the corresponding target computation unit.

[0114] Execution optimization module 304: is used to generate an executable instruction stream for the in-memory computing device based on the allocation result of at least one subtask of the task allocation module 303, and output the executed instruction stream and the current deployment configuration file.

[0115] based on Figure 3 The large model compilation optimization system shown is based on in-memory computing devices, such as Figure 4 As shown, before the splitting module 301, a preprocessing module 305 is further included;

[0116] The preprocessing module 305 is used to perform:

[0117] Parse the large model configuration file and extract the operator type, tensor dimension, and data dependencies of the large model configuration file;

[0118] Based on the extracted operator type, tensor dimension, and data dependency, a list of operator features is generated.

[0119] The hardware parameters of the in-memory computing device are detected, and a hardware capability list is generated, wherein the hardware parameters include: storage array size, number of parallel paths, instruction set, and precision range.

[0120] like Figure 4 As shown, the task allocation module 303 is used to perform:

[0121] Based on the operator feature list and the hardware capability list, a task granularity evaluation function is constructed using the task granularity evaluation model;

[0122] The task granularity evaluation function is: G = α × C + β × P + γ × D;

[0123] Where G is the optimal task granularity for each operator, α is the computational complexity weight coefficient, C is the computational complexity, β is the parallel path weight coefficient, P is the parallel path, γ is the data volume weight coefficient, and D is the data volume.

[0124] Calculate the optimal task granularity for each operator based on the task granularity evaluation function;

[0125] Based on the optimal task granularity of each operator, the mapping relationship between the at least one sub-tensor, the large model operator, and the in-memory computation instructions, the target computation task is broken down into the at least one sub-task and assigned to the corresponding target computation unit.

[0126] The computational priority of the at least one subtask is set based on the degree of data localization.

[0127] like Figure 4 As shown, the execution optimization module 304 is used to perform:

[0128] Based on the allocation result of the at least one subtask, generate and execute a binary instruction stream executable by the in-memory computing device;

[0129] The accuracy and latency data of the large model are collected in real time, and the accuracy data and latency data are determined to be necessary for optimization based on preset accuracy thresholds and preset latency thresholds, respectively.

[0130] When the accuracy data is lower than the preset accuracy threshold, the current hierarchical quantization parameter is adjusted;

[0131] When the latency data is higher than the preset latency threshold, the current task granularity is adjusted, and the high-dimensional tensor of the large model splitting in the execution step is returned to at least one sub-tensor adapted to the storage array. The weight data of the large model with a reuse rate greater than the preset threshold is preloaded into the storage area of ​​the in-memory computing device.

[0132] Output the optimized binary instruction stream and the current deployment configuration file, wherein the current deployment configuration file includes: the current hierarchical quantization parameters and the current task granularity.

[0133] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the large model compilation and optimization system based on in-memory computing devices. In other embodiments of the present invention, the large model compilation and optimization system based on in-memory computing devices may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0134] The information interaction and execution process between the various units in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0135] This invention also provides a large model compilation optimization system based on in-memory computing devices, comprising: at least one memory and at least one processor;

[0136] At least one memory for storing machine-readable programs;

[0137] At least one processor is configured to invoke a machine-readable program to execute the data retrieval method in any embodiment of the present invention.

[0138] This invention also provides a computer-readable medium storing computer instructions, which, when executed by a processor, cause the processor to perform the large model compilation optimization method based on in-memory computing devices according to any embodiment of this invention.

[0139] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0140] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0141] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0142] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0143] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0144] The various embodiments of the present invention have at least the following beneficial effects:

[0145] 1. In this embodiment of the invention, a method for optimizing the compilation of large models based on in-memory computing devices is provided. This method first decomposes the high-dimensional tensors of the large model into sub-tensors adapted to the storage array, avoiding waste of storage resources due to data size mismatch. Simultaneously, weight data with a reuse rate greater than a preset threshold is preloaded into the device storage area, reducing storage bandwidth usage caused by repeated weight data transfer. Then, based on a preset operator-instruction matching table, a mapping relationship between large model operators and in-memory computing instructions is constructed, ensuring that device instruction resources can be efficiently invoked by large model operators, avoiding idle instruction resources. Next, based on the sub-tensor data volume and the mapping relationship between operators and instructions, the target computing task is decomposed into sub-tasks and allocated to target computing units, achieving precise matching between computing unit resources and task types, avoiding computing resource mismatch. Finally, an executable instruction stream is generated based on the allocation results, ensuring that the storage resources, instruction resources, and computing unit resources of the optimized in-memory computing device are all adapted to the computing requirements of the large model, thereby significantly improving device resource utilization.

[0146] 2. In this embodiment of the invention, by parsing the configuration file of a large model, the core computational features of the large model are systematically extracted. The operator types cover the types of operators used to implement key operations such as matrix multiplication, attention mechanisms, and vector addition in the large model. The tensor dimensions correspond to the specific size of high-dimensional data blocks in the large model. The data dependencies clarify the calling order and data flow logic between each operator and tensor. The extracted operator types, tensor dimensions, and data dependencies are generated into an operator feature list, which clearly presents the computational requirements and data characteristics of the large model, providing a model-side basis for subsequent adaptation and optimization. Then, hardware parameters of the in-memory computing device are obtained through hardware probing, including the storage array size that determines data storage adaptability, the number of parallel paths that affects parallel computing capabilities, the instruction set that restricts operator instruction mapping, and the accuracy range that ensures computational precision. These parameters are integrated into a hardware capability list, which clarifies the hardware-side adaptation boundaries and capability limits. Through the operator feature list and the hardware capability list, a high degree of adaptation between the computational requirements of the large model and the hardware capabilities of the in-memory computing device can be achieved, laying the foundation for subsequent data layout reconstruction, operator instruction mapping, and task granularity allocation.

[0147] 3. In this embodiment of the invention, a task granularity evaluation model is constructed based on the preprocessed operator feature list and hardware capability list, thereby establishing a task granularity evaluation function and calculating the optimal task granularity for each operator. This ensures that the task granularity matches both the operator's own computational requirements and the hardware's parallel processing capabilities. Furthermore, by combining the mapping relationship between at least one sub-tensor of the already split adaptive storage array, large model operators, and in-memory computation instructions, the target computation task is split into at least one sub-task according to the optimal task granularity. Large-granularity sub-tasks (e.g., matrix multiplication sub-tasks with high computational complexity and large data volume) are allocated to array computing units, while small-granularity sub-tasks (e.g., vector operation sub-tasks with low computational complexity and small data volume) are allocated to digital computing units, avoiding resource waste. Simultaneously, computational priorities can be set for each sub-task based on the degree of data localization. Sub-tasks with high data localization (e.g., matrix multiplication sub-tasks involving preloaded weights) are given high priority, allowing them to occupy computational resources preferentially and avoiding hardware idleness due to data waiting.

[0148] It should be noted that not all steps and modules in the above processes and system structure diagrams are mandatory; some steps or modules can be omitted as needed. The execution order of each step is not fixed and can be adjusted as required. The system structure described in the above embodiments can be a physical structure or a logical structure. That is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or they may be jointly implemented by certain components in multiple independent devices.

[0149] In the above embodiments, the hardware units can be implemented mechanically or electrically. For example, a hardware unit may include permanent dedicated circuitry or logic (such as a dedicated processor, FPGA, or ASIC) to perform the corresponding operation. The hardware unit may also include programmable logic or circuitry (such as a general-purpose processor or other programmable processor), which can be temporarily configured by software to perform the corresponding operation. The specific implementation method (mechanical, dedicated permanent circuitry, or temporarily configured circuitry) can be determined based on cost and time considerations.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for large model compilation optimization based on in-memory computing devices, characterized in that, The method comprises: splitting the high-dimensional tensor of the large model into at least one sub-tensor adapted to the storage array, and preloading weight data with a large model multiplexing rate greater than a preset threshold to the storage area of the in-memory computing device; based on a preset operator instruction matching table, constructing a mapping relationship between the large model operator and the in-memory computing instruction; based on the at least one sub-tensor, the mapping relationship between the large model operator and the in-memory computing instruction, splitting the target computing task into at least one sub-task and assigning it to the corresponding target computing unit; according to the assignment result of the at least one sub-task, generating an instruction stream executable by the in-memory computing device, and outputting the optimized instruction stream and the current deployment configuration file.

2. The method of claim 1, wherein, before the high-dimensional tensor of the large model is split into at least one sub-tensor adapted to the storage array, and the weight data with a large model multiplexing rate greater than a preset threshold is preloaded to the storage area of the in-memory computing device, further comprising: parsing the large model configuration file and extracting the operator type, tensor dimension and data dependency relationship of the large model configuration file; based on the extracted operator type, tensor dimension and data dependency relationship, generating an operator feature list; detecting the hardware parameters of the in-memory computing device and generating a hardware capability list, wherein the hardware parameters include: storage array size, parallel path number, instruction set, precision range.

3. The method of claim 2, wherein, based on the at least one sub-tensor, the mapping relationship between the large model operator and the in-memory computing instruction, splitting the target computing task into at least one sub-task and assigning it to the corresponding target computing unit, comprises: based on the operator feature list and the hardware capability list, using a task granularity evaluation model to construct a task granularity evaluation function; the task granularity evaluation function is: G=α×C+β×P+γ×D; wherein G is the optimal task granularity of each operator, α is the calculation complexity weight coefficient, C is the calculation complexity, β is the parallel path number weight coefficient, P is the parallel path number, γ is the data volume weight coefficient, and D is the data volume; according to the task granularity evaluation function, calculating the optimal task granularity of each operator; based on the optimal task granularity of each operator, the at least one sub-tensor, the mapping relationship between the large model operator and the in-memory computing instruction, splitting the target computing task into the at least one sub-task and assigning it to the corresponding target computing unit; based on the data localization degree, setting the corresponding computing priority for the at least one sub-task.

4. The method of claim 1, wherein, according to the assignment result of the at least one sub-task, generating an instruction stream executable by the in-memory computing device, and outputting the executed instruction stream and the current deployment configuration file, comprises: according to the assignment result of the at least one sub-task, generating a binary instruction stream executable by the in-memory computing device and executing it; real-time collection of precision data and delay data of the large model, and determination of whether the precision data and the delay data need to be optimized based on a preset precision threshold and a preset delay threshold, respectively. ​ ​ ​ adjust the current hierarchical quantization parameter when the precision data is lower than the preset precision threshold; adjust the current task granularity when the delay data is higher than the preset delay threshold, return to split the high-dimensional tensor of the large model into at least one sub-tensor that fits the storage array, and preload weight data with a large model multiplexing rate greater than a preset threshold to a storage area of an in-memory computing device; output the optimized binary instruction stream and the current deployment configuration file, wherein the current deployment configuration file includes the current hierarchical quantization parameter and the current task granularity.

5. A large model compilation optimization system based on in-memory computing devices, characterized by, The system comprises: a splitting module for splitting a high-dimensional tensor of a large model into at least one sub-tensor that fits a storage array, and preloading weight data with a large model multiplexing rate greater than a preset threshold to a storage area of an in-memory computing device; an instruction mapping module for constructing a mapping relationship between a large model operator and an in-memory computing instruction based on a preset operator instruction matching table; a task allocation module for splitting a target computing task into at least one sub-task based on the at least one sub-tensor split by the splitting module, the mapping relationship between the large model operator and the in-memory computing instruction constructed by the instruction mapping module, and allocating the target computing task to a corresponding target computing unit; an execution optimization module for generating an instruction stream executable by the in-memory computing device according to the allocation result of the at least one sub-task of the task allocation module, and outputting the optimized instruction stream and a current deployment configuration file.

6. The system of claim 5, further comprising: a preprocessing module before the splitting module; the preprocessing module is configured to perform: parsing a large model configuration file and extracting an operator type, a tensor dimension, and a data dependency relationship of the large model configuration file; generating an operator feature list based on the extracted operator type, tensor dimension, and data dependency relationship; detecting hardware parameters of the in-memory computing device and generating a hardware capability list, wherein the hardware parameters include a storage array size, a parallel path number, an instruction set, and a precision range.

7. The system of claim 6, wherein the task allocation module is configured to perform: constructing a task granularity evaluation function using a task granularity evaluation model based on the operator feature list and the hardware capability list; the task granularity evaluation function is G = α × C + β × P + γ × D; wherein G is the optimal task granularity of each operator, α is a calculation complexity weight coefficient, C is a calculation complexity, β is a parallel path number weight coefficient, P is a parallel path number, γ is a data volume weight coefficient, and D is a data volume; calculating the optimal task granularity of each operator according to the task granularity evaluation function; splitting the target computing task into the at least one sub-task based on the optimal task granularity of each operator, the at least one sub-tensor, the mapping relationship between the large model operator and the in-memory computing instruction, and allocating the target computing task to a corresponding target computing unit; setting a corresponding computing priority for the at least one sub-task based on a data localization degree.

8. The system of claim 5, further comprising: a task execution module for executing the at least one sub-task based on the current deployment configuration file. The execution optimization module is configured to perform: generating a binary instruction stream executable by the in-memory computing device and executing according to an allocation result of the at least one subtask; collecting precision data and latency data of the large model in real time, and determining whether the precision data and the latency data need to be optimized based on a preset precision threshold and a preset latency threshold, respectively; adjusting a current layered quantization parameter when the precision data is lower than the preset precision threshold; adjusting a current task granularity when the latency data is higher than the preset latency threshold, returning to the step of splitting the high-dimensional tensor of the large model into at least one sub-tensor adapted to the storage array, and preloading weight data with a large model multiplexing rate greater than a preset threshold to a storage area of the in-memory computing device; outputting the optimized binary instruction stream and a current deployment configuration file, wherein the current deployment configuration file includes the current layered quantization parameter and the current task granularity.

9. A large model compilation optimization system based on in-memory computing devices, characterized in that, comprise: at least one memory and at least one processor; the at least one memory is configured to store a machine-readable program; the at least one processor is configured to call the machine-readable program and execute the method of any one of claims 1 to 4.

10. A computer readable medium characterized by The computer readable medium stores computer instructions, and the computer instructions make the processor execute the method of any one of claims 1 to 4 when executed by the processor.

Images