Data processing method and device for large model parameters, equipment and medium

Abstract

Embodiments of the present application provide a data processing method and device for large model parameters, equipment and medium, belonging to the field of artificial intelligence. The method comprises: obtaining a quantization matrix and meta parameters of a target large model from different partitions of a first data storage area of a memory, and reading a memory access granularity representing the data processing capability of a processor, determining the size of the padding area between the inserted meta parameters and the quantization matrix based on the memory access granularity; dividing a second data storage area different from the first data storage area in the memory, sequentially writing the meta parameters, the preset padding information matching the size of the padding area, and the quantization matrix in the second data storage area to obtain an initial data atomic block; and arranging all the initial data atomic blocks in the second data storage area to obtain a target data atomic block. The present application optimizes the data storage mode, improves the memory bandwidth utilization of the processor in the inference process, and further improves the data calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a data processing method and device for large model parameters, equipment and medium. BACKGROUND

[0002] In recent years, with the revolutionary progress of large models (such as large language models) based on the Transformer architecture in multiple fields, the huge model size of large language models has brought unprecedented challenges to computing infrastructure. In the related art, a block quantization technology is used to compress model parameters, and the quantized matrix and meta parameters after block are mapped to different physical memory layouts.

[0003] However, in the related art, multiple non-merged read operations are initiated from the global memory to read the physically separated low-precision quantized data and high-precision meta data, respectively, during inference using model parameters. When completing the dequantization calculation, the data of different precisions will cause the data storage volume to expand sharply. In order to make the data meet the stringent input layout requirements of the calculation unit, the high-precision data that has just expanded several times must be written back to the on-chip shared memory for temporary storage and rearrangement. In this way, repeated read and write operations reduce the memory bandwidth utilization and data calculation efficiency of the processor. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a data processing method, device, equipment and medium for large model parameters, which aims to optimize the data storage method, improve the memory bandwidth utilization of the processor during inference, and further improve the data calculation efficiency.

[0005] To achieve the above purpose, the first aspect of the embodiments of the present application proposes a data processing method for large model parameters, applied to a processor, the processor being integrated with a memory, and the data processing method for large model parameters comprising:

[0006] obtaining a quantized matrix obtained by compressing parameters of a target large model, and meta parameters used to restore the quantized matrix, from different partitions of a first data storage area of the memory;

[0007] reading a memory access granularity representing a data processing capability of the processor from a configuration data storage area of the processor, and determining a padding area size between the meta parameters and the quantized matrix based on the memory access granularity;

[0008] dividing a second data storage area different from the first data storage area in the memory, and sequentially writing the meta parameters, preset padding information matching the padding area size, and the quantized matrix in the second data storage area to obtain an initial data atomic block;

[0009] arranging all the initial data atomic blocks in the second data storage area in sequence to obtain a target data atomic block.

[0010] In some embodiments, the padding area size between the insertion parameter and the quantization matrix is determined based on a memory access granularity, including:

[0011] determining a sum of data lengths of the insertion parameter and the quantization matrix as a target data length;

[0012] if the target data length is an integer multiple of the memory access granularity, determining the padding area size as zero;

[0013] if the target data length is less than the memory access granularity, calculating a first difference value between the memory access granularity and the target data length, and determining the padding area size as the first difference value;

[0014] if the target data length is greater than the memory access granularity, performing a modulo operation on the target data length and the memory access granularity to obtain a modulo result, calculating a second difference value between the memory access granularity and the modulo result, and determining the padding area size as the second difference value.

[0015] In some embodiments, the insertion parameter includes a scaling factor and an offset parameter;

[0016] writing the insertion parameter, preset padding information matching the padding area size, and the quantization matrix in the second data storage area in sequence to obtain the initial data atomic block, including:

[0017] writing the scaling factor, the offset parameter, preset padding information matching the padding area size, and the quantization matrix in the second data storage area in sequence to obtain the initial data atomic block, wherein the preset padding information is any one of preset zero data, a preset identifier, or a preset check code, and a data length of the initial data atomic block is an integer multiple of the memory access granularity.

[0018] In some embodiments, the processor is integrated with a register unit;

[0019] After arranging all the initial data atomic blocks in the second data storage area in sequence to obtain the target data atomic block, the method further includes:

[0020] when the processor receives a data processing signal, the processor loads the target data atomic block stored in the second data storage area into the register unit;

[0021] parallelly accessing the target data atomic block located in the register unit to implement data processing on parameters of the target large model, and obtaining a target data processing result.

[0022] In some embodiments, the target data atomic block located in the register unit is accessed in parallel to implement data processing on parameters of the target large model, to obtain a target data processing result, including:

[0023] A preset thread block is started, wherein the thread block includes a plurality of threads, and each thread includes corresponding index information;

[0024] Each thread determines a corresponding fixed offset position based on the respective corresponding index information, and reads a corresponding meta parameter and a quantization matrix corresponding to the meta parameter based on the fixed offset position;

[0025] Based on the read scaling factor and offset parameter, the quantization matrix is dequantized to obtain a target dequantization matrix;

[0026] Based on the target dequantization matrix, matrix operation processing is performed to obtain a target data processing result.

[0027] In some embodiments, based on the read scaling factor and offset parameter, the quantization matrix is dequantized to obtain a target dequantization matrix, including:

[0028] The quantization matrix and the scaling factor are multiplied to obtain an initial dequantization matrix;

[0029] The initial dequantization matrix and the offset parameter are added to obtain a target dequantization matrix.

[0030] In some embodiments, based on the read scaling factor and offset parameter, the quantization matrix is dequantized to obtain a target dequantization matrix, further including:

[0031] If the quantization matrix is represented in UINT8 format, the quantization matrix includes a high-bit quantization matrix and a low-bit quantization matrix;

[0032] The high-bit quantization matrix and the scaling factor are multiplied to obtain an initial high-bit dequantization matrix, and the initial high-bit dequantization matrix and the offset parameter are added to obtain a high-bit dequantization matrix;

[0033] The low-bit quantization matrix and the scaling factor are multiplied to obtain an initial low-bit dequantization matrix, and the initial low-bit dequantization matrix and the offset parameter are added to obtain a low-bit dequantization matrix;

[0034] Based on the high-bit dequantization matrix and the low-dimensional dequantization matrix, a target dequantization matrix is determined.

[0035] In some embodiments, the meta parameter includes a plurality of hierarchical meta parameters at different precision levels;

[0036] The target data atomic block located in the register unit is accessed in parallel to implement data processing on parameters of the target large model, to obtain a target data processing result, further including:

[0037] The performance monitoring unit based on the preset processor determines internal processing pressure information of the processor, and the sensor unit based on the preset processor determines external processing pressure information of the processor;

[0038] Obtain static fault tolerance rate information of the target large model, wherein the static fault tolerance rate information is determined based on a target processing task of the target large model in advance;

[0039] Obtain a historical data processing result, and perform dynamic error accumulation monitoring on the historical data processing result to obtain dynamic fault tolerance rate information;

[0040] Determine a target processing pressure score based on the internal processing pressure information and the external processing pressure information, and determine a target fault tolerance rate score based on the static fault tolerance rate information and the dynamic fault tolerance rate information;

[0041] Assign a first weight to the target processing pressure score and a second weight to the target fault tolerance rate score, and determine a target score based on the first weight, the second weight, the target processing pressure score, and the target fault tolerance rate score;

[0042] Parallelly access the target data atom block located in the register unit to obtain the meta parameter, select a target meta parameter from the plurality of hierarchical meta parameters based on the target score, and perform data processing on the parameters of the target large model based on the target meta parameter to obtain a target data processing result.

[0043] In some embodiments, assigning a first weight to the target processing pressure score and a second weight to the target fault tolerance rate score includes:

[0044] If the target processing pressure score does not reach a preset pressure threshold and the dynamic fault tolerance rate information includes abnormal information, the first weight is assigned to the target processing pressure score and the second weight is assigned to the target fault tolerance rate score, wherein the second weight is greater than the first weight;

[0045] If the target processing pressure score reaches the preset pressure threshold and the dynamic fault tolerance rate information includes abnormal information, the weight assignment operation is stopped and target warning information is generated.

[0046] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application proposes a data processing device for large model parameters, comprising:

[0047] An obtaining module is configured to obtain a quantization matrix obtained by parameter compression of a target large model and a meta parameter used to restore the quantization matrix from different partitions of a first data storage area of a memory;

[0048] The insertion module is configured to read a memory access granularity representing a data processing capability of the processor from a configuration data storage area of the processor, determine a padding area size between the insertion parameter and the quantization matrix based on the memory access granularity, and determine a padding area size between the insertion parameter and the quantization matrix based on the memory access granularity.

[0049] The data atom block determination module is configured to divide a second data storage area different from the first data storage area in the memory, and sequentially write the insertion parameter, preset padding information matched with the padding area size, and the quantization matrix in the second data storage area to obtain an initial data atom block.

[0050] The target storage module is configured to arrange all the initial data atom blocks in the second data storage area in sequence to obtain a target data atom block.

[0051] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the data processing method for large model parameters of the first aspect when executing the computer program.

[0052] To achieve the above object, a fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data processing method for large model parameters of the first aspect.

[0053] The data processing method, device, equipment and medium for large model parameters provided by the present application obtain the quantization matrix obtained after parameter compression of the target large model and the insertion parameter used for restoring the quantization matrix from different partitions of the first data storage area of the memory, read the memory access granularity representing the data processing capability of the processor from the configuration data storage area of the processor, determine the padding area size between the insertion parameter and the quantization matrix based on the memory access granularity, divide the second data storage area different from the first data storage area in the memory, sequentially write the insertion parameter, the preset padding information matched with the padding area size, and the quantization matrix in the second data storage area to obtain the initial data atom block, and arrange all the initial data atom blocks in the second data storage area in sequence to obtain the target data atom block. That is, the present application determines the padding area size according to the memory access granularity of the processor itself, and then packs the insertion parameter, the padding area and the quantization matrix into an initial data atom block which is physically continuous and address-aligned. The fundamental reshaping of the structure makes the originally required multiple and inefficient discrete memory access into a single and efficient combined memory access to obtain all the data required for a computing task, thus solving the memory bandwidth waste problem caused by unreasonable data layout from the source, greatly improving the memory bandwidth utilization, and further improving the data computing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a schematic diagram of an optional implementation environment of the data processing apparatus for large model parameters provided by the embodiments of the present application;

[0055] Figure 2 is an optional flowchart of the data processing method for large model parameters provided by the embodiments of the present application;

[0056] Figure 3 is an optional implementation flowchart of step 102 in Figure 2

[0057] Figure 4 is an optional generation schematic diagram of an initial data atomic block of the data processing method for large model parameters provided by the embodiments of the present application;

[0058] Figure 5 is an optional implementation flowchart of step 104 in Figure 2

[0059] Figure 6 is an optional implementation flowchart of step 104.1.2 in Figure 5

[0060] Figure 7 is an optional implementation flowchart of step A.3 in Figure 6

[0061] Figure 8 is another optional implementation flowchart of step A.3 in Figure 6

[0062] Figure 9 is a processing flowchart of an optional conventional data processing method;

[0063] Figure 10 is an optional processing flowchart of the data processing method for large model parameters provided by the embodiments of the present application;

[0064] Figure 11 is another optional implementation flowchart of step 104.1.2 in Figure 5

[0065] Figure 12 is an optional implementation flowchart of step D.5 in Figure 11

[0066] Figure 13 is an optional device module schematic diagram of the data processing apparatus for large model parameters provided by the embodiments of the present application;

[0067] Figure 14 ​​​​​​​Fig. 1 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0069] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0071] In recent years, with the revolutionary progress of large models (such as large language models) based on the Transformer architecture in multiple fields, the huge model size of large language models has brought unprecedented challenges to computing infrastructure. In the related art, the model parameters are compressed by using block-wise quantization technology, and the quantized matrix and meta-parameters after block division are mapped to different physical memory layouts.

[0072] Among them, block-wise quantization is also called group-wise quantization, and its core idea is to divide a large weight tensor into several smaller blocks or groups, and calculate the quantization parameters independently for each block to obtain meta-parameters, including scale and zero-point. In this way, when an abnormal value occurs, the influence of the abnormal value is limited within the block it is in, and does not affect the quantization accuracy of other blocks, thereby achieving high compression ratio while preserving the original performance of the model to the greatest extent.

[0073] In addition, the performance of modern Artificial Intelligence (AI) accelerators, especially Neural Processing Units (NPUs), is not only determined by the peak computing capability, but also largely limited by the efficiency of their memory subsystems. A typical NPU memory architecture is hierarchical, including a High Bandwidth Memory (HBM) located off-chip but tightly integrated with the processor through high-density through-silicon via technology, and on-chip multi-level caches (L1 Cache, L2 Cache) and shared memory that can be directly controlled by programmers.

[0074] Among them, the HBM provides a massive memory bandwidth at the level of terabytes per second (TB / s), which is the key to supporting large model inference. However, to fully utilize this bandwidth, a specific memory access pattern must be followed: the memory controller of the NPU interacts with the HBM in "transactions", and each transaction can read data of a cache line size (e.g., 32, 64, or 128 bytes). When all threads in a warp (usually containing 32 threads) access a physically contiguous and aligned memory address, these access requests can be coalesced by the memory controller into one or a few memory transactions, thereby achieving an effective bandwidth close to the hardware peak.

[0075] On the contrary, if the threads within a warp access non-contiguous, discrete, or large-strided memory addresses, the memory controller will not be able to coalesce and must initiate independent, narrow memory transactions for each or each group of threads. This will result in a large number of bus cycles being wasted on address transmission and command overhead, resulting in a much lower actual memory bandwidth than the theoretical value, forming a so-called "memory access bottleneck" that severely restricts the overall performance of the NPU.

[0076] After understanding the above knowledge background, the following introduces the data processing method adopted by the related technology:

[0077] When using the block quantization technology, the related technology generally follows the principle of separating quantized data and meta-parameters. As shown in Figure 1 Figure 1 ​is a schematic diagram of an optional implementation environment of the data processing apparatus for large model parameters provided by the embodiments of the present application. Specifically, a weight matrix after block quantization logically contains two parts of information: 1) a quantization matrix containing a large number of low-bit-width quantized integer values (for example, an array composed of INT4 values); 2) a parameter matrix containing high-precision parameter values associated with each quantization block (for example, a scale factor in FP16 format and an optional zero point corresponding to each block).

[0078] In the related art, when storing quantized data and parameters, this logical separation is directly mapped to the physical memory layout: all quantized weight values are packed into a large, continuous memory block, and all metadata (such as all scales and all zero points) are organized into one or more independent, smaller arrays and stored in another memory area. This means that to restore the true value of any data block, the computing core (such as the processor) needs to obtain information from two completely different and physically non-adjacent memory addresses. Due to its excellent accuracy retention capability, block quantization has become the de facto standard in the field of large model training quantization (Post-Training Quantization, PTQ), and is adopted by many advanced quantization algorithms (such as GPTQ, AWQ) and frameworks (such as LLaMA.cpp, bitsandbytes).

[0079] Taking the GGUF (Georgi Gerganov Universal Format) file format in the current popular LLaMA.cpp ecosystem as an example, the design goal is to provide a scalable single file container containing complete model information (including tokenizer, model structure, weights, etc.), and to optimize disk storage and memory mapping efficiency. Among them, the metadata (Metadata Key-Value Store), tensor information (Tensor Info) and tensor data (Tensor Data) corresponding to the GGUF file are stored in segments: for block-quantized tensors, such as Q4_K_M type, although each data block is logically associated with a Scale value, these Scale values and corresponding quantized data are not guaranteed to be physically continuous at the micro level (i.e., the granularity of a combined memory access) in file layout and final loading.

[0080] That is, when the loader such as LLaMA.cpp parses the GGUF file, its main task is to correctly load the data of different tensors into different regions of the NPU's corresponding HBM. The optimization goal of the memory layout is usually macroscopic, such as allocating different layers or tensors to different NPUs or memory regions, rather than deeply optimizing the micro-access pattern for specific kernels.

[0081] INT4, where INT4 is a data format that uses 4-bit binary numbers to represent integers. Due to the use of only 4 bits, INT4 can represent a relatively small range of values, typically from -8 to 7 (for signed integers) or from 0 to 15 (for unsigned integers). This low-bit-width data representation is mainly used in scenarios where precision requirements are not high but storage and computing efficiency requirements are high, such as weight parameter quantization in deep learning. By reducing the number of bits required for each value, storage requirements and computing speed are reduced, while accepting some loss of precision.

[0082] FP16, where FP16 is a data type that uses 16 bits (two bytes) to represent floating-point numbers: it includes 1 bit of sign bit, 5 bits of exponent bit, and 10 bits of mantissa bit (also known as decimal part or significant digits), which can provide higher numerical range and precision than INT4. FP16 is commonly used in applications that require high computational accuracy, such as graphics processing, machine learning inference, etc. It can significantly reduce memory usage and bandwidth requirements while ensuring sufficient accuracy, thereby improving computing efficiency and reducing energy consumption.

[0083] GPTQ (Gradient-based Precision Quantization), which is a gradient-based quantization method that aims to convert the weights in large language models from high-precision formats (such as FP16 or FP32) to low-precision formats (such as INT4) to reduce model size and speed up inference.

[0084] AWQ (Activation-aware Weight Quantization), which refers to a weight quantization technique that is aware of activation values. It not only considers the distribution characteristics of model weights, but also pays special attention to the numerical range and distribution of activation function outputs. This method can dynamically adjust the quantization strategy according to actual runtime data, ensuring that the model performance is maintained while reducing the demand for computing resources as much as possible.

[0085] LLaMA.cpp is an open-source project that focuses on supporting efficient inference for the LLaMA series of large language models released by Meta. This project employs various optimization techniques such as quantization, memory management, and multi-threading processing to enable the LLaMA model to run faster and more efficiently in resource-constrained environments, such as personal computers.

[0086] bitsandbytes is a library that focuses on numerical computation optimization, particularly in handling low-precision data types (such as INT8, INT4) for efficient operations. This library provides a rich set of tools and functions for quantization of model weights and activation values, mixed-precision training and inference, aiming to reduce the computational resource requirements of deep learning models while maintaining high prediction accuracy.

[0087] After understanding the traditional data processing methods used in the related art, the drawbacks of the related art are introduced:

[0088] However, based on the current data storage method used in the related art, the processor needs to initiate multiple non-merged read operations from the global memory to read the physically separated low-precision quantized data and high-precision metadata during inference using model parameters. When completing the dequantization calculation, the data of different precisions will cause the data storage volume to expand dramatically. In order to make the data meet the stringent input layout requirements of the calculation unit, the high-precision data that has just expanded several times must be written back from the register to the on-chip shared memory for temporary storage and rearrangement. In this way, repeated read-write operations reduce the data computation efficiency and memory bandwidth utilization of the processor.

[0089] Based on this, the present application proposes a data processing method, device, equipment and medium for large model parameters to solve the problem that in the traditional data processing method, the quantized matrix and meta parameters of the target large model after block quantization are stored separately in the physical memory, which causes the processor to have to initiate multiple non-continuous and non-merged memory read operations during inference, seriously reducing the actual utilization rate of high-bandwidth memory (such as HBM), thus forming a memory access bottleneck.

[0090] Specifically, after obtaining the separate quantization matrix and meta-parameters, this application precisely calculates and inserts a padding region based on the processor's own memory access granularity (such as cache line size), packaging the three (meta-parameters, padding region, and quantization matrix) into a physically contiguous and address-aligned "initial data atom block." This fundamental structural reshaping transforms the previously inefficient, multiple discrete memory accesses into a single, efficient merged memory access that retrieves all the data required for a computational task. Thus, it fundamentally solves the problem of wasted memory bandwidth caused by unreasonable data layout, greatly improving memory bandwidth utilization and consequently increasing data computation efficiency.

[0091] Meanwhile, since the data is processed directly in the register after loading, the step of writing back to the on-chip shared memory after data expansion following dequantization is avoided, eliminating the bottleneck of on-chip data flow, thereby significantly improving the overall data computing efficiency of the processor and ultimately accelerating the inference speed of large language models.

[0092] It should be noted that in this application embodiment, when information related to user characteristics, such as basic user information or user identity, is required, the user's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain sensitive personal information of a user, the user's individual permission or consent will be obtained first. Only after obtaining the user's individual permission or consent will the necessary data for the normal operation of this application embodiment be obtained. For example, before obtaining the quantization parameters and corresponding meta-parameters of the target large model, the authorization or consent of relevant personnel will be obtained first. Furthermore, other relevant data obtained in this application embodiment are all authorized data, and will not be elaborated upon here.

[0093] In this application embodiment, the description will focus on a data processing device (hereinafter referred to as "data processing device") for large model parameters, which can be integrated into a processor. For example... Figure 2 As shown, Figure 2 This is an optional flowchart of a data processing method for large model parameters provided in an embodiment of this application. Figure 2 The method may include, but is not limited to, the following steps 101 to 104. When the data processing device executes the data processing method for large model parameters (which can also be simply referred to as the "data processing method" for ease of description), the specific process is as follows. It should be noted first that this data processing method is applied to the processor and the processor is associated with the memory; in addition, the embodiments of this application... Figure 2 The order of steps 101 to 104 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0094] At step 101, a quantization matrix obtained by parameter compression of a target large model and meta parameters used to restore the quantization matrix are obtained from different partitions of a first data storage area of a memory.

[0095] The step 101 is described in detail below.

[0096] In some embodiments, the target large model (hereinafter also referred to as "model" for ease of description) refers to a deep learning model that needs to be compressed, stored and efficiently restored for inference, such as Meta Large Language Model (LLaMA), Qwen, ChatGLM, etc. The parameter quantity of the model is usually between hundreds of millions and tens of billions. Since such models are large in size after training, direct deployment on general-purpose devices will face problems such as insufficient memory, high latency and high power consumption. Therefore, when storing the model parameters of such models, it is necessary to convert them into a compact form through parameter compression technology (such as quantization).

[0097] Further, the stored model parameters of the target large model include two core parts: one is the "quantization matrix", which is a low-precision integer matrix (such as INT4 or INT8) obtained by compressing the original high-precision model weight (such as FP16) through a quantization algorithm, and its purpose is to reduce the size of the original model; the other is the "meta parameter", which is a high-precision data necessary for restoring the precision of the quantization matrix, and the meta parameter usually includes a scaling factor and a zero point, etc. The meta parameter is logically closely related to the corresponding quantization matrix, and the traditional method usually stores the quantization matrix and the meta parameter in different memory addresses.

[0098] Among them, the memory refers to a hardware device used to save data and programs, which undertakes the function of long-term or temporary storage of information in the computer system where the processor is located. Common memories include main memories such as Double Data Rate (DDR), High Bandwidth Memory (HBM), caches and external storage (such as hard disks). In the application scenario of large models, the memory is mainly used to store model-related parameters such as weights, activation values and intermediate calculation results. The memory works with the processor to provide necessary data support for the computer system.

[0099] Further, the first data storage region in the embodiments of the present application is a predefined, structured, and logically divided region in the memory, which is specially used to store the compressed large model parameters. It is usually configured during the initialization of the related computer system or the model loading stage. The region includes multiple different partitions. In general, the quantization matrix and the meta-parameter will be stored in different partitions of the first data storage region respectively. It should be noted that "first" here is only for differentiation and has no other special meaning.

[0100] The processor is the core hardware in the computer system responsible for executing instructions and processing data. It generally refers to chips capable of computing and controlling, including central processing units (CPUs), graphics processing units (GPUs), and neural processing units (NPUs). Specifically, the CPU is a traditional general-purpose processor that is good at sequentially executing complex tasks and system management. The GPU has a large number of parallel computing cores and is particularly suitable for processing graphics rendering and large-scale parallel computing tasks such as scientific simulation and deep learning training. The NPU is a special-purpose processor designed for artificial intelligence tasks, which is highly optimized for matrix operations, low-precision computing, and neural network inference, and has higher energy efficiency and computing density.

[0101] In step 102, the memory access granularity representing the data processing capability of the processor is read from the configuration data storage region of the processor, and the padding area size between the inserted meta-parameter and the quantization matrix is determined based on the memory access granularity.

[0102] The step 102 is described in detail below.

[0103] In some embodiments, the memory access granularity is a core hardware characteristic parameter of the processor, which represents the size of the continuous data block that the processor (such as NPU or GPU) can read at most once, for example, the length of a cache line, which may be 64 bytes or 128 bytes. By reading this granularity, the corresponding padding area size can be accurately calculated. The role of the padding area is to serve as a structural placeholder in the subsequent steps to ensure that the total length of the final data unit is exactly an integer multiple of the memory access granularity, thereby laying the foundation for efficient merged memory access.

[0104] That is, the size of the initial data atomic block determined in the subsequent embodiments of the present application is not arbitrarily set, but follows the "Kernel-Aware" design principle: the size of the initial data atomic block is determined according to the data processing capability of its final consumer, i.e., the parallel computing kernel on the NPU.

[0105] In some embodiments, as shown in Figure 3 , the size of the initial data atomic block is determined according to the data processing capability of the parallel computing kernel on the NPU. Figure 3 ​Figure 2 An optional implementation flowchart of step 102 in FIG. 1, determining the padding size between the inserted meta-parameter and the quantization matrix based on the memory access granularity, includes the following steps:

[0106] 102.1.1 determining the sum of the data length of the meta-parameter and the quantization matrix as a target data length;

[0107] 102.1.2 if the target data length is an integer multiple of the memory access granularity, determining the padding size as zero;

[0108] 102.1.3 if the target data length is less than the memory access granularity, calculating a first difference between the memory access granularity and the target data length, and determining the padding size as the first difference;

[0109] 102.1.4 if the target data length is greater than the memory access granularity, performing a modulo operation on the target data length and the memory access granularity to obtain a modulo result, calculating a second difference between the memory access granularity and the modulo result, and determining the padding size as the second difference.

[0110] The following describes steps 102.1.1 to 102.1.4 in detail.

[0111] In some embodiments, the target data length refers to the total number of original bytes occupied by a logical data unit (i.e., a quantization matrix and its corresponding meta-data) in memory without adding any padding information, so as to subsequently determine alignment and padding based on the target data length and the memory access granularity. The padding size determination method is as follows:

[0112] (1) if the target data length is an integer multiple of the memory access granularity, i.e., when the calculated target data length can be evenly divided by the memory access granularity, it means that the start and end addresses of the data unit are naturally aligned with the boundary of efficient hardware reading, and no additional padding is required to achieve optimal memory access. Therefore, in this case, the padding size is determined to be zero.

[0113] (2) if the target data length is less than the memory access granularity, i.e., when the target data length is insufficient for one memory access granularity, in order to avoid the processor initiating a low-efficiency memory read operation with insufficient width, the data unit needs to be "supplemented" to a complete memory access granularity. Specifically, the exact number of bytes required for padding is obtained by calculating the difference (first difference) between the memory access granularity and the target data length, thereby ensuring that the final data block can fill a complete hardware memory transaction unit, so as to convert a potential low-efficiency access into an efficient access.

[0114] (3) If the target data length is greater than the memory access granularity, i.e. when the target data length spans at least one memory access granularity but fails to fill up the last one, then the modulo operation is performed on the target data length and the memory access granularity to obtain a modulo result, and a second difference between the memory access granularity and the modulo result is calculated as the padding size. This step handles the most common and complex case, i.e. the data unit spans multiple memory access granularities but fails to fill up the last one. The "modulo operation" here is a core mathematical tool, and the "modulo result" calculated by it represents the number of bytes that the data unit overflows or occupies in the last incomplete memory access granularity. In order to align the end address of the entire data unit to the boundary of the next memory access granularity, the number of bytes that need to be padded is the second difference between the "memory access granularity" and the "modulo result". This operation ensures that even large data blocks that span multiple hardware read units can maintain strict memory alignment as a whole.

[0115] Step 103: Divide a second data storage area in the memory, which is different from the first data storage area, and write the meta parameter, the preset padding information matching the padding size, and the quantization matrix in the second data storage area in sequence to obtain an initial data atomic block.

[0116] The following describes step 103 in detail.

[0117] In some embodiments, step 103 writes the physically separated meta parameter and quantization matrix obtained from step 101, and the padding information determined according to the padding size calculated in step 102, in a new memory area (the second data storage area) in a determined order of "meta parameter-padding information-quantization matrix" in sequence. The result of this operation is to generate a corresponding initial data atomic block (Quantized Data-Metadata Atomic Block, QDMAB), which is physically continuous, and each initial data atomic block corresponds to a logical quantization block in the original matrix. The QDMAB encapsulates all the information required to process the block, making it a self-contained, indivisible atomic operation unit. And due to the existence of the padding area, the overall size of multiple initial data atomic blocks is aligned.

[0118] Exemplarily, a certain matrix multiplication operation requires a 16x16 FP16 weight block, i.e., 16*16*2=512 bytes, if INT4 quantization is adopted, the data area size is 16*16*0.5=128 bytes, assuming that the corresponding metadata (an FP32 Scale and an FP32 Zero-Point) occupies 8 bytes, then the quantized matrix and the metadata will occupy a total of 136 bytes; assuming that the memory access granularity of the processor is 128 bytes, in order to achieve 128-byte multiple alignment, the total data size of the QDMAB can be set to 256 bytes, so the padding area size can be calculated as 256-136=120 bytes. Alternatively, the size of the quantized matrix can be adjusted so that the quantized matrix, the metadata and the padding area exactly fill one or more 128-byte memory segments to obtain the initial data atomic block.

[0119] It can be understood that the embodiments of the present application ensure that the total length of each data atomic block finally constructed by the device will be a strict integer multiple of the memory access granularity regardless of the original data size of the metadata and the quantized matrix, and such deterministic structural guarantee is the fundamental prerequisite for realizing efficient hardware merging memory access, which lays a solid foundation for subsequent processor to read and process large model parameters at high speed and without obstacles.

[0120] In some embodiments, the metadata, preset padding information matching the padding area size, and the quantized matrix are sequentially written in the second data storage area to obtain the initial data atomic block, including the following steps:

[0121] 103.1.1 sequentially writing the scale factor, the offset parameter, preset padding information matching the padding area size, and the quantized matrix in the second data storage area to obtain the initial data atomic block, wherein the preset padding information is any one of preset zero data, a preset identifier or a preset check code, and the data length of the initial data atomic block is an integer multiple of the memory access granularity.

[0122] The step 103.1.1 is described in detail below.

[0123] In the embodiments of the present application, the metadata includes the scale factor and the offset parameter. Then, the scale factor, the offset parameter, preset padding information matching the padding area size, and the quantized matrix are sequentially written in the second data storage area to obtain the initial data atomic block. As shown in Figure 4 Figure 4 is an optional initial data atomic block generation schematic diagram provided by the data processing method for large model parameters according to the embodiments of the present application, Figure 4 ​The image above shows a high-precision raw matrix (FLOAT32 format). The part highlighted in red represents a quantization matrix to be processed, which includes the corresponding scaling factor and offset parameters. Figure 4 (The original meta-parameters are not shown in the text). Figure 4 The image below shows the new data arrangement formed in a storage unit (such as HBM) after the quantization matrix and its corresponding meta-parameters have been processed; that is, a physically contiguous initial data atom block. In this final data structure, the meta-parameters used to recover the data (including the scaling factor Scale=100 and the offset parameter Offset=0) are centrally located in the "metadata area" at the beginning; while the quantization matrix obtained after parameter compression in the original matrix ( Figure 4 The data within the red box (such as integers 22, 81, 5, etc.) is linearly arranged in the "data area". Crucially, a "padding area" (0 ​​in this example) is precisely calculated and inserted between the metadata area and the data area based on the processor's memory access granularity. This ensures that the total length of the initial data atom block is an integer multiple of the memory access granularity, laying the structural foundation for subsequent efficient one-time merging of memory accesses.

[0124] The various parts of this data structure are described below:

[0125] (1) Metadata area

[0126] This region, located at the beginning of QDMAB, contains all the parameters required for dequantization of this block of data. The specific content and size of the meta-parameters are configurable to accommodate different quantization algorithms. A typical configuration includes:

[0127] Scaling factor: One or more high-precision floating-point values ​​used to reconstruct the quantized value. A common data type is a 32-bit single-precision floating-point number (FP32). For more complex quantization schemes, multiple scale values ​​may exist.

[0128] Zero / Offset Parameter: One or more integer or floating-point values ​​used to handle asymmetric quantization. Its data type is the same high-precision floating-point type as Scale.

[0129] (2) Filling area

[0130] Padding region is located after metadata region, which is used to ensure that the total size of the entire QDMAB meets the specific hardware alignment requirements. The padding region is usually filled with padding information (such as fixed value 0), the purpose of which is to make the total byte number of each QDMAB an integer multiple of a value optimal for hardware access, such as 64 bytes, 128 bytes or 256 bytes, to ensure that the starting address of each QDMAB is aligned with the boundary of cache line or memory transaction, avoiding the performance loss caused by cross-line access, and laying the foundation for perfect merge access; in addition, the padding region composed of fixed values provides an excellent opportunity for data compression when stored offline.

[0131] Among them, the preset zero data refers to a sequence of all-zero bytes (fixed value 0) written in the padding region, which mainly functions to achieve memory alignment without introducing additional semantic information. Zero data is the simplest data, which will not interfere with the subsequent analysis of valid data (such as scaling factor, quantization matrix), nor will it cause hardware or software exceptions.

[0132] Among them, the preset identifier refers to a fixed data pattern (such as 0xAABBCCDD or the string QMAT) with specific meaning, which is used to mark the type, version, source or belonging module of the data block. Writing the preset identifier as padding information in the padding region not only achieves memory alignment, but also quickly identifies the purpose or structure of the data block in the debugging, loading or verification stage, for example, the inference engine can judge whether the atomic block is an INT4 quantization weight or belongs to which layer network by reading the identifier, thereby improving system maintainability and flexibility of runtime scheduling.

[0133] Among them, the preset check code refers to a piece of check information pre-calculated and written in the padding region according to the metadata (such as scaling factor, offset parameter, quantization matrix), common forms include cyclic redundancy check (CRC), checksum or hash fragment. Taking the check code as padding information not only meets the memory alignment requirement, but also can be used for subsequent verification of data integrity, for example, after model loading or transmission is completed, the system can recalculate the check value and compare it with the preset check code to detect whether there is storage error, transmission interference or memory damage, which is particularly suitable for model deployment in high reliability scenarios (such as vehicle-mounted AI, medical equipment).

[0134] (3) Data region

[0135] The data area follows the padding area and contains all the low-precision quantized values of the quantized block. The quantization matrix can be an array composed of quantization weights in multiple low-precision integers (such as INT8, INT4, INT3, INT2, etc.). Moreover, in order to maximize the spatial efficiency, these non-byte-aligned weight values (such as INT4) are tightly packed in memory. For example, two INT4 values can be packed and stored in one byte (UINT8).

[0136] It should be noted that the meta-parameter can further include a quantization mode identifier, a sparsity marker, and verification information; the quantization mode identifier is used to help the inference engine correctly select the dequantization formula and processing logic, the sparsity marker is used to skip invalid calculations to improve the calculation efficiency when the model is pruned or sparsified, and the verification information is used to enhance the reliability and security of the model parameters. That is, the specific content included in the meta-parameter can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0137] It can be understood that the embodiments of the present application convert a logically associated but physically separated data set into a highly optimized structure and self-contained information data set. Specifically, by prepositioning and accurately inserting the scaling factor and offset parameters required for dequantization into the padding information to obtain the initial data atomic block, the deterministic and aligned data structure of the initial data atomic block greatly simplifies the access logic of the subsequent processor kernel, so that it no longer needs to grab data from different addresses through multiple pointers, in order to maximize the utilization of memory bandwidth, thereby improving the data calculation efficiency.

[0138] Step 104: sequentially arranging all the initial data atomic blocks in the second data storage area to obtain a target data atomic block.

[0139] The step 104 is described in detail below.

[0140] In some embodiments, all generated independent initial data atomic blocks are linearly arranged one after another without gaps in memory, finally forming a macroscopically continuous target data atomic block. It can be understood that the traditional data processing method separates and stores the quantization matrix and the meta parameter, which causes the processor to have to initiate multiple non-continuous and inefficient access requests to the memory when performing a computing task, which causes serious waste of memory bandwidth and introduces unnecessary processing delay. In comparison, the embodiments of the present application reconfigure and bind logically associated data at the physical level to form a memory-aligned and self-contained initial data atomic block. Then, the initial data atomic blocks are arranged continuously to form a target data atomic block under a new data structure, so that the processor can obtain the quantization matrix and the corresponding meta parameter required for a computing task through a single and efficient combined memory access operation. In this way, the embodiments of the present application fundamentally eliminate the memory access bottleneck caused by unreasonable data layout, greatly improve the utilization rate of memory bandwidth, simplify the data acquisition logic of the processor core, reduce the turnover overhead of on-chip resources, and thus significantly improve the processing efficiency of large model parameters and the overall inference performance.

[0141] In some embodiments, as shown in Figure 5 Figure 5 is Figure 2 an optional implementation flowchart of step 104 in

[0142] 104.1.1 When the processor receives the data processing signal, the processor loads the target data atomic block stored in the second data storage area into the register unit;

[0143] 104.1.2 Parallel access to the target data atomic block located in the register unit to implement data processing on the parameters of the target large model, obtaining a target data processing result.

[0144] The following describes steps 104.1.1 to 104.1.2 in detail.

[0145] The data processing signal is a control instruction that triggers the processor to start executing a data computing task, which can come from the operating system, the driver, the runtime system or the upper application. For example, when the user inputs a text to request a reply, the inference engine sends a signal to the processor to start the inference, which is usually transmitted in the form of an interrupt, a register write or a command queue submission, indicating the start of the model inference process; After the processor detects the data processing signal, the target data atomic block stored in the second data storage area is loaded into the register unit for subsequent data processing. ​

[0146] Further, when the computing task is triggered, the processor will load the previously prepared, macroscopically continuous target data atomic block from the second data storage area into its internal integrated register unit at one time and in its entirety. The register unit is the fastest storage component inside the processor core, and data located here means that it can be directly accessed by the computing unit with minimal delay. Since the target data atomic block itself is continuous and each sub-block within it is aligned, this loading operation can be one or a few efficient combined memory accesses, thereby maximizing the utilization of the memory bus bandwidth.

[0147] Further, after the target data atomic block is loaded into the register unit, multiple parallel computing units (such as threads in a GPU) of the processor can simultaneously perform parallel access processing on the target data atomic block. Moreover, since the target data atomic block is composed of a series of initial data atomic blocks of uniform size and identical structure, each computing unit can accurately and conflict-free locate and read the quantization matrix and the corresponding meta-parameter it is responsible for processing through simple address offset calculation. This seamless parallel data acquisition enables all computing units to be fully mobilized to efficiently perform dequantization and subsequent calculations and ultimately obtain the target data processing result.

[0148] In some embodiments, as shown in Figure 6 , Figure 6 is Figure 5 an optional implementation flowchart of step 104.1.2 in

[0149] A.1 start a preset thread block, wherein the thread block includes multiple threads, and each thread includes corresponding index information;

[0150] A.2 each thread determines a corresponding fixed offset position based on the respective corresponding index information, and reads the corresponding meta-parameter and the quantization matrix corresponding to the meta-parameter based on the fixed offset position;

[0151] A.3 perform dequantization processing on the quantization matrix based on the read scaling factor and offset parameter to obtain a target dequantization matrix;

[0152] A.4 perform matrix operation processing based on the target dequantization matrix to obtain a target data processing result.

[0153] The following describes steps A.1 to A.4 in detail.

[0154] In the parallel processing, a thread block is a basic scheduling unit, which is composed of a group of threads that execute the same program code (i.e., a computing kernel) simultaneously. Specifically, the device assigns a unique "index information" (e.g., threadIdx) to each thread, which is the identity of the corresponding thread within the thread block and serves as the basis for subsequent task allocation and data positioning.

[0155] Further, after the data has been loaded into the register unit, each thread uses its unique index information and the pre-known uniform size of the initial data atomic block to instantly calculate the fixed offset position of the data block corresponding to its responsibility in the register unit through simple multiplication and addition operations (e.g., base address + index information * uniform size of the initial data atomic block). In addition, since the internal structure of each initial data atomic block is also fixed (meta parameters first, quantization matrix second), once the thread locates the fixed offset position, it can directly and accurately read the required meta parameters, without any locks or complex synchronization mechanisms.

[0156] Further, each thread independently uses the scaling factor and offset parameter in its own computing unit and register to perform inverse quantization processing on the corresponding quantization matrix to obtain the target inverse quantization matrix, which is a data that has been restored in precision and can be used for standard floating-point operations. The entire conversion process is completed entirely within the cache or register (register unit) of the processor core, avoiding interaction with slower memory.

[0157] Further, after obtaining the high-precision target inverse quantization matrix, each thread immediately uses it to perform the core matrix operation processing, which is usually the most computationally intensive part of the neural network. The matrix operation processing can be, for example, matrix multiplication or matrix convolution. After all threads complete their respective matrix operations in parallel, the results are integrated or processed, and the final target data processing result of the entire computing task is obtained, such as the output of a neural network layer.

[0158] In some embodiments, as shown in Figure 7 , the method further comprises the following steps: Figure 7 , the method further comprises the following steps: Figure 6 , the method further comprises the following steps:

[0159] B.1 multiplying the quantization matrix and the scaling factor to obtain an initial inverse quantization matrix;

[0160] B.2 adding the initial inverse quantization matrix and the offset parameter to obtain the target inverse quantization matrix.

[0161] The following describes Steps B.1 to B.2 in detail.

[0162] In some embodiments, to restore the quantization matrix to the original numerical range, first multiply the quantization matrix and the scale factor to obtain an initial dequantization matrix that is remapped back to the floating-point number domain; then add the initial dequantization matrix and the offset parameter to obtain a high-precision floating-point matrix (target dequantization matrix) that has been restored in both numerical scale and position. This process can be represented by the following formula:

[0163] FP_value = INT_value * Scale + offset <1>

[0164] where FP_value represents the target dequantization matrix; INT_value represents the quantization matrix; Scale represents the scale factor; and offset represents the offset parameter.

[0165] In some embodiments, as shown in Figure 8 , Figure 8 is Figure 6 another optional implementation flowchart of Step A.3 in

[0166] C.1 If the quantization matrix is represented in the UINT8 format, the quantization matrix includes a high-bit quantization matrix and a low-bit quantization matrix;

[0167] C.2 Multiply the high-bit quantization matrix and the scale factor to obtain an initial high-bit dequantization matrix, and add the initial high-bit dequantization matrix and the offset parameter to obtain a high-bit dequantization matrix;

[0168] C.3 Multiply the low-bit quantization matrix and the scale factor to obtain an initial low-bit dequantization matrix, and add the initial low-bit dequantization matrix and the offset parameter to obtain a low-bit dequantization matrix;

[0169] C.4 Determine the target dequantization matrix based on the high-bit dequantization matrix and the low-bit dequantization matrix.

[0170] The following describes Steps C.1 to C.4 in detail.

[0171] In some embodiments, when the quantization matrix adopts INT4 (4-bit integer) for weight quantization, since the basic unit of hardware storage and processing is usually byte (i.e. 8 bits), it is not possible to directly and efficiently store and access data types smaller than one byte, and therefore it is necessary to pack two INT4 quantization data into one UINT8 byte. In this case, the quantization matrix is stored in the memory in the format of UINT8, and the high 4 bits and the low 4 bits of each byte store an INT4 value (i.e. high-bit quantization matrix and low-bit quantization matrix) respectively, so as to realize efficient use of storage space.

[0172] Further, first, the high-bit quantization matrix is extracted from the physical byte, and then a standard dequantization process is performed thereon: first, the value scale is restored by multiplication with the scaling factor to obtain an initial high-bit dequantization matrix representing one of the intermediate results; then, the value offset is corrected by addition with the offset parameter, and finally the high-bit dequantization matrix with full restored precision is obtained. At this point, the dequantization operation is completed for half of the data in the physical byte.

[0173] Further, step C.3 extracts the low-bit quantization matrix from the physical byte in a similar manner to step C.2, and applies the same dequantization calculation logic thereto: first, multiplication with the scaling factor obtains an initial low-bit dequantization matrix, and then addition with the offset parameter obtains the high-precision recovery result of the other half of the data, i.e. the low-bit dequantization matrix. Step C.3 can be executed in parallel with step C.2 to improve data processing efficiency. Further, the two high-precision dequantization matrices processed independently are recombined into one logically unified matrix to obtain the target dequantization matrix. It can be understood that the target dequantization matrix is already in the register, and its layout naturally corresponds to the distribution of threads within the thread block, so it can be directly fed into the corresponding calculation kernel for matrix operation processing without the need for any data transfer, transposition, rearrangement or reloading from the memory, etc.

[0174] For the convenience of the reader to understand, as shown in Figure 9 Figure 9 ​is a processing flow diagram of an optional traditional data processing method. The traditional data processing method indicated in the flow diagram is described as follows: in order to perform a Kernel calculation, the processor must first initiate a memory request to "read the data of the A matrix" (i.e., the quantized matrix), and then initiate two independent memory requests to "read the Scale of the A matrix" and "read the ZeroPoint of the A matrix" (i.e., the meta parameters) respectively. Then, the series of operations are repeated for the B matrix, and a total of six discrete memory accesses are required to obtain all the input data required for the calculation. This data layout results in a large number of non-merged memory transactions, which seriously reduces the utilization of memory bandwidth and introduces significant read delay. Similarly, after the calculation is completed, three separate write operations are required to write back the result C matrix, further exacerbating the Input / Output (I / O) bottleneck, and the entire process is limited in performance due to low data access efficiency.

[0175] Compared with the prior art, the data processing method provided by the embodiments of the present application has the advantages of Figure 10 as shown in the following table: Figure 10 is a processing flow diagram of an optional data processing method provided by the embodiments of the present application. The data processing method indicated in the flow diagram is described as follows: before the core calculation, the system has packaged the quantized matrix and its meta parameters (Scale and ZeroPoint) into an initial data atomic block (i.e., QDMAB) aligned in memory through preprocessing. Therefore, in the core calculation stage, the processor only needs to initiate a request to "read the QDMAB block of the A matrix", and through an efficient merged memory access, all the information (quantized matrix and meta parameters) required for the matrix calculation can be loaded into the register unit. Similarly, the operation on the B matrix is simplified to a read operation. In this way, the embodiments of the present application fundamentally eliminate the multiple discrete access bottleneck in the prior art, greatly improve the utilization of memory bandwidth, and after the data is loaded, the Kernel calculation, the dequantization processing and the write back of the final result ("write back the QDMAB data block of the C matrix") are all performed on the structured and atomic data unit, so that the entire data processing link is highly smooth and efficient, thereby significantly improving the overall calculation performance and speed.

[0176] It can be understood that the quantization matrix and the corresponding meta-parameter are loaded from the HBM to the register unit, and the dequantization and subsequent calculation preparation are completely performed in the register, which completely avoids the step of writing back the dequantized data to the shared memory in the traditional data processing method; and since the complex data in the shared memory no longer needs to be managed, the design of the Kernel is greatly simplified. In the actual large model inference process (especially in the scene of small batch and long sequence generation), the performance bottleneck is mainly in the memory bandwidth, and the embodiments of the present application can directly improve the key performance indicators by maximizing the bandwidth utilization and reducing the calculation pause.

[0177] In some embodiments, as shown in Figure 11 , Figure 11 is Figure 5 Step 104.1.2 in another optional implementation flowchart, the target data atomic block located in the register unit is accessed in parallel to realize data processing on the parameters of the target large model, and a target data processing result is obtained, and further comprising the following steps:

[0178] D.1 Based on the pre-set performance monitoring unit of the processor, the internal processing pressure information of the processor is determined, and based on the pre-set sensor unit of the processor, the external processing pressure information of the processor is determined;

[0179] D.2 Obtain the static fault tolerance rate information of the target large model, wherein the static fault tolerance rate information is obtained based on the target processing task of the target large model in advance;

[0180] D.3 Obtain the historical data processing result, and perform dynamic error accumulation monitoring on the historical data processing result to obtain dynamic fault tolerance rate information;

[0181] D.4 Determine the target processing pressure score based on the internal processing pressure information and the external processing pressure information, and determine the target fault tolerance rate score based on the static fault tolerance rate information and the dynamic fault tolerance rate information;

[0182] D.5 Assign a first weight to the target processing pressure score, and assign a second weight to the target fault tolerance rate score, determine a target score based on the first weight, the second weight, the target processing pressure score and the target fault tolerance rate score;

[0183] D.6 Access the meta-parameter located in the target data atomic block in the register unit in parallel, select the target meta-parameter from the plurality of hierarchical meta-parameters based on the target score, and realize data processing on the parameters of the target large model based on the target meta-parameter to obtain a target data processing result.

[0184] The steps D.1 to D.6 are described in detail below.

[0185] In some embodiments, the meta-parameters include a plurality of hierarchical meta-parameters at different precision levels. The hierarchical meta-parameters refer to a plurality of sets of meta-parameters (such as scaling factors, offset parameters) corresponding to different precision levels (such as INT8, INT4, NF4, etc.) prepared in advance for the same quantization matrix, thereby forming a parameter set at multiple precision levels or compression rates. During the dynamic running of the computer system, the device will adaptively select the most suitable hierarchical meta-parameters according to the current processing pressure and fault tolerance capability (evaluated by the target score): for example, in high-load or low-fault-tolerance scenarios, high-precision levels (such as meta-parameters corresponding to INT8) are selected to ensure model accuracy; while in resource-constrained or high-fault-tolerance conditions, lower precision or higher compression levels (such as INT4 or sparsification levels) are switched to, thereby improving computational efficiency and throughput speed.

[0186] Further, the internal processing pressure information is derived from a performance monitoring unit, which is a hardware inside the processor for counting information such as instruction cycle number, cache hit rate, etc. The internal processing pressure reflects the busy degree of the computing core. The external processing pressure information is derived from a sensor unit, which reflects the overall physical running state of the chip; wherein the sensor unit can be a temperature sensor, a power consumption monitor, etc. The sensor unit can be specifically set according to actual conditions, and the embodiments of the present application do not limit this.

[0187] Further, the static fault tolerance rate information is a pre-evaluated parameter, which represents the inherent tolerance of the target large model to the loss of calculation accuracy when executing a specified "target processing task". The static fault tolerance rate depends on the analysis of the essential characteristics of the target processing task, and its value does not change with the change of the runtime environment. For example, in tasks such as open-domain dialogue, creative writing, etc., the model output has multiple reasonable forms, so the system has a high tolerance to accuracy loss, and the static fault tolerance rate is determined to be high; while in tasks such as mathematical calculation, code generation or logical reasoning, the accuracy requirement of the result is extremely high, and a slight error may cause the output to be completely wrong, so its static fault tolerance rate is low.

[0188] Further, the dynamic fault tolerance rate information is determined by continuously monitoring a series of historical data processing results previously generated by the computer system. This "dynamic error accumulation monitoring" can find the trend that the error may gradually magnify due to continuous use of low-precision calculation and eventually affect the quality of the result. It reflects the actual performance of the model in the current running state in real time, and is a dynamic and real-time supplement and correction to the static fault tolerance rate.

[0189] Furthermore, the target processing stress score is a comprehensive quantitative assessment of the processor's current operating load. It is calculated by fusing internal processing stress information (such as metrics collected by the performance monitoring unit, including computing unit utilization, memory bandwidth usage, and cache contention) and external processing stress information (such as physical environment data obtained by the sensor unit, including chip temperature, power consumption, and power supply stability). The target processing stress score reflects the overall stress level experienced by the hardware system when executing tasks. A higher value indicates that the processor is closer to its resource limits, potentially leading to overheating, high latency, or decreased energy efficiency, thus prompting the system to adopt load reduction or energy-saving strategies. The target fault tolerance score is a comprehensive assessment of the target large model's ability to tolerate computational errors under the current task. It combines static and dynamic fault tolerance information. A higher target fault tolerance score indicates that the model output is less sensitive to accuracy loss, and consequently, the computer system has greater flexibility in adjusting computational accuracy to adapt to resource constraints during inference.

[0190] Furthermore, configurable hyperparameters can be introduced: a first weight and a second weight. In this way, the device can flexibly adjust the relative importance of these two factors according to the actual application scenario, and then combine the target processing pressure score and the target fault tolerance score to determine the target score. Next, the device will select a target meta-parameter from multiple hierarchical meta-parameters based on the target score. Finally, the selected target meta-parameter is used to complete inverse quantization and data processing.

[0191] It is understood that the device proposed in this application embodiment no longer employs a fixed quantization strategy like traditional data processing methods. Instead, it dynamically and intelligently determines the most suitable computational precision by sensing the internal and external pressure of the processor in real time and continuously monitoring the error accumulation of the model output. When the processor load is high or the task fault tolerance is high, it automatically selects a lower precision quantization scheme (such as INT4) to maximize processing speed; while when the processor load is low or a decrease in precision is detected, it can seamlessly switch to a higher precision scheme (such as INT8 or higher) to ensure the accuracy of the results. This closed-loop adaptive adjustment capability enables the device proposed in this application embodiment to always operate at the optimal operating point under various complex and variable workloads, greatly improving its energy efficiency ratio and overall resource utilization efficiency.

[0192] In some embodiments, such as Figure 12 As shown, Figure 12 yes Figure 11 Step D.5, an optional implementation flowchart, assigns a first weight to the target processing stress score and a second weight to the target fault tolerance score, including the following steps:

[0193] E.1 If the target processing pressure score does not reach the preset pressure threshold, and the dynamic fault tolerance rate information includes abnormal information, assign a first weight to the target processing pressure score, and assign a second weight to the target fault tolerance rate score, wherein the second weight is greater than the first weight;

[0194] E.2 If the target processing pressure score reaches the preset pressure threshold, and the dynamic fault tolerance rate information includes abnormal information, stop the weight assignment operation and generate target warning information.

[0195] The following describes steps E.1 to E.2 in detail.

[0196] In some embodiments, when the computer system has performance redundancy (i.e., the target processing pressure score does not reach the preset pressure threshold), and the output accuracy of the model has begun to decline and triggered an alarm (i.e., the dynamic fault tolerance rate information includes abnormal information), a second weight greater than the first weight is forcibly assigned to the target fault tolerance rate score, so that the data processing device can prioritize model accuracy when making decisions, so as to fully exert the advantage of hardware redundancy, thereby timely suppressing error accumulation, improving output stability, effectively recovering and maintaining the inference quality of the model without sacrificing performance, and achieving optimal synergy of resources and accuracy.

[0197] The preset pressure threshold is a critical value for measuring the degree of processor load, which is set in advance during system design, and is used to determine whether the current hardware is in a high-pressure running state. This threshold is usually set based on performance monitoring data of the processor (such as CPU utilization, memory bandwidth occupancy) and external physical indicators (such as temperature, power consumption). When the target processing pressure score is below this threshold, it indicates that the processor computing resources are still abundant and can bear a certain degree of high-precision computing load. Once the threshold is reached or exceeded, it means that the processor is close to the performance limit or the heat dissipation boundary, and further increasing the load may lead to increased latency, decreased energy efficiency, or even system frequency reduction, so protective scheduling strategies need to be adopted.

[0198] The abnormal information refers to a signal detected during dynamic error accumulation monitoring of historical data processing results that the model output quality has significantly decreased or deviated from the normal range, which is used to represent unstable or unreliable state in the model inference process. For example, when consecutive multiple inference results appear logical contradictions, semantic confusion, output repetition, or too large deviation from the reference output, the system determines that the error accumulation has exceeded the reasonable range, triggering an abnormal flag. The presence of abnormal information reflects that the model has failed to maintain the expected accuracy under the current accuracy configuration or running environment, prompting the device to adjust the computing strategy to restore the output quality.

[0199] Further, if the processor is running at full load or overload (the target processing pressure score reaches the preset pressure threshold) while the output accuracy of the model also deteriorates unacceptably (the dynamic fault tolerance rate information includes abnormal information), that is, the data processing device cannot trade off accuracy for performance or relieve pressure by sacrificing accuracy in this case, the data processing device will stop the weight distribution operation decisively to avoid making any automatic decisions that may worsen the situation, and immediately generate target warning information to report this conflict state that cannot be automatically resolved to the upper management system or manual operator for external intervention.

[0200] As shown in Figure 13 , Figure 13 is an optional device module schematic diagram of the data processing device for large model parameters provided by the embodiments of the present application. The data processing device for large model parameters can include the following modules 201 to 204:

[0201] The acquisition module 201 is configured to acquire the quantization matrix obtained by parameter compression of the target large model and the meta-parameter used to restore the quantization matrix from different partitions of the first data storage area of the memory.

[0202] The insertion module 202 is configured to read the memory access granularity representing the data processing capability of the processor from the configuration data storage area of the processor, and determine the padding area size between the meta-parameter and the quantization matrix based on the memory access granularity.

[0203] The data atom block determination module 203 is configured to divide a second data storage area different from the first data storage area in the memory, and sequentially write the meta-parameter, the preset padding information matched with the padding area size, and the quantization matrix in the second data storage area to obtain an initial data atom block.

[0204] The target storage module 204 is configured to arrange all the initial data atom blocks in the second data storage area in sequence to obtain a target data atom block.

[0205] The specific implementation of the data processing device for large model parameters is basically the same as the above-mentioned specific embodiments of the data processing method for large model parameters, and will not be repeated here.

[0206] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned data processing method for large model parameters when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0207] As shown in Figure 14 , Figure 14is a hardware structure schematic diagram of an electronic device provided by an embodiment of the present application, and the electronic device comprises:

[0208] The processor 301 can be implemented in the form of a general central processing unit, a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0209] The memory 302 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 302 and are called and executed by the processor 301 to implement the data processing method for large model parameters of the embodiments of the present application.

[0210] The input / output interface 303 is used to realize information input and output.

[0211] The communication interface 304 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0212] The bus 305 is used to transmit information between various components (for example, the processor 301, the memory 302, the input / output interface 303, and the communication interface 304) of the device.

[0213] The processor 301, the memory 302, the input / output interface 303, and the communication interface 304 are connected to each other through the bus 305 to realize communication connection between them in the device.

[0214] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned data processing method for large model parameters.

[0215] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely from the processor, which can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0216] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0217] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps or different steps.

[0218] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0219] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0220] The terms "first", "second", "third", "fourth" and the like used in the specification of the present application and the above-described drawings (if any) are used to distinguish similar objects, and do not necessarily have to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0221] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0222] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0223] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0224] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0225] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0226] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A data processing method for large model parameters, characterized in that, Applied to a processor associated with memory, the data processing method for large model parameters includes: From different partitions of the first data storage area of ​​the memory, obtain the quantization matrix of the target large model after parameter compression, and the meta-parameters used to recover the quantization matrix; From the configuration data storage area of ​​the processor, read the memory access granularity characterizing the data processing capability of the processor, and determine the size of the padding area between the meta-parameter and the quantization matrix based on the memory access granularity; A second data storage area, different from the first data storage area, is divided in the memory. The meta-parameters, preset filling information matching the size of the filling area, and the quantization matrix are sequentially written into the second data storage area to obtain an initial data atom block. All the initial data atom blocks are arranged consecutively in the second data storage area to obtain the target data atom block; The meta-parameters include scaling factor and offset parameter; The step of sequentially writing the meta-parameters, preset padding information matching the size of the padding area, and the quantization matrix into the second data storage area to obtain an initial data atom block includes: The scaling factor, the offset parameter, the preset padding information matching the size of the padding area, and the quantization matrix are sequentially written into the second data storage area to obtain the initial data atom block. The preset padding information is any one of preset zero data, preset identifier, or preset checksum. The data length of the initial data atom block is an integer multiple of the memory access granularity.

2. The data processing method for large model parameters according to claim 1, characterized in that, Determining the size of the padding region between the meta-parameters and the quantization matrix based on the memory access granularity includes: The sum of the data lengths of the meta-parameters and the quantization matrix is ​​determined to be the target data length; If the length of the target data is an integer multiple of the memory access granularity, the size of the filling area is determined to be zero; If the target data length is less than the memory access granularity, calculate the first difference between the memory access granularity and the target data length, and determine the padding size as the first difference; If the target data length is greater than the memory access granularity, perform a modulo operation on the target data length and the memory access granularity to obtain a modulo result, and calculate a second difference between the memory access granularity and the modulo result to determine the size of the filling area as the second difference.

3. The data processing method for large model parameters according to claim 1, characterized in that, The processor integrates a register unit; After arranging all the initial data atom blocks consecutively in the second data storage area to obtain the target data atom block, the method further includes: When the processor receives a data processing signal, the processor loads the target data atom block stored in the second data storage area into the register unit; Parallel access to the target data atom block located in the register unit is used to perform data processing on the parameters of the target large model, and the target data processing result is obtained.

4. The data processing method for large model parameters according to claim 3, characterized in that, The parallel access to the target data atomic block located in the register unit to perform data processing on the parameters of the target large model, and to obtain the target data processing result, includes: Start a preset thread block, wherein the thread block includes multiple threads, and each thread includes corresponding index information; Each thread determines a corresponding fixed offset position based on its respective index information, and reads the corresponding meta-parameter and the quantization matrix corresponding to the meta-parameter based on the fixed offset position; Based on the read scaling factor and offset parameter, the quantization matrix is ​​dequantized to obtain the target dequantization matrix; Matrix operations are performed on the target inverse quantization matrix to obtain the target data processing result.

5. The data processing method for large model parameters according to claim 4, characterized in that, The step of performing dequantization processing on the quantization matrix based on the read scaling factor and offset parameter to obtain the target dequantized matrix includes: Multiplying the quantization matrix and the scaling factor yields the initial inverse quantization matrix; The initial dequantization matrix and the offset parameter are added together to obtain the target dequantization matrix.

6. The data processing method for large model parameters according to claim 5, characterized in that, The step of performing dequantization processing on the quantization matrix based on the read scaling factor and offset parameter to obtain the target dequantization matrix further includes: If the quantization matrix is ​​represented in UINT8 format, the quantization matrix includes a high-order quantization matrix and a low-order quantization matrix; Multiply the high-order quantization matrix and the scaling factor to obtain the initial high-order inverse quantization matrix, and add the initial high-order inverse quantization matrix and the offset parameter to obtain the high-order inverse quantization matrix; Multiply the low-order quantization matrix and the scaling factor to obtain the initial low-order inverse quantization matrix, and add the initial low-order inverse quantization matrix and the offset parameter to obtain the low-order inverse quantization matrix. The target dequantization matrix is ​​determined based on the high-order dequantization matrix and the low-order dequantization matrix.

7. The data processing method for large model parameters according to claim 3, characterized in that, The meta-parameters include hierarchical meta-parameters at multiple different precision levels; The parallel access to the target data atom block located in the register unit to perform data processing on the parameters of the target large model and obtain the target data processing result further includes: Based on the processor's pre-set performance monitoring unit, the processor's internal processing pressure information is determined, and based on the processor's pre-set sensor unit, the processor's external processing pressure information is determined. Obtain the static fault tolerance information of the target large model, wherein the static fault tolerance information is determined in advance based on the target processing task of the target large model; Obtain historical data processing results, perform dynamic error accumulation monitoring on the historical data processing results, and obtain dynamic fault tolerance information; The target processing pressure score is determined based on the internal processing pressure information and the external processing pressure information, and the target fault tolerance score is determined based on the static fault tolerance information and the dynamic fault tolerance information. A first weight is assigned to the target processing pressure score, and a second weight is assigned to the target fault tolerance score. Based on the first weight, the second weight, the target processing pressure score, and the target fault tolerance score, the target score is determined. The target data atomic blocks located in the register unit are accessed in parallel to obtain the meta-parameters. Based on the target score, a target meta-parameter is selected from multiple hierarchical meta-parameters. Based on the target meta-parameters, data processing of the parameters of the target large model is performed to obtain the target data processing result.

8. The data processing method for large model parameters according to claim 7, characterized in that, The process of assigning a first weight to the target processing pressure score and assigning a second weight to the target fault tolerance score includes: If the target processing pressure score does not reach the preset pressure threshold and the dynamic fault tolerance information includes abnormal information, then a first weight is assigned to the target processing pressure score and a second weight is assigned to the target fault tolerance score, wherein the second weight is greater than the first weight. If the target processing pressure score reaches the preset pressure threshold and the dynamic fault tolerance information includes abnormal information, then the weight allocation operation is stopped and a target warning message is generated.

9. A data processing device for large model parameters, characterized in that, include: The acquisition module is used to acquire, from different partitions of the first data storage area of ​​the memory, the quantization matrix obtained after parameter compression of the target large model, and the meta-parameters used to recover the quantization matrix; An insertion module is used to read the memory access granularity characterizing the processor's data processing capability from the processor's configuration data storage area, and determine the size of the fill area between the meta-parameters and the quantization matrix based on the memory access granularity. The data atom block determination module is used to divide a second data storage area in the memory that is different from the first data storage area, and to write the meta-parameters, preset filling information matching the size of the filling area, and the quantization matrix sequentially in the second data storage area to obtain an initial data atom block; The target storage module is used to continuously arrange all the initial data atom blocks in the second data storage area to obtain the target data atom block; The meta-parameters include scaling factor and offset parameter; The step of sequentially writing the meta-parameters, preset padding information matching the size of the padding area, and the quantization matrix into the second data storage area to obtain an initial data atom block includes: The scaling factor, the offset parameter, the preset padding information matching the size of the padding area, and the quantization matrix are sequentially written into the second data storage area to obtain the initial data atom block. The preset padding information is any one of preset zero data, preset identifier, or preset checksum. The data length of the initial data atom block is an integer multiple of the memory access granularity.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data processing method for large model parameters as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method for large model parameters as described in any one of claims 1 to 8.

Images