Data processing method and device for large model parameters, equipment and medium

Abstract

The embodiment of the invention provides a data processing method and device for large model parameters, equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining a quantization matrix and a meta-parameter of a target large model from different partitions of a first data storage area of a memory, reading memory access granularity representing the data processing capability of a processor, and determining the size of a filling area between the insertion meta-parameter and the quantization matrix based on the memory access granularity; dividing a second data storage area different from the first data storage area in the memory, and sequentially writing the meta-parameter, preset filling information matched with the filling area in size and the quantization matrix into the second data storage area to obtain an initial data atomic block; and continuously arranging all the initial data atomic blocks in the second data storage area to obtain a target data atomic block. By optimizing the data storage mode, the memory bandwidth utilization rate of the processor in the reasoning process is improved, and then the data calculation efficiency of the processor is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a data processing method and device for large model parameters, equipment and medium. BACKGROUND

[0002] In recent years, with the revolutionary progress of large models (such as large language models) based on the Transformer architecture in multiple fields, the huge model size of large language models has brought unprecedented challenges to computing infrastructure. In the related art, a block quantization technology is used to compress model parameters, and the quantized matrix and meta parameters after block are mapped to different physical memory layouts.

[0003] However, in the related art, multiple non-merged read operations need to be initiated from the global memory to read the physically separated low-precision quantized data and high-precision meta data respectively during the inference process using model parameters. When the reverse quantization calculation is completed, the data of different precisions will cause the data storage volume to expand sharply. In order to make the data meet the stringent input layout requirements of the calculation unit, the high-precision data that has just expanded several times must be written back from the register to the on-chip shared memory for temporary storage and rearrangement. In this way, repeated read-write operations reduce the memory bandwidth utilization and data calculation efficiency of the processor. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a data processing method, device, equipment and medium for large model parameters, which aims to optimize the data storage mode, improve the memory bandwidth utilization of the processor during the inference process, and further improve the data calculation efficiency.

[0005] To achieve the above purpose, the first aspect of the embodiments of the present application proposes a data processing method for large model parameters, applied to a processor, the processor is integrated with a memory, and the data processing method for large model parameters comprises: Obtaining a quantized matrix obtained by compressing parameters of a target large model and meta parameters used to restore the quantized matrix from different partitions of a first data storage area of the memory; Reading a memory access granularity representing a data processing capability of the processor from a configuration data storage area of the processor, and determining a padding area size between the meta parameters and the quantized matrix based on the memory access granularity; Dividing a second data storage area different from the first data storage area in the memory, and sequentially writing the meta parameters, preset padding information matched with the padding area size, and the quantized matrix in the second data storage area to obtain an initial data atomic block; Arranging all the initial data atomic blocks continuously in the second data storage area to obtain a target data atomic block.

[0006] In some embodiments, the padding area size between the inserted meta-parameter and the quantization matrix is determined based on a memory access granularity, comprising: determining a sum of data lengths of the meta-parameter and the quantization matrix as a target data length; if the target data length is an integer multiple of the memory access granularity, determining the padding area size as zero; if the target data length is less than the memory access granularity, calculating a first difference between the memory access granularity and the target data length, and determining the padding area size as the first difference; if the target data length is greater than the memory access granularity, performing a modulo operation on the target data length and the memory access granularity to obtain a modulo result, calculating a second difference between the memory access granularity and the modulo result, and determining the padding area size as the second difference.

[0007] In some embodiments, the meta-parameter comprises a scaling factor and an offset parameter; writing the meta-parameter, preset padding information matching the padding area size, and the quantization matrix in the second data storage area in sequence to obtain an initial data atomic block, comprising: writing the scaling factor, the offset parameter, preset padding information matching the padding area size, and the quantization matrix in the second data storage area in sequence to obtain an initial data atomic block, wherein the preset padding information is any one of preset zero data, a preset identifier, or a preset check code, and a data length of the initial data atomic block is an integer multiple of the memory access granularity.

[0008] In some embodiments, the processor is integrated with a register unit; after arranging all the initial data atomic blocks in the second data storage area in sequence to obtain a target data atomic block, further comprising: when the processor receives a data processing signal, the processor loads the target data atomic block stored in the second data storage area into the register unit; parallelly accessing the target data atomic block located in the register unit to implement data processing on parameters of the target large model, to obtain a target data processing result.

[0009] In some embodiments, parallelly accessing the target data atomic block located in the register unit to implement data processing on parameters of the target large model, to obtain a target data processing result, comprising: starting a preset thread block, wherein the thread block comprises a plurality of threads, and each thread comprises corresponding index information; each thread determines a corresponding fixed offset position based on the respective corresponding index information, and reads the corresponding meta-parameter and the quantization matrix corresponding to the meta-parameter based on the fixed offset position; Based on the read scaling factor and offset parameter, the quantization matrix is dequantized to obtain a target dequantization matrix; Based on the target dequantization matrix, matrix operation processing is performed to obtain a target data processing result.

[0010] In some embodiments, based on the read scaling factor and offset parameter, the quantization matrix is dequantized to obtain a target dequantization matrix, including: The quantization matrix and the scaling factor are multiplied to obtain an initial dequantization matrix; The initial dequantization matrix and the offset parameter are added to obtain the target dequantization matrix.

[0011] In some embodiments, based on the read scaling factor and offset parameter, the quantization matrix is dequantized to obtain a target dequantization matrix, further including: If the quantization matrix is represented in UINT8 format, the quantization matrix includes a high-bit quantization matrix and a low-bit quantization matrix; The high-bit quantization matrix and the scaling factor are multiplied to obtain an initial high-bit dequantization matrix, and the initial high-bit dequantization matrix and the offset parameter are added to obtain a high-bit dequantization matrix; The low-bit quantization matrix and the scaling factor are multiplied to obtain an initial low-bit dequantization matrix, and the initial low-bit dequantization matrix and the offset parameter are added to obtain a low-bit dequantization matrix; Based on the high-bit dequantization matrix and the low-dimensional dequantization matrix, the target dequantization matrix is determined.

[0012] In some embodiments, the meta parameter includes a plurality of hierarchical meta parameters at different precision levels; Parallelly accessing the target data atomic block located in the register unit to realize data processing on the parameters of the target large model to obtain a target data processing result, further including: Based on the pre-set performance monitoring unit of the processor, the internal processing pressure information of the processor is determined, and based on the pre-set sensor unit of the processor, the external processing pressure information of the processor is determined; Obtain the static fault tolerance rate information of the target large model, wherein the static fault tolerance rate information is determined based on the target processing task of the target large model in advance; Obtain the historical data processing result, and dynamically accumulate and monitor the error of the historical data processing result to obtain dynamic fault tolerance rate information; Based on the internal processing pressure information and the external processing pressure information, a target processing pressure score is determined, and based on the static fault tolerance rate information and the dynamic fault tolerance rate information, a target fault tolerance rate score is determined; assign a first weight to the target processing pressure score and a second weight to the target fault tolerance rate score, determine a target score based on the first weight, the second weight, the target processing pressure score, and the target fault tolerance rate score; The target data atomic block located in the storage unit is accessed to obtain meta parameters, the target meta parameter is selected from the plurality of hierarchical meta parameters based on the target score, and the data processing of the parameters of the target large model is implemented based on the target meta parameter to obtain a target data processing result.

[0013] In some embodiments, assigning a first weight to the target processing pressure score and a second weight to the target fault tolerance rate score includes: If the target processing pressure score does not reach the preset pressure threshold and the dynamic fault tolerance rate information includes abnormal information, the first weight is assigned to the target processing pressure score and the second weight is assigned to the target fault tolerance rate score, wherein the second weight is greater than the first weight. If the target processing pressure score reaches the preset pressure threshold and the dynamic fault tolerance rate information includes abnormal information, the weight assignment operation is stopped and target warning information is generated.

[0014] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application proposes a data processing device for large model parameters, comprising: The acquisition module is configured to acquire, from different partitions of a first data storage area of a memory, a quantization matrix obtained by parameter compression of a target large model and a meta parameter used to restore the quantization matrix; The insertion module is configured to read, from a configuration data storage area of a processor, a memory access granularity representing a data processing capability of the processor, and determine a padding area size between the inserted meta parameter and the quantization matrix based on the memory access granularity; The data atomic block determination module is configured to divide a second data storage area different from the first data storage area in the memory, and sequentially write the meta parameter, the preset padding information matching the padding area size, and the quantization matrix in the second data storage area to obtain an initial data atomic block. The target storage module is configured to arrange all the initial data atomic blocks in the second data storage area in sequence to obtain a target data atomic block.

[0015] To achieve the above-mentioned purpose, a third aspect of the embodiments of the present application proposes an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the data processing method for large model parameters of the first aspect when executing the computer program.

[0016] To achieve the above object, a computer readable storage medium is provided in a fourth aspect of the embodiments of the present application, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the data processing method for large model parameters in the first aspect.

[0017] The data processing method, device, equipment and medium for large model parameters provided by the present application obtain the quantization matrix obtained by parameter compression of the target large model and the meta parameter used to restore the quantization matrix from different partitions of the first data storage area of the memory; read the memory access granularity representing the data processing capability of the processor from the configuration data storage area of the processor, determine the padding area size between the inserted meta parameter and the quantization matrix based on the memory access granularity; divide a second data storage area different from the first data storage area in the memory, sequentially write the meta parameter, the preset padding information matching the padding area size, and the quantization matrix in the second data storage area to obtain an initial data atomic block; and arrange all the initial data atomic blocks in the second data storage area in sequence to obtain a target data atomic block. That is, the present application determines the padding area size according to the memory access granularity of the processor itself, and then packs the meta parameter, the padding area and the quantization matrix into an initial data atomic block which is physically continuous and address-aligned; the fundamental reshaping of this structure changes the original multiple and inefficient discrete memory access into a single and efficient combined memory access to obtain all the data required for a computing task, thus solving the memory bandwidth waste problem caused by unreasonable data layout from the source, greatly improving the memory bandwidth utilization, and further improving the data computing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a schematic diagram of an optional implementation environment of the data processing device for large model parameters provided by the embodiments of the present application; Figure 2 is an optional flowchart of the data processing method for large model parameters provided by the embodiments of the present application; Figure 3 is an optional implementation flowchart of step 102 in Figure 2 Figure 4 is a generation schematic diagram of an initial data atomic block of the data processing method for large model parameters provided by the embodiments of the present application; Figure 5 is an optional implementation flowchart of step 104 in Figure 2 Figure 6 is an optional implementation flowchart of step 104.1.2 in Figure 5 Figure 7 ​​​is Figure 6 is a flow chart of another optional implementation of step A.3 in Figure 8 is Figure 6 is a flow chart of yet another optional implementation of step A.3 in Figure 9 is a schematic diagram of a processing flow of an optional conventional data processing method; Figure 10 is a schematic diagram of a processing flow of an optional data processing method for large model parameters provided by the embodiments of the present application; Figure 11 is Figure 5 is a flow chart of yet another optional implementation of step 104.1.2 in Figure 12 is Figure 11 is a flow chart of another optional implementation of step D.5 in Figure 13 is a schematic diagram of an optional device module of a data processing apparatus for large model parameters provided by the embodiments of the present application; Figure 14 is a schematic diagram of a hardware structure of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0019] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not intended to limit the present application.

[0020] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flow chart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0022] In recent years, with the revolutionary progress of large models based on the Transformer architecture (e.g., large language models) in multiple fields, the huge model size of large language models has brought unprecedented challenges to computing infrastructure. In related technologies, block-wise quantization technology is used to compress model parameters, and the quantized matrix and meta-parameters after block division are mapped to different physical memory layouts.

[0023] Among them, block-wise quantization is also called group-wise quantization, and its core idea is to divide a large weight tensor into several smaller blocks or groups, and calculate the quantization parameters independently for each block to obtain meta-parameters, including scale and zero-point. In this way, when an abnormal value occurs, the influence of the abnormal value is limited within the block it is in, and does not affect the quantization accuracy of other blocks, thereby achieving high compression ratio while preserving the original performance of the model to the greatest extent.

[0024] In addition, the performance of modern artificial intelligence (AI) accelerators, especially neural processing units (NPUs), is not only determined by peak computing power, but also to a large extent by the efficiency of their storage subsystems. The typical NPU storage architecture is hierarchical, including high-bandwidth memory (HBM) located off-chip but tightly integrated with the processor through high-density through-silicon via technology, as well as on-chip multi-level cache (L1 Cache, L2 Cache) and shared memory (Shared Memory) that can be directly controlled by programmers.

[0025] Among them, HBM provides terabytes per second (TB / s) level of massive memory bandwidth, which is the key to supporting large model inference. However, to fully utilize this bandwidth, a specific memory access pattern must be followed: the memory controller of the NPU interacts with the HBM in "transactions", and a transaction can read a cache line size of data (e.g., 32, 64, or 128 bytes); when all threads in a warp (usually containing 32 threads) access a physically contiguous and aligned memory address simultaneously, these access requests can be coalesced by the memory controller into one or a few memory transactions, thereby achieving an effective bandwidth close to the hardware peak.

[0026] Conversely, if the threads within a warp access non-contiguous, discrete, or Strided memory addresses, the memory controller will be unable to coalesce and must initiate independent, narrow memory transactions for each or small groups of threads. This will result in a large number of bus cycles being wasted on address transmission and command overhead, causing the actual obtained memory bandwidth to be much lower than the theoretical value, forming a so-called "memory access bottleneck", which severely restricts the overall performance of the NPU.

[0027] After understanding the above knowledge background, the following introduces the data processing method adopted by the related technology: The related technology generally follows the principle of separating quantized data and meta-parameters when using block quantization technology. As shown in Figure 1 Figure 1 is a schematic diagram of an optional implementation environment of a data processing device for large model parameters provided by an embodiment of the present application. Specifically, a weight matrix after block quantization, the information of which logically contains two parts: 1) a quantization matrix containing a large number of low-bit-width quantized integer values (for example, an array composed of INT4 values); 2) a meta-parameter matrix containing high-precision meta-parameters associated with each quantization block (for example, a scale factor and an optional zero point in FP16 format for each block).

[0028] When the related technology stores quantized data and meta-parameters, this logical separation is directly mapped to the physical memory layout: all quantized weight values are packed into a large, contiguous memory block, while all meta-data (such as all Scales and all Zero-Points) are organized into one or more independent, smaller arrays and stored in another memory area. This means that to restore the true value of any data block, the computing core (such as the processor) needs to obtain information from two completely different and physically non-adjacent memory addresses. Due to its excellent precision retention capability, block quantization has become the de facto standard in the field of large model training quantization (Post-Training Quantization, PTQ), and is adopted by many advanced quantization algorithms (such as GPTQ, AWQ) and frameworks (such as LLaMA.cpp, bitsandbytes).

[0029] ​For example, the GGUF (Georgi Gerganov Universal Format) file format in the current popular LLaMA.cpp ecosystem aims to provide an extensible single file container that contains complete model information, including tokenizers, model structures, weights, and optimizes disk storage and memory mapping efficiency. Among them, the metadata (Metadata Key-Value Store), tensor information (Tensor Info), and tensor data (Tensor Data) corresponding to the GGUF file are stored in segments: for block quantized tensors, such as Q4_K_M type, although each data block is logically associated with a Scale value, these Scale values and corresponding quantized data are not guaranteed to be physically continuous at the micro level (i.e., the granularity of a single consolidated memory access) when the file layout and final loading are considered.

[0030] That is, when parsing the GGUF file, the main task of the LLaMA.cpp loader is to correctly load the data of different tensors into different regions of the NPU's HBM. The optimization goal of its memory layout is usually macroscopic, such as allocating different layers or tensors to different NPUs or memory regions, rather than deeply optimizing the micro-access pattern for specific kernels.

[0031] Among them, INT4 is a data format that uses 4-bit binary numbers to represent integers. Due to the use of only 4 bits, INT4 can represent a relatively small range of values, typically from -8 to 7 (for signed integers) or from 0 to 15 (for unsigned integers). This low-bit-width data representation is mainly used in scenarios where precision requirements are not high but storage and computing efficiency requirements are high, such as weight parameter quantization in deep learning. By reducing the number of bits required for each value, it reduces storage requirements and speeds up the calculation process, while accepting a certain degree of precision loss.

[0032] Among them, FP16 format, also known as half-precision floating-point format, is a data type that uses 16 bits (two bytes) to represent floating-point numbers: it includes 1-bit sign bit, 5-bit exponent bit, and 10-bit mantissa bit (also known as decimal part or significant digits), which can provide higher numerical range and precision than INT4. FP16 is commonly used in applications that require high computational accuracy, such as graphics processing, machine learning inference, etc. It can significantly reduce memory usage and bandwidth requirements while ensuring sufficient accuracy, thereby improving computational efficiency and reducing energy consumption.

[0033] GPTQ (Gradient-based Precision Quantization) is a gradient-based quantization method that aims to convert the weights in large language models from high-precision formats (such as FP16 or FP32) to low-precision formats (such as INT4) to reduce model size and speed up inference.

[0034] AWQ (Activation-aware Weight Quantization) refers to a weight quantization technique that takes into account not only the distribution characteristics of model weights but also the numerical range and distribution of activation function outputs. This method dynamically adjusts the quantization strategy based on actual runtime data to ensure that the model performance is maintained while reducing the demand for computing resources as much as possible.

[0035] LLaMA.cpp is an open-source project that supports the efficient inference of Meta's LLaMA series of large language models. By using various optimization techniques such as quantization, memory management, and multi-threading processing, the LLaMA model can run faster and more efficiently in resource-constrained environments (such as personal computers).

[0036] bitsandbytes is a library focused on numerical computation optimization, particularly in handling low-precision data types (such as INT8, INT4) for efficient operations. The library provides a rich set of tools and functions for model weight and activation value quantization, mixed-precision training and inference, aiming to reduce the demand for computing resources while maintaining high prediction accuracy.

[0037] After understanding the traditional data processing methods used in related technologies, the drawbacks of related technologies are introduced: However, based on the current data storage methods used in related technologies, the processor needs to initiate multiple non-merged read operations from global memory to read physically separated low-precision quantized data and high-precision metadata during inference using model parameters. When completing the dequantization calculation, the different precision data will cause the data storage volume to expand dramatically. In order to make the data meet the stringent input layout requirements of the computing unit, the high-precision data that has just expanded several times must be written back to the on-chip shared memory for temporary storage and rearrangement. In this way, repeated read-write operations reduce the data computing efficiency and memory bandwidth utilization of the processor.

[0038] Based on this, the present application proposes a data processing method, device, equipment and medium for large model parameters in view of the problem that in the traditional data processing method, the quantization matrix and the meta parameter after the target large model is divided and quantized are stored separately in the physical memory, which causes the processor to have to initiate multiple non-continuous and non-merged memory reading operations during inference, seriously reducing the actual utilization rate of high-bandwidth memory (such as HBM), thereby forming a memory access bottleneck. Specifically, after obtaining the separated quantization matrix and meta parameter, the present application accurately calculates and inserts a padding area according to the memory access granularity (such as cache line size) of the processor itself, and packs the three (meta parameter, padding area, quantization matrix) into an "initial data atomic block" which is physically continuous and address-aligned; this fundamental reshaping of the structure changes the originally multiple and inefficient discrete memory access into a single and efficient combined memory access to obtain all the data required for a calculation task. In this way, the problem of memory bandwidth waste caused by unreasonable data layout is solved from the source, greatly improving the memory bandwidth utilization rate and thus improving the data calculation efficiency.

[0039] At the same time, since the data is loaded and processed directly in the register, the step of data inflation after dequantization and writing back to the on-chip shared memory is avoided, eliminating the bottleneck of on-chip data flow, thereby significantly improving the overall data calculation efficiency of the processor and ultimately accelerating the inference speed of the large language model.

[0040] It should be noted that in the embodiments of the present application, when it is necessary to obtain user's basic information or information related to user characteristics such as user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained first, and after obtaining the separate permission or separate consent of the user, the necessary data for enabling the embodiments of the present application to operate normally will be obtained, for example, the embodiments of the present application will first obtain the authorization or consent of the relevant personnel before obtaining the quantization parameters of the target large model and the corresponding meta parameters. In addition, other related data obtained by the embodiments of the present application are all authorized data, which will not be described one by one here.

[0041] In the embodiments of the present application, the data processing device for large model parameters (for the sake of description, hereinafter can also be referred to as "data processing device") will be described from the dimension of the device, which can be integrated in the processor. As shown in Figure 2 Figure 2 is an optional flowchart of the data processing method for large model parameters provided by the embodiments of the present application, Figure 2 ​The method in the method can include but is not limited to the following steps 101 to step 104, when the data processing device executes the data processing method for the large model parameter (for the convenience of description, it can also be referred to as "data processing method" simply), the specific process is as follows, it needs to be pointed out that the data processing method is applied to the processor and the processor is associated with the memory; in addition, the embodiment of the present application does not make specific limitation to the order of steps 101 to step 104 in the method, and the order of steps can be adjusted or some steps can be reduced or increased according to actual needs. Figure 2 The order of steps 101 to step 104 in the method is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or increased according to actual needs.

[0042] Step 101, obtaining the quantization matrix obtained by parameter compression of the target large model and the meta parameter used to recover the quantization matrix from different partitions of the first data storage area of the memory.

[0043] The step 101 is described in detail below.

[0044] In some embodiments, the target large model (for the convenience of description, hereinafter can also be referred to as "model") refers to a deep learning model that needs to be compressed, stored and efficiently recovered inference ability, such as Meta large language model (Large Language Model Meta AI, LLaMA), Qwen, ChatGLM, etc., the parameter quantity of the model is usually between hundreds of millions to tens of billions. Because the volume of such model is large after training, directly deploying on general devices will face problems such as insufficient memory, high delay and large power consumption, therefore, when storing the model parameters of such model, it is necessary to convert it into a compact form through parameter compression technology (such as quantization).

[0045] Further, the stored model parameters of the target large model include two core parts: one is "quantization matrix", which is a low-precision integer matrix (such as INT4 or INT8) obtained by compressing the original high-precision model weight (such as FP16) through quantization algorithm, its purpose is to reduce the size of the original model; the other is "meta parameter", which is the high-precision data necessary to recover the precision of the quantization matrix, the meta parameter usually includes scaling factor and zero point, etc., the meta parameter is logically closely related to the corresponding quantization matrix, and the traditional method usually stores the quantization matrix and the meta parameter in different memory addresses.

[0046] The memory refers to a hardware device for saving data and programs, and undertakes the function of long-term or temporary storage of information in the computer system where the processor is located. Common memories include main memories such as double data rate synchronous dynamic random access memory (DDR), high bandwidth memory (HBM), cache and external storage (such as hard disk). In the application scenario of large models, the memory is mainly used to store model-related parameters such as weights, activation values, intermediate calculation results, etc. The memory works cooperatively with the processor to provide necessary data support for the computer system.

[0047] Further, the first data storage area in the embodiment of the present application is a pre-defined, structured and divided logical area in the memory, which is specially used to store large model parameters after compression processing. It is usually configured during the initialization or model loading stage of the related computer system. The area includes multiple different partitions. In general, the quantization matrix and the meta parameter will be stored in different partitions of the first data storage area respectively. It should be noted that "first" here is only for differentiation and has no other special meaning.

[0048] The processor is the core hardware in the computer system responsible for executing instructions and processing data, and generally refers to a chip capable of computing and controlling, which includes central processing unit (CPU), graphics processing unit (GPU) and neural network processing unit (NPU) etc. Specifically, the CPU as a traditional general-purpose processor is good at sequential execution of complex tasks and system management; the GPU has a large number of parallel computing cores, and is particularly suitable for processing graphics rendering and large-scale parallel computing tasks such as scientific simulation and deep learning training; and the NPU is a special-purpose processor designed for artificial intelligence tasks, which is highly optimized for matrix operations, low-precision calculations and neural network inference, and has higher energy efficiency and computing density.

[0049] Step 102, reading the memory access granularity representing the data processing capability of the processor from the configuration data storage area of the processor, and determining the padding area size between the inserted meta parameter and the quantization matrix based on the memory access granularity.

[0050] The step 102 is described in detail as follows.

[0051] In some embodiments, memory access granularity is a core hardware characteristic parameter of the processor. It characterizes the size of the continuous data block that the processor (such as an NPU or GPU) can read at a time, such as the length of a cache line, which may be 64 bytes or 128 bytes. By reading this granularity, the embodiments of the present application can accurately calculate the size of the corresponding padding area. The role of the padding area is to serve as a structural placeholder in subsequent steps to ensure that the total length of the data unit finally formed is exactly an integer multiple of the memory access granularity, thereby laying the foundation for achieving efficient merged memory access.

[0052] That is to say, the size of the initial data atomic block determined subsequently in the embodiment of the present application is not set arbitrarily, but follows the "Kernel-Aware" design principle: the size of the initial data atomic block is determined based on the data processing capability of its final consumer, the parallel computing core on the NPU.

[0053] In some embodiments, as Figure 3 As shown, Figure 3 yes Figure 2 An optional implementation flowchart of step 102 in FIG. 100 is provided, wherein the size of the padding area between the inserted meta-parameters and the quantization matrix is ​​determined based on the memory access granularity, including the following steps: 102.1.1 Determine the sum of the data lengths of the meta-parameters and the quantization matrix as the target data length; 102.1.2 If the target data length is an integer multiple of the memory access granularity, the padding area size is determined to be zero; 102.1.3 If the target data length is less than the memory access granularity, calculate a first difference between the memory access granularity and the target data length, and determine the padding area size as the first difference; 102.1.4 If the target data length is greater than the memory access granularity, perform a modulo operation on the target data length and the memory access granularity to obtain a modulo result, calculate a second difference between the memory access granularity and the modulo result, and determine the size of the padding area as the second difference.

[0054] Steps 102.1.1 to 102.1.4 are described in detail below.

[0055] In some embodiments, the target data length refers to the total number of original bytes occupied by a logical data unit (i.e., a quantization matrix and its corresponding metadata) in memory without adding any padding information, so as to perform alignment judgment and padding calculation based on the target data length and memory access granularity. The method for determining the padding area size is as follows: (1) If the target data length is an integer multiple of the memory access granularity, that is, when the calculated target data length can be divided by the memory access granularity, it means that the start and end addresses of the data unit are naturally aligned with the boundary of the hardware efficient read, and no additional padding is needed to achieve optimal memory access. Therefore, in this case, the padding area size is determined to be zero.

[0056] (2) If the target data length is less than the memory access granularity, that is, when the target data length is less than one memory access granularity, in order to avoid the processor initiating a low-efficiency memory read operation with insufficient width, the data unit needs to be "supplemented" to a complete memory access granularity. Specifically, by calculating the difference (first difference) between the memory access granularity and the target data length, the exact number of bytes needed for padding is obtained, so as to ensure that the final data block can fill a complete hardware memory transaction unit, so that the processor can convert a potential inefficient access into an efficient access.

[0057] (3) If the target data length is greater than the memory access granularity, that is, when the target data length spans at least one memory access granularity, but fails to fill the last granularity, then the target data length and the memory access granularity are subjected to a modulo operation to obtain a modulo result, and a second difference between the memory access granularity and the modulo result is calculated, which is taken as the padding area size. This step handles the most common and complex case, that is, the data unit spans multiple memory access granularities, but fails to fill the last granularity. The "modulo operation" here is a core mathematical tool, and the "modulo result" calculated by it represents the number of bytes overflowed or occupied by the data unit in the last incomplete memory access granularity. In order to align the end address of the entire data unit to the boundary of the next memory access granularity, the number of bytes needed for padding is the second difference between the "memory access granularity" and the "modulo result". This operation ensures that even a large data block spanning multiple hardware read units can maintain strict memory alignment as a whole.

[0058] Step 103, dividing a second data storage area different from the first data storage area in the memory, and sequentially writing the meta-parameter, the preset padding information matching the padding area size, and the quantization matrix into the second data storage area to obtain an initial data atomic block.

[0059] The following describes step 103 in detail.

[0060] In some embodiments, step 103 writes the physically separated meta-parameter and quantization matrix obtained from step 101, and the padding information determined according to the padding region size calculated in step 102, in a new memory region (second data storage region) in a close and sequential order of "meta-parameter-padding information-quantization matrix". The result of this operation is to generate a corresponding initial data atomic block (Quantized Data-Metadata Atomic Block, QDMAB), which is physically continuous, and each initial data atomic block corresponds to a logical quantization block in the original matrix. The QDMAB encapsulates all the information required to process the block, making it a self-contained, indivisible atomic operation unit. And due to the existence of the padding region, the overall size of multiple initial data atomic blocks is aligned.

[0061] For example, a certain matrix multiplication and addition operation requires a 16x16 FP16 weight block, i.e. 16*16*2=512 bytes, if INT4 quantization is used, the data region size is 16*16*0.5=128 bytes, assuming that the corresponding metadata (a FP32 Scale and a FP32 Zero-Point) occupies 8 bytes, then the quantization matrix and meta-parameter will occupy a total of 136 bytes; assuming that the memory access granularity of the processor is 128 bytes, in order to achieve 128 byte multiple alignment, the total data size of the QDMAB can be set to 256 bytes, so the padding region size can be calculated as 256-136=120 bytes. Alternatively, the size of the quantization matrix can be adjusted so that the quantization matrix, meta-parameter and padding region exactly fill one or more 128 byte memory segments to obtain the initial data atomic block.

[0062] It can be understood that the embodiments of the present application ensure that the total length of each data atomic block finally constructed by the device will be a strict integer multiple of the memory access granularity regardless of the original data size of the meta-parameter and the quantization matrix. This deterministic structural guarantee is the fundamental prerequisite for realizing hardware efficient merging memory access, and lays a solid foundation for subsequent processor high-speed and barrier-free reading and processing of large model parameters.

[0063] In some embodiments, the meta-parameter, the preset padding information matching the padding region size, and the quantization matrix are sequentially written in the second data storage region to obtain the initial data atomic block, including the following steps: 103.1.1 sequentially write the scaling factor, the offset parameter, preset padding information matching the padding area size, and the quantization matrix in the second data storage area to obtain an initial data atomic block, wherein the preset padding information is any one of preset zero data, a preset identifier, or a preset check code, and the data length of the initial data atomic block is an integer multiple of the memory access granularity.

[0064] The following describes step 103.1.1 in detail.

[0065] In the embodiments of the present application, the meta parameters include the scaling factor and the offset parameter. Then, the scaling factor, the offset parameter, preset padding information matching the padding area size, and the quantization matrix are sequentially written in the second data storage area to obtain the initial data atomic block. As shown in Figure 4 , the initial data atomic block is obtained. Figure 4 is an optional initial data atomic block generation diagram provided by the data processing method for large model parameters in the embodiments of the present application, Figure 4 As shown in the upper part of Figure 4 , it is a high-precision original matrix (FLOAT32 format), and the part selected by the red box represents a quantization matrix to be processed. The quantization matrix includes corresponding scaling factor and offset parameter Figure 4 (not shown in the original meta parameter part in Figure 4 ); the lower part shows the new data arrangement formed in the storage unit (such as HBM) after the quantization matrix and the corresponding meta parameters are processed, that is, a physically continuous initial data atomic block. In this final data structure, the meta parameters (including the scaling factor Scale = 100 and the offset parameter Offset = 0) used to restore the data are placed in the “meta data area” at the head; and the quantization matrix (the data in the red box in , such as the parameters 22, 81, 5, etc.) obtained after the parameter compression in the original matrix is linearly arranged in the “data area”. It is crucial that a “padding area” (the padding value is 0 in this example) is accurately calculated and inserted between the meta data area and the data area according to the memory access granularity of the processor, so as to ensure that the total length of the entire initial data atomic block is an integer multiple of the memory access granularity, thereby laying a structural foundation for subsequent one-time efficient merging memory access.

[0066] The various parts in the data structure are introduced as follows: This area is located at the starting position of the QDMAB and contains all the parameters required for the inverse quantization of the block data. The specific content and size of the meta parameters are configurable to adapt to different quantization algorithms. A typical configuration includes: Scale Factor: one or more high-precision floating-point values, which is the dimension for restoring quantized values. The common data type is 32-bit single-precision floating-point (FP32). For more complex quantization schemes, there may be multiple scale values; Zero Point / Offset Parameter: one or more integer or floating-point values, used to handle asymmetric quantization, whose data type is the same high-precision floating-point type as Scale.

[0067] (2) Padding Area The padding area is located after the metadata area, which is used to ensure that the total size of the entire QDMAB meets the specific hardware alignment requirements. The padding area is usually filled with padding information (such as fixed value 0), the purpose of which is to make the total byte number of each QDMAB an integer multiple of a value optimal for hardware access, such as 64 bytes, 128 bytes, or 256 bytes, to ensure that the starting address of each QDMAB is aligned with the boundary of the cache line or memory transaction, avoiding performance loss caused by cross-line access, and laying the foundation for perfect merge access; in addition, when storing offline, the padding area composed of fixed values provides an excellent opportunity for data compression.

[0068] Among them, the preset zero data refers to a sequence of all-zero bytes (fixed value 0) written in the padding area, which mainly functions to achieve memory alignment without introducing additional semantic information. Zero data is the simplest data, which will not interfere with the subsequent analysis of valid data (such as scale factor, quantization matrix), nor will it cause hardware or software exceptions.

[0069] Among them, the preset identifier refers to a fixed data pattern (such as 0xAABBCCDD or the string QMAT) with specific meaning, which is used to mark the type, version, source, or belonging module of the data block. Writing the preset identifier as padding information in the padding area not only achieves memory alignment, but also quickly identifies the purpose or structure of the data block in the debugging, loading, or verification stage, for example, the inference engine can determine whether the atomic block is an INT4 quantized weight or belongs to which layer network by reading the identifier, thereby improving system maintainability and flexibility of runtime scheduling.

[0070] Among them, the preset checksum is a piece of verification information calculated in advance according to the metadata (such as scale factor, offset parameter, quantization matrix) and written into the padding area. Common forms include cyclic redundancy check (CRC), checksum, or hash fragment. Using the checksum as padding information not only meets the memory alignment requirement, but also can be used for subsequent verification of data integrity, for example, after model loading or transmission is completed, the system can recalculate the checksum and compare it with the preset checksum to detect whether there is storage error, transmission interference, or memory damage, which is particularly suitable for model deployment in high-reliability scenarios (such as vehicle-mounted AI, medical devices).

[0071] (3) Data area The data area follows the padding area and contains all the low-precision quantized values of the quantization block. The quantization matrix can be an array composed of quantization weights in multiple low-precision integers (such as INT8, INT4, INT3, INT2, etc.). Moreover, in order to maximize the space efficiency, these non-byte-aligned weight values (such as INT4) are tightly packed in memory. For example, two INT4 values can be packed and stored in a byte (UINT8).

[0072] It should be noted that the meta-parameter can further include a quantization mode identifier, a sparsity marker, and check information; the quantization mode identifier is used to help the inference engine correctly select the dequantization formula and processing logic, the sparsity marker is used to skip invalid calculations to improve the calculation efficiency when the model is pruned or sparsified, and the check information is used to enhance the reliability and security of the model parameters. That is, the specific content included in the meta-parameter can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0073] It can be understood that the embodiments of the present application convert a logically associated but physically separated data set into a highly optimized structure and self-contained information data set. Specifically, by prepositioning and accurately inserting the scaling factor and offset parameters required for dequantization into the padding information to obtain the initial data atomic block, the deterministic and aligned data structure of the initial data atomic block greatly simplifies the access logic of the subsequent processor kernel, so that it no longer needs to grab data from different addresses through multiple pointers, in order to maximize the utilization of memory bandwidth, thereby improving the data calculation efficiency.

[0074] Step 104: sequentially arranging all the initial data atomic blocks in the second data storage area to obtain the target data atomic block.

[0075] The step 104 is described in detail below.

[0076] In some embodiments, all generated independent initial data atomic blocks are linearly arranged one after another without gaps in memory, finally forming a macroscopically continuous target data atomic block. It can be understood that the traditional data processing method separates the quantization matrix and the meta parameter for storage, resulting in the processor having to initiate multiple non-continuous and inefficient access requests to the memory when performing a calculation task, which causes serious waste of memory bandwidth and introduces unnecessary processing delay. In comparison, the embodiments of the present application reconfigure and bind logically associated data at the physical level to form a memory-aligned and self-contained initial data atomic block. Then, the initial data atomic blocks are arranged continuously to form a target data atomic block under a new data structure, so that the processor can obtain the quantization matrix and the corresponding meta parameter required for a calculation task through a single and efficient combined memory access operation. In this way, the embodiments of the present application fundamentally eliminate the memory access bottleneck caused by unreasonable data layout, greatly improve the utilization rate of memory bandwidth, simplify the data acquisition logic of the processor core, reduce the turnover overhead of on-chip resources, and thus significantly improve the processing efficiency of large model parameters and the overall inference performance.

[0077] In some embodiments, as shown in Figure 5 , Figure 5 is Figure 2 an optional implementation flowchart of step 104, after arranging all initial data atomic blocks continuously in the second data storage area to obtain the target data atomic block, the following steps are further included: 104.1.1 when the processor receives a data processing signal, the processor loads the target data atomic block stored in the second data storage area into the register unit; 104.1.2 accessing the target data atomic block located in the register unit in parallel to implement data processing on the parameters of the target large model, obtaining a target data processing result.

[0078] The following describes steps 104.1.1 to 104.1.2 in detail.

[0079] The data processing signal is a control instruction that triggers the processor to start executing a data calculation task, which can come from the operating system, the driver, the runtime system or the upper application. For example, when the user inputs a text to request a reply, the inference engine sends a signal to the processor to start the inference, which is usually transmitted in the form of an interrupt, a register write or a command queue submission, indicating the start of the model inference process; after the processor detects the data processing signal, it loads the target data atomic block stored in the second data storage area into the register unit for subsequent data processing.

[0080] Further, when the computing task is triggered, the processor will load the previously prepared, macroscopically continuous target data atomic block from the second data storage area into its internal integrated register unit at one time and in its entirety. The register unit is the fastest storage component inside the processor core, and data located here means that it can be directly accessed by the computing unit with minimal delay. Since the target data atomic block itself is continuous and each sub-block within it is aligned, this loading operation can be one or a few efficient combined memory accesses, thereby maximizing the utilization of the memory bus bandwidth.

[0081] Further, after the target data atomic block is loaded into the register unit, multiple parallel computing units (such as threads in a GPU) of the processor can simultaneously perform parallel access processing on the target data atomic block. Moreover, since the target data atomic block is composed of a series of initial data atomic blocks of uniform size and identical structure, each computing unit can accurately and conflict-free locate and read the quantization matrix and the corresponding meta-parameter it is responsible for processing through simple address offset calculation. This seamless parallel data acquisition enables all computing units to be fully mobilized to efficiently perform dequantization and subsequent calculations, and ultimately obtain the target data processing result.

[0082] In some embodiments, as shown in Figure 6 , Figure 6 is Figure 5 an optional implementation flowchart of step 104.1.2 in , parallel access to the target data atomic block located in the register unit to implement data processing on the parameters of the target large model, and obtain the target data processing result, including the following steps: A.1 start a preset thread block, wherein the thread block includes multiple threads, and each thread includes corresponding index information; A.2 each thread determines a corresponding fixed offset position based on the respective corresponding index information, and reads the corresponding meta-parameter and the quantization matrix corresponding to the meta-parameter based on the fixed offset position; A.3 perform dequantization processing on the quantization matrix based on the read scaling factor and offset parameter to obtain a target dequantization matrix; A.4 perform matrix operation processing based on the target dequantization matrix to obtain the target data processing result.

[0083] The following describes steps A.1 to A.4 in detail.

[0084] In the parallel processing, a thread block is a basic scheduling unit, which is composed of a group of threads that execute the same program code (i.e., a computing kernel) simultaneously. Specifically, the device assigns a unique "index information" (e.g., threadIdx) to each thread, which is the identity of the corresponding thread within the thread block and serves as the basis for subsequent task allocation and data positioning.

[0085] Further, after the data has been loaded into the register unit, each thread uses its unique index information and the pre-known uniform size of the initial data atomic block to instantly calculate the fixed offset position of the data block corresponding to its responsibility in the register unit through simple multiplication and addition operations (e.g., base address + index information * uniform size of the initial data atomic block). In addition, since the internal structure of each initial data atomic block is also fixed (meta parameters first, quantization matrix second), once the thread locates the fixed offset position, it can directly and accurately read the required meta parameters, without any locks or complex synchronization mechanisms.

[0086] Further, each thread independently uses the scaling factor and offset parameter in its own computing unit and register to perform inverse quantization processing on the corresponding quantization matrix to obtain the target inverse quantization matrix, which is a data that has been restored in precision and can be used for standard floating-point operations. The entire conversion process is completed entirely within the cache or register (register unit) of the processor core, avoiding interaction with slower memory.

[0087] Further, after obtaining the high-precision target inverse quantization matrix, each thread immediately uses it to perform the core matrix operation processing, which is usually the most computationally intensive part of the neural network. The matrix operation processing can be, for example, matrix multiplication or matrix convolution. After all threads complete their respective matrix operations in parallel, the results are integrated or processed, and the final target data processing result of the entire computing task is obtained, such as the output of a neural network layer.

[0088] In some embodiments, as shown in Figure 7 , the step A.3 is an optional implementation flowchart for performing inverse quantization processing on the quantization matrix based on the read scaling factor and offset parameter to obtain the target inverse quantization matrix, including the following steps: Figure 7 B.1 multiplying the quantization matrix and the scaling factor to obtain an initial inverse quantization matrix; Figure 6 B.2 adding the initial inverse quantization matrix and the offset parameter to obtain the target inverse quantization matrix. The steps B.1 to B.2 are described in detail below.

[0089] The steps B.1 to B.2 are described in detail below. ​

[0090] In some embodiments, in order to restore the quantization matrix to the original numerical range, first multiply the quantization matrix and the scaling factor to obtain an initial dequantization matrix that is remapped back to the floating-point number domain; then add the initial dequantization matrix and the offset parameter to obtain a high-precision floating-point matrix (target dequantization matrix) that has been restored in numerical scale and position. This process can be represented by the following formula: FP_value = INT_value * Scale + offset <1> Wherein, FP_value represents the target dequantization matrix; INT_value represents the quantization matrix; Scale represents the scaling factor; offset represents the offset parameter.

[0091] In some embodiments, as shown in Figure 8 , Figure 8 is Figure 6 Step A.3 in another optional implementation flowchart, based on the read scaling factor and offset parameter, the quantization matrix is dequantized to obtain the target dequantization matrix, which further includes the following steps: C.1 If the quantization matrix is represented in UINT8 format, the quantization matrix includes a high-bit quantization matrix and a low-bit quantization matrix; C.2 Multiply the high-bit quantization matrix and the scaling factor to obtain an initial high-bit dequantization matrix, and add the initial high-bit dequantization matrix and the offset parameter to obtain a high-bit dequantization matrix; C.3 Multiply the low-bit quantization matrix and the scaling factor to obtain an initial low-bit dequantization matrix, and add the initial low-bit dequantization matrix and the offset parameter to obtain a low-bit dequantization matrix; C.4 Determine the target dequantization matrix based on the high-bit dequantization matrix and the low-dimensional dequantization matrix.

[0092] The following describes steps C.1 to C.4 in detail.

[0093] In some embodiments, when the quantization matrix adopts INT4 (4-bit integer) for weight quantization, since the basic unit of hardware storage and processing is usually a byte (i.e. 8 bits), it is not possible to directly and efficiently store and access data types smaller than a byte, therefore it is necessary to pack two INT4 quantization data into a UINT8 byte. In this case, the quantization matrix is stored in UINT8 format in memory, and the high 4 bits and the low 4 bits of each byte store an INT4 value (i.e. high-bit quantization matrix and low-bit quantization matrix) respectively, thereby achieving efficient use of storage space.

[0094] Furthermore, first, the high-bit quantization matrix is ​​extracted from the physical byte, and then a standard inverse quantization process is performed on it: first, the numerical scale is restored by multiplying it with the scaling factor to obtain an initial high-bit inverse quantization matrix represented as one of the intermediate results; then, the numerical offset is corrected by adding it to the offset parameter, and finally a high-bit inverse quantization matrix with fully restored accuracy is obtained. At this point, the inverse quantization operation is completed on half of the data in the physical byte.

[0095] Furthermore, step C.3 extracts the low-bit quantization matrix from the physical byte in a similar manner to step C.2, and applies the same inverse quantization calculation logic to it: first multiplying it with the scaling factor to obtain the initial low-bit inverse quantization matrix, and then adding it to the offset parameter to obtain the high-precision recovery result of the other half of the data, that is, the low-bit inverse quantization matrix. Among them, step C.3 can be executed in parallel with step C.2 to improve data processing efficiency. Further, the two independently processed high-precision inverse quantization matrices are re-merged into a logically unified matrix to obtain the target inverse quantization matrix. It can be understood that the target inverse quantization matrix is ​​already in the register, and its layout naturally corresponds to the distribution of threads in the thread block, so it can be directly sent to the corresponding computing kernel to perform matrix operation processing without performing any data handling, transposition, rearrangement or reloading from memory.

[0096] To facilitate readers’ understanding, Figure 9 As shown, Figure 9 This flowchart illustrates an alternative traditional data processing method. To perform a kernel computation, the processor must first initiate a memory request to read the data of matrix A (i.e., the quantization matrix), followed by two separate memory requests to read the scale of matrix A and the zeropoint of matrix A (i.e., the meta-parameters). This sequence of operations is then repeated for matrix B, requiring a total of six discrete memory accesses to gather all the input data required for the computation. This data layout results in a large number of unmerged memory transactions, severely reducing memory bandwidth utilization and introducing significant read latency. Similarly, after the computation is complete, writing back the result matrix C requires three separate write operations, further exacerbating the input / output (I / O) bottleneck. This inefficient data access hinders performance throughout the entire process.

[0097] In comparison, Figure 10 As shown, Figure 10is an optional processing flow diagram of the data processing method for large model parameters provided by the embodiments of the present application. The data processing method indicated in the flowchart is described: before the core calculation, the system has packaged the quantization matrix and its element parameters (Scale and ZeroPoint) into the initial data atomic block (i.e. QDMAB) aligned in memory through preprocessing. Therefore, in the core calculation stage, the processor only needs to initiate a request to "read the QDMAB block of the A matrix", and through a high-efficiency combined memory access, all the information (quantization matrix and element parameters) required for the matrix calculation can be completely loaded into the register unit; similarly, the operation of the B matrix is simplified to a read operation. In this way, the embodiments of the present application fundamentally eliminate the multiple discrete access bottlenecks in the prior art, greatly improve the memory bandwidth utilization, and after the data is loaded, the Kernel calculation, the dequantization processing and the final result write-back ("write back the QDMAB data block of the C matrix") are all performed on the structured and atomic data unit, making the entire data processing link highly smooth and efficient, thereby significantly improving the overall calculation performance and speed.

[0098] It can be understood that after the quantization matrix and the corresponding element parameters are loaded from the HBM to the register unit, the dequantization and subsequent calculation preparation work will be completely performed in the register, which completely avoids the step of writing back the dequantized data to the shared memory in the traditional data processing method; and since there is no need to manage the temporary storage, rearrangement and synchronization of complex data in the shared memory, the design of the Kernel is greatly simplified. In the actual large model inference process (especially in the scene of small batch and long sequence generation), the performance bottleneck is mainly in the memory bandwidth. The embodiments of the present application can directly improve the key performance indicators by maximizing the bandwidth utilization and reducing the calculation pause.

[0099] In some embodiments, as shown in Figure 11 , the step 104.1.2 in Figure 11 is Figure 5 another optional implementation flowchart, and the target data atomic block located in the register unit is accessed in parallel to implement data processing on the parameters of the target large model to obtain a target data processing result, and further comprising the following steps: D.1 Based on the pre-set performance monitoring unit of the processor, determine the internal processing pressure information of the processor, and based on the pre-set sensor unit of the processor, determine the external processing pressure information of the processor; D.2 Obtain the static fault tolerance rate information of the target large model, wherein the static fault tolerance rate information is obtained based on the target processing task of the target large model in advance; D.3 Obtain the historical data processing result, and dynamically monitor the error accumulation of the historical data processing result to obtain dynamic fault tolerance rate information; D.4 determining a target processing pressure score based on the internal processing pressure information and the external processing pressure information, and determining a target fault tolerance rate score based on the static fault tolerance rate information and the dynamic fault tolerance rate information; D.5 assigning a first weight to the target processing pressure score and a second weight to the target fault tolerance rate score, and determining a target score based on the first weight, the second weight, the target processing pressure score, and the target fault tolerance rate score; D.6 accessing the target data atomic block located in the register unit to obtain meta parameters, selecting a target meta parameter from the plurality of hierarchical meta parameters based on the target score, and performing data processing on parameters of the target large model based on the target meta parameter to obtain a target data processing result.

[0100] The following describes steps D.1 to D.6 in detail.

[0101] In some embodiments, the meta parameters include hierarchical meta parameters at different precision levels. The hierarchical meta parameters refer to a plurality of sets of meta parameters (such as scaling factors and offset parameters) corresponding to different precision levels (such as INT8, INT4, and NF4) prepared in advance for the same quantization matrix, thereby forming a parameter set at multiple precision levels or compression rates. During dynamic running of the computer system, the device adaptively selects the most suitable hierarchical meta parameter according to the current processing pressure and fault tolerance capability (evaluated through the target score): for example, in a high-load or low-fault-tolerance scenario, a high-precision level (such as the meta parameter corresponding to INT8) is selected to ensure model accuracy; and in a resource-constrained or high-fault-tolerance condition, a lower-precision or higher-compression-rate level (such as INT4 or a sparsification level) is switched to, thereby improving the calculation efficiency and throughput speed.

[0102] Further, the internal processing pressure information is derived from a performance monitoring unit, which is a hardware inside the processor for counting information such as instruction cycle number and cache hit rate, and the internal processing pressure reflects the busy degree of the computing core. The external processing pressure information is derived from a sensor unit, which reflects the overall physical running state of the chip; the sensor unit can be a temperature sensor, a power consumption monitor, etc., and the sensor unit can be specifically set according to actual conditions, which is not limited in the embodiments of the present application.

[0103] Further, the static fault tolerance rate information is a pre-evaluated parameter representing the inherent tolerance of the target large model to the loss of calculation accuracy when performing the specified "target processing task". The static fault tolerance rate depends on the analysis of the essential characteristics of the target processing task, and its value does not change with the change of the runtime environment. For example, in open-domain dialogue, creative writing and other semantic flexible tasks, the model output has multiple reasonable forms, so the system has high tolerance to precision loss, and the static fault tolerance rate is determined to be high. However, in mathematical calculation, code generation or logical reasoning tasks, the accuracy of the result is extremely high, and a small error may cause the output to be completely wrong, so its static fault tolerance rate is low.

[0104] Further, the dynamic fault tolerance rate information is determined by continuously monitoring a series of historical data processing results previously generated by the computer system. This "dynamic error accumulation monitoring" can discover the trend that the error may gradually amplify due to continuous use of low-precision calculation and eventually affect the quality of the result. It reflects the actual performance of the model in the current running state in real time, and is a dynamic and real-time supplement and correction to the static fault tolerance rate.

[0105] Further, the target processing pressure score is a comprehensive quantitative evaluation of the current running load of the processor, which is calculated based on internal processing pressure information (such as calculation unit utilization, memory bandwidth occupation, cache contention, etc. collected by the performance monitoring unit) and external processing pressure information (such as chip temperature, power consumption, power supply stability, etc. physical environment data obtained by the sensor unit); the target processing pressure score reflects the overall pressure level of the hardware system when performing the task, and the higher the value, the closer the processor is to the resource limit, which may have the risk of overheating, high latency or energy efficiency decline, prompting the system to take load reduction or energy saving strategies. The target fault tolerance rate score is a comprehensive evaluation of the tolerance of the target large model to calculation errors under the current task, which combines static fault tolerance rate information and dynamic fault tolerance rate information; the higher the target fault tolerance rate score, the less sensitive the model output is to precision loss, and thus the computer system has greater flexibility to adjust the calculation accuracy to adapt to resource constraints during inference.

[0106] Further, configurable hyperparameters, first weight and second weight, can be introduced. In this way, the device can flexibly adjust the relative importance of the two factors according to the actual application scenario, and then determine the target score by combining the target processing pressure score and the target fault tolerance rate score; then, the device will select the target meta-parameter from the multiple hierarchical meta-parameters according to the target score; finally, the selected target meta-parameter is used to complete the dequantization and data processing.

[0107] It can be understood that the device proposed in the embodiments of the present application no longer adopts a fixed quantization strategy as in the traditional data processing method, but dynamically and intelligently determines the most appropriate calculation precision in real time by sensing the internal and external pressure of the processor and continuously monitoring the error accumulation of the model output. When the processor load is high or the task fault tolerance rate is high, it will automatically select a lower precision quantization scheme (such as INT4) to maximize the processing speed; and when the processor load is low or the precision is monitored to be reduced, it can seamlessly switch to a higher precision scheme (such as INT8 or higher) to ensure the accuracy of the results. This closed-loop adaptive adjustment capability enables the device proposed in the embodiments of the present application to always operate at the optimal operating point under various complex and variable workloads, greatly improving its energy efficiency ratio and overall resource utilization efficiency.

[0108] In some embodiments, as shown in Figure 12 , Figure 12 is Figure 11 an optional implementation flowchart of step D.5, assigning a first weight to the target processing pressure score and a second weight to the target fault tolerance rate score, comprising the following steps: E.1 If the target processing pressure score does not reach the preset pressure threshold, and the dynamic fault tolerance rate information includes abnormal information, assign a first weight to the target processing pressure score and a second weight to the target fault tolerance rate score, wherein the second weight is greater than the first weight; E.2 If the target processing pressure score reaches the preset pressure threshold, and the dynamic fault tolerance rate information includes abnormal information, stop the weight assignment operation and generate target warning information.

[0109] The steps E.1 to E.2 are described in detail below.

[0110] In some embodiments, when the computer system has performance redundancy (i.e., the target processing pressure score does not reach the preset pressure threshold), and the output precision of the model has started to decline and triggered an alarm (i.e., the dynamic fault tolerance rate information includes abnormal information), a second weight greater than the first weight is forcibly assigned to the target fault tolerance rate score, so that the data processing device can prioritize model precision when making decisions, so as to fully utilize the advantages of hardware redundancy, thereby timely suppressing error accumulation, improving output stability, effectively recovering and maintaining the inference quality of the model without sacrificing performance, and achieving optimal synergy between resources and precision.

[0111] The preset stress threshold refers to a critical value set during system design to measure the processor load level and is used to determine whether the hardware is currently operating under high stress. This threshold is typically set based on a combination of processor performance monitoring data (such as compute unit utilization and memory bandwidth occupancy) and external physical indicators (such as temperature and power consumption). When the target processing stress score is below this threshold, it indicates that the processor has sufficient computing resources and can handle a certain degree of high-precision computing load. However, once this threshold is reached or exceeded, it means that the processor is approaching its performance limit or heat dissipation boundary. Further increasing the load may lead to increased latency, decreased energy efficiency, or even system frequency reduction, necessitating a protective scheduling strategy.

[0112] Abnormal information refers to signals that indicate a significant decrease in model output quality or deviation from the normal range during the dynamic error accumulation monitoring of historical data processing results. It is used to characterize the unstable or unreliable state of the model reasoning process. For example, when multiple consecutive reasoning results exhibit logical contradictions, semantic confusion, repeated outputs, or excessive deviations from the reference output, the system determines that the error accumulation has exceeded a reasonable range and triggers an abnormal flag. The appearance of abnormal information reflects that the model can no longer maintain the expected accuracy under the current precision configuration or operating environment, prompting the device to adjust its calculation strategy to restore output quality.

[0113] Furthermore, if the processor is already fully loaded or overloaded (the target processing pressure score reaches the preset pressure threshold) and the output accuracy of the model also deteriorates unacceptably (the dynamic fault tolerance information includes abnormal information), that is, the data processing device cannot sacrifice performance in exchange for accuracy, nor can it relieve pressure by sacrificing accuracy. At this time, the data processing device will decisively stop the weight allocation operation, avoid making any automatic decisions that may cause the situation to worsen, and immediately generate a target warning message, reporting this conflict state that cannot be automatically resolved to the upper management system or manual operator to request external intervention.

[0114] like Figure 13 As shown, Figure 13 20 is an optional device module diagram of a data processing device for large model parameters provided in an embodiment of the present application. The data processing device for large model parameters may include the following modules 201 to 204: An acquisition module 201 is configured to acquire, from different partitions of a first data storage area of ​​a memory, a quantization matrix of a target large model obtained after parameter compression, and meta-parameters for restoring the quantization matrix; An insertion module 202 is configured to read a memory access granularity representing the data processing capability of the processor from a configuration data storage area of ​​the processor, and determine a size of a padding area between the insertion meta-parameter and the quantization matrix based on the memory access granularity; The data atom block determination module 203 is configured to divide a second data storage area different from the first data storage area in the memory, and sequentially write the meta parameter, the preset padding information matched with the padding area size, and the quantization matrix in the second data storage area to obtain an initial data atom block. The target storage module 204 is configured to arrange all the initial data atom blocks in the second data storage area in sequence to obtain a target data atom block.

[0115] The specific implementation of the data processing apparatus for large model parameters is basically the same as the specific embodiments of the data processing method for large model parameters described above, and will not be repeated here.

[0116] The embodiments of the present application also provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the data processing method for large model parameters when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0117] As shown in Figure 14 , the hardware structure of the electronic device provided by the embodiments of the present application is shown in Figure 14 . The electronic device includes: The processor 301 can be implemented in the form of a general central processing unit, a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application. The memory 302 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 302 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are saved in the memory 302 and are called and executed by the processor 301 to implement the data processing method for large model parameters provided by the embodiments of the present application. The input / output interface 303 is used to realize information input and output. The communication interface 304 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.). A bus 305 transmits information between various components (e.g., the processor 301, the memory 302, the input / output interface 303, and the communication interface 304) of the device. The processor 301, the memory 302, the input / output interface 303, and the communication interface 304 are communicatively connected to each other within the device through the bus 305.

[0118] The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the above-mentioned data processing method for large model parameters.

[0119] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0121] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0122] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0123] Those skilled in the art can understand that all or some of the steps in the above-mentioned method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware, and appropriate combinations thereof.

[0124] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a recited step or its integral sub-steps or additional steps whether or not readily ascertainable from the description or the like. Further, the words "a" or "an", as used herein in the disclosure and elsewhere, are used indiscriminately and are to be interpreted in the same way, i.e. as meaning "one or more".

[0125] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are only A, only B, and A and B at the same time. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or the like means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0126] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0127] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0128] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0129] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0130] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A data processing method for large model parameters, characterized in that: Applied to a processor, the processor being associated with a memory, the data processing method for large model parameters includes: Obtaining, from different partitions of the first data storage area of ​​the memory, a quantization matrix obtained after parameter compression of the target large model and meta-parameters used to restore the quantization matrix; Reading a memory access granularity representing a data processing capability of the processor from a configuration data storage area of ​​the processor, and determining a size of a padding area inserted between the meta-parameter and the quantization matrix based on the memory access granularity; dividing a second data storage area different from the first data storage area in the memory, and sequentially writing the meta-parameters, preset padding information matching the size of the padding area, and the quantization matrix into the second data storage area to obtain an initial data atomic block; All the initial data atomic blocks are continuously arranged in the second data storage area to obtain target data atomic blocks.

2. The data processing method for large model parameters according to claim 1, characterized in that: The determining, based on the memory access granularity, a size of a padding area inserted between the meta-parameter and the quantization matrix comprises: Determining a sum of the meta-parameter and the data length of the quantization matrix as a target data length; If the target data length is an integer multiple of the memory access granularity, determining the size of the padding area to be zero; If the target data length is smaller than the memory access granularity, calculating a first difference between the memory access granularity and the target data length, and determining the size of the padding area to be the first difference; If the target data length is greater than the memory access granularity, a modulo operation is performed on the target data length and the memory access granularity to obtain a modulo result, and a second difference between the memory access granularity and the modulo result is calculated to determine the filling area size as the second difference.

3. The data processing method for large model parameters according to claim 1, characterized in that: The meta parameters include scaling factors and offset parameters; The step of sequentially writing the meta-parameters, the preset filling information matching the size of the filling area, and the quantization matrix into the second data storage area to obtain an initial data atomic block includes: The scaling factor, the offset parameter, the preset filling information matching the size of the filling area, and the quantization matrix are written sequentially into the second data storage area to obtain the initial data atomic block, wherein the preset filling information is any one of preset zero data, a preset identifier, or a preset check code, and the data length of the initial data atomic block is an integer multiple of the memory access granularity.

4. The data processing method for large model parameters according to claim 3, characterized in that: The processor is integrated with a register unit; After all the initial data atomic blocks are continuously arranged in the second data storage area to obtain target data atomic blocks, the method further includes: When the processor receives the data processing signal, the processor loads the target data atomic block stored in the second data storage area into the register unit; The target data atomic blocks located in the register unit are accessed in parallel to realize data processing of the parameters of the target large model and obtain target data processing results.

5. The data processing method for large model parameters according to claim 4, characterized in that: The parallel access to the target data atomic block located in the register unit to implement data processing of the parameters of the target large model to obtain a target data processing result includes: Starting a preset thread block, wherein the thread block includes a plurality of threads, and each of the threads includes corresponding index information; Each of the threads determines a corresponding fixed offset position based on the corresponding index information, and reads the corresponding meta-parameter and the quantization matrix corresponding to the meta-parameter based on the fixed offset position; Based on the read scaling factor and the offset parameter, performing inverse quantization processing on the quantization matrix to obtain a target inverse quantization matrix; Matrix operation processing is performed based on the target inverse quantization matrix to obtain the target data processing result.

6. The data processing method for large model parameters according to claim 5, characterized in that: The step of performing inverse quantization processing on the quantization matrix based on the read scaling factor and the offset parameter to obtain a target inverse quantization matrix includes: Multiplying the quantization matrix and the scaling factor to obtain an initial inverse quantization matrix; The initial inverse quantization matrix and the offset parameter are added to obtain the target inverse quantization matrix.

7. The data processing method for large model parameters according to claim 6, characterized in that: The method further includes: performing inverse quantization processing on the quantization matrix based on the read scaling factor and the offset parameter to obtain a target inverse quantization matrix; If the quantization matrix is ​​represented in UINT8 format, the quantization matrix includes a high-bit quantization matrix and a low-bit quantization matrix; Multiplying the high-order quantization matrix and the scaling factor to obtain an initial high-order inverse quantization matrix, and adding the initial high-order inverse quantization matrix and the offset parameter to obtain a high-order inverse quantization matrix; Multiplying the low-bit quantization matrix and the scaling factor to obtain an initial low-bit inverse quantization matrix, and adding the initial low-bit inverse quantization matrix and the offset parameter to obtain a low-bit inverse quantization matrix; The target inverse quantization matrix is ​​determined based on the high-order inverse quantization matrix and the low-dimensional inverse quantization matrix.

8. The data processing method for large model parameters according to claim 4, characterized in that: The meta-parameters include hierarchical meta-parameters at multiple different precision levels; The parallel access to the target data atomic block located in the register unit to implement data processing of the parameters of the target large model to obtain a target data processing result further includes: determining internal processing pressure information of the processor based on a preset performance monitoring unit of the processor, and determining external processing pressure information of the processor based on a preset sensor unit of the processor; Obtaining static fault tolerance information of the target large model, wherein the static fault tolerance information is determined in advance based on a target processing task of the target large model; Obtaining historical data processing results, performing dynamic error accumulation monitoring on the historical data processing results, and obtaining dynamic fault tolerance rate information; determining a target processing pressure score based on the internal processing pressure information and the external processing pressure information, and determining a target fault tolerance score based on the static fault tolerance information and the dynamic fault tolerance information; Assigning a first weight to the target processing pressure score and a second weight to the target fault tolerance score, and determining a target score based on the first weight, the second weight, the target processing pressure score, and the target fault tolerance score; The target data atomic block located in the storage unit is accessed in parallel to obtain the meta-parameters, target meta-parameters are selected from the multiple hierarchical meta-parameters based on the target scores, and data processing of the parameters of the target large model is performed based on the target meta-parameters to obtain the target data processing results.

9. The data processing method for large model parameters according to claim 8, characterized in that: The assigning a first weight to the target processing pressure score and assigning a second weight to the target fault tolerance score includes: If the target processing pressure score does not reach the preset pressure threshold and the dynamic fault tolerance rate information includes abnormal information, assigning a first weight to the target processing pressure score and assigning a second weight to the target fault tolerance rate score, wherein the second weight is greater than the first weight; If the target processing pressure score reaches a preset pressure threshold and the dynamic fault tolerance information includes abnormal information, the weight allocation operation is stopped and a target warning message is generated.

10. A data processing device for large model parameters, characterized in that: include: an acquisition module, configured to acquire, from different partitions of a first data storage area of ​​a memory, a quantization matrix of a target large model obtained after parameter compression, and meta-parameters for restoring the quantization matrix; an insertion module, configured to read a memory access granularity representing a data processing capability of the processor from a configuration data storage area of ​​the processor, and determine a size of a padding area inserted between the meta-parameter and the quantization matrix based on the memory access granularity; a data atomic block determination module, configured to divide a second data storage area different from the first data storage area in the memory, and sequentially write the meta-parameters, preset padding information matching the size of the padding area, and the quantization matrix into the second data storage area to obtain an initial data atomic block; The target storage module is configured to continuously arrange all the initial data atomic blocks in the second data storage area to obtain target data atomic blocks.

11. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the data processing method for large model parameters according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method for large model parameters according to any one of claims 1 to 9 is implemented.

Images