Large model reasoning method, large model reasoning system and computer storage medium

Abstract

The invention relates to the field of artificial intelligence, and relates to a large model reasoning method, a large model reasoning system and a computer storage medium, and the large model reasoning method comprises the steps: dynamically determining the weight quantization precision required by a current network layer in a preset reasoning model according to current input data; obtaining target weight data corresponding to the current network layer according to the weight quantization precision required by the current network layer; according to the target weight data and the current input data, executing forward calculation of the current network layer, generating intermediate output data, and taking the intermediate output data as input data of the next network layer; and processing each network layer in the preset reasoning model in sequence, and when the last network layer in the preset reasoning model is reached, obtaining target output data. According to the method, the weight quantization precision of each network layer is dynamically determined based on the current input, efficient and accurate layer-by-layer reasoning is realized, the model reasoning efficiency is improved, and the output quality is ensured.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a large-scale model reasoning method, a large-scale model reasoning system, and a computer storage medium. Background Technology

[0002] In current technologies, with the development of deep learning, the parameter scale of large-scale pre-trained models (commonly known as "large models") has reached hundreds of millions or even billions, requiring significant computational resources and storage space for inference. To improve inference efficiency, existing solutions typically employ a uniform quantization strategy, compressing all model weights with a fixed low bit width (e.g., 8 bits or 4 bits) to reduce data size and computational burden. However, this "one-size-fits-all" precision setting cannot meet the diverse precision requirements of different inputs or subtasks: using high precision for simple inputs leads to resource waste; for complex inputs, low precision significantly affects the accuracy of the results. Some solutions attempt to achieve switching by pre-preparing multiple models with different precision configurations, but this not only increases memory usage and maintenance costs but also lacks a mechanism for dynamically adjusting precision based on specific inputs.

[0003] Furthermore, due to the large number of parameters in large models, traditional deployment methods require loading the entire model into high-speed GPU memory at once to ensure real-time inference speed, making GPU memory capacity a bottleneck. Although weights can be kept resident in low-speed storage such as CPU memory and loaded on demand during inference, the lack of an effective scheduling strategy will lead to frequent data exchanges, increasing latency and reducing system performance. Summary of the Invention

[0004] One objective of this invention is to provide a large-model inference method, a large-model inference system, and a computer storage medium to solve the technical problems in the prior art of how to dynamically adjust the model calculation accuracy, reduce peak memory usage and invalid weight residency, while avoiding a significant increase in memory access latency.

[0005] In a first aspect, embodiments of the present invention provide a reasoning method for large models, the method comprising: Based on the current input data, dynamically determine the required weight quantization precision for the current network layer in the preset inference model; Based on the required weight quantization precision of the current network layer, the target weight data corresponding to the current network layer is obtained; Based on the target weight data and the current input data, perform the forward computation of the current network layer to generate intermediate output data, and use the intermediate output data as the input data of the next network layer; Each network layer in the preset inference model is processed sequentially, and the target output data is obtained when the last network layer in the preset inference model is reached.

[0006] In a second aspect, a large model inference system is provided, the large model inference system including an electronic device, the electronic device including a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor, when executing the one or more computer programs, causing the large model inference system to implement the large model inference method as described in the first aspect.

[0007] In a third aspect, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the reasoning method for a large model as described in the first aspect.

[0008] In a fourth aspect, a large-scale model inference apparatus is provided, the large-scale model inference apparatus comprising: The determination unit is used to dynamically determine the weight quantization precision required for the current network layer in the preset inference model based on the current input data. The determining unit is further configured to obtain the target weight data corresponding to the current network layer based on the weight quantization precision required by the current network layer. The processing unit is configured to perform forward computation of the current network layer based on the target weight data and the current input data, generate intermediate output data, and use the intermediate output data as input data for the next network layer. The processing unit is also used to process each network layer in the preset inference model in sequence, and when the last network layer in the preset inference model is reached, the target output data is obtained.

[0009] In the embodiments implemented by the aforementioned large-scale model inference method, large-scale model inference system, and computer storage medium, this embodiment dynamically determines the required weight quantization precision of each network layer in the preset inference model based on the current input data. The system can achieve adaptive precision allocation for different inputs, thereby reducing unnecessary computational resource consumption while ensuring model inference accuracy. Based on the determined weight quantization precision, the corresponding target weight data is accurately obtained, making the forward computation of each network layer more efficient and accurate. Furthermore, by processing and transmitting intermediate output data layer by layer, the continuity and accuracy of the model inference process are ensured, ultimately obtaining high-quality target output data. Therefore, this solution effectively improves inference efficiency, optimizes computational resource utilization, and maintains the model's inference performance. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the structure of a large model inference system according to an embodiment of the present invention; Figure 2 This is a software architecture diagram of a large model inference system according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of another large-scale model inference system in an embodiment of the present invention; Figure 4 This is a flowchart illustrating a reasoning method for a large model in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a large-scale inference device according to an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0013] It should be noted that, unless otherwise specified, the various features in the embodiments of this invention can be combined with each other, all of which are within the protection scope of this invention. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this invention do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0014] In existing technologies, large-scale model acceleration solutions primarily employ a uniform quantization strategy, representing all model weights using a fixed low bit width (e.g., 8 bits or 4 bits). However, this "one-size-fits-all" precision strategy has significant drawbacks; it cannot flexibly adapt to the varying precision requirements of different inputs or subtasks. Specifically, for simple inputs, using a fixed high precision leads to unnecessary waste of computational resources; while for complex inputs, a fixed low precision may significantly impair the accuracy of the results. Furthermore, another approach attempts to pre-prepare multiple models with different precisions for switching, but this not only significantly increases memory usage and maintenance costs but also lacks the ability to dynamically adjust precision based on specific inputs.

[0015] On the other hand, the massive parameter scale of large models also presents a memory bottleneck for their deployment. Traditional deployment methods require loading the entire model into high-speed video memory (such as GPU memory) at once to ensure real-time inference speed, but this makes GPU memory capacity a key factor limiting model deployment. Although model weights can be stored in slower storage such as CPU memory and loaded on demand during inference, frequent data exchanges will introduce significant latency overhead without an effective scheduling strategy, which will actually reduce the overall system performance. Therefore, how to reduce the long-term residence of invalid weight data in video memory, reduce peak memory usage, and without significantly increasing memory access latency has become a key problem that urgently needs to be solved in the field of large model inference.

[0016] Therefore, this embodiment dynamically determines the required weight quantization precision for each network layer in the preset inference model based on the current input data. The system can achieve adaptive precision allocation for different inputs, thereby reducing unnecessary computational resource consumption while ensuring model inference accuracy. Based on the determined weight quantization precision, the corresponding target weight data is accurately obtained, making the forward computation of each network layer more efficient and accurate. Furthermore, by processing and passing intermediate output data layer by layer, the continuity and accuracy of the model inference process are ensured, ultimately obtaining high-quality target output data. Therefore, this solution effectively improves inference efficiency, optimizes computational resource utilization, and maintains the model's inference performance.

[0017] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a large-scale model inference system. Figure 1 In the large model inference system 10, there is an electronic device 20, which includes at least one processor 201 and a memory 202.

[0018] Among them, electronic equipment 20 can be servers, network devices (such as routers and switches), programmable logic controllers (PLCs), monitoring and management equipment, etc., and is not limited to one specific device.

[0019] The processor 201 can be a central processing unit (CPU), a graphics processing unit (GPU), or other computing unit suitable for performing neural network inference. The memory 202 includes a multi-level storage structure such as main memory (e.g., CPU memory) and GPU video memory, used to store program code and data.

[0020] The processor 201 is configured to support the large-model inference system in executing the corresponding functions of the large-model inference method in the above-described method embodiments. The processor 201 can be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0021] The memory 202 is used to store program code and common storage components, including graph databases, vector databases, MySQL, Redis, and MQ, to ensure data storage and management. The memory 202 may include volatile memory (VM), such as random access memory (RAM); it may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it may include a combination of the above types of memory.

[0022] Specifically, the processor 201 may include a transmitting card, a receiving card, and a driver chip.

[0023] At the software level, such as Figure 2 As shown, Figure 2Figure 2 shows the software architecture of the large model inference system, which includes a model weight storage module 30 and a precision scheduling module 40. The large model inference system also implements an on-demand loading function to dynamically manage the scheduling of weight data across different storage levels.

[0024] Specifically, the model weight storage module 30 uses bit-plane compression to store all parameters of the large model. This involves splitting the model's full-precision weights into multiple bit planes based on the importance of each binary bit, and storing these bit-plane data in main memory (e.g., CPU memory or other external storage media). This decomposition method allows the weights to have multiple precision representations, and the system can flexibly combine different numbers of bit planes as needed to achieve dynamic precision adjustment.

[0025] The core purpose of bit-plane decomposition is very clear: to decompose a complete weight tensor into multiple binary "bit planes", thereby decoupling the storage of weights from the bit width used in computation.

[0026] The precision scheduling module 40 includes a trainable router unit embedded before the key computing unit. This router analyzes the characteristics of the current input data in real time and outputs a control signal indicating the number of activated bit planes required for each layer. Through training, the router can dynamically select appropriate weights to quantize precision based on input complexity and the importance of model layers, thereby optimizing the use of computing resources while ensuring inference accuracy.

[0027] For example, for a given network layer, the router determines whether that layer needs higher-precision weights to participate in the computation when processing the current input. If the input is relatively simple or the layer has little impact on the final result, the router can choose to use only a few high-order bit planes (low precision) to meet the requirements. Conversely, if the input is complex or the layer has a significant impact on the model's accuracy, the router will instruct the loading of more bit planes (increasing precision) to participate in the layer's computation. The router's decision logic is obtained through learning from a large number of samples during the training phase, enabling it to comprehensively weigh computational costs and accuracy loss, and adaptively select the optimal precision configuration layer by layer.

[0028] The on-demand loading function dynamically schedules the migration of weight data between storage layers based on the router's precision requirements. Specifically, CPU memory serves as the main storage unit, holding all bit-plane weight data, while GPU memory acts as a high-speed cache, responsible for storing the weight bit-planes required for the current inference. During inference, the on-demand loading mechanism checks whether the GPU memory has already loaded weight data that meets the precision requirements of the current layer. If some bit-planes are not yet loaded, the corresponding data is retrieved from CPU memory and transferred to GPU memory; conversely, unused bit-planes remain in CPU memory, avoiding the occupation of valuable GPU memory space. After computation, the system promptly releases or reverts unused weight data, improving memory utilization efficiency.

[0029] Through the above architecture design and functional implementation, the large model inference system 10 can achieve dynamic precision scheduling and efficient storage resource management while ensuring model inference accuracy, significantly improving inference efficiency and reducing GPU memory pressure.

[0030] For example, you can refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a large-scale model inference system; Figure 3 In this model, Input: The input data is the starting point for the entire large-scale inference process, such as user questions or text to be analyzed; Embedding: The embedding layer is the first layer of the model, responsible for converting discrete data (such as words) into continuous, high-dimensional vector representations so that the neural network can process them; Router: The router is the core component of the precision scheduling module 40, a lightweight, trainable neural network. Its role is to intelligently determine the precision (i.e., how many bit planes) that each subsequent computational unit should use for computation based on the characteristics of the input data.

[0031] The Transformer consists of multiple repeating "blocks". Each block typically contains a multi-head attention mechanism (MHA) and a multilayer perceptron (MLP), normalized together using RMS Norm. The diagram illustrates the composition of multiple computational units (such as attention modules and multilayer perceptron modules) within a single block in a typical Transformer architecture. Multiple blocks are stacked sequentially layer by layer within the Transformer, with the input of each subsequent block being the output of the previous block.

[0032] Furthermore, MHA (Multi-Head Attention) is a core component of the Transformer model, used to capture dependencies between different positions in the input sequence.

[0033] Among them, MLP (Multi-Layer Perceptron) usually refers to a neural network composed of multiple fully connected layers, which is used to perform nonlinear transformations on the output of the attention mechanism.

[0034] The structure includes: RMS Norm (Root Mean Square Layer Normalization), a commonly used neural network normalization technique for stabilizing the training process; Linear layer, also known as a fully connected layer, used to perform linear transformations on the input; Softmax function, typically used in the last layer of a neural network to convert the output into a probability distribution; Weights, which are parameters in a neural network and represent the knowledge learned by the model; and Bit-Planes, which are multiple independent binary matrices formed by splitting the weight data into its binary bits. Each matrix represents a single bit (0 or 1) of the weight value. From top to bottom (MSB to LSB), they represent the most significant bit to the least significant bit. The MSB (Most Significant Bit) is the leftmost bit in a binary number and has the greatest impact on the value. The LSB (Least Significant Bit) is the rightmost bit in a binary number and has the least impact on the value.

[0035] The weight pool, the storage area on the right side of the diagram, represents the model weight storage module 30. It stores the bit-plane data of all weight parameters of the model. CPU memory: Central Processing Unit memory. As the system's main memory, it stores the complete, bit-plane-split model weight data. GPU memory: Graphics Processing Unit memory. As high-speed memory, it loads the bit-plane weights required for the current computation layer only when needed, allowing the GPU to perform high-speed calculations.

[0036] Therefore, the input data in this scheme first passes through the Embedding layer, where it is converted into a vector representation that the model can understand. This vector representation is then fed into the Router. The Router analyzes the features of the current input and determines the precision requirements of subsequent computational units such as MHA and MLP. It outputs a control signal for each computational unit, deciding how many bit planes that unit should use. For example, in the diagram, the Router decides on 5-bit precision for the MHA unit on the left, 4-bit precision for the MLP unit on the right, 8-bit precision for another MHA unit, and 3-bit precision for another MLP unit. This decision-making process is real-time and may differ for each input. On-demand weight loading (executed by the on-demand loading function): Based on the Router's decision, the on-demand loading function begins its work. It checks whether the GPU memory has already loaded weight data that meets the precision requirements of the current computational unit. If not, it loads only the high-importance bit planes selected by the Router from the weight pool in CPU memory (e.g., for 5-bit precision, loading the MSB to the 5th bit plane). The dashed arrows in the diagram illustrate this "on-demand" data flow: only selected bit planes flow from CPU memory to GPU memory. Weights of a specific precision loaded into GPU memory, along with the current input data, are fed into the corresponding computation unit (such as MHA or MLP) for forward computation. The computation results (intermediate output data) continue to the next layer, repeating the process until all network layers have been computed. Furthermore, after a computation unit completes its task, less important bit plane weights that are no longer needed can be promptly released from GPU memory or migrated back to CPU memory to free up valuable GPU memory space for potentially higher-precision computations later. Finally, after computation across all layers, the model outputs a result, typically converted into a probability distribution by a Softmax layer, yielding the final inference result (as shown by the arrows at the top of the diagram).

[0037] See Figure 4 , Figure 4 A flowchart illustrating a large-scale model inference method provided in an embodiment of the present invention is shown. The method includes the following steps: S10. Based on the current input data, dynamically determine the weight quantization accuracy required for the current network layer in the preset inference model.

[0038] The current input data refers to the input information passed to the network layer currently being processed during inference. The current input data includes initial input data and intermediate output data.

[0039] Specifically, initial input data refers to the raw data provided by the user at the beginning, such as a sentence or an image. Intermediate output data refers to the result generated after a certain layer of the model has completed its calculation. This result will continue to be passed as the current input data for the next layer. For example, after the first layer has finished processing, its output becomes the current input data for the second layer.

[0040] Here, the current network layer refers to the specific layer in the model that is currently performing inference computation. The network layers are arranged sequentially, with the first network layer being the first layer near the input layer of the model, and subsequent layers processing the output of the previous layer in turn.

[0041] Specifically, if the current input data is the initial input data, the current network layer is the first network layer; if the current input data is the intermediate output data, the current network layer is a network layer other than the first network layer.

[0042] The pre-set inference model refers to a neural network model that has been trained and deployed in a large-scale inference system to perform inference tasks (such as image recognition, natural language processing, etc.). This model consists of multiple network layers, each containing a certain number of parameters (weights).

[0043] In this context, weight quantization precision refers to the number of binary bits used for model weights during calculation. Higher precision (more bits) results in more accurate values ​​and generally more precise calculation results, but also requires more computation and memory. Conversely, lower precision (fewer bits) results in more accurate values ​​and greater memory usage.

[0044] Dynamically determining the weight quantization precision refers to adjusting the weight quantization precision used by the current network layer in real time based on the characteristics of the current input data, rather than using the same precision at a fixed time. The aim is to optimize computational efficiency and resource utilization while ensuring inference performance.

[0045] The required weight quantization precision for the current network layer refers to the bit width used for quantizing weights, which is dynamically determined for the current network layer, such as 2-bit, 4-bit, 8-bit, etc., and determines the precision of the weight data representation.

[0046] The specific process of S10 can be combined with S110-S120, and will not be described again here.

[0047] As can be seen, through the aforementioned dynamic adjustment mechanism in this embodiment, the system can flexibly determine the required weight quantization precision for each network layer based on the characteristics of the current input data (whether it is the initial input or intermediate output). This not only ensures the accuracy of inference but also effectively reduces the consumption of computing and storage resources, improving overall inference efficiency and system performance.

[0048] S110. In one embodiment, the current input data is initial input data, and the current network layer is a first network layer. The step of dynamically determining the required weight quantization precision of the current network layer in a preset inference model based on the current input data includes: preprocessing the initial input data to obtain a feature vector corresponding to the initial input data; calculating the first network layer based on the feature vector to obtain a first importance score corresponding to each candidate bit width in the first network layer; performing probability transformation on the first importance score corresponding to each candidate bit width based on a preset temperature parameter and a preset function to obtain a first probability distribution corresponding to the first network layer, the first probability distribution including the probability value corresponding to each candidate bit width; and determining the required weight quantization precision of the first network layer based on the first probability distribution.

[0049] Preprocessing refers to a series of transformations and feature extraction operations performed on the initial input data, converting it into a format more suitable for router analysis, usually a high-dimensional vector, i.e., a feature vector.

[0050] Among them, the feature vector is a numerical array obtained from the preprocessing step, representing the key feature information of the input data, which facilitates subsequent calculation and decision-making.

[0051] Among them, the candidate bit width refers to a predefined set of optional weight quantization precisions, such as 2-bit, 4-bit, 8-bit, etc. The system will select the quantization precision suitable for the current network layer from these candidates.

[0052] The first importance score is a score calculated for each candidate bit width. This score reflects the importance or contribution to the final result if the current network layer uses this bit width. The higher the score, the more suitable the bit width is for the current input.

[0053] The preset temperature parameter (represented by α) is a parameter used to control the sharpness of the probability distribution. The higher the temperature (the smaller α), the flatter the probability distribution, indicating that the router has less confidence in choosing which bit width to select and tends to explore; the lower the temperature (the larger α), the sharper the probability distribution, indicating that the router has great confidence in choosing a specific bit width.

[0054] The preset function here refers to the Softmax function. It is a mathematical function that transforms any real vector into a probability distribution, with the sum of its output values ​​being 1, and each value representing a probability.

[0055] The probability transformation refers to the process of mapping importance scores to probability distributions, usually using softmax or similar functions, so that the sum of the probabilities of all candidate bit widths is 1.

[0056] The first probability distribution represents the set of probabilities that a candidate bit width is selected, with each candidate bit width corresponding to a probability value.

[0057] Specifically, the input data (such as an image) is first preprocessed to extract vectors describing its features. For example, the image is processed through a convolutional or feature extraction network to obtain a fixed-length feature vector.

[0058] Specifically, in the process of calculating the first importance score corresponding to each candidate bit width in the first network layer based on the feature vector, the following formula can be used: For a given input query x, first evaluate its value in the model block. Hidden representation at the location And calculate an importance score. This score reflects how sensitive the computation of that block is to quantization errors with respect to the current input. The formula for calculation is: in It is a lightweight multilayer perceptron (MLP) that makes decisions based on the internal state of the model.

[0059] Specifically, in the process of performing probability transformation on the first importance score corresponding to each candidate bit width according to the preset temperature parameter and preset function to obtain the first probability distribution corresponding to the first network layer, the following formula can be used: The importance score is transformed into a value with respect to different candidate bit widths using the softmax function. The probability distribution of ) Here, α is a temperature parameter used to control the "sharpness" of the probability distribution. A high α produces a "sharp" low-entropy distribution, meaning that the router is very confident in its choice of a specific level of precision; while a low α produces a "flat" distribution, indicating higher uncertainty and spreading the probability across multiple bit widths.

[0060] Optionally, determining the required weight quantization precision of the first network layer based on the first probability distribution includes: filtering within the first probability distribution to determine the candidate bit width corresponding to the largest probability value as the target bit width; determining the target bit width as the required weight quantization precision of the first network layer; or, randomly sampling the first probability distribution to obtain a sampling result; and determining the sampling result as the required weight quantization precision of the first network layer.

[0061] The target bit width refers to the bit width that is ultimately selected in the probability distribution, which is the quantization precision that the current network layer will actually use.

[0062] Among them, selecting the maximum probability value means directly choosing the candidate bit width with the highest probability as the final quantization precision.

[0063] Random sampling refers to randomly selecting a candidate bit width based on a probability distribution.

[0064] Specifically, in the process of filtering in the first probability distribution and determining the candidate bit width corresponding to the largest probability value as the target bit width, the candidate bit width with the largest probability value in the probability distribution is directly selected as the final decision.

[0065] Specifically, in the process of randomly sampling the first probability distribution to obtain the sampling result, the probability distribution is treated as a discrete probability mass function, and a random sampling is performed. The sampling result is the bit width of the final decision. This strategy introduces randomness, which can sometimes help the model explore different combinations of precision during training and avoid getting trapped in local optima.

[0066] For example, suppose the first network layer has four candidate bit widths: 2-bit, 4-bit, 6-bit, and 8-bit.

[0067] The input is an image of a cat. After preprocessing, a feature vector is obtained, such as [0.3, 0.7, 0.2, 0.5]. Using this feature vector, four candidate bit width importance scores are calculated: [1.2, 2.5, 1.0, 0.8]. The importance scores are then probabilistically transformed using the softmax function and a preset temperature parameter (e.g., temperature = 0.5), resulting in the following probability distribution: 2-bit: 0.15, 4-bit: 0.60, 6-bit: 0.18, 8-bit: 0.07. This indicates that the system considers 4-bit quantization precision to be the most suitable for the current input data and the first network layer. Therefore, the 4-bit quantization precision with the highest probability is directly selected as the weight quantization precision for the first network layer.

[0068] As can be seen, in this embodiment, the quantization precision is dynamically adjusted according to the characteristics of different input data, avoiding a one-size-fits-all approach and improving the accuracy and efficiency of model inference; by dynamically selecting an appropriate bit width, unnecessary computation and storage overhead is reduced.

[0069] S120. In one embodiment, the current input data is the intermediate output data, and the current network layer is a network layer other than the first network layer. The step of dynamically determining the required weight quantization precision of the current network layer in the preset inference model based on the current input data includes: calculating the second importance score corresponding to each candidate bit width in the current network layer based on the intermediate output data; performing a probability transformation on the second importance score corresponding to each candidate bit width based on preset temperature parameters and a preset function to obtain a second probability distribution corresponding to the current network layer, the second probability distribution including the probability value corresponding to each candidate bit width; and determining the required weight quantization precision of the current network layer based on the second probability distribution.

[0070] The second importance score is similar to the first importance score, but it is an adaptive score for each candidate bit width of the current network layer (not the first layer) under the current input data (intermediate output data).

[0071] The second probability distribution is obtained by applying a probability transformation function to the second importance score, and represents the probability that each candidate bit width is selected.

[0072] Specifically, the current network layer receives the output from the previous network layer. This data has undergone feature extraction and transformation, containing abstract information from previous layers. Using the current input data, the performance and applicability of each candidate bitwidth in the current network layer are evaluated under the current input conditions, yielding a corresponding second importance score. This calculation process may involve a scoring model or empirical function. The second importance scores of all candidate bitwidths are then transformed into a second probability distribution using a preset function (such as softmax) and a temperature parameter. The temperature parameter controls the "smoothness" of the distribution, affecting the determinism of the selection.

[0073] Specifically, in the process of calculating the second importance score corresponding to each candidate bit width in the current network layer based on the intermediate output data, the specific description in S110 of calculating the first importance score corresponding to each candidate bit width in the first network layer based on the feature vector can be referred to, which will not be repeated here.

[0074] S120 is a subsequent network layer processing procedure following S110.

[0075] Specifically, the process of performing probability transformation on the second importance score corresponding to each candidate bit width according to the preset temperature parameter and preset function to obtain the second probability distribution corresponding to the current network layer can be referred to in S110 for the specific description of performing probability transformation on the first importance score corresponding to each candidate bit width according to the preset temperature parameter and preset function to obtain the first probability distribution corresponding to the first network layer, which will not be repeated here.

[0076] Optionally, determining the required weight quantization precision of the current network layer based on the second probability distribution includes: filtering within the second probability distribution to determine the candidate bit width corresponding to the largest probability value as the target bit width; determining the target bit width as the required weight quantization precision of the current network layer; or, randomly sampling the second probability distribution to obtain a sampling result; and determining the sampling result as the required weight quantization precision of the current network layer.

[0077] The specific implementation process of determining the required weight quantization accuracy of the current network layer according to the second probability distribution can be referred to in the specific description of determining the required weight quantization accuracy of the first network layer according to the first probability distribution in S110, and will not be repeated here.

[0078] For example, suppose the current network layer is the third layer, with four candidate bit widths: 2-bit, 4-bit, 6-bit, and 8-bit. The third layer receives the output feature vector from the second layer, such as [0.5, 0.1, 0.3, 0.7]. The importance scores for the four candidate bit widths are calculated using a scoring mechanism as: [0.9, 1.8, 1.5, 0.6]. Using the softmax function and a temperature parameter (e.g., 0.7), these scores are converted into probability distributions: 2-bit: 0.18, 4-bit: 0.45, 6-bit: 0.30, 8-bit: 0.07. Therefore, 4-bit is chosen as the quantization precision for the third layer.

[0079] As can be seen, this embodiment analyzes the characteristics of intermediate output data and dynamically adjusts the quantization bit width of the current layer to more precisely adapt to different levels and input characteristics of the model. Furthermore, by reasonably selecting the bit width, it avoids over-quantization leading to accuracy loss and also avoids wasting resources by using excessively high bit widths, thus achieving a balance between performance and resources.

[0080] S20. Obtain the target weight data corresponding to the current network layer according to the weight quantization accuracy required by the current network layer.

[0081] The target weight data refers to the weight data that, after quantization, is ultimately used for calculation in the current network layer. The target weight data meets the required quantization precision.

[0082] In one embodiment, obtaining the target weight data corresponding to the current network layer based on the required weight quantization precision of the current network layer includes: determining multiple weight components corresponding to the required weight quantization precision of the current network layer; if all of the multiple weight components exist in a preset storage area in the current network layer, combining the multiple weight components to generate the target weight data corresponding to the current network layer; or, if not all of the multiple weight components exist in the preset storage area in the current network layer, retrieving the missing weight components from a preset weight database to the preset storage area, and combining all the weight components to generate the target weight data corresponding to the current network layer.

[0083] In this context, a weight component refers to the multiple basic parts into which the weight data is divided. Each component typically corresponds to a bit plane, which is a binary matrix at a specific bit of the weight. For example, to obtain a 5-bit weight, 5 weight components (i.e., 5 bit planes) are required.

[0084] The preset storage area refers to the storage space where weight data is actually loaded for computation during model inference. In this scheme, the preset storage area can be GPU memory.

[0085] The preset weight database refers to a database that stores the complete model weights. The weights are stored in bit-plane format, and different bit-width weight components can be retrieved on demand. In this solution, the preset weight database is the "weight pool" located in CPU memory, where the weight data is stored in the form of multiple bit-planes.

[0086] Specifically, when determining the weight quantization precision required by the current network layer, among the multiple weight components corresponding to that precision, since each weight component represents a bit plane, determining 5-bit precision means requiring a total of 5 bit planes from the least significant bit to the most significant bit (e.g., MSB, ..., the 5th bit plane). For example, if the current network layer requires 5-bit precision, then the system needs the MSB, the 2nd bit, the 3rd bit, the 4th bit, and the 5th bit.

[0087] Furthermore, the system checks whether these five specific bit planes have been loaded into the GPU memory. For example, the system checks the GPU memory and finds that the MSB, bit planes 2, 3, and 4 already exist, but bit plane 5 is missing.

[0088] Furthermore, if all weight components already exist, these bit planes are directly combined to reconstruct the target weight data. For example, if the system finds that all five required bit planes have been loaded, they are directly combined to obtain a 5-bit weight matrix for subsequent use.

[0089] Alternatively, if some weight components are missing, the missing weight components (bit plane) will be retrieved as needed from the preset weight database (weight pool in CPU memory) and loaded into the preset storage area (GPU memory). Combination will only be performed after all necessary components have been loaded.

[0090] For example, if the system detects a missing fifth bit plane, it requests the data for that fifth bit plane from the weight pool in CPU memory and transfers it to GPU memory. After loading, the system then combines these five bit planes into the target weight data.

[0091] In one embodiment, the preset weight database includes all weight parameters of the preset inference model. The weight parameters are stored in the form of multiple bit planes, each bit plane being a weight component. Each bit plane is a binary matrix, and the multiple bit planes can be combined to generate weight data with different bit widths.

[0092] In this context, a bit plane refers to a two-dimensional matrix composed of all the weight values ​​corresponding to each bit in the binary representation of the weight data. For example, an 8-bit weight has 8 bit planes, arranged sequentially from the least significant bit to the most significant bit.

[0093] Specifically, the core purpose of bit-plane decomposition is very clear: to decompose a complete weight tensor into multiple binary "bit planes," thereby decoupling the storage of weights from the bit width used in computation. Mathematically, a primitive full-precision weight matrix W can be represented as a weighted sum of multiple binary matrices W^(b). The formula is as follows: Where B is the maximum bit width (e.g., 8), and Then it represents the b-th bit plane, which is a binary matrix containing only 0s and 1s.

[0094] As can be seen, this embodiment stores weights in bit-plane form, supporting flexible combinations and avoiding the need to store weights separately for each quantization bit width, thus saving storage space. Furthermore, it dynamically retrieves the required weight components according to the current layer's needs, improving system response speed and resource utilization. By combining different numbers of bit-planes, it supports multiple weight quantization strategies from low bit width to high bit width, meeting different precision requirements. In addition, a preset storage area caches some weight components, avoiding repeated readings from the database and improving access efficiency.

[0095] S30. Based on the target weight data and the current input data, perform the forward computation of the current network layer to generate intermediate output data, and use the intermediate output data as the input data of the next network layer.

[0096] Forward computation refers to the data flow process from input to output in a neural network. The current network layer uses the input data and weight parameters to perform calculations and generate the output result.

[0097] For example, while processing the second layer of the model, using 5-bit precision, the on-demand loading function has already loaded 5-bit weights onto the GPU. Simultaneously, the computation result of the first layer (intermediate output data 1) has also been passed to the second layer as the current input data.

[0098] Specifically, the current network layer receives two parts of data: the output (or original input) of the previous layer and the quantized weights of the current layer. Matrix multiplication, convolution, and other calculations are performed using the input data and target weights. The calculation process typically includes: multiplying the input data and weights, adding a bias (if applicable), and performing a non-linear transformation using an activation function (such as ReLU). The result is the output feature map or vector of the current network layer, called intermediate output data. This intermediate output data is used as the input data for the next network layer to continue the calculation of subsequent layers.

[0099] For example, suppose the current network layer is a convolutional layer: Input data: a feature map with shape (batch size = 1, number of channels = 3, height = 32, width = 32); Target weight data: quantized 4-bit convolutional kernel weights with shape (output channel number = 16, input channel number = 3, kernel size = 3x3). A convolution operation is performed on the input data and weights, calculating the feature map for each output channel. The convolution result is then biased and passed through the ReLU activation function to obtain an output feature map with shape (1, 16, 30, 30) (assuming no padding and a stride of 1). This output feature map is used as the input data for the next layer of the network for further computation.

[0100] As can be seen, the forward computation in this embodiment ensures that information is passed from the input layer to the output layer layer by layer. Furthermore, by quantizing the weights, the target weight data not only reduces computational resource consumption and accelerates inference speed, but also maintains model accuracy as much as possible. Moreover, each layer can independently complete computation and output results, facilitating dynamic adjustment of the quantization strategy and model structure. In addition, this process is applicable to various network layer types such as convolutional layers, fully connected layers, and attention layers, supporting diverse network structures.

[0101] In one embodiment, after performing forward computation of the current network layer based on the target weight data and the current input data to generate intermediate output data, the method further includes: removing the target weight component corresponding to the current network layer, wherein the target weight component is a preset low importance weight component.

[0102] Among them, the preset low importance weight components refer to weight components that are predefined or determined by a certain algorithm to have a small impact on network performance, and these components contribute a limited amount to the final output. The preset low importance weight components are the least significant bit (LSB) plane. For example, in a 5-bit weight, the 5th bit plane (counting from the MSB) is the low importance component.

[0103] Specifically, removal refers to removing or migrating unwanted weight data from high-speed memory (GPU memory) to free up video memory space. Removal is also often referred to as "unloading" or "migration".

[0104] For example, suppose the current layer uses 5-bit weights (composed of the MSB, the 2nd, 3rd, 4th, and 5th bit planes). After computation, the system identifies the 5th bit plane as a low-importance component and removes it from GPU memory. At this point, only the MSB, the 2nd, 3rd, and 4th bit planes may remain in GPU memory.

[0105] Specifically, the current network layer has completed its calculations using the target weight data and the current input data, and has output intermediate output data. At this point, the computational task of this layer is complete, and the weight data it used is no longer needed. The system knows which bit planes the target weight data consists of (e.g., MSB, bits 2, 3, 4, and 5). According to preset rules, the system identifies the low-importance components (e.g., bit 5) and performs a removal operation, removing these low-importance components from a preset storage area (GPU memory).

[0106] Optionally, for the removed low-importance weight components, the system can choose to migrate them back to main memory (CPU memory) or mark them as reclaimable in GPU memory, waiting for subsequent unified release.

[0107] If the system chooses to migrate back, it will transfer these bit planes from GPU memory back to the weight pool in CPU memory via the data bus. If the system chooses to release, it will directly release the video memory they occupy on the GPU.

[0108] As can be seen, removing low-importance weight components in this embodiment can improve computational efficiency, reduce invalid computations, and thus save computational resources and time; at the same time, it simplifies weight data, reduces storage requirements, and is suitable for resource-constrained devices; and by accurately identifying low-importance components, it minimizes the degradation of model performance and achieves efficient pruning.

[0109] S40. Process each network layer in the preset inference model in sequence. When the last network layer in the preset inference model is reached, the target output data is obtained.

[0110] The last network layer refers to the final computation layer in the model structure, such as the output layer (e.g., the Softmax layer) in classification tasks.

[0111] The target output data is the final result generated by the model after calculations across all layers. For example, for a question-answering task, it might be the probability distribution of the answers; for a text generation task, it might be the predicted probability of the next word.

[0112] Specifically, during neural network inference, the pre-defined inference model consists of multiple network layers sequentially. Each network layer receives input data, combines it with its internal weight parameters, performs calculations, and generates output data, which then serves as the input for the next layer. This process begins with the first layer of the model and proceeds layer by layer until the last network layer in the model has been processed.

[0113] In conjunction with S10-S40, the implementation process of this embodiment can be one of the following descriptions, which are only examples: A1. Preprocess the initial input data to obtain the feature vector corresponding to the initial input data; A2. Based on the feature vector, calculate the first importance score corresponding to each candidate bit width in the first network layer; A3. Based on the preset temperature parameters and preset functions, perform probability transformation on the first importance score corresponding to each candidate bit width to obtain the first probability distribution corresponding to the first network layer. The first probability distribution includes the probability value corresponding to each candidate bit width. A4. Filter the first probability distribution to determine the candidate bit width corresponding to the largest probability value as the target bit width, and determine the target bit width as the weight quantization accuracy required by the first network layer. A5. Based on the required weight quantization precision of the first network layer, determine multiple weight components corresponding to the required weight quantization precision of the first network layer. A6. If all of the multiple weight components already exist in the preset storage area of ​​the first network layer, combine the multiple weight components to generate the target weight data corresponding to the first network layer. A7. Based on the target weight data and the initial input data, perform the forward computation of the first network layer to generate intermediate output data, and use the intermediate output data as the input data of the second network layer. A8. Based on the intermediate output data in A7, calculate the second importance score for each candidate bit width in the second network layer. A9. Based on the preset temperature parameters and preset functions, perform probability transformation on the second importance score corresponding to each candidate bit width to obtain the second probability distribution corresponding to the second network layer. The second probability distribution includes the probability value corresponding to each candidate bit width. A10. Filter in the second probability distribution to determine the candidate bit width corresponding to the largest probability value as the target bit width, and determine the target bit width as the weight quantization accuracy required for the second network layer. A11. Based on the required weight quantization precision of the second network layer, determine the multiple weight components corresponding to the required weight quantization precision of the second network layer. A12. If not all of the multiple weight components are present in the preset storage area of ​​the second network layer, the missing weight components are retrieved from the preset weight database and added to the preset storage area, and all weight components are combined to generate the target weight data corresponding to the second network layer. A13. Based on the target weight data in A12 and the intermediate input data in A7, perform the forward computation of the second network layer to generate intermediate output data, and use the intermediate output data as the input data of the third network layer. Repeat steps A8-A13 to perform dynamic precision decision-making, on-demand loading, and computation execution on each layer of the model. When the last network layer in the preset inference model is reached, the target output data corresponding to the initial data is output.

[0114] This embodiment dynamically determines the required weight quantization precision for each network layer in the preset inference model based on the current input data. The system can achieve adaptive precision allocation for different inputs, thereby reducing unnecessary computational resource consumption while ensuring model inference accuracy. Based on the determined weight quantization precision, the corresponding target weight data is accurately obtained, making the forward computation of each network layer more efficient and accurate. Furthermore, by processing and passing intermediate output data layer by layer, the continuity and accuracy of the model inference process are ensured, ultimately obtaining high-quality target output data. Therefore, this solution effectively improves inference efficiency, optimizes computational resource utilization, and maintains the model's inference performance.

[0115] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0116] As another aspect of the embodiments of this application, this application provides a large-scale model inference apparatus. The large-scale model inference apparatus can be a software module, which includes several instructions stored in a memory. A processor can access the memory, invoke the instructions for execution, and complete the large-scale model inference methods described in the various embodiments above.

[0117] See Figure 5 , Figure 5 This is a schematic diagram of the structure of a large-scale model inference device provided in an embodiment of this application. Figure 5 As shown, the inference device 500 of this large model includes: The determining unit 501 is used to dynamically determine the weight quantization accuracy required for the current network layer in the preset inference model based on the current input data. The determining unit 501 is further configured to obtain the target weight data corresponding to the current network layer based on the weight quantization accuracy required by the current network layer. Processing unit 502 is configured to perform forward computation of the current network layer according to the target weight data and the current input data, generate intermediate output data, and use the intermediate output data as input data for the next network layer; The processing unit 502 is also used to process each network layer in the preset inference model in sequence, and when it reaches the last network layer in the preset inference model, it obtains the target output data.

[0118] This embodiment dynamically determines the required weight quantization precision for each network layer in the preset inference model based on the current input data. The system can achieve adaptive precision allocation for different inputs, thereby reducing unnecessary computational resource consumption while ensuring model inference accuracy. Based on the determined weight quantization precision, the corresponding target weight data is accurately obtained, making the forward computation of each network layer more efficient and accurate. Furthermore, by processing and passing intermediate output data layer by layer, the continuity and accuracy of the model inference process are ensured, ultimately obtaining high-quality target output data. Therefore, this solution effectively improves inference efficiency, optimizes computational resource utilization, and maintains the model's inference performance.

[0119] It should be noted that the aforementioned large-scale model inference apparatus can execute the large-scale model inference method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the large-scale model inference apparatus can be found in the large-scale model inference method provided in the embodiments of this application.

[0120] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the large model inference method as described in the foregoing embodiments.

[0121] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0122] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A reasoning method for large models, characterized in that, The method includes: Based on the current input data, dynamically determine the required weight quantization precision for the current network layer in the preset inference model; Based on the required weight quantization precision of the current network layer, the target weight data corresponding to the current network layer is obtained; Based on the target weight data and the current input data, perform the forward computation of the current network layer to generate intermediate output data, and use the intermediate output data as the input data of the next network layer; Each network layer in the preset inference model is processed sequentially, and the target output data is obtained when the last network layer in the preset inference model is reached.

2. The method according to claim 1, characterized in that, The current input data is the initial input data, and the current network layer is the first network layer; The step of dynamically determining the required weight quantization precision for the current network layer in the preset inference model based on the current input data includes: The initial input data is preprocessed to obtain the feature vector corresponding to the initial input data; Based on the feature vector, the first importance score corresponding to each candidate bit width in the first network layer is calculated. Based on preset temperature parameters and preset functions, the first importance score corresponding to each candidate bit width is probabilistically transformed to obtain the first probability distribution corresponding to the first network layer. The first probability distribution includes the probability value corresponding to each candidate bit width. Based on the first probability distribution, determine the required weight quantization precision for the first network layer.

3. The method according to claim 2, characterized in that, The step of determining the required weight quantization precision for the first network layer based on the first probability distribution includes: The candidate bit width corresponding to the highest probability value is determined as the target bit width by filtering within the first probability distribution. The target bit width is determined to be the weight quantization precision required by the first network layer; or, Random sampling is performed on the first probability distribution to obtain the sampling results; The sampling result is determined to be the weight quantization accuracy required for the first network layer.

4. The method according to claim 1, characterized in that, The current input data is the intermediate output data, and the current network layer is a network layer other than the first network layer; The step of dynamically determining the required weight quantization precision for the current network layer in the preset inference model based on the current input data includes: Based on the intermediate output data, the second importance score corresponding to each candidate bit width in the current network layer is calculated. Based on preset temperature parameters and preset functions, the second importance score corresponding to each candidate bit width is probabilistically transformed to obtain the second probability distribution corresponding to the current network layer. The second probability distribution includes the probability value corresponding to each candidate bit width. Based on the second probability distribution, the required weight quantization precision for the current network layer is determined.

5. The method according to claim 4, characterized in that, The step of determining the required weight quantization precision for the current network layer based on the second probability distribution includes: The candidate bit width corresponding to the highest probability value is determined as the target bit width by filtering in the second probability distribution. The target bit width is determined to be the weight quantization precision required by the current network layer; or, Random sampling is performed on the second probability distribution to obtain the sampling results; The sampling result is determined to be the weight quantization precision required for the current network layer.

6. The method according to claim 1, characterized in that, The step of obtaining the target weight data corresponding to the current network layer according to the weight quantization precision required by the current network layer includes: Based on the required weight quantization precision of the current network layer, determine multiple weight components corresponding to the required weight quantization precision of the current network layer. If all the multiple weight components already exist in the preset storage area of ​​the current network layer, combine the multiple weight components to generate the target weight data corresponding to the current network layer; or... If not all of the multiple weight components exist in the preset storage area of ​​the current network layer, the missing weight components are retrieved from the preset weight database and added to the preset storage area. All weight components are then combined to generate the target weight data corresponding to the current network layer.

7. The method according to claim 6, characterized in that, The preset weight database includes all weight parameters of the preset inference model. The weight parameters are stored in the form of multiple bit planes, each bit plane being a weight component. Each bit plane is a binary matrix, and multiple bit planes can be combined to generate weight data with different bit widths.

8. The method according to claim 1, characterized in that, After performing forward computation of the current network layer based on the target weight data and the current input data to generate intermediate output data, the method further includes: Remove the target weight component corresponding to the current network layer. The target weight component is a preset low-importance weight component.

9. A large-scale model reasoning system, characterized in that, The system includes an electronic device comprising a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, wherein, when executing the one or more computer programs, the processor causes the large model inference system to implement the large model inference method as described in any one of claims 1-8.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform the reasoning method for the large model as described in any one of claims 1-8.

Images