Method for determining reasoning result through large language model and electronic equipment

Abstract

The invention discloses a method for determining a reasoning result through a big language model and electronic equipment, and relates to the technical field of computers, the big language model performs forward calculation on cue words and divides the cue words into a plurality of sub-models to determine part of reasoning results, and each sub-model is responsible for calculating a sub-reasoning result of a corresponding target digit; and finally, a final inference result is obtained by collecting sub-inference results of each sub-model, inference tasks with different calculated amounts can be allocated according to different performances of different computing devices on the premise that the model performance is not reduced, all the computing devices are effectively utilized, and a use threshold is reduced for application of a large language model. Meanwhile, as the reasoning calculation task of each sub-model is independent, the sub-model does not need to communicate with other sub-models before final calculation is completed, the model communication time is saved, and the reasoning efficiency is greatly improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and electronic device for determining inference results using a large language model. Background Technology

[0002] Training and inference of large language models rely on powerful computing power. If the inference computation of large language models adopts a variety of parallel computing strategies such as data parallelism, model parallelism and pipelined parallelism, the model computation process can be distributed to multiple computing devices. Data parallelism copies the model to multiple cards and synchronizes gradients after processing different data. Model parallelism splits the model into layers or tensors and distributes it to multiple cards to reduce GPU memory and coordinate computation. Pipeline parallelism executes the model in a segmented pipeline according to layers to improve throughput and reduce peak GPU memory usage. However, all these parallel solutions require communication between the various computing devices during the computation process, which requires a lot of communication and synchronization time. Summary of the Invention

[0003] This application provides a method and electronic device for determining inference results using a large language model. While maintaining the original model structure, the large language model is divided into multiple sub-models to calculate a portion of the final result. This solves the problem in related technologies where parallel schemes require communication between various computing devices during the calculation process, resulting in a large amount of communication and synchronization time.

[0004] This application provides a method for determining inference results using a large language model, including: In response to the large language model performing forward computation on the prompt words, the original model weights of the large language model are obtained, the tensor data format of the inference results output by the large language model is identified, and the tensor bit width of the tensor data format is determined. Multiple sub-models are used to determine some inference results, and the target bit lengths corresponding to the multiple sub-models are allocated according to the order of the tensor bit lengths; The scaling factor of each of the sub-models is determined according to the target bit depth of each sub-model, and the scaling factors of the sub-models are arranged in order of the tensor bit depth. The original model weights are adjusted according to the scaling factor of the multiple sub-models to obtain the weights of each sub-model. The weights of each sub-model are used to replace the original model weights of the large language model to determine each sub-model, and the sub-inference results of the target number of bits corresponding to each sub-model are obtained. The inference result output by the large language model is determined based on the sub-inference results corresponding to the target bit length of multiple sub-models and the order of the tensor bit length.

[0005] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described methods for determining inference results using a large language model. In response to the large language model performing forward computation on the prompt words, the original model weights of the large language model are obtained, the tensor data format of the inference results output by the large language model is identified, and the tensor bit width of the tensor data format is determined. Multiple sub-models are used to determine some inference results, and the target bit lengths corresponding to the multiple sub-models are allocated according to the order of the tensor bit lengths; The scaling factor of each of the sub-models is determined according to the target bit depth of each sub-model, and the scaling factors of the sub-models are arranged in order of the tensor bit depth. The original model weights are adjusted according to the scaling factor of the multiple sub-models to obtain the weights of each sub-model. The weights of each sub-model are used to replace the original model weights of the large language model to determine each sub-model, and the sub-inference results of the target number of bits corresponding to each sub-model are obtained. The inference result output by the large language model is determined based on the sub-inference results corresponding to the target bit length of multiple sub-models and the order of the tensor bit length.

[0006] This application divides the forward computation of prompt words in a large language model into multiple sub-models, each determining a portion of the inference result. Each sub-model is responsible for calculating the sub-inference result corresponding to the target number of digits. Finally, the final inference result is obtained by collecting the sub-inference results from each sub-model. Without compromising model performance, inference tasks with varying computational loads can be allocated based on the different performance levels of various computing devices, effectively utilizing all computing resources and lowering the barrier to entry for large language models. Furthermore, since each sub-model's inference computation task is independent, there is no need to communicate with other sub-models before the final computation is completed, saving model communication time and significantly improving inference efficiency. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1 This is an application environment diagram of the method for determining inference results using a large language model in one embodiment of this application; Figure 2This is a schematic diagram of the reasoning process in which the large language model uses the original model weights to perform forward calculation on the prompt words in one embodiment of this application; Figure 3 This is a schematic diagram of the inference process of performing forward computation of prompt words by a large language model performing different scaling and quantization operations in one embodiment of this application; Figure 4 This is a flowchart illustrating a method for determining inference results using a large language model in one embodiment of this application; Figure 5 This is a diagram illustrating the principle of splitting the total number of digits of the original model weights into sub-model weights in one embodiment of this application; Figure 6 This is a structural block diagram of a device for determining inference results using a large language model in one embodiment of this application; Figure 7 This is an internal structural diagram of a computer device in one embodiment of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0010] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0011] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0012] With the rapid development of large language model technology, it has been widely applied in various industries, profoundly promoting the intelligent transformation of various fields. In the field of natural language processing, large language models can now generate high-quality text, covering a variety of tasks such as news reporting, creative writing, and code generation. However, as the scale of the models continues to expand, they face severe challenges in terms of computing resources and hardware devices, significantly raising the application threshold.

[0013] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on large datasets. The core idea is to learn patterns and structures of natural language through large-scale unsupervised training, mimicking human language cognition and generation processes to some extent.

[0014] Training and inference of large language models rely on powerful computing capabilities. For example, training a model with billions of parameters typically requires a large number of high-performance GPUs or dedicated accelerators (such as TPUs). These devices are not only expensive but also have stringent requirements for power and cooling systems, placing a heavy burden on most enterprises and research institutions in terms of procurement and maintenance. At the same time, the ever-increasing scale of models is gradually making hardware performance a bottleneck, necessitating continuous upgrades to meet computing demands.

[0015] Computational efficiency is another key challenge. Training large language models often takes days or even weeks, consuming significant computational resources and extending development and deployment cycles. To improve efficiency, researchers have tried various parallel computing strategies, such as data parallelism, model parallelism, and pipelined parallelism, but their implementation and optimization are highly complex, requiring sophisticated scheduling and communication mechanisms. Furthermore, performance differences between hardware platforms pose further challenges to efficiency improvements.

[0016] Parallelism schemes in related technologies, such as tensor parallelism, pipelined parallelism, expert parallelism, data parallelism, and sequence parallelism, involve distributing the model computation process across multiple computing devices. Data parallelism replicates the model across multiple GPUs, synchronizing gradients after processing different data. Model parallelism splits the model across multiple GPUs by layer or tensor, reducing GPU memory and enabling collaborative computation. Pipeline parallelism executes the model in a segmented pipeline by layer, improving throughput and reducing peak GPU memory usage. These strategies are often used in combination to fully utilize hardware resources and improve the training efficiency and scalability of large models. All of these parallelism schemes require communication between the various computing devices during computation, resulting in significant time spent on communication and synchronization.

[0017] One related technology involves directly quantizing model weights: Model quantization is a technique for optimizing deep learning models. It reduces model storage space and computation by converting model parameters from high precision (e.g., 32-bit floating-point numbers) to low precision (e.g., 8-bit integers). The quantization process includes post-training quantization, quantization-aware training, and dynamic quantization. Post-training quantization directly quantizes the pre-trained model, which may result in a loss of accuracy.

[0018] The method for determining inference results using large language models provided in this application can be applied to, for example... Figure 1In the application environment shown, terminal 102 and server 104 communicate via a network. Terminal 102 inputs prompt words to server 104, and server 104 executes a large language model to perform forward computation on the prompt words to obtain the inference result. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0019] like Figure 2 As shown, the large language model uses the original model weights to perform forward computation on the prompt words to obtain the sub-inference results (the computation results in the figure), which are the inference logits. The logits are tensors, and the dimension of a tensor is equal to its order. A 0-dimensional tensor is a scalar, a 1-dimensional tensor is a vector, a 2-dimensional tensor is a matrix, and 3 dimensions and above are tensors.

[0020] This application provides a method for determining inference results using large language models. This multi-device collaborative inference method distributes the computational load across multiple devices while maintaining model accuracy, significantly reducing single-device memory requirements, decreasing reliance on high-end hardware, and improving overall computational speed. This method effectively lowers the application threshold for large-scale models, providing technical support for wider deployment and use. The main function of this method is that a single computing device only calculates a portion of the final result, and the final forward inference result is obtained by weighted summation of multiple sub-inference results.

[0021] Traditional compression methods (such as quantization, pruning, or distillation) can reduce the memory consumption of model weights, but they often lead to a decrease in expressive power, which in turn affects inference performance. To address this bottleneck, this application divides the large model into multiple sub-models while maintaining the original model structure. Through collaborative inference, the overall accuracy is maintained, significantly reducing the memory requirements of a single device.

[0022] To further enhance the flexibility and scalability of the method, this application also proposes a learnable weighted fusion mechanism that adaptively fuses the outputs of different sub-models. This ensures the accuracy and stability of the final inference result while being compatible with multiple compression strategies. This method allows users to freely choose compression methods based on actual resource constraints, providing a new technical path for the efficient deployment of large language models.

[0023] like Figure 3 As shown, after the large language model is trained, in order to fully utilize existing inference computing resources and improve inference efficiency, we specify the number of sub-models based on the specific computing resources available. Let's assume we set the number of sub-models to n, which is consistent with the number of computing devices used for subsequent inference. Next, we copy the model weights n times, each copy being a replica of the original model weights.

[0024] For these n original model weights, we will perform different scaling and quantization operations on them respectively. In this way, we can obtain multiple sub-weights, each used to calculate a different number of bits in the inference logits (tensor). Although the model structure of each sub-weight is exactly the same as the original model structure, they are only responsible for calculating a specific part of the final logits. This means that although the model architecture remains unchanged, each sub-weight only undertakes a portion of the computational task during inference, thus achieving parallel computing and efficient resource utilization.

[0025] Since each sub-weight is only responsible for calculating a portion of the result, the required data format (enhancing diversity) can be more flexible. Specifically, the data format can be changed from the original 32-bit floating-point number (FP32) to less space-consuming 16-bit floating-point number (FP16), 8-bit integer (INT8), or even 4-bit integer (INT4). This quantization operation not only reduces the memory space required by the model during inference but also significantly improves inference speed, especially on resource-constrained devices such as edge devices or mobile devices. In this way, we can maximize the use of limited computing resources while maintaining model performance, achieving efficient and fast inference computation, such as... Figure 2 As shown.

[0026] After obtaining multiple sub-models, joint fine-tuning is required to ensure that these sub-models maintain the accuracy of the overall model when working together. This process involves finely adjusting the weights of each sub-model to ensure that they accurately output the expected results when working together. Unlike the traditional single-model calculation process, each sub-model processed by this invention is only responsible for calculating a portion of the final logits. This means that to obtain the final logits tensor, the outputs of multiple sub-models need to be multiplied by their corresponding weighted values, and then these weighted results are summed, such as... Figure 3 As shown.

[0027] Figure 4 As shown, embodiments of this application provide a method for determining inference results using a large language model, comprising the following steps: Step S1: In response to the large language model performing forward computation on the prompt words, obtain the original model weights of the large language model, identify the tensor data format of the reasoning results output by the large language model, and determine the tensor bit width of the tensor data format. Step S2: Use multiple sub-models to determine part of the inference results, and allocate the target bit length of the multiple sub-models according to the order of tensor bit length; Step S3: Determine the scaling factor of multiple sub-models according to the target bit depth of each sub-model, and arrange the scaling factors of multiple sub-models in order of tensor bit depth. Step S4: Adjust the original model weights according to the scaling factor of multiple sub-models to obtain the weights of each sub-model. Replace the original model weights of the large language model with the weights of each sub-model to determine each sub-model and obtain the sub-inference results of the target bit length corresponding to each sub-model. Step S5: Determine the inference result output by the large language model based on the sub-inference results of the target bit length corresponding to multiple sub-models and the order of tensor bit length.

[0028] This approach divides the forward computation of prompt words in the large language model into multiple sub-models, each determining a portion of the inference result. Each sub-model is responsible for calculating the sub-inference result corresponding to the target number of digits. Finally, the final inference result is obtained by collecting the sub-inference results from each sub-model. Without compromising model performance, this method allocates inference tasks with varying computational loads based on the different performance levels of various computing devices, effectively utilizing all computing resources and lowering the barrier to entry for large language models. Furthermore, since each sub-model's inference computation task is independent, there is no need to communicate with other sub-models before the final computation is completed, saving model communication time and significantly improving inference efficiency.

[0029] In this embodiment, identifying the tensor data format of the inference results output by the large language model and determining the tensor bit width of the tensor data format includes: Obtain the inference results of the large language model's forward computation of prompt words, identify the tensor data format of the inference results, and determine the number of tensor bits included in the tensor data format; For tensor data formats, the number of bits in multiple tensors is labeled sequentially, with each bit corresponding to one tensor bit.

[0030] In this process, by analyzing the tensor format of the inference results of the large language model and obtaining the number of tensor bits contained therein, and then labeling these bits in order, the system can clearly grasp the numerical range and weight position corresponding to each bit. This provides an accurate basis for allocating target bits to the sub-model, ensuring that subsequent scaling, inference and recombination can accurately correspond to the tensor bit order, avoiding numerical errors or displacement errors, and improving the correctness and stability of compressed inference.

[0031] In this embodiment, multiple sub-models are used to determine partial inference results, and the target bit depth corresponding to the multiple sub-models is allocated according to the order of tensor bit depth, including: Get the number of sub-models; Divide the tensor bit length into a number of sub-model bit arrays according to the number of consecutive bit numbers. Associate each bit array with a sub-model, and the bit array associated with each sub-model is used as its corresponding target bit length.

[0032] Specifically, by dividing the tensor bit depth into bit arrays equal to the number of sub-models using consecutive bit numbers, each sub-model has a clearly defined bit computation range, ensuring that the inference tasks of each sub-model do not overlap and that the computational load is evenly distributed bit by bit. This not only ensures clear computational logic but also reduces cross-bit interference and redundant computation, improves the efficiency of multi-sub-model collaborative inference, and lays the foundation for constructing a residual progression method.

[0033] Specifically, the tensor bit depth is divided into sub-model number arrays according to the number of consecutive bit positions, including: Get the number of sub-models n, and get the total bit index m of the tensor bit; Calculate the average value of the total number of tensor bits m divided by the number of sub-models n, and determine whether the average value is an integer. If the average value is an integer, then the tensor bit length is divided into n bit arrays according to the method of consecutive bit numbers, and each bit array contains m / n consecutive bit numbers; If the average value is not an integer, the bit number contained in each bit array is randomly assigned, and the corresponding bit array is matched according to the computational performance of the sub-model. The comprehensive score of the bit array is obtained by assigning the bit array to multiple sub-models, and the bit array of the highest comprehensive score is selected to assign bit arrays to multiple sub-models.

[0034] If the average value is not an integer, the bit arrays for multiple sub-models are formed by rounding m / n down and up, using consecutive bit numbers of the rounded values.

[0035] In this embodiment, the scaling factor of multiple sub-models is determined according to the target bit depth of each sub-model, and the scaling factors of the multiple sub-models are arranged in order of tensor bit depth, including: Identify the decimal point in the weights of the original model, and determine the number of digits before and after the decimal point. Obtain the first target number of digits of the sub-inference result used to calculate the target number of digits corresponding to the first sub-model in the original model weights. With the first target number of digits ... Obtain the second target numerical digits of the sub-inference result used to calculate the target digits corresponding to the second sub-model in the original model weights. After excluding the first target numerical digits from the original model weights, take the second target numerical digits as the target and determine the second number of decimal places to move the decimal point forward or backward. Determine the scaling factor corresponding to the second sub-model based on the second number of decimal places. Continue in this manner until the scaling factors of multiple sub-models are determined. The scaling factors of multiple sub-models are obtained in the order of tensor bit depth, and then arranged sequentially according to the order in which the scaling factors of multiple sub-models are obtained.

[0036] Specifically, by detecting the change in the number of decimal places before and after the decimal point in the weights of the original model, and calculating the displacement required to move the decimal point based on the target number of decimal places, the scaling factor of each sub-model is determined, enabling each sub-model to perform inference calculations with the corresponding bit width precision. Arranging the scaling factors in bit order ensures that the sub-model calculations are consistent with the tensor bit position, thereby achieving layered calculations based on precision, reducing floating-point calculation errors, and making the overall inference accuracy of the combined multi-sub-models closer to the original model.

[0037] In this embodiment, the original model weights are adjusted according to the scaling factors of multiple sub-models to obtain the weights of each sub-model. The weights of each sub-model replace the original model weights of the large language model to determine each sub-model. The sub-inference results corresponding to the target bit depth for each sub-model are then obtained, including: The weights of the first sub-model are obtained based on the weights of the original model and the first scaling factor. The first sub-model is determined based on the weights of the first sub-model, and the sub-inference result of the target number of bits corresponding to the first sub-model is obtained. The first difference is obtained by subtracting the product of the first sub-model weight and the first scaling factor from the original model weight. The second sub-model weight is obtained based on the first difference and the second scaling factor. The second sub-model is determined based on the second sub-model weight, and the sub-inference result of the target number of bits corresponding to the second sub-model is obtained. This process continues until the sub-inference results for the target number of bits corresponding to all sub-models are obtained.

[0038] In this embodiment, the weights of the first sub-model are obtained based on the original model weights and the first scaling factor. The first sub-model is then determined based on these weights, and the sub-inference result corresponding to the target bit depth of the first sub-model is obtained, including: Obtain the number of sub-models preset for calculating the inference results, and copy the weights of the original model multiple times based on the number of sub-models; Obtain the weight values ​​of multiple parameters in the large language model included in the original model weights; The weights of the first sub-model are obtained by dividing the weights of multiple parameters in a copy of the original model by the first scaling factor. The original model weights of the large language model are replaced with the weights of the first sub-model to form the first sub-model; The first sub-model performs forward computation on the prompt words to obtain the sub-inference result corresponding to the target number of words in the first sub-model.

[0039] Specifically, by copying the weights of the original model and scaling them by division according to the first scaling factor, a first sub-model is constructed. This sub-model can still simulate the calculation of some bits in the original model under the condition of reduced precision, and obtains the corresponding bit width partial results after forward inference. This operation lays the initial foundation for subsequent residual calculation and sub-model progressive inference, allowing the entire compressed inference process to start from high-bit or specific bit segments, which is beneficial for subsequent layer-by-layer compensation and accurate reconstruction.

[0040] In this embodiment, the first difference is obtained by subtracting the product of the first sub-model weight and the first scaling factor from the original model weights. The second sub-model weights are then obtained based on the first difference and the second scaling factor. The second sub-model is determined based on these weights, and the sub-inference result corresponding to the target bit depth of the second sub-model is obtained, including: The first difference is obtained by subtracting the product of the weights of the first sub-model and the scaling factor of the first sub-model from the weights of the original model copied from it. Divide the first difference by the scaling factor of the second sub-model to obtain the weight of the second sub-model; The original model weights of the large language model are replaced with the weights of the second sub-model to form the second sub-model; The second sub-model performs forward computation on the prompt words to obtain the sub-inference result corresponding to the target number of words in the second sub-model.

[0041] The scaling factor of each sub-model is equal to the reciprocal of the scaling factor of the respective sub-model. Let the weight of the original model be Q, and the weight of the i-th sub-model be Qi. i The scaling factor corresponding to the weight of the i-th sub-model is α. i Scaling factor α i It is equal to the reciprocal of the scaling factor of the i-th sub-model; the scaling factor corresponding to the weight of the (i+1)-th sub-model is α. i+1 If the total sub-weights of t sub-models have been obtained The weights of the remaining sub-models are then... .

[0042] Specifically, a first difference is obtained by subtracting the inverse-scaled weights of the first sub-model from the original weights. This difference is then used to construct a second sub-model at a second scaling factor, compensating for bit segments or precision areas not covered by the first sub-model. This residual progression method allows each sub-model to progressively approximate the complete expression of the original weights in a hierarchical manner. Thus, even with limited precision in each sub-model, complete inference accuracy can still be reconstructed segmentally, significantly improving the overall usability of the bit-by-bit split model.

[0043] In this embodiment, the inference result output by the large language model is determined based on the sub-inference results corresponding to the target bit length of multiple sub-models and the order of tensor bit length, including: Obtain the sub-inference result corresponding to the target number of bits for each sub-model; Determine the position of the target bit length in the tensor bit length for each sub-model, and determine the weighted value corresponding to the sub-inference result of the target bit length for each sub-model based on the corresponding position; The sub-inference results of the target number of multiple sub-models are multiplied by their corresponding weighted values, and then summed to obtain the final tensor. The final tensor is used as the inference result of the large language model for forward computation of the prompt words.

[0044] Specifically, by acquiring the bitwise inference results of all sub-models, assigning weighted values ​​according to the corresponding positions of the tensor bits, and then combining them in a weighted summation manner to obtain the final output, the inference results of different bits scattered in each sub-model can be strictly reconstructed in bit order. This process ensures that the reconstructed tensor is mathematically consistent with the original inference, achieving accurate inference restoration through "segmented calculation of sub-models - overall reconstruction," and significantly reducing the information loss caused by compressed inference.

[0045] Specifically, suppose we have n sub-models, each of which outputs a tensor as the sub-inference result, denoted as M. i (i=1,2,...,n). Each sub-model outputs a sub-inference result with a corresponding weighted value β. i These weights are the same as the scaling factors for each sub-model, and the final tensor M can be obtained using the following formula: .

[0046] In this embodiment, the method further includes: Obtain the total loss value of the large language model; The total loss value is allocated to each sub-model based on the weighted value of each sub-model; Each sub-model performs backpropagation calculations based on its assigned loss value to obtain the corresponding gradient and update the sub-model's parameters.

[0047] This process ensures that each sub-model is optimized for the part of bits it is responsible for, thereby maintaining consistency and accuracy in collaborative work throughout the model.

[0048] This mechanism involves obtaining the total loss value of the original large model and allocating the loss according to the contribution weight of each sub-model in the final inference tensor. This allows each sub-model to perform independent backpropagation training based on its assigned bit segment, thus preventing individual sub-models from failing to update effectively due to unclear responsibilities. This ensures that all sub-models collaboratively optimize during inference accuracy reconstruction, enabling the compressed running model to continuously optimize towards overall accuracy consistency.

[0049] In this embodiment, allocating the total loss value to each sub-model based on the weighted value of each sub-model includes: Let the total loss of the large language model be L, and the sub-loss of the i-th sub-model be L. i The scaling factor corresponding to the sub-loss value of the i-th sub-model is γ. i ; The scaling factor γ corresponding to the sub-loss value of the (i+1)th sub-model is obtained. i+1 If the total sub-loss value of t sub-models has been obtained The sub-loss value of the remaining sub-model is .

[0050] Specifically, by using the reciprocal of the scaling factor of the sub-model as the scaling coefficient, and calculating the loss value of the remaining sub-models based on the sub-model index and the sum of existing losses, the loss allocation process is made computable and mathematically closed. This method ensures that the loss partitioning among sub-models conforms to the scaling structure logic while avoiding imbalance in loss allocation, improving the stability and consistency of training, and mathematically guaranteeing the reliable training effect of multi-sub-model compressed inference.

[0051] The original model weights, after being processed through a scaling quantization mechanism, still require fine-tuning to ensure inference accuracy. Unlike traditional post-quantization fine-tuning, this application scales and distributes the loss values ​​obtained during fine-tuning, similar to weight scaling. Each sub-model only needs to obtain its own loss value (based on its number of bits), then performs reverse calculations to update its parameter values. During post-quantization fine-tuning, multiple sub-models are jointly fine-tuned. This distribution method processes the jointly calculated loss values ​​according to a scaling mechanism and then distributes them to different sub-models to update their parameters.

[0052] The core function of the scaling quantization mechanism in this application is to enable the processed weights to correctly calculate the logits result with a specified number of bits. For example... Figure 5 As shown, the specific processing flow of the method for determining inference results using a large language model is as follows: Calculate the number of digits in the upper and lower limits of the original model weights, and then copy the original model weights n times according to the required number of model segments. Select a copy of the weights, and according to the number of logits to be calculated, divide each part of the weights by the scaling factor (e.g., the original model weights have 7 decimal places and 2 decimal places, and the sub-model weights to be processed first are used to calculate the first 3 decimal places, so the scaling factor is 10000.0). Truncate the processed weights, convert them to a low-precision format, and save them as the weights of sub-model 1. Details of weight truncation and low-precision conversion: When performing weight truncation, since a sub-model is only responsible for calculating a few digits, the quantization of the sub-model needs to prioritize precision over representation length. If the fp8 precision format is used, 8-bit floating-point numbers, suitable for scenarios with high precision requirements for decimal numbers, are used to more accurately represent small values. During the conversion process, values ​​exceeding the retention range are truncated, and the weight values ​​are mapped to the corresponding numerical range according to the selected precision format, completing the low-precision conversion. For the weights of sub-model 2, the original model weights need to be subtracted from the product of the weights of sub-model 1 and the scaling factor of sub-model 1. Then, the difference is divided by the scaling factor of the computational bit depth of sub-model 2 (e.g., 10.0). The processed weights are truncated, converted to a low-precision format, and saved as the weights of sub-model 2. The weights of sub-model 3 need to be obtained by subtracting the product of the weights of sub-model 1 and the scaling factor of sub-model 1, and the weights of sub-model 2 and the scaling factor of sub-model 2 from the original model weights in sequence. Then, the lookup value is divided by the scaling factor corresponding to sub-model 3. The processed weights are truncated, converted to a low-precision format, and saved as the weights of sub-model 2. This process continues until all sub-models have been processed.

[0053] like Figure 5 As shown, if the original model weights are [1234567.89, 2345678.91, 3456789.1, 67891.23], then the total number of digits in the original model weights is [1234567.89, 2345678.91, 3456789.1, 0067891.23]. The scaling factor of sub-model 1 weights is 1000, which means that the number of digits in the original model weights selected by sub-model 1 weights is [123, 234, 345, 006]. The scaling factor of sub-model 2 weights is 10, which means that the number of digits in the original model weights selected by sub-model 1 weights is [456, 567, 678, 789]. The scaling factor of sub-model 3 weights is 0.01, which means that the number of digits in the original model weights selected by sub-model 3 weights is [789, 891, 910, 123].

[0054] In the aforementioned method for determining inference results using a large language model, the forward computation of the prompt words by the large language model is divided into multiple sub-models, each determining a portion of the inference result. Each sub-model is responsible for calculating the sub-inference result corresponding to the target number of digits. Finally, the final inference result is obtained by collecting the sub-inference results from each sub-model. Without reducing model performance, inference tasks with varying computational loads can be allocated according to the different performance levels of various computing devices, effectively utilizing all computing resources and lowering the barrier to entry for large language models. Furthermore, since each sub-model's inference computation task is independent, there is no need to communicate with other sub-models before the final computation is completed, saving model communication time and significantly improving inference efficiency.

[0055] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0056] In one embodiment, such as Figure 6 As shown, a device 10 for determining inference results using a large language model is provided, including: a bit identification module 1, a bit allocation module 2, a scaling determination module 3, a sub-inference result acquisition module 4, and a result reconstruction module 5.

[0057] The bit-count recognition module 1 is used to respond to the large language model's forward computation of prompt words, obtain the original model weights of the large language model, identify the tensor data format of the inference results output by the large language model, and determine the tensor bit count of the tensor data format.

[0058] The bit allocation module 2 is used to determine part of the inference results using multiple sub-models, and allocate the target bit number corresponding to the multiple sub-models according to the order of tensor bit number.

[0059] The scaling determination module 3 is used to determine the scaling factor of multiple sub-models according to the target bit depth of the multiple sub-models, and arrange the scaling factors of the multiple sub-models in the order of tensor bit depth.

[0060] The sub-inference result acquisition module 4 is used to adjust the weights of the original model according to the scaling factor of multiple sub-models to obtain the weights of each sub-model, replace the original model weights of the large language model with the weights of each sub-model to determine each sub-model, and obtain the sub-inference result of the target number of bits corresponding to each sub-model.

[0061] The result reconstruction module 5 is used to determine the inference result output by the large language model based on the sub-inference results of the target bit length corresponding to multiple sub-models and the order of tensor bit length.

[0062] In this embodiment, identifying the tensor data format of the inference results output by the large language model and determining the tensor bit width of the tensor data format includes: Obtain the inference results of the large language model's forward computation of prompt words, identify the tensor data format of the inference results, and determine the number of tensor bits included in the tensor data format; For tensor data formats, the number of bits in multiple tensors is labeled sequentially, with each bit corresponding to one tensor bit.

[0063] In this embodiment, multiple sub-models are used to determine partial inference results, and the target bit depth corresponding to the multiple sub-models is allocated according to the order of tensor bit depth, including: Get the number of sub-models; Divide the tensor bit length into a number of sub-model bit arrays according to the number of consecutive bit numbers. Associate each bit array with a sub-model, and the bit array associated with each sub-model is used as its corresponding target bit length.

[0064] In this embodiment, the scaling factor of multiple sub-models is determined according to the target bit depth of each sub-model, and the scaling factors of the multiple sub-models are arranged in order of tensor bit depth, including: Identify the decimal point in the weights of the original model, and determine the number of digits before and after the decimal point. Obtain the first target number of digits of the sub-inference result used to calculate the target number of digits corresponding to the first sub-model in the original model weights. With the first target number of digits ... Obtain the second target numerical digits of the sub-inference result used to calculate the target digits corresponding to the second sub-model in the original model weights. After excluding the first target numerical digits from the original model weights, take the second target numerical digits as the target and determine the second number of decimal places to move the decimal point forward or backward. Determine the scaling factor corresponding to the second sub-model based on the second number of decimal places. Continue in this manner until the scaling factors of multiple sub-models are determined. The scaling factors of multiple sub-models are obtained in the order of tensor bit depth, and then arranged sequentially according to the order in which the scaling factors of multiple sub-models are obtained.

[0065] In this embodiment, the original model weights are adjusted according to the scaling factors of multiple sub-models to obtain the weights of each sub-model. The weights of each sub-model replace the original model weights of the large language model to determine each sub-model. The sub-inference results corresponding to the target bit depth for each sub-model are then obtained, including: The weights of the first sub-model are obtained based on the weights of the original model and the first scaling factor. The first sub-model is determined based on the weights of the first sub-model, and the sub-inference result of the target number of bits corresponding to the first sub-model is obtained. The first difference is obtained by subtracting the product of the first sub-model weight and the first scaling factor from the original model weight. The second sub-model weight is obtained based on the first difference and the second scaling factor. The second sub-model is determined based on the second sub-model weight, and the sub-inference result of the target number of bits corresponding to the second sub-model is obtained. This process continues until the sub-inference results for the target number of bits corresponding to all sub-models are obtained.

[0066] In this embodiment, the weights of the first sub-model are obtained based on the original model weights and the first scaling factor. The first sub-model is then determined based on these weights, and the sub-inference result corresponding to the target bit depth of the first sub-model is obtained, including: Obtain the number of sub-models preset for calculating the inference results, and copy the weights of the original model multiple times based on the number of sub-models; Obtain the weight values ​​of multiple parameters in the large language model included in the original model weights; The weights of the first sub-model are obtained by dividing the weights of multiple parameters in a copy of the original model by the first scaling factor. The original model weights of the large language model are replaced with the weights of the first sub-model to form the first sub-model; The first sub-model performs forward computation on the prompt words to obtain the sub-inference result corresponding to the target number of words in the first sub-model.

[0067] In this embodiment, the first difference is obtained by subtracting the product of the first sub-model weight and the first scaling factor from the original model weights. The second sub-model weights are then obtained based on the first difference and the second scaling factor. The second sub-model is determined based on these weights, and the sub-inference result corresponding to the target bit depth of the second sub-model is obtained, including: The first difference is obtained by subtracting the product of the weights of the first sub-model and the scaling factor of the first sub-model from the weights of the original model copied from it. Divide the first difference by the scaling factor of the second sub-model to obtain the weight of the second sub-model; The original model weights of the large language model are replaced with the weights of the second sub-model to form the second sub-model; The second sub-model performs forward computation on the prompt words to obtain the sub-inference result corresponding to the target number of words in the second sub-model.

[0068] In this embodiment, the inference result output by the large language model is determined based on the sub-inference results corresponding to the target bit length of multiple sub-models and the order of tensor bit length, including: Obtain the sub-inference result corresponding to the target number of bits for each sub-model; Determine the position of the target bit length in the tensor bit length for each sub-model, and determine the weighted value corresponding to the sub-inference result of the target bit length for each sub-model based on the corresponding position; The sub-inference results of the target number of multiple sub-models are multiplied by their corresponding weighted values, and then summed to obtain the final tensor. The final tensor is used as the inference result of the large language model for forward computation of the prompt words.

[0069] In this embodiment, as Figure 6 As shown, the large language model determining inference result device 10 also includes: a loss control module 6. The loss control module 6 is used for: Obtain the total loss value of the large language model; The total loss value is allocated to each sub-model based on the weighted value of each sub-model; Each sub-model performs backpropagation calculations based on its assigned loss value to obtain the corresponding gradient and update the sub-model's parameters.

[0070] In this embodiment, allocating the total loss value to each sub-model based on the weighted value of each sub-model includes: Let the total loss of the large language model be L, and the sub-loss of the i-th sub-model be L. i The scaling factor corresponding to the sub-loss value of the i-th sub-model is α. i The scaling factor is equal to the reciprocal of the scaling factor of the i-th sub-model; The scaling factor α corresponding to the sub-loss value of the (i+1)th sub-model is obtained. i+1 If the total sub-loss value of t sub-models has been obtained The sub-loss value of the remaining sub-model is .

[0071] In the aforementioned large language model-based inference result determination device, the large language model's forward computation of prompt words is divided into multiple sub-models, each determining a portion of the inference result. Each sub-model is responsible for calculating the sub-inference result corresponding to the target number of digits. Finally, the final inference result is obtained by collecting the sub-inference results from each sub-model. Without reducing model performance, inference tasks with varying computational loads can be allocated according to the different performance levels of various computing devices, effectively utilizing all computing resources and lowering the barrier to entry for large language models. Furthermore, since each sub-model's inference computation task is independent, there is no need to communicate with other sub-models before the final computation is completed, saving model communication time and significantly improving inference efficiency.

[0072] For a description of the features in the embodiment corresponding to the device for determining inference results using a large language model, please refer to the relevant descriptions in the embodiment corresponding to the method for determining inference results using a large language model; they will not be repeated here.

[0073] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the method for determining inference results using a large language model.

[0074] In one embodiment, the electronic device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores large language model determination data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for determining the results of large language model inference.

[0075] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the method for determining inference results using a large language model: In response to the large language model performing forward computation on prompt words, the original model weights of the large language model are obtained, the tensor data format of the inference results output by the large language model is identified, and the tensor bit width of the tensor data format is determined. Multiple sub-models are used to determine some inference results, and the target bit length of the multiple sub-models is allocated according to the order of tensor bit length. The scaling factor of each sub-model is determined based on the target bit depth of each sub-model, and the scaling factors of the sub-models are arranged in order of tensor bit depth. The original model weights are adjusted according to the scaling factor of multiple sub-models to obtain the weights of each sub-model. The weights of each sub-model are used to replace the original model weights of the large language model to determine each sub-model. The sub-inference results of the target bit length corresponding to each sub-model are obtained. The inference result output by the large language model is determined based on the sub-inference results of the target bit length corresponding to multiple sub-models and the order of tensor bit length.

[0076] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0077] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the method for determining inference results using a large language model: In response to the large language model performing forward computation on prompt words, the original model weights of the large language model are obtained, the tensor data format of the inference results output by the large language model is identified, and the tensor bit width of the tensor data format is determined. Multiple sub-models are used to determine some inference results, and the target bit length of the multiple sub-models is allocated according to the order of tensor bit length. The scaling factor of each sub-model is determined based on the target bit depth of each sub-model, and the scaling factors of the sub-models are arranged in order of tensor bit depth. The original model weights are adjusted according to the scaling factor of multiple sub-models to obtain the weights of each sub-model. The weights of each sub-model are used to replace the original model weights of the large language model to determine each sub-model. The sub-inference results of the target bit length corresponding to each sub-model are obtained. The inference result output by the large language model is determined based on the sub-inference results of the target bit length corresponding to multiple sub-models and the order of tensor bit length.

[0078] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps in any of the above embodiments of the method for determining inference results using a large language model: In response to the large language model performing forward computation on prompt words, the original model weights of the large language model are obtained, the tensor data format of the inference results output by the large language model is identified, and the tensor bit width of the tensor data format is determined. Multiple sub-models are used to determine some inference results, and the target bit length of the multiple sub-models is allocated according to the order of tensor bit length. The scaling factor of each sub-model is determined based on the target bit depth of each sub-model, and the scaling factors of the sub-models are arranged in order of tensor bit depth. The original model weights are adjusted according to the scaling factor of multiple sub-models to obtain the weights of each sub-model. The weights of each sub-model are used to replace the original model weights of the large language model to determine each sub-model. The sub-inference results of the target bit length corresponding to each sub-model are obtained. The inference result output by the large language model is determined based on the sub-inference results of the target bit length corresponding to multiple sub-models and the order of tensor bit length.

[0079] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0080] The foregoing has provided a detailed description of the method and electronic device for determining inference results using a large language model, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for determining inference results using a large language model, characterized in that, include: In response to the large language model performing forward computation on the prompt words, the original model weights of the large language model are obtained, the tensor data format of the inference results output by the large language model is identified, and the tensor bit width of the tensor data format is determined. Multiple sub-models are used to determine some inference results, and the target bit lengths corresponding to the multiple sub-models are allocated according to the order of the tensor bit lengths; The scaling factor of each of the sub-models is determined according to the target bit depth of each sub-model, and the scaling factors of the sub-models are arranged in order of the tensor bit depth. The original model weights are adjusted according to the scaling factor of the multiple sub-models to obtain the weights of each sub-model. The weights of each sub-model are used to replace the original model weights of the large language model to determine each sub-model, and the sub-inference results of the target number of bits corresponding to each sub-model are obtained. The inference result output by the large language model is determined based on the sub-inference results of the target bit length corresponding to each sub-model and the order of the tensor bit length.

2. The method for determining inference results using a large language model according to claim 1, characterized in that, The step of identifying the tensor data format of the inference result output by the large language model and determining the tensor bit width of the tensor data format includes: Obtain the inference result output by the large language model performing forward computation on the prompt words, identify the tensor data format of the inference result, and determine the number of tensor bits included in the tensor data format; The tensor data format is labeled with position numbers for multiple tensor bits in sequence, with each position number corresponding to one tensor bit.

3. The method for determining inference results using a large language model according to claim 2, characterized in that, The step of using multiple sub-models to determine partial inference results and allocating the target bit depth corresponding to the multiple sub-models according to the order of the tensor bit depth includes: Obtain the number of the sub-models; The tensor bit depth is divided into a number of sub-model bit arrays according to the number of consecutive bit numbers. Each bit array is associated with a sub-model, and the bit array associated with each sub-model is used as its corresponding target bit depth.

4. The method for determining inference results using a large language model according to claim 1, characterized in that, The step of determining the scaling factor of multiple sub-models based on the target bit depth of each sub-model, and arranging the scaling factors of the multiple sub-models in order of tensor bit depth, includes: Identify the decimal point in the weights of the original model, and determine the number of digits before and after the decimal point. Obtain the first target number of digits of the sub-inference result used to calculate the target number of digits corresponding to the first sub-model in the original model weights, take the decimal point after the first target number of digits as the target, determine the first number of digits to move the decimal point forward or backward, and determine the scaling factor corresponding to the first sub-model based on the first number of digits to move. Obtain the second target numerical digits of the sub-inference result used to calculate the target digits corresponding to the second sub-model in the original model weights. After excluding the first target numerical digits from the original model weights, take the second target numerical digits as the target and determine the second number of decimal places to move the decimal point forward or backward. Determine the scaling factor corresponding to the second sub-model based on the second number of decimal places. Continue in this manner until the scaling factors of multiple sub-models are determined. The scaling factors of the multiple sub-models are obtained in the order of the tensor bit depth, and the scaling factors of the multiple sub-models are arranged sequentially according to the order in which the scaling factors of the multiple sub-models are obtained.

5. The method for determining inference results using a large language model according to claim 1, characterized in that, The process of obtaining the first sub-model weights based on the original model weights and the first scaling factor, determining the first sub-model based on the first sub-model weights, and obtaining the sub-inference result corresponding to the target bit depth of the first sub-model includes: Obtain the number of sub-models preset for calculating the inference results, and copy the weights of the original model multiple times according to the number of sub-models; Obtain the weight values ​​of multiple parameters in the large language model included in the original model weights; The weight values ​​of multiple parameters in a copy of the original model weights are divided by the first scaling factor to obtain the weights of the first sub-model. The original model weights of the large language model are replaced with the weights of the first sub-model to form the first sub-model; The first sub-model performs forward calculation on the prompt word to obtain the sub-inference result corresponding to the target number of bits of the first sub-model.

6. The method for determining inference results using a large language model according to claim 5, characterized in that, The process of subtracting the product of the first sub-model weight and the first scaling factor from the original model weights to obtain a first difference, obtaining the second sub-model weights based on the first difference and a second scaling factor, determining the second sub-model based on the second sub-model weights, and obtaining the sub-inference result of the target bit depth corresponding to the second sub-model includes: The first difference is obtained by subtracting the product of the weights of the first sub-model and the scaling factor of the first sub-model from the weights of the original model copied from the original model. Divide the first difference by the scaling factor of the second sub-model to obtain the weight of the second sub-model; The original model weights of the large language model are replaced with the weights of the second sub-model to form the second sub-model; The second sub-model performs forward calculations on the prompt words to obtain the sub-inference result corresponding to the target number of words in the second sub-model.

7. The method for determining inference results using a large language model according to claim 1, characterized in that, The step of adjusting the original model weights according to the scaling factors of the multiple sub-models to obtain the weights of each sub-model, replacing the original model weights of the large language model with the weights of each sub-model to determine each sub-model, and obtaining the sub-inference result of the target bit depth corresponding to each sub-model includes: The weight of the first sub-model is obtained based on the original model weight and the first scaling factor. The first sub-model is determined based on the weight of the first sub-model, and the sub-inference result of the target number of bits corresponding to the first sub-model is obtained. The first difference is obtained by subtracting the product of the first sub-model weight and the first scaling factor from the original model weight. The second sub-model weight is obtained based on the first difference and the second scaling factor. The second sub-model is determined based on the second sub-model weight, and the sub-inference result of the target number of bits corresponding to the second sub-model is obtained. This process continues until the sub-inference results for the target number of bits corresponding to all sub-models are obtained.

8. The method for determining inference results using a large language model according to claim 7, characterized in that, The step of determining the inference result output by the large language model based on the sub-inference results corresponding to the target bit length of multiple sub-models and the order of the tensor bit length includes: Obtain the sub-inference result corresponding to the target number of bits for each of the sub-models; Determine the position of the target bit length corresponding to each sub-model in the tensor bit length, and determine the weighted value corresponding to the sub-inference result of the target bit length corresponding to each sub-model based on the corresponding position; The sub-inference results corresponding to the target number of the multiple sub-models are multiplied by their corresponding weighted values, and then summed to obtain the final tensor. The final tensor is used as the inference result of the large language model performing forward calculation on the prompt words.

9. The method for determining inference results using a large language model according to claim 8, characterized in that, The method further includes: Obtain the total loss value of the large language model; The total loss value is allocated to each sub-model based on the weighted value of each sub-model; Each sub-model performs backpropagation calculations based on its assigned loss value to obtain the corresponding gradient and updates the parameters of the sub-model.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the method for determining inference results using a large language model as described in any one of claims 1 to 9 when executing the computer program.

Images