Model obtaining method, model reasoning method and electronic equipment

Abstract

The invention provides a model obtaining method, a model reasoning method and electronic equipment. The model obtaining method comprises the following steps: performing quantization processing on an initial weight parameter of an initial model to obtain a first weight parameter; and performing model fine tuning according to the first weight parameter to obtain a first low-rank weight parameter. And combining the first low-rank weight parameter with the first weight parameter to obtain a second weight parameter. And performing quantization processing on the second weight parameter to obtain a third weight parameter. And performing decomposition processing on a parameter difference value between the third weight parameter and the first weight parameter to obtain a calibrated low-rank weight parameter. And obtaining a target model based on the calibrated low-rank weight parameter and the initial weight parameter.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a model acquisition method, a model reasoning method, and an electronic device. Background Art

[0002] In recent years, large models have been widely used in human society and have become a powerful tool for facilitating people's work and lives. In many scenarios, it is necessary to fine-tune pre-trained large language models on specific datasets. Furthermore, when performing inference on large models on end devices, low-bit quantization of the model weights is often required due to limitations in memory resources, graphics card bandwidth, and inference efficiency.

[0003] However, there is currently a problem of low accuracy when combining fine-tuning and quantization techniques. Summary of the Invention

[0004] One aspect of the present disclosure provides a model acquisition method, comprising: quantizing initial weight parameters of an initial model to obtain a first weight parameter; fine-tuning the model based on the first weight parameter to obtain a first low-rank weight parameter; merging the first low-rank weight parameter with the first weight parameter to obtain a second weight parameter; quantizing the second weight parameter to obtain a third weight parameter; decomposing a parameter difference between the third weight parameter and the first weight parameter to obtain a calibrated low-rank weight parameter; and obtaining a target model based on the calibrated low-rank weight parameter and the initial weight parameter.

[0005] Optionally, the model acquisition method also includes: fine-tuning the initial quantization model corresponding to the first weight parameter based on the calibrated low-rank weight parameter and the first weight parameter to obtain a second low-rank weight parameter. Merging the second low-rank weight parameter and the first weight parameter to obtain a fourth weight parameter. Quantizing the fourth weight parameter to obtain a fifth weight parameter. Decomposing the parameter difference between the fifth weight parameter and the first weight parameter to obtain the calibrated low-rank weight parameter of the current iteration. Continue to execute the process of fine-tuning the calibrated low-rank weight parameter and the first weight parameter of the current iteration to the decomposition process until the calibrated low-rank weight parameter of the last iteration is obtained. Obtaining the target model based on the calibrated low-rank weight parameter and the initial weight parameter includes: obtaining the target model based on the calibrated low-rank weight parameter and the initial weight parameter of the last iteration.

[0006] Optionally, quantizing the initial weight parameters of the initial model to obtain first weight parameters includes: calibrating and quantizing the initial weight parameters to obtain first quantized weight parameters. Dequantizing the first quantized weight parameters to obtain the first weight parameters. The precision of the data type of the first weight parameters is less than or equal to the precision of the data type of the initial weight parameters. Obtaining a target model based on the calibrated low-rank weight parameters and the initial weight parameters includes: obtaining the target model based on the calibrated low-rank weight parameters and the first quantized weight parameters.

[0007] Optionally, the initial weight parameters are calibrated and quantized to obtain first quantized weight parameters, including: determining inverse weight parameters corresponding to the initial weight parameters. The initial weight parameters are divided into blocks to obtain multiple weight parameter blocks. Part of the weight parameters in any weight parameter block are quantized, and the quantization error is determined. Based on the quantization error and the inverse weight parameter, the unquantized weight parameters in the weight parameter block are adjusted. After completing the quantization of the weight parameter block, the inverse weight parameter is adjusted. Repeat the steps of quantizing part of the weight parameters to adjusting the inverse weight parameter until the quantization of multiple weight parameter blocks is completed to obtain the first quantized weight parameter.

[0008] Optionally, fine-tuning the model according to the first weight parameter to obtain the first low-rank weight parameter includes: obtaining a fine-tuning dataset and initial low-rank weight parameters. Processing the fine-tuning dataset using the initial fine-tuning model to obtain a fine-tuning loss, wherein the initial low-rank weight parameters are fused to an initial quantization model corresponding to the first weight parameter to obtain an initial fine-tuning model. Based on the fine-tuning loss, the initial low-rank weight parameters are adjusted until the first low-rank weight parameters are obtained.

[0009] Optionally, under offline conditions, the first low-rank weight parameter and the first weight parameter are merged to obtain a second weight parameter.

[0010] Optionally, quantizing the second weight parameter to obtain the third weight parameter includes: calibrating and quantizing the second weight parameter using a fine-tuning data set to obtain the second quantized weight parameter, and dequantizing the second quantized weight parameter to obtain the third weight parameter.

[0011] Optionally, decomposing a parameter difference between the third weight parameter and the first weight parameter to obtain a calibrated low-rank weight parameter includes: determining a weight parameter difference between the third weight parameter and the first weight parameter; and decomposing the weight parameter difference according to a rank of the first low-rank weight parameter to obtain the calibrated low-rank weight parameter.

[0012] Another aspect of the present disclosure provides a model inference method, comprising: obtaining input data of at least one modality, the modality including at least one of text, image, voice, and video. Using a target model to infer the input data to obtain output data, the output data including at least data in the text modality. The weight parameters of the target model are obtained based on the initial weight parameters of the initial model and the calibrated low-rank weight parameters. The calibrated low-rank weight parameters are obtained based on decomposing the parameter difference between the first weight parameter and the third weight parameter. The first weight parameter is obtained based on quantizing the initial weight parameter. The third weight parameter is obtained based on quantizing the second weight parameter. The second weight parameter is obtained based on merging the first low-rank weight parameter with the first weight parameter. And the first low-rank weight parameter is obtained based on fine-tuning the model of the first weight parameter.

[0013] Another aspect of the present disclosure provides an electronic device, including: a memory, at least one processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0015] Figure 1 Schematically illustrates an application scenario diagram of the model acquisition method according to an embodiment of the present disclosure;

[0016] Figure 2 The flowchart of the model acquisition method according to the embodiment of the present disclosure is schematically shown;

[0017] Figure 3 A flowchart of a model acquisition method according to another embodiment of the present disclosure is schematically shown;

[0018] Figure 4 A flowchart of a method for quantizing an initial model according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 5 The flowchart of the method for calibrating and quantizing the initial weight parameters according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 6 A flowchart of a method for fine-tuning a model according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 7 A flowchart of a method for quantizing a second weight parameter according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 8A flow chart schematically illustrates a method for decomposing parameter differences according to an embodiment of the present disclosure;

[0023] Figure 9 The flowchart of the model reasoning method according to the embodiment of the present disclosure is schematically shown;

[0024] Figure 10 Schematically shows a structural block diagram of a model obtaining device according to an embodiment of the present disclosure;

[0025] Figure 11 A block diagram of an electronic device suitable for implementing the model acquisition method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely illustrative and are not intended to limit the scope of the present disclosure. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0029] The accompanying drawings show some block diagrams and / or flow charts. It should be understood that some blocks in the block diagrams and / or flow charts, or combinations thereof, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when these instructions are executed by the processor, they can create a device for implementing the functions / operations described in the block diagrams and / or flow charts.

[0030] Therefore, the techniques of the present disclosure can be implemented in the form of hardware and / or software (including firmware, microcode, etc.). In addition, the techniques of the present disclosure can take the form of a computer program product on a computer-readable medium having stored thereon instructions, which can be used by or in conjunction with an instruction execution system. In the context of the present disclosure, a computer-readable medium can be any medium that can contain, store, convey, propagate, or transmit instructions. For example, a computer-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. Specific examples of computer-readable media include: magnetic storage devices, such as magnetic tape or hard disk drives (HDDs); optical storage devices, such as compact disks (CD-ROMs); memory, such as random access memory (RAM) or flash memory; and / or wired or wireless communication links.

[0031] Embodiments of the present disclosure provide a model acquisition method for improving the fine-tuning and quantization of large models. The model acquisition method includes: quantizing the initial weight parameters of the initial model to obtain a first weight parameter; fine-tuning the model based on the first weight parameter to obtain a first low-rank weight parameter; merging the first low-rank weight parameter with the first weight parameter to obtain a second weight parameter; quantizing the second weight parameter to obtain a third weight parameter; decomposing the difference between the third weight parameter and the first weight parameter to obtain a calibrated low-rank weight parameter; and obtaining a target model based on the calibrated low-rank weight parameter and the initial weight parameter. The present disclosure calibrates the first low-rank weight parameter related to fine-tuning through quantization, fine-tuning, low-rank decomposition, and calibration. When merging the first low-rank weight parameter for online inference is required, it can be directly merged with the quantized initial weight parameter (or after data type conversion). This approach does not affect inference speed or excessively consume device resources, nor does it significantly reduce the performance of the fine-tuning adapter. The accuracy performance on the corresponding fine-tuning task is far superior to existing solutions in the industry.

[0032] Figure 1 The application scenario diagram of the model acquisition method according to an embodiment of the present disclosure is schematically shown.

[0033] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0034] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as large model applications, security applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0035] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0036] Server 105 may be a server that provides various services, such as a backend management server (for example only) that supports websites browsed or applications logged into by users using terminal devices 101, 102, and 103. The backend management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal device to process the large model used by the user.

[0037] It should be noted that the model acquisition method provided in the embodiment of the present disclosure can generally be executed by the server 105 or by the terminal device. Accordingly, the model acquisition device provided in the embodiment of the present disclosure can generally be set in the server 105. The model acquisition method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the model acquisition device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0038] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0039] The following will be based on Figure 1 The scene described by Figures 2 to 9 The model acquisition method of the disclosed embodiment is described in detail.

[0040] Figure 2 The flowchart of the model acquisition method according to the embodiment of the present disclosure is schematically shown.

[0041] According to the embodiments of the present disclosure, Figure 2As shown, the model obtaining method of this embodiment includes, for example, operations S210 to S260.

[0042] In operation S210 , the initial weight parameters of the initial model are quantized to obtain first weight parameters.

[0043] Quantization refers to converting floating-point weight parameters into low-precision representations (such as 8-bit integers) to reduce the storage and computational complexity of the model.

[0044] You can use linear or nonlinear quantization. For example, linear quantization uniformly maps the floating-point range to the integer range, while nonlinear quantization (such as logarithmic quantization) uses a non-uniform mapping based on the distribution of weights.

[0045] Assuming the initial weight parameters are 32-bit floating-point numbers, they are converted to 8-bit integers through linear quantization. For example, the weight values are mapped from [-1.0, 1.0] to [0, 255].

[0046] In operation S220 , the model is fine-tuned according to the first weight parameter to obtain a first low-rank weight parameter.

[0047] Model fine-tuning refers to adjusting the weight parameters through training based on the quantized weights to adapt to specific data processing tasks. Low-rank weight parameters refer to the weight matrix obtained through low-rank decomposition (such as singular value decomposition).

[0048] The first weight parameter can be decomposed using singular value decomposition (SVD) or other low-rank decomposition methods, retaining the first k singular values to obtain a first low-rank weight parameter. The first weight parameter may be, for example, a weight parameter of a pre-trained basic large model (or a general large model), or a weight parameter of an adapter applied to a specific scenario (e.g., medical, legal, or other fields).

[0049] For example, perform SVD decomposition on the weight matrix, retain the first k singular values and corresponding singular vectors, and obtain a low-rank approximate matrix.

[0050] In operation S230 , the first low-rank weight parameter is combined with the first weight parameter to obtain a second weight parameter.

[0051] The merging process can use weighted averaging, concatenation, or other fusion methods. For example, the low-rank weight parameter and the original quantized weight parameter are weighted and summed in a certain ratio. For example, the first low-rank weight parameter and the first weight parameter are weighted and summed in a ratio of 0.5:0.5 to obtain the second weight parameter.

[0052] In operation S240 , the second weight parameter is quantized to obtain a third weight parameter.

[0053] The purpose of the re-quantization process is to further compress the weight parameters and reduce the storage and computational complexity of the model. The same quantization method as in operation S210 may be used, but a different quantization range or precision may be used. For example, the second weight parameter may be further quantized from an 8-bit integer to a 4-bit integer to obtain a third weight parameter.

[0054] In operation S250, a parameter difference between the third weight parameter and the first weight parameter is decomposed to obtain a calibrated low-rank weight parameter.

[0055] Decomposition processing refers to performing low-rank decomposition on the parameter difference to extract the main information.

[0056] For example, SVD or other low-rank decomposition methods can be used to calculate the difference matrix between the third weight parameter and the first weight parameter, perform SVD decomposition on it, retain the first k singular values, and obtain the calibrated low-rank weight parameters.

[0057] In operation S260 , a target model is obtained based on the calibrated low-rank weight parameters and the initial weight parameters.

[0058] The final target model is obtained by merging the calibrated low-rank weight parameters with the quantized initial weight parameters to obtain the weight parameters corresponding to the efficient and highly accurate target model.

[0059] The calibrated low-rank weight parameters and the quantized initial weight parameters can be weighted averaged or otherwise combined.

[0060] In some embodiments, the initial model can be, for example, a large language model (LLM). These models typically contain billions or even hundreds of billions of parameters, resulting in very high storage and computational costs. The weight parameters of the initial model are floating-point numbers (such as FP32 or FP16), which consume a large amount of memory and computing resources. Using methods such as quantization and low-rank decomposition, the model size and computational requirements can be significantly reduced while minimizing performance.

[0061] For example, the initial model can be a vision model. These models are used for tasks such as image classification and object detection and typically contain millions to billions of parameters. The weight parameters of the initial model are also floating-point numbers. Quantization can significantly reduce the model size, making it easier to deploy on mobile devices or embedded systems.

[0062] For another example, the initial model can be a multimodal model. These models process multimodal data, such as text and images, and typically have large parameter sizes. Quantization and low-rank decomposition can reduce the model's storage and computational requirements while maintaining multimodal understanding capabilities.

[0063] For example, the initial model could be a speech model. These models are used for tasks like speech synthesis and speech recognition and have large parameter sizes. Quantization can significantly reduce the model size, making it easier to deploy on edge devices such as mobile phones and smart speakers.

[0064] Another example is a recommendation system model. These models are used for personalized recommendation tasks and typically contain numerous embedding and fully connected layers, resulting in a large number of parameters. Quantization and low-rank decomposition can reduce the model size and improve the real-time performance of the recommendation system.

[0065] This embodiment achieves calibration of the first low-rank weight parameters related to fine-tuning through steps such as quantization, fine-tuning, low-rank decomposition and calibration. When merged inference of the first low-rank weight parameters is required, they can be directly merged with the quantized initial weight parameters (or after data type conversion). This does not affect the inference speed or occupy too many device resources, and will not significantly reduce the performance of the fine-tuning adapter. The accuracy performance on the corresponding fine-tuning tasks is far superior to existing solutions in the industry.

[0066] Figure 3 The flowchart of the model acquisition method according to another embodiment of the present disclosure is schematically shown.

[0067] According to the embodiments of the present disclosure, Figure 3 In addition to the above reference Figure 2 In addition to operations S210 to S260 described above, the method of this embodiment also includes, for example, operations S310 to S350. Accordingly, based on the calibrated low-rank weight parameters and the initial weight parameters, a target model is obtained, including operation S360. For the sake of brevity, the description of operations S210 to S260 is omitted here, and subsequent related method embodiments are similar and will not be repeated.

[0068] In operation S310 , an initial quantization model corresponding to the first weight parameter is fine-tuned based on the calibrated low-rank weight parameter and the first weight parameter to obtain a second low-rank weight parameter.

[0069] After the first calibration, the initial quantized model is fine-tuned using the calibrated low-rank weight parameters and the first weight parameters to further adapt to the specific task.

[0070] For example, the model is trained using a fine-tuning dataset for a specific task (such as text classification, image classification, etc.). During the training process, some weights (such as the first weight parameter) are fixed, and only the low-rank weight parameters are adjusted.

[0071] Assume that the initial quantized model is based on the INT8 quantized version of the large language model, and the fine-tuning dataset is text classification data for a specific domain. Based on the calibrated low-rank weight parameters, the fine-tuning dataset is used for training to obtain the second low-rank weight parameters.

[0072] In operation S320, the second low-rank weight parameter and the first weight parameter are combined to obtain a fourth weight parameter.

[0073] The fine-tuned second low-rank weight parameter is merged with the first weight parameter to combine the information of the two.

[0074] Weighted averaging, concatenation or other fusion methods may be used. In operation S330, the fourth weight parameter is quantized to obtain a fifth weight parameter.

[0075] The merged fourth weight parameter is quantized to further compress the model.

[0076] Use the same quantization method as before (e.g. linear or logarithmic).

[0077] For example, the fourth weight parameter is quantized from FP16 to INT8 to obtain the fifth weight parameter.

[0078] In operation S340, a parameter difference between the fifth weight parameter and the first weight parameter is decomposed to obtain a calibrated low-rank weight parameter for the current iteration.

[0079] The difference between the fifth weight parameter and the first weight parameter is calculated, and the difference is subjected to low-rank decomposition to extract main information.

[0080] Singular value decomposition (SVD) or other low-rank decomposition methods can be used. For example, the difference matrix between the fifth weight parameter and the first weight parameter is calculated, and the SVD decomposition is performed on it, and the first k singular values are retained to obtain the calibrated low-rank weight parameters for the current iteration.

[0081] In operation S350 , the process of fine-tuning the calibrated low-rank weight parameters and the first weight parameters of the current iteration to the decomposition process is continued until the calibrated low-rank weight parameters of the last iteration are obtained.

[0082] Repeat steps S310 to S340 to gradually optimize the model until a preset number of iterations or convergence condition is reached. Each iteration fine-tunes, merges, quantizes, and decomposes the low-rank weight parameters and the first weight parameters after the previous calibration. The number of iterations can be adjusted based on task requirements and computing resources.

[0083] Assume that the number of iterations is 3, and each iteration is optimized based on the result of the previous iteration, and finally the calibrated low-rank weight parameters of the last iteration are obtained.

[0084] In operation S360 , a target model is obtained based on the calibrated low-rank weight parameters of the last iteration and the initial weight parameters.

[0085] The calibrated low-rank weight parameters of the last iteration are merged with the quantized initial weight parameters to obtain the final target model.

[0086] Figure 4 The flowchart of the method for quantizing the initial model according to the embodiment of the present disclosure is schematically shown.

[0087] According to the embodiments of the present disclosure, Figure 4 As shown, for example, the initial weight parameters of the initial model are quantized through operations S411 to S412 to obtain the first weight parameters. Accordingly, based on the calibrated low-rank weight parameters and the initial weight parameters, the target model is obtained, including operation S461.

[0088] In operation S411 , calibration quantization is performed on the initial weight parameters to obtain first quantized weight parameters.

[0089] Calibrated quantization, for example, means that during the quantization process, the quantization range can be dynamically adjusted according to the distribution of weight parameters to reduce the quantization error.

[0090] For example, a calibration dataset (e.g., a subset of the training set) is used to calculate the range (e.g., maximum and minimum values) of the weight parameters, and linear quantization or nonlinear quantization is performed based on the calculated range.

[0091] Assuming that the initial weight parameter is FP32, its range is calculated using the calibration dataset and quantized to INT8 to obtain the first quantized weight parameter.

[0092] In operation S412, the first quantized weight parameter is dequantized to obtain a first weight parameter, wherein the precision of the data type of the first weight parameter is less than or equal to the precision of the data type of the initial weight parameter.

[0093] Dequantization refers to converting the quantized weight parameters back to floating-point representation for subsequent fine-tuning and calculation.

[0094] Use the range parameter during quantization to map the quantized weight parameters back to the floating point range. For example, dequantize the first quantized weight parameter of INT8 to FP16 to obtain the first weight parameter.

[0095] In operation S461 , a target model is obtained based on the calibrated low-rank weight parameter and the first quantization weight parameter.

[0096] Based on the dequantized first weight parameters, the model is fine-tuned to adapt to specific tasks.

[0097] The model is trained using a fine-tuning dataset for a specific task. During training, some weights (such as the first weight parameter) are fixed, and only the low-rank weight parameters are adjusted.

[0098] Assuming the initial model is a large language model and the fine-tuning dataset is legal text classification data, fine-tune the first weight parameter to obtain the first low-rank weight parameter.

[0099] Then, the fine-tuned low-rank weight parameters are merged with the first weight parameters to combine the information of the two.

[0100] The combined second weight parameters are quantized to further compress the model.

[0101] The second weight parameter may be quantized from FP16 to INT8 using the same calibration quantization method as operation S411 to obtain a third weight parameter.

[0102] The difference between the third weight parameter and the first weight parameter is calculated, and the difference is subjected to low-rank decomposition to extract main information.

[0103] Singular value decomposition (SVD) or other low-rank decomposition methods can be used. For example, the difference matrix between the third weight parameter and the first weight parameter is calculated, and SVD decomposition is performed on it. The first k singular values are retained to obtain the calibrated low-rank weight parameters.

[0104] Finally, the calibrated low-rank weight parameters are merged with the first quantized weight parameters to obtain the final target model.

[0105] A weighted average or other fusion method may be used. For example, the calibrated low-rank weight parameter and the first quantized weight parameter are weighted and summed in a ratio of 0.7:0.3 to obtain the weight parameter of the target model.

[0106] This embodiment further optimizes the model acquisition method by introducing calibration quantization and dequantization processes, which can significantly reduce the storage and computing requirements of the model while maintaining model performance, and is suitable for the efficient deployment of large models on specific tasks.

[0107] Figure 5 The flowchart of the method for calibrating and quantizing initial weight parameters according to an embodiment of the present disclosure is schematically shown.

[0108] According to the embodiments of the present disclosure, Figure 5 As shown, for example, the initial weight parameters are calibrated and quantized through operations S5111 to S5116 to obtain first quantized weight parameters.

[0109] In operation S5111, inverse weight parameters corresponding to the initial weight parameters are determined.

[0110] The inverse weight parameter refers to, for example, an auxiliary parameter used to adjust the quantization error, and is usually calculated using the initial weight parameter.

[0111] The inverse matrix of the initial weight parameters or other forms of auxiliary parameters can be used.

[0112] Assuming the initial weight parameter is matrix W, calculate its inverse matrix W -1 as the inverse weight parameter.

[0113] In operation S5112, the initial weight parameters are divided into blocks to obtain a plurality of weight parameter blocks.

[0114] Blocking refers to dividing the initial weight parameter matrix into multiple smaller sub-matrices (blocks) so that quantization can be performed block by block.

[0115] The matrix can be divided into blocks according to rows or columns, or the block size can be customized according to task requirements.

[0116] For example, the initial weight parameter matrix W is divided into four sub-matrices W1, W2, W3, and W4.

[0117] In operation S5113, part of the weight parameters in any one weight parameter block is quantized, and a quantization error is determined.

[0118] Part of the weight parameters in each weight parameter block are quantized, and the error between the quantized weight parameters and the original weight parameters is calculated.

[0119] For example, the first 50% weight parameters in the submatrix W1 are quantized to obtain the quantized weight parameters W1 ^ , and calculate the quantization error E1=W1-W1 ^ .

[0120] In operation S5114, unquantized weight parameters in the weight parameter block are adjusted based on the quantization error and the inverse weight parameter.

[0121] During the quantization process, each time a weight is quantized, a certain amount of quantization error is introduced. To compensate for this error, for example, GPTQ (Generative Pre-trained Transformer Quantization) can be used to adjust the remaining unquantized weights. This adjustment is based on information from the inverse Hessian matrix, which reflects the interdependencies between weights. Through global updates, GPTQ ensures that quantization errors do not accumulate, keeping the model output as close to the original model as possible.

[0122] The unquantized weight parameters are adjusted using the quantization error and the inverse weight parameter to reduce the overall quantization error.

[0123] Use the inverse weight parameter to compensate for the quantization error and adjust the unquantized weight parameter. For example, according to the quantization error E1 and the inverse weight parameter W -1 , adjust the unquantized weight parameters in the submatrix W1 to obtain the adjusted weight parameters W1′.

[0124] In operation S5115, after the quantization of the weight parameter block is completed, the inverse weight parameters are adjusted.

[0125] After completing the quantization of a block of weight parameters, the inverse weight parameters are updated to reflect the changes after quantization.

[0126] Recalculate the inverse weight parameters based on the quantized weight parameter block. For example, based on the quantized submatrix W1 ^ And the adjusted weight parameter W1′, update the inverse weight parameter W -1 .

[0127] In operation S5116, the steps of quantizing part of the weight parameters to adjusting the inverse weight parameters are repeated until the quantization of the plurality of weight parameter blocks is completed to obtain the first quantized weight parameters.

[0128] Operations S5113 to S5115 are repeated for each weight parameter block until all blocks are quantized.

[0129] Each sub-matrix is quantized, error calculated, unquantized weight parameter adjusted, and inverse weight parameter updated in turn. For example, the above process is repeated for sub-matrices W2, W3, and W4, and finally the first quantized weight parameter W is obtained. Q .

[0130] Then, the calibrated low-rank weight parameters are merged with the first quantized weight parameters to obtain the final target model.

[0131] A weighted average or other fusion method may be used. For example, the calibrated low-rank weight parameter and the first quantized weight parameter are weighted and summed in a ratio of 0.7:0.3 to obtain the weight parameter of the target model.

[0132] In some embodiments, for example, the GPTQ quantization method can be used to perform 4-bit calibration quantization on the pre-trained large language model on the calibration set to obtain an integer first quantization weight matrix (i.e., a first quantization weight parameter) corresponding to the calibrated and quantized large language model. For example, the initial weight of the pre-trained large language model can be W, and the integer first quantization weight matrix corresponding to the calibrated and quantized large language model can be WQ , then the GPTQ calibration quantization process can be expressed as WW Q .

[0133] Then, the first quantization weight matrix W Q Dequantize (W Q )) to obtain the FP16 large language model for fine-tuning. For example, the inverse quantized FP16 large language model for fine-tuning can be expressed as .

[0134] This embodiment further optimizes the model acquisition method by introducing block quantization and inverse weight parameter adjustment process, which can significantly reduce the storage and computing requirements of the model while maintaining the model performance, and is suitable for the efficient deployment of large models on specific tasks.

[0135] Figure 6 A flow chart of a method for fine-tuning a model according to an embodiment of the present disclosure is schematically shown.

[0136] According to the embodiments of the present disclosure, Figure 6 As shown, for example, by performing operations S621 to S623, the model is fine-tuned according to the first weight parameter to obtain the first low-rank weight parameter.

[0137] In operation S621 , a fine-tuning dataset and initial low-rank weight parameters are obtained.

[0138] The fine-tuning dataset can be data for a specific task (such as text classification, image classification, etc.), and the initial low-rank weight parameters are the initial low-rank representations obtained by low-rank decomposition or other methods.

[0139] Fine-tuning datasets can be obtained from specific tasks (such as legal texts, medical images, etc.). The initial low-rank weight parameters can be obtained by performing singular value decomposition (SVD) on the first weight parameters.

[0140] Assuming that the fine-tuning dataset is legal text classification data, the initial low-rank weight parameter is obtained by performing SVD decomposition on the first weight parameter.

[0141] In operation S622 , the fine-tuning dataset is processed using the initial fine-tuning model to obtain a fine-tuning loss, wherein the initial low-rank weight parameter is fused to the initial quantization model corresponding to the first weight parameter to obtain an initial fine-tuning model.

[0142] The initial low-rank weight parameters are combined with the first weight parameters to construct an initial fine-tuned model. The first weight parameters are, for example, the weight parameters of a pre-trained basic large model (or a general large model), and the initial low-rank weight parameters are, for example, the weights of an adapter applied to a specific scenario (e.g., medical, legal, or other fields).

[0143] Weighted averaging, concatenation or other fusion methods can be used.

[0144] Use the initial fine-tuned model to perform inference or training on the fine-tuning dataset and calculate task-related losses (such as cross entropy loss, mean squared error, etc.).

[0145] Perform forward propagation on the fine-tuning dataset and calculate the loss. For example, perform inference on the initial fine-tuned model using the legal text classification data and calculate the cross-entropy loss.

[0146] In operation S623 , the initial low-rank weight parameters are adjusted based on the fine-tuning loss until first low-rank weight parameters are obtained.

[0147] According to the fine-tuning loss, the initial low-rank weight parameters are adjusted by gradient descent or other optimization methods to minimize the loss.

[0148] Use the backpropagation algorithm to calculate the gradient and update the initial low-rank weight parameters. For example, based on the fine-tuning loss, use the Adam optimizer to iteratively adjust the initial low-rank weight parameters until the loss converges to obtain the first low-rank weight parameters.

[0149] Then, the fine-tuned first low-rank weight parameter is merged with the first weight parameter to obtain a second weight parameter.

[0150] The combined second weight parameters are quantized to further compress the model.

[0151] Using the same quantization method as before (such as linear quantization or logarithmic quantization), the second weight parameter is quantized from FP16 to INT8 to obtain the third weight parameter.

[0152] The difference between the third weight parameter and the first weight parameter is calculated, and the difference is subjected to low-rank decomposition to extract main information.

[0153] Use singular value decomposition (SVD) or other low-rank decomposition methods. For example, calculate the difference matrix between the third weight parameter and the first weight parameter, perform SVD decomposition on it, retain the first k singular values, and obtain the calibrated low-rank weight parameters.

[0154] Finally, the calibrated low-rank weight parameters are merged with the quantized initial weight parameters to obtain the final target model.

[0155] In some embodiments, for example, the FP16 large language model obtained by dequantization (weight matrix corresponding to ) are fine-tuned to obtain low-rank matrices loraA and loraB.

[0156] Then, the weight matrix of the FP16 large language model obtained by the dequantization and the low-rank matrices loraA and loraB are merged to obtain the weight matrix corresponding to the FP16 fine-tuned large language model. For example, the weight matrix corresponding to the FP16 fine-tuned large language model can be expressed as , then the weight matrix of the FP16 large language model obtained by dequantization and the low-rank matrices loraA and loraB are merged and can be expressed as:

[0157] .

[0158] This embodiment further optimizes the model acquisition method by introducing the initial low-rank weight parameter fusion and fine-tuning loss optimization process. It can significantly reduce the storage and computing requirements of the model while maintaining the model performance, and is suitable for the efficient deployment of large models on specific tasks.

[0159] According to an embodiment of the present disclosure, under an offline condition, the first low-rank weight parameter is merged with the first weight parameter to obtain a second weight parameter.

[0160] Offline conditions refer to the pre-completion of weight parameter merging processing without relying on real-time computing resources.

[0161] Weighted averaging, concatenation, or other fusion methods can be used. Offline processing can be done on high-performance computing devices to reduce the computational burden during deployment.

[0162] Then, the merged second weight parameter is quantized using the same quantization method as before (such as linear quantization or logarithmic quantization) to quantize the second weight parameter from FP16 to INT8 to obtain the third weight parameter.

[0163] The difference between the third weight parameter and the first weight parameter is calculated, and the difference is subjected to low-rank decomposition to extract main information.

[0164] Use singular value decomposition (SVD) or other low-rank decomposition methods. For example, calculate the difference matrix between the third weight parameter and the first weight parameter, perform SVD decomposition on it, retain the first k singular values, and obtain the calibrated low-rank weight parameters.

[0165] Finally, the calibrated low-rank weight parameters are merged with the quantized initial weight parameters to obtain the final target model.

[0166] This embodiment further optimizes the model acquisition method by introducing offline weight parameter merging processing, which can significantly reduce the computational burden during deployment while maintaining model performance. It is suitable for efficient deployment of large models in resource-constrained environments.

[0167] Figure 7 A flowchart of a method for quantizing the second weight parameter according to an embodiment of the present disclosure is schematically shown.

[0168] According to the embodiments of the present disclosure, Figure 7 As shown, for example, the second weight parameter is quantized through operations S741-S742 to obtain a third weight parameter.

[0169] In operation S741 , the second weight parameter is calibrated and quantized using the fine-tuning data set to obtain a second quantized weight parameter.

[0170] Calibrated quantization, for example, refers to dynamically adjusting the quantization range according to the distribution of the fine-tuning dataset to reduce the quantization error.

[0171] Calculate the range (such as maximum and minimum values) of the second weight parameter using the fine-tuning dataset.

[0172] Perform linear quantization or nonlinear quantization based on the calculated range. Assuming the second weight parameter is FP16, calculate its range using the fine-tuning dataset and quantize it to INT8 to obtain the second quantized weight parameter.

[0173] In operation S742 , the second quantization weight parameter is inversely quantized to obtain a third weight parameter.

[0174] Dequantization refers to converting the quantized weight parameters back to floating-point representation for subsequent fine-tuning and calculation.

[0175] Use the range parameter during quantization to map the quantized weight parameters back to the floating point range. For example, dequantize the second quantized weight parameter of INT8 to FP16 to obtain the third weight parameter.

[0176] Then, the difference between the third weight parameter and the first weight parameter is calculated, and the difference is subjected to low-rank decomposition to extract the main information.

[0177] Singular value decomposition (SVD) or other low-rank decomposition methods can be used. For example, the difference matrix between the third weight parameter and the first weight parameter is calculated, and SVD decomposition is performed on it. The first k singular values are retained to obtain the calibrated low-rank weight parameters.

[0178] Finally, the calibrated low-rank weight parameters are merged with the quantized initial weight parameters to obtain the final target model.

[0179] In some embodiments, for example, the weight matrix corresponding to the fine-tuned large language model of FP16 obtained by offline merging can be calibrated and quantized. For example, the GPTQ quantization method is also used to add the fine-tuning data set related to lora fine-tuning to the calibration set to perform 4-bit calibration quantization on the weight matrix. The weight matrix after calibration and quantization can be expressed as , then the calibration and quantization of the weight matrix corresponding to the fine-tuned large language model of FP16 obtained by offline merging can be expressed as .

[0180] Then, the calibrated quantized weight matrix Perform inverse quantization to obtain the inverse quantization weight matrix corresponding to the large language model after fine-tuning FP16. For example, the inverse quantization weight matrix corresponding to the large language model after fine-tuning FP16 can be expressed as .

[0181] This embodiment introduces calibration quantization and dequantization processes and performs calibration quantization based on the fine-tuning data set, so that the quantization before and after fine-tuning is consistent, that is, the change in data type is consistent, which facilitates the subsequent calibration of low-rank weight parameters.

[0182] Figure 8 A flow chart of a method for decomposing parameter differences according to an embodiment of the present disclosure is schematically shown.

[0183] According to the embodiments of the present disclosure, Figure 8 As shown, for example, the parameter difference between the third weight parameter and the first weight parameter is decomposed through operations S851~S852 to obtain the calibrated low-rank weight parameter.

[0184] In operation S851, a weight parameter difference between the third weight parameter and the first weight parameter is determined.

[0185] The weight parameter difference refers to the difference between the third weight parameter and the first weight parameter, and reflects the error between the quantization weight after fine-tuning and the quantization weight before fine-tuning.

[0186] Directly calculate the difference matrix between the third weight parameter and the first weight parameter. Assume that the third weight parameter is W3 and the first weight parameter is W1, and calculate the difference matrix ΔW=W3-W1.

[0187] In operation S852, the weight parameter difference is decomposed according to the rank of the first low-rank weight parameter to obtain a calibrated low-rank weight parameter.

[0188] According to the rank of the first low-rank weight parameter (ie, the dimension of the low-rank decomposition), the difference matrix is low-rank decompositioned to extract the main information.

[0189] Use singular value decomposition (SVD) or other low-rank decomposition methods. Keep the first k singular values, where k is the rank of the first low-rank weight parameter. For example, perform SVD decomposition on the difference matrix ΔW, keep the first k singular values, and obtain the calibrated low-rank weight parameter ΔW low-rank .

[0190] Finally, the calibrated low-rank weight parameters are merged with the quantized initial weight parameters to obtain the final target model.

[0191] Weighted averaging or other fusion methods can be used. For example, the calibrated low-rank weight parameter ΔW low-rank The weighted sum is calculated with the quantized initial weight parameter W1 in a ratio of 0.7:0.3 to obtain the weight parameter of the target model.

[0192] In some embodiments, for example, the dequantized weight matrix corresponding to the FP16 fine-tuned large language model can be subtracted from the dequantized weight matrix corresponding to the FP16 large language model used for fine-tuning to obtain a weight difference matrix corresponding to loraB∙loraA. For example, the weight difference matrix can be expressed as ΔW, then:

[0193] .

[0194] Then, the weight difference matrix ΔW is decomposed by SVD, and the first r singular values and singular vectors are selected according to the rank r of the lora adapter to decompose (loraA) ̃ and (loraB) ̃. (loraA) ̃ and (loraB) ̃ are used as nodes to jump to the fine-tuning step to start a new round of lora fine-tuning until the maximum training iteration round n is met, and the final calibrated loraA and loraB are obtained.

[0195] Finally, the calibrated loraA, loraB can be combined with the quantized initial weight parameters to obtain the target model. For example, loraA, loraB can be directly combined with the quantized weight matrix W corresponding to the FP16 large language model used for fine-tuning. Q Merge to get the weight matrix corresponding to the target model. For example, based on the formula:

[0196] , merge directly.

[0197] After multiple rounds of training and calibration using the above solution, loraB and loraA can be directly merged when lora merge inference is needed online. This does not affect the inference speed or occupy too many device resources, and will not significantly reduce the performance of the lora adapter. The accuracy performance on the corresponding fine-tuning tasks is far superior to existing solutions in the industry.

[0198] This embodiment introduces a weight parameter difference decomposition process, decomposes the difference matrix based on the rank of the low-rank weight parameters, and realizes calibration of the low-rank weight matrix of the fine-tuning model.

[0199] Figure 9 The flowchart of the model reasoning method according to the embodiment of the present disclosure is schematically shown.

[0200] According to the embodiments of the present disclosure, Figure 9 As shown, the model inference method of this embodiment includes, for example, operations S910 to S920.

[0201] In operation S910 , input data of at least one modality is acquired, where the modality includes at least one of text, image, voice, and video.

[0202] The input data can be in at least one of the following modalities: text, image, voice, and video.

[0203] Data can be obtained from user input, sensors, or other data sources. The input data is preprocessed (such as text segmentation, image normalization, speech feature extraction, etc.).

[0204] Assume that the input data is a piece of text: "Please analyze this legal text", and perform word segmentation and encoding on it.

[0205] In operation S920, the target model is used to infer the input data to obtain output data, and the output data includes at least data in the text modality. The weight parameters of the target model are obtained based on the initial weight parameters of the initial model and the calibrated low-rank weight parameters. The calibrated low-rank weight parameters are obtained based on the decomposition processing of the parameter difference between the first weight parameter and the third weight parameter. The first weight parameter is obtained based on the quantization processing of the initial weight parameter. The third weight parameter is obtained based on the quantization processing of the second weight parameter. The second weight parameter is obtained based on the merging processing of the first low-rank weight parameter and the first weight parameter. And the first low-rank weight parameter is obtained based on model fine-tuning of the first weight parameter.

[0206] The target model is an optimized model based on the initial model and the calibrated low-rank weight parameters, which is used to reason about the input data.

[0207] The weight parameters of the target model are used to perform forward propagation calculations on the input data. The output data can be text-based data (such as classification results, generated text, etc.).

[0208] For example, the encoded text is input into the target model and the output data is obtained: "This text belongs to the legal field."

[0209] The weight parameters of the target model are obtained by, for example, merging the initial weight parameters and the calibrated low-rank weight parameters.

[0210] The calibrated low-rank weight parameter is obtained by performing low-rank decomposition on the difference between the first weight parameter and the third weight parameter.

[0211] Calculate the difference matrix ΔW = W3-W1. Perform singular value decomposition (SVD) on the difference matrix, retain the first k singular values, and obtain the calibrated low-rank weight parameter ΔW low-rank .

[0212] The first weight parameter is obtained by quantizing the initial weight parameter.

[0213] Use linear quantization or nonlinear quantization method. For example, quantize the initial weight parameter from FP32 to INT8 to obtain the first weight parameter W1.

[0214] The third weight parameter is obtained by quantizing the second weight parameter.

[0215] The same quantization method as that used to quantize the initial weight parameter is used. For example, the second weight parameter is quantized from FP16 to INT8 to obtain the third weight parameter W3.

[0216] The second weight parameter is obtained by combining the first low-rank weight parameter with the first weight parameter.

[0217] Weighted averaging, concatenation, or other fusion methods may be used. For example, the first low-rank weight parameter and the first weight parameter are weighted and summed in a ratio of 0.5:0.5 to obtain the second weight parameter W2.

[0218] The first low-rank weight parameter is obtained by fine-tuning the first weight parameter.

[0219] Use the fine-tuning dataset to train the model and adjust the low-rank weight parameters. For example, fine-tune the first weight parameter on the legal text classification data to obtain the first low-rank weight parameter.

[0220] This embodiment implements an efficient model inference method by combining methods such as quantization, fine-tuning, low-rank decomposition, and difference decomposition. It can significantly reduce the storage and computing requirements of the model while maintaining model performance, and is suitable for efficient inference of large models on multimodal tasks.

[0221] Based on the above method, the present disclosure also provides a model acquisition device. Figure 10 The model acquisition device is described in detail.

[0222] Figure 10The structural block diagram of the model acquisition device according to an embodiment of the present disclosure is schematically shown.

[0223] like Figure 10 As shown, the model acquisition device 1000 of this embodiment includes, for example: a first quantization module 1010, a fine-tuning module 1020, a merging module 1030, a second quantization module 1040, a decomposition module 1050 and a determination module 1060. The model acquisition device 1000 can execute the above reference Figures 2 to 9 The method described in this paper can achieve high-precision fine-tuning and quantization of large models.

[0224] Specifically, the first quantization module 1010 is used to quantize the initial weight parameters of the initial model to obtain the first weight parameters. In one embodiment, the first quantization module 1010 can be used to perform the operation S210 described above, which will not be repeated here.

[0225] The fine-tuning module 1020 is configured to fine-tune the model according to the first weight parameter to obtain the first low-rank weight parameter. In one embodiment, the fine-tuning module 1020 may be configured to perform the operation S220 described above, which will not be described in detail herein.

[0226] The merging module 1030 is configured to merge the first low-rank weight parameter with the first weight parameter to obtain a second weight parameter. In one embodiment, the merging module 1030 may be configured to perform the operation S230 described above, which will not be described in detail herein.

[0227] The second quantization module 1040 is used to quantize the second weight parameter to obtain a third weight parameter. In one embodiment, the second quantization module 1040 can be used to perform the operation S240 described above, which will not be repeated here.

[0228] The decomposition module 1050 is used to decompose the parameter difference between the third weight parameter and the first weight parameter to obtain the calibrated low-rank weight parameter. In one embodiment, the decomposition module 1050 can be used to perform the operation S250 described above, which will not be repeated here.

[0229] The determination module 1060 is used to obtain the target model based on the calibrated low-rank weight parameters and the initial weight parameters. In one embodiment, the determination module 1060 can be used to perform the operation S260 described above, which will not be repeated here.

[0230] It is understood that the first quantization module 1010, the fine-tuning module 1020, the merging module 1030, the second quantization module 1040, the decomposition module 1050, and the determination module 1060 can be implemented in a single module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present disclosure, at least one of the first quantization module 1010, the fine-tuning module 1020, the merging module 1030, the second quantization module 1040, the decomposition module 1050, and the determination module 1060 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware in any other reasonable manner for integrating or packaging circuits, or in an appropriate combination of software, hardware, and firmware. Alternatively, at least one of the first quantization module 1010, the fine-tuning module 1020, the merging module 1030, the second quantization module 1040, the decomposition module 1050 and the determination module 1060 can be at least partially implemented as a computer program module, and when the program is run by a computer, the function of the corresponding module can be executed.

[0231] Based on the above method, the present disclosure also provides a model reasoning device, comprising:

[0232] an acquisition module, configured to acquire input data in at least one modality, the modality comprising at least one of text, image, voice, and video;

[0233] An inference model of the target model, configured to infer the input data to obtain output data, wherein the output data includes at least data in a textual modality;

[0234] The weight parameters of the target model are obtained based on the initial weight parameters of the initial model and the calibrated low-rank weight parameters;

[0235] The calibrated low-rank weight parameter is obtained by decomposing a parameter difference between the first weight parameter and the third weight parameter;

[0236] The first weight parameter is obtained by quantizing the initial weight parameter;

[0237] The third weight parameter is obtained based on quantizing the second weight parameter;

[0238] The second weight parameter is obtained by merging the first low-rank weight parameter and the first weight parameter; and

[0239] The first low-rank weight parameter is obtained based on model fine-tuning of the first weight parameter.

[0240] Figure 11 A block diagram of an electronic device suitable for implementing a model acquisition method or a model inference method according to an embodiment of the present disclosure is schematically shown.

[0241] like Figure 11 As shown, the electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage portion 1108 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1101 may also include onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0242] Various programs and data required for the operation of the electronic device 1100 are stored in the RAM 1103. The processor 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. The processor 1101 performs various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 1102 and / or the RAM 1103. It should be noted that the programs may also be stored in one or more memories other than the ROM 1102 and the RAM 1103. The processor 1101 may also perform various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0243] According to an embodiment of the present disclosure, electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to bus 1104. Electronic device 1100 may also include one or more of the following components connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 1108 including a hard disk; and a communication section 1109 including a network interface card such as a LAN card or modem. Communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 1110 as needed, so that computer programs read from the removable media can be installed into storage section 1108 as needed.

[0244] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently without being incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the model acquisition method or model inference method according to the embodiments of the present disclosure.

[0245] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above, and / or one or more memories other than ROM 1102 and RAM 1103.

[0246] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to cause the computer system to implement the model acquisition method or model reasoning method provided by the embodiments of the present disclosure.

[0247] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 1101. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0248] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 1109, and / or installed from removable media 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0249] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1109 and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0250] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0251] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0252] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0253] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A model acquisition method, comprising: Quantizing the initial weight parameters of the initial model to obtain a first weight parameter; Fine-tune the model according to the first weight parameter to obtain a first low-rank weight parameter; Merging the first low-rank weight parameter with the first weight parameter to obtain a second weight parameter; quantizing the second weight parameter to obtain a third weight parameter; Decomposing a parameter difference between the third weight parameter and the first weight parameter to obtain a calibrated low-rank weight parameter; A target model is obtained based on the calibrated low-rank weight parameters and the initial weight parameters.

2. The method according to claim 1, wherein The method further comprises: Fine-tuning the initial quantization model corresponding to the first weight parameter based on the calibrated low-rank weight parameter and the first weight parameter to obtain a second low-rank weight parameter; Combining the second low-rank weight parameter and the first weight parameter to obtain a fourth weight parameter; quantizing the fourth weight parameter to obtain a fifth weight parameter; Decomposing a parameter difference between the fifth weight parameter and the first weight parameter to obtain a calibrated low-rank weight parameter for a current iteration; Continue to fine-tune the low-rank weight parameters calibrated in the current iteration and the first weight parameters to the decomposition process until the low-rank weight parameters calibrated in the last iteration are obtained; The obtaining of a target model based on the calibrated low-rank weight parameters and the initial weight parameters includes: The target model is obtained based on the calibrated low-rank weight parameters of the last iteration and the initial weight parameters.

3. The method according to claim 1 or 2, wherein: The quantizing of the initial weight parameters of the initial model to obtain the first weight parameters includes: Calibrate and quantize the initial weight parameter to obtain a first quantized weight parameter; Dequantizing the first quantization weight parameter to obtain the first weight parameter; Wherein, the precision of the data type of the first weight parameter is less than or equal to the precision of the data type of the initial weight parameter; The obtaining of a target model based on the calibrated low-rank weight parameters and the initial weight parameters includes: The target model is obtained based on the calibrated low-rank weight parameters and the first quantization weight parameters.

4. The method according to claim 3, wherein: Calibration and quantization are performed on the initial weight parameter to obtain a first quantized weight parameter, including: determining an inverse weight parameter corresponding to the initial weight parameter; Dividing the initial weight parameter into blocks to obtain a plurality of weight parameter blocks; quantizing a portion of weight parameters in any one of the weight parameter blocks and determining a quantization error; adjusting unquantized weight parameters in the weight parameter block based on the quantization error and the inverse weight parameter; After completing the quantization of the weight parameter block, adjusting the inverse weight parameter; Repeat the steps of quantizing the part of the weight parameters to adjusting the inverse weight parameters until the quantization of the multiple weight parameter blocks is completed to obtain the first quantized weight parameters.

5. The method according to claim 1, wherein Fine-tuning the model according to the first weight parameter to obtain the first low-rank weight parameter includes: Get the fine-tuning dataset and initial low-rank weight parameters; Processing the fine-tuning dataset using an initial fine-tuning model to obtain a fine-tuning loss, wherein the initial low-rank weight parameter is fused to an initial quantization model corresponding to the first weight parameter to obtain the initial fine-tuning model; Based on the fine-tuning loss, the initial low-rank weight parameter is adjusted until the first low-rank weight parameter is obtained.

6. The method according to claim 1, wherein Under offline conditions, the first low-rank weight parameter and the first weight parameter are merged to obtain the second weight parameter.

7. The method according to claim 1, wherein The quantizing the second weight parameter to obtain a third weight parameter includes: calibrating and quantizing the second weight parameter using a fine-tuning data set to obtain a second quantized weight parameter; Dequantize the second quantization weight parameter to obtain the third weight parameter.

8. The method according to claim 1, wherein The decomposing the parameter difference between the third weight parameter and the first weight parameter to obtain the calibrated low-rank weight parameter includes: determining a weight parameter difference between the third weight parameter and the first weight parameter; The weight parameter difference is decomposed according to the rank of the first low-rank weight parameter to obtain the calibrated low-rank weight parameter.

9. A model inference method comprising: Obtaining input data in at least one modality, the modality comprising at least one of text, image, voice, and video; Using a target model to reason on the input data to obtain output data, the output data at least including data in a text modality; The weight parameters of the target model are obtained based on the initial weight parameters of the initial model and the calibrated low-rank weight parameters; The calibrated low-rank weight parameter is obtained by decomposing a parameter difference between the first weight parameter and the third weight parameter; The first weight parameter is obtained by quantizing the initial weight parameter; The third weight parameter is obtained based on quantizing the second weight parameter; The second weight parameter is obtained by merging the first low-rank weight parameter and the first weight parameter; and The first low-rank weight parameter is obtained based on model fine-tuning of the first weight parameter.

10. An electronic device comprising: A memory, at least one processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images