Model quantization method, apparatus, device, storage medium, and program product

Abstract

According to embodiments of the present disclosure, a model quantization method, apparatus, device, storage medium, and program product are provided. The method includes: determining a quantifiable model component in a machine learning model; dividing a weight parameter of the quantifiable model component into a plurality of parameter blocks; performing quantization on the plurality of parameter blocks by using a plurality of scaling factors corresponding to the plurality of parameter blocks to obtain quantized weight parameters, the quantized weight parameters including a plurality of quantized parameter blocks corresponding to the plurality of parameter blocks; and determining a quantized machine learning model based at least on the quantized weight parameters and the plurality of scaling factors.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of information technology, and particularly to model quantization methods, apparatus, devices, computer-readable storage media, and computer program products. Background Technology

[0002] With the rapid development of artificial intelligence technology, machine learning models are becoming increasingly powerful in terms of reasoning, generalization, and expressive abilities. Significant achievements have been made in various fields such as computer vision, speech recognition, and natural language processing. However, as model performance improves, the number of model parameters also increases, placing higher demands on the computing power and memory capacity of the deployed devices. Summary of the Invention

[0003] In a first aspect of this disclosure, a model quantization method is provided. The method includes: identifying quantizable model components in a machine learning model; dividing the weight parameters of the quantizable model components into multiple parameter blocks; performing quantization on the multiple parameter blocks using multiple scaling factors corresponding to the multiple parameter blocks to obtain quantized weight parameters, the quantized weight parameters including multiple quantized parameter blocks corresponding to the multiple parameter blocks; and determining a quantized machine learning model based at least on the quantized weight parameters and the multiple scaling factors.

[0004] In a second aspect of this disclosure, an apparatus for model quantization is provided. The apparatus includes: a first determining module configured to determine quantizable model components in a machine learning model; a partitioning module configured to partition weight parameters of the quantizable model components into multiple parameter blocks; a quantization module configured to perform quantization on the multiple parameter blocks using multiple scaling factors corresponding to the multiple parameter blocks to obtain quantized weight parameters, the quantized weight parameters including multiple quantized parameter blocks corresponding to the multiple parameter blocks; and a second determining module configured to determine a quantized machine learning model based at least on the quantized weight parameters and the multiple scaling factors.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method of the first aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program that can be executed by a processor to perform the methods of the first aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1A A block diagram of an application environment according to an exemplary implementation of this disclosure is shown; Figure 1B A schematic diagram of an example architecture of a machine learning model according to an embodiment of the present disclosure is shown; Figure 1C A schematic diagram of an example environment for model training and application according to some embodiments of the present disclosure is shown; Figure 2 A flowchart illustrating the model quantization process according to some embodiments of the present disclosure is shown; Figure 3 A schematic structural block diagram of an apparatus for model quantization according to some embodiments of the present disclosure is shown; and Figure 4 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0010] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0011] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0012] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0013] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0014] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0017] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0018] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0019] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0020] Figure 1A A block diagram 100A of an application environment according to an exemplary implementation of this disclosure is shown. For example... Figure 1A As shown, robot device 110 and user 120 can be located in environment 160, and user 120 can control robot device 110 to perform a variety of tasks.

[0021] Environment 160 may include, but is not limited to, one or more objects. For example, in a kitchen / dining environment, environment 160 may include, but is not limited to, various items, such as edible food, kitchen utensils, etc., or a combination of one or more of the above. In a wardrobe environment, environment 160 may include, but is not limited to, various types of clothing. Environment 160 may include, but is not limited to, one or more rooms. For example, in a home environment, environment 160 may include, but is not limited to, a living room, bedroom, study, kitchen, toilet, etc., or a combination of one or more of the above.

[0022] like Figure 1AAs shown, user 120 can instruct robot device 110 to manipulate various objects in environment 160 and complete corresponding tasks. Here, objects can be various items in environment 160. For example, user 120 can instruct robot device 110 to find a specific object in environment 160. Or, user 120 can instruct robot device 110 to place the found object in a designated location, and so on. In some examples, user 120 can instruct robot device 110 to perform a task in environment 160. For example, in a life service scenario, robot device 110 can perform tasks such as indoor cleaning, item organization, home control, and voice interaction in environment 160. In an industrial manufacturing scenario, robot device 110 can perform tasks such as industrial inspection and production operations in environment 160. However, these are merely examples, and this disclosure does not limit the scope of the invention.

[0023] like Figure 1A As shown, the robot device 110 may include multiple parts. For example, the control unit 111 can serve as the control center of the robot device 110, and an application can be loaded into the control unit 111 to control the various parts of the robot device. The user 120 can use the interaction unit 112 to interact with the robot device 110, for example, by inputting control commands to the robot device 110 to perform desired tasks. The robot device 110 may include an arm 113 for performing actions such as grasping and releasing. For example, the arm 113 can grasp an object and move it to a desired position, and so on.

[0024] Alternatively and / or additionally, the robot device 110 may also include a data acquisition unit 114. Here, the data acquisition unit 114 may include various types, such as an image acquisition unit, a sound acquisition unit, etc. Alternatively and / or additionally, the robot device 110 may further include a sensing unit for detecting surrounding objects, for example, detecting the distance between the robot and surrounding objects based on laser light, etc. The robot device 110 may also include a drive unit 115, for example, the drive unit 115 can drive the robot device 110 to move along a desired path.

[0025] Environment 160 may include one or more acquisition units 130, ..., and 132. For example, one or more image acquisition devices may be deployed in a room to acquire images of the room from various angles. Environment 160 may include a control device 140, which can control one or more acquisition units 130, ..., and 132, etc., via a network (not shown). Alternatively and / or additionally, in a smart home environment, control device 140 can control various electrical devices in environment 160.

[0026] Alternatively and / or additionally, a machine learning model (e.g., machine learning model 150) may be provided to manage environment 160. It should be understood that, although... Figure 1A The machine learning model 150 is shown to be located within environment 160, but it can also be deployed in robotic device 110. Alternatively and / or additionally, the machine learning model 150 can also be located at a remote device outside environment 160, and the control device 140, robotic device 110, or other device can access the remote machine learning model 150 via a network.

[0027] Machine learning model 150 may include one or more models. If machine learning model 150 includes multiple models, these multiple models may include multiple types of models. Machine learning model 150 may include a Vision-Language-Action (VLA) model. A VLA model is a multimodal model that integrates vision, language, and action. Machine learning model 150 may, for example, include at least a Vision-Language Model (VLM) and an action model. The Vision-Language Model has the ability to process images and natural language. The Action Model can control the robotic device 110 to perform various actions. Machine learning model 150 may also include, for example, an image processing model, a text processing model, and so on.

[0028] In some embodiments, the machine learning model 150 may include a content-generating model capable of generating corresponding outputs based on model inputs. In some implementations, the machine learning model may receive text-modal model inputs (e.g., natural language and / or machine language) and / or non-text-modal model inputs (e.g., images, speech, video, etc.), and may obtain corresponding model outputs based on the model inputs and prompts, thereby completing the task execution.

[0029] Machine learning model 150 may include an encoder module 151, a backbone network 152, and a decoder module 156. The encoder module 151 can perform feature encoding on the model input (e.g., encoding an image or video into a feature space). The encoder module 151 may include, but is not limited to, a video encoder, a speech encoder, and an action encoder. The backbone network 152 can perform feature extraction on the encoded model input to obtain a feature representation 155 for performing a task. In some embodiments, the backbone network 152 may include one or more attention modules (e.g., transformer blocks) capable of performing attention-based processing on the encoded model input to extract the feature representation 155. The decoder module 156 then decodes the output feature representation 155 to obtain the model output. The decoder module may include, for example, an action decoder. For example, for a VLA model, the model output may be, for example, the next action the robot will perform. It should be understood that... Figure 1A The illustration shows an example application of machine learning model 150, which is not limited in this disclosure.

[0030] Figure 1B A schematic diagram of an example architecture of a machine learning model 150 according to an embodiment of this disclosure is shown. Figure 1B As shown, the machine learning model 150 of an embodiment of this disclosure may include a VLA model. A VLA model is a multimodal model that integrates vision, language, and action. Given visual input 153a (e.g., an image or video, etc.) and language input 153b (e.g., text or voice commands) in the environment space (e.g., environment 160) where the robot device 110 is located, the VLA model can output action commands 157. The action commands 157 can be used to control the robot device 110 to perform the task requested by the visual input 153a or the language input 153b.

[0031] In some embodiments, such as Figure 1B As shown, the VLA model may include a visual encoder 151a, a language encoder 151b, a backbone network 152, and an action decoder 156c. The visual encoder 151a performs feature encoding on the visual input 153a (e.g., encoding an image or video into a feature space) to obtain a visual representation 154a. The language encoder 151b performs feature encoding on the language input 153b, such as text commands or voice commands, to obtain a language representation 154b. The backbone network 152 performs feature extraction based on the visual representation 154a and the language representation 154b to obtain a feature representation 155 for action decision-making. Then, the action decoder 156c performs feature decoding on the feature representation 155 to obtain an action command 157 (or a sequence of action commands).

[0032] The motion command 157 here can be implemented in several ways. As an example, motion command 157 can indicate the relative change in the end-effector posture of the robotic arm (i.e., robot device 110). Based on this relative change using the inverse dynamics model of the robotic arm, motion parameters such as angles, velocities, and accelerations for multiple degrees of freedom of the robotic arm (e.g., joints or end-effector grippers) can be determined. Alternatively, motion command 157 can also indicate the motion parameters for multiple degrees of freedom of the robot device 110. In other words, motion decoder 156c can also directly output the motion parameters for specific degrees of freedom of the robot device 110, allowing the robot device 110 controller to directly drive its movement based on these motion parameters.

[0033] In some embodiments, the VLA model may further include a motion encoder 151c. The model input of the VLA model may also include motion input 153c, which may include, but is not limited to, motion commands 157 provided to the robot device 110 in the previous control cycle or motion sensing results of the robot device 110 in the previous control cycle. The motion encoder 151c may perform feature encoding on the motion input 153c to obtain a motion representation 154c. The backbone network 152 may perform feature extraction based on the visual representation 154a, language representation 154b, and motion representation 154c to obtain a feature representation 155. Of course, the model input of the VLA model described above is merely exemplary. In practical applications, the model input of the VLA model may also include, for example, depth maps, point clouds, force feedback information, or haptic feedback information. In this case, the VLA model may also include, for example, a depth map encoder, a point cloud encoder, a force encoder, or a haptic encoder.

[0034] In some embodiments, the backbone network 152 can perform multimodal fusion and feature extraction on the visual representation 154a, language representation 154b, and action representation 154c based on a cross-attention mechanism to obtain feature representation 155. In some embodiments, the visual encoder 151a, language encoder 151b, and backbone network 152 can be implemented based on a pre-trained VLM model. In other words, a VLA model can be constructed by combining the action encoder 151c and action decoder 156c on the basis of a pre-trained VLM model. Of course, the backbone network 152 described above is only exemplary. In practical applications, any other suitable multimodal model can be used to construct the backbone network 152. The embodiments of this disclosure do not limit this.

[0035] Figure 1C A schematic diagram of an example environment 100C for model training and application according to some embodiments of the present disclosure is shown. Figure 1C The example environment 100C illustrates three distinct phases of the machine learning model 150: a pre-training phase 172, a fine-tuning phase 174, and an application phase 176. A testing phase, not shown in the figure, may also occur after the pre-training phase 172 or the fine-tuning phase 174.

[0036] Example environment 100C involves a model pre-training system 192, a model fine-tuning system 194, and a model application system 196. In the pre-training phase 172, the model pre-training system 192 is configured to perform pre-training of a machine learning model 150 using a pre-training dataset 182. At the start of pre-training, the individual components of the machine learning model 150 may have initial parameter values. The pre-training process involves updating the parameter values ​​of the machine learning model 150 to desired values ​​based on data from the pre-training dataset 182. The pre-training task is used to assist in updating the parameters of the machine learning model 150.

[0037] During the pre-training phase 172, the machine learning model 150 can learn strong generalization capabilities using a pre-training dataset 182 containing a large amount of data. After pre-training, the parameter values ​​of the machine learning model 150 have been updated to include the pre-trained parameter values. In some embodiments, the machine learning model 150 may include a VLA model, and the pre-training dataset 182 used to train the machine learning model 150 may include multiple pre-training samples, each including sample language input, sample visual input, and sample action instructions. The pre-trained machine learning model 120 can generate corresponding action instructions relatively accurately based on the received language input and visual input.

[0038] The pre-trained machine learning model 150 can be provided to the fine-tuning stage 174, where the model fine-tuning system 194 fine-tunes it for different downstream tasks. In the fine-tuning stage 174, the parameter values ​​of the machine learning model 150 are further adjusted using the training dataset 184. During fine-tuning, the parameters of the machine learning model 150 are also updated and adjusted using the corresponding training algorithm. Since the model has learned a great deal from the training data in the pre-training stage, a downstream task model that meets expectations can be obtained using only a small amount of training data in the fine-tuning stage 174. The training samples included in the training dataset 184 can be of the same type as the pre-training samples included in the pre-training dataset 182; that is, the training dataset 184 can include multiple training samples, each of which can include sample language input, sample visual input, and sample action instructions. In some cases, the pre-training dataset 182 can include multiple pre-training samples for a general scenario, while the training dataset 184 can include multiple training samples for a specific scenario or specific task.

[0039] In some embodiments, a testing phase may be included after the fine-tuning phase 174, where the performance of the machine learning model 150 can be further tested using a test dataset. The dataset used in the testing phase is of the same type as that used in the fine-tuning phase.

[0040] In application phase 176, the obtained machine learning model 150, with trained parameter values, can be provided to the model application system 196 for use. In application phase 176, the machine learning model 150 can be used to process the corresponding model input 186 in the real-world scenario and provide the corresponding model output 188. This is merely an example, combined with... Figure 1B As shown, model input 186 may include language input 153b and visual input 153a, and model output 188 may include corresponding action instructions 157.

[0041] exist Figure 1C In this system, the model pre-training system 192, the model fine-tuning system 194, and the model application system 196 can be deployed on any suitable electronic device. This electronic device can be any type of computing-capable device, including terminal devices or servers. In some examples, the model application system 196 can be deployed on the server side and on the client side of the robot device 110. The machine learning model 150 can be deployed on the server, and the robot device 110 can interact between the client and the server, requesting the machine learning model 150 to perform tasks. Alternatively, the machine learning model 150 can also be deployed on the robot device 110. As an example, the server side of the model application system 196 can perform quantization processing on the trained machine learning model 150 to obtain a quantized machine learning model 150. Then, based on the communication connection between the server and the robot device 110, the quantized machine learning model 150 is provided to the robot device 110.

[0042] Terminal devices can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.

[0043] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. A server may include, for example, computing systems / servers such as mainframes, edge computing nodes, computing devices in a cloud environment, etc. It is understood that the model pre-training system 192, the model fine-tuning system 194, and the model application system 196 can be deployed on the same electronic device or on different electronic devices; this disclosure does not limit this.

[0044] It should be understood that Figure 1A , Figure 1B , Figure 1C The components and arrangements shown in the example environment are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model pre-training system 192, the model fine-tuning system 194, and the model application system 196 may be integrated in the same system or device. For example, at least the model pre-training system 192 and the model fine-tuning system 194 may be integrated in a model training system or device. Implementations of this disclosure are not limited in this respect.

[0045] As mentioned above, with the rapid development of artificial intelligence technology, machine learning models are becoming increasingly powerful in terms of reasoning, generalization, and expressive abilities. Significant achievements have been made in various fields such as computer vision, speech recognition, and natural language processing. However, as model performance improves, the number of model parameters also increases, placing higher demands on the computing power and memory capacity of the deployed devices.

[0046] For example, driven by both artificial intelligence (AI) and robotics, Embodied Artificial Intelligence (EAI) has experienced rapid development. EAI integrates AI into physical entities such as robots to create intelligent agents with perception, decision-making, and execution capabilities. Visual Language Model (VLM) and Visual Language Action (VLA) models, which are widely used in EAI, have relatively large parameter counts. These models face challenges in terms of computing power and memory during pre-training, fine-tuning, post-training, edge deployment, and deployment on robotic devices.

[0047] To compress machine learning models and reduce their computational and memory requirements, some traditional techniques employ dynamic quantization. Dynamic quantization calculates quantization parameters (e.g., scaling factors) in real-time during model inference, increasing inference latency. Other traditional techniques use quantization methods such as INT4 (4-bit Integer) or FP8 (8-bit Floating Point) to quantize the model. INT4 quantization converts floating-point parameters in the model to 4-bit positive numbers, with each parameter value represented by only 4 bits. FP8 quantization performs per-tensor or per-channel quantization to convert floating-point parameters (e.g., 32-bit or 16-bit) to 8-bit floating-point numbers. In other words, FP8 quantization determines the scaling factor for the entire tensor or for the entire channel in the weight matrix. Then, the scaling factor is used to transform the parameters of the entire tensor or the entire channel. While INT4 and FP8 quantization methods offer high compression ratios, they are prone to destroying key feature representations of the model and struggle to capture local outliers in the parameters, leading to a decrease in task accuracy and success rate. Therefore, traditional quantization methods cannot maintain both high compression ratios and task accuracy.

[0048] In view of this, an improved scheme for model quantization is provided in the embodiments of this disclosure. In this improved scheme, quantizable model components in a machine learning model are identified, and the weight parameters of the quantizable model components are divided into multiple parameter blocks. Quantization is performed on the multiple parameter blocks using multiple scaling factors corresponding to the multiple parameter blocks to obtain quantized weight parameters. The quantized weight parameters include multiple quantized parameter blocks corresponding to the multiple parameter blocks. Then, a quantized machine learning model is determined based at least on the quantized weight parameters and the multiple scaling factors.

[0049] According to embodiments of this disclosure, quantifiable model components are divided into fine-grained parameter blocks, and a scaling factor is determined for each parameter block. This not only suppresses the impact of local outliers on task accuracy but also has a high compression ratio, reducing the model's requirements for memory and computing power, improving the model's inference speed, and reducing energy consumption during the model's inference process.

[0050] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0051] Figure 2A flowchart of a model quantization process 200 according to some embodiments of the present disclosure is shown. For ease of discussion, process 200 will be described hereinafter from the perspective of model application system 196, but this is merely exemplary. After the machine learning model 150 has been trained, post-training quantization (PTQ) can be performed on the machine learning model 150 using model application system 196. Process 200 can also be implemented on a server, terminal device, or other device if quantization is performed on the model during model pre-training, fine-tuning, or other stages. For example, during model training, process 200 can be implemented in model training system 192, and quantization can be performed on the machine learning model 150 using model pre-training system 192. As another example, during model fine-tuning, process 200 can be implemented in model fine-tuning system 194, and quantization can be performed on the machine learning model 150 using model fine-tuning system 194. Embodiments of the present disclosure are not limiting in this regard.

[0052] In box 210, model application system 196 can determine quantizable model components in machine learning model 150. Machine learning model 150 can be any suitable machine learning model. In some embodiments, machine learning model 150 may include a VLA model or a VLM model. Of course, the machine learning models described above are merely exemplary. Embodiments of this disclosure can be applied to perform quantization processing on any suitable machine learning model. Embodiments of this disclosure do not limit the type of model to be quantized. As an example, after model pre-training system 192 and model fine-tuning system 194 perform pre-training and fine-tuning on a VLA model, post-training quantization (PTQ) can be performed on the trained VLA model. The quantized VLA model is then deployed on robotic device 110 or at the edge.

[0053] Quantizable model components can include functional modules, functional layers, or other suitable components from machine learning model 150. It is understood that some model components in machine learning model 150 are relatively sensitive to accuracy; performing quantization on these components will significantly degrade the task accuracy of machine learning model 150. Other model components in machine learning model 150 are relatively insensitive to accuracy; performing quantization on these components will not easily cause a significant degradation in the task accuracy of machine learning model 150. To balance the task accuracy and compression ratio of machine learning model 150, suitable quantizable model components can be selected from machine learning model 150. Quantizable model components can include model components that, after quantization, will not significantly degrade the task accuracy and have a high compression ratio.

[0054] In some embodiments, the quantizable model component may include a linear layer in the machine learning model 150. This linear layer may include any suitable linear layer, including but not limited to fully connected layers, convolutional layers, and projection layers in attention modules, such as query projection layers (q_proj), key projection layers (k_proj), value projection layers (v_proj), or gated projection layers (gate_proj), etc. Since linear layers account for a relatively high proportion of parameters in the machine learning model 150, and linear layers are relatively less sensitive to numerical precision, performing quantization on linear layers can achieve a balance between task accuracy and compression ratio.

[0055] In some embodiments, the model application system 196 can utilize functions to determine quantifiable model components in the machine learning model 150. As an example, the machine learning model 150 may include a VLA model. The model application system 196 can utilize, for example, a `should_quantize_weight` function to determine quantifiable model components in the VLA model. For instance, at least a portion of the linear layers in a language model (LM) can be identified as quantifiable model components, while visual modules, layer normalization modules, or embedding layers, which are relatively sensitive to numerical precision, can be excluded from the VLA model. In this way, not only can the precision of the visual modules in the VLA model be maintained, but the data volume of the machine learning model can be significantly compressed by quantizing, for example, the linear layers in the language model, and precision alignment of the multimodal model can be achieved.

[0056] It should be understood that the quantifiable model components described above are merely exemplary. Quantifiable model components may also include any other suitable model components in the machine learning model. For example, for the vision module of a VLA model, a functional layer that can be divided into multiple sub-blocks of equal size can be selected from the vision module. It should also be noted that in practical applications, any other suitable method can be used to determine the quantifiable model components. For example, a predetermined type of quantifiable model component can be given in advance, and the model application system 196 can select a functional component matching the predetermined type from the machine learning model 150 as the quantifiable model component. The embodiments of this disclosure do not limit the type and determination method of the quantifiable model components.

[0057] In box 220, the model application system 196 can divide the weight parameters of the quantifiable model components into multiple parameter blocks. Weight parameters, also known as learnable parameters, indicate the mapping relationship between model inputs and model outputs, used to transform model inputs into model outputs. In some embodiments, weight parameters may include multidimensional arrays, such as two-dimensional matrices or multidimensional tensors. Of course, the data format of the weight parameters may differ depending on the type of machine learning model 150 or the type of quantifiable model components.

[0058] A parameter block can be understood as a structured unit formed by dividing weight parameters, also known as a "sub-block". Dividing quantifiable model components into "sub-blocks" can accurately suppress the impact of local outliers in the weight parameters on accuracy. In some embodiments, the model application system 196 can divide the weight parameters into multiple parameter blocks according to a predetermined size. That is, the model application system 196 divides the weight parameters into multiple parameter blocks of a predetermined size. As an example, the weight parameters of the quantifiable model component may include a weight matrix W. The model application system 196 can divide the weight matrix into multiple N×N (e.g., 128×128, 64×64, or 256×256, etc.) parameter blocks, where N is a positive integer greater than 1. Alternatively, the model application system 196 can also divide the weight parameters into multiple parameter blocks of different sizes according to the data structure of the weight parameters. The embodiments of this disclosure do not limit this.

[0059] In box 230, model application system 196 can perform quantization on multiple parameter blocks using multiple scaling factors corresponding to multiple parameter blocks to obtain quantized weight parameters. The quantized weight parameters include multiple quantized parameter blocks corresponding to the multiple parameter blocks.

[0060] A scaling factor is used to convert the original parameter block into a quantized parameter block. The scaling factor indicates the conversion ratio between the parameter values ​​in the original parameter block and the parameter values ​​in the quantized parameter block. Each parameter block in a set of parameters can have a corresponding scaling factor; for example, parameter block block1 can have a corresponding scaling factor S1, parameter block2 can have a corresponding scaling factor S2, and so on. In some embodiments, the quantizable model component can have a corresponding scaling factor matrix, which can include multiple scaling factors, and the distribution of these scaling factors in the scaling factor matrix corresponds to the distribution of the multiple parameter blocks in the weight parameters of the quantizable model component. Of course, these scaling factors can also be stored in any other suitable data structure.

[0061] In practical applications, scaling factors can be determined in various ways. The following examples illustrate these methods. In some embodiments, the model application system 196 can determine multiple scaling factors corresponding to multiple parameter blocks based on the distribution of parameter values ​​within those blocks. In other words, for each parameter block, the model application system 196 can determine the distribution characteristics or value patterns of the parameter values ​​within that block. The scaling factor for that parameter block is then determined based on these distribution characteristics or value patterns. For example, the scaling factor S2 is determined based on the distribution characteristics or value patterns of parameter block sub-block2.

[0062] In some embodiments, the distribution of parameter values ​​in a parameter block can indicate the maximum absolute value of the parameter values ​​in the parameter block. For each parameter block among multiple parameter blocks, the model application system 196 can determine the maximum absolute value of the parameter values ​​in the parameter block. The model application system 196 can also determine the maximum finite value corresponding to the quantization format based on the quantization format of the quantizable model component. Subsequently, the model application system 196 can determine the scaling factor corresponding to the parameter block based on the maximum absolute value and the maximum finite value.

[0063] The quantization format indicates the data structure of the quantized parameter values. This quantization format can be a pre-defined format or a specific format that matches the machine learning model 150 or the quantizable model components. For example, the quantization format of the quantizable model components can be determined based on the metadata of the machine learning model 150 or information related to the quantizable model components (such as the type of the model components, the data structure of the weight parameters, etc.).

[0064] In some examples, the quantization format may include, but is not limited to, FP8, FP6 (6-bit Floating Point), or INT8 (8-bit Integer) formats. The FP8 format uses 8 bits (i.e., 1 byte) to represent a floating-point number; that is, the quantized parameter value can be a floating-point number represented by 8 bits. FP8 formats include E4M3 and E5M2 formats. E4M3 includes 1 sign bit, 4 exponent bits, and 3 mantissa bits; E5M2 includes 1 sign bit, 5 exponent bits, and 2 mantissa bits. The FP6 format uses 6 bits to represent a floating-point number (i.e., a parameter value), including a sign bit, exponent bit, and mantissa bit. The INT8 format uses 8 bits to represent an integer value; that is, the quantized parameter value can be an integer value represented by 8 bits. It is understood that different quantization formats have different maximum finite values; for example, the maximum finite value of the E4M3 format in FP8 is 448, and the maximum finite value of the INT8 format is 127, and so on.

[0065] In some examples, model application system 196 can determine the scaling factor corresponding to the parameter block based on the ratio between the maximum absolute value and the maximum finite value. As an example, model application system 196 can determine the scaling factor using the following formula: (1) in Indicates the scaling factor. Indicates a parameter block. Represents the maximum absolute value. This represents the maximum finite value of the quantization format corresponding to the quantizable model component. Assuming the quantization format is E4M3, then the range of the E4M3 format is [-448, 448]. .

[0066] In some embodiments, the distribution of parameter values ​​in a parameter block can also indicate the numerical range of parameter values ​​within the parameter block; for example, the distribution can indicate the maximum and minimum parameter values ​​within the parameter block. The model application system 196 can determine a first absolute value corresponding to the maximum parameter value and a second absolute value corresponding to the minimum parameter value, and determine the maximum absolute value between the first and second absolute values. Then, the model application system 196 can determine a scaling factor based on the maximum absolute value and the maximum finite value of the quantization format of the quantizable model component. Alternatively, the model application system 196 can determine a typical numerical range of the parameter block (e.g., the range in which 99% or 99.9% of the parameter values ​​fall), and determine the maximum value within the typical numerical range as the maximum effective value. Then, the model application system 196 can determine the scaling factor corresponding to the parameter block based on the ratio between the maximum effective value and the maximum finite value of the quantization format. It should be understood that the above method for determining the scaling factor is merely exemplary, and any appropriate method can be selected to determine the scaling factor according to actual needs. The embodiments of this disclosure do not limit this.

[0067] In some embodiments, after determining multiple scaling factors corresponding to multiple parameter blocks, the model application system 196 can perform quantization on the multiple parameter blocks using the multiple scaling factors to obtain quantized weight parameters. Specifically, for each parameter block among the multiple parameter blocks, the model application system 196 can perform quantization processing on the parameter block using the scaling factor corresponding to that parameter block to obtain a quantized parameter block. For example, the model application system 196 can perform quantization processing on parameter block block1 using scaling factor S1 to obtain quantized parameter block block1. The model application system 196 can also perform quantization processing on parameter block block2 using scaling factor S2 to obtain quantized parameter block block2.

[0068] In some examples, model application system 196 can perform quantization on parameter blocks using the following formula: (2) in This represents the quantized parameter block. This indicates the quantization format. This represents the transformation function. Combining formula (2), it can be seen that the model application system 196 can determine the ratio of the parameter block to the scaling factor. Then, the model application system 196 can utilize the transformation function... The ratio is converted into a quantization format suitable for quantizable model components (e.g., FP8 or INT8) to obtain quantized parameter blocks. It is understood that the model application system 196 can perform quantization on each parameter block based on the above formula and using the scaling factor corresponding to each parameter block, to obtain multiple quantized parameter blocks as quantized weight parameters. Of course, the above quantization process is merely exemplary. In practical applications, any other suitable quantization method can be used to quantize the parameter blocks. The embodiments of this disclosure do not limit this.

[0069] In box 240, the model application system 196 determines the quantized machine learning model 150 based at least on the quantized weight parameters and multiple scaling factors.

[0070] In some embodiments, the model application system 196 may replace the weight parameters in the machine learning model with quantized weight parameters and multiple scaling factors to obtain a quantized machine learning model. The model application system 196 may determine tagging information related to the quantization of quantizable model components. Then, the model application system 196 may determine the metadata of the quantized machine learning model based on the tagging information related to the quantization of the quantizable model components. The tagging information may indicate at least one of the following: quantizable model components, unquantized model components in the quantized machine learning model 150, the quantization format of the quantizable model components, or the size of the parameter block. For example, the tagging information may include the identifier of the quantizable model components, the identifier of the unquantized model components in the machine learning model 150, an indication of the quantization format (e.g., FP8, INT8, etc.) or the size of the parameter block (e.g., N×N), etc.

[0071] As an example, model application system 196 can replace the weight parameters of the quantizable model components in machine learning model 150 with quantized weight parameters to obtain a quantized machine learning model 150. Model application system 196 can package the scaling factor matrix and the quantized machine learning model 150 into a predefined format (e.g., safetensors format or other formats). Then, model application system 196 can generate a configuration file for the quantized machine learning model 150 based on the labeling information. The configuration file can contain metadata about the quantized machine learning model 150. For example, model application system 196 can use a function (e.g., the `quantize_model` function) to generate a configuration file such as `config.json` based on the labeling information. In this way, it is convenient to perform inference using the scaling factor and quantized weight parameters during the model inference process, or to perform dequantization on the quantized weight parameters.

[0072] In practical applications, some robotic devices 110 or other terminal devices deploying the machine learning model 150 may have the ability to directly perform inference using the scaling factor and quantized weight parameters. In this case, the model application system 196 may not need to perform dequantization on the quantized machine learning model 150. Conversely, some robotic devices 110 or terminal devices may not have the ability to directly perform inference using the scaling factor and quantized weight parameters. In this case, the model application system 196 can perform dequantization on the machine learning model 150 to obtain a dequantized machine learning model 150. In this way, the machine learning model 150 can be adapted to these robotic devices 110 or terminal devices.

[0073] In some embodiments, the model application system 196 can perform dequantization on multiple quantized parameter blocks in the quantized machine learning model 150 using multiple scaling factors to obtain dequantized weight parameters. Then, the model application system 196 can replace the multiple quantized parameter blocks in the quantized machine learning model 150 with the dequantized weight parameters to obtain a dequantized machine learning model 150. For example, for each parameter block in the multiple quantized parameter blocks, the model application system 196 can perform dequantization on the parameter block using the following formula: (3) in This represents the dequantized parameter block. It is understood that the model application system 196 can use the above formula (3) to perform dequantization on multiple quantized parameter blocks respectively, for example, restoring an FP8 format parameter block to FP16, FP32, or other formats. Alternatively, the model application system 196 can also determine multiple inverse scaling factors based on multiple scaling factors. The inverse scaling factors are used to perform dequantization (or "restoration") on the quantized parameter blocks. Afterwards, the model application system 196 can use the multiple inverse scaling factors to perform dequantization on multiple quantized parameter blocks to obtain multiple dequantized parameter blocks.

[0074] In summary, the embodiments of this disclosure divide the quantifiable model components into fine-grained parameter blocks and determine a scaling factor for each parameter block. This not only suppresses the impact of local outliers on task accuracy but also achieves a high compression ratio, thereby reducing the model's requirements for memory and computing power, improving the model's inference speed, and reducing energy consumption during the model's inference process.

[0075] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 3 A schematic structural block diagram of an apparatus 300 for model quantization according to some embodiments of the present disclosure is shown. The apparatus 300 may be implemented as or included in a model application system 196. Various modules / components in the apparatus 300 may be implemented by hardware, software, firmware, or any combination thereof.

[0076] The apparatus 300 includes: a first determining module 310 configured to determine quantizable model components in a machine learning model; a partitioning module 320 configured to partition the weight parameters of the quantizable model components into multiple parameter blocks; a quantization module 330 configured to perform quantization on the multiple parameter blocks using multiple scaling factors corresponding to the multiple parameter blocks to obtain quantized weight parameters, the quantized weight parameters including multiple quantized parameter blocks corresponding to the multiple parameter blocks; and a second determining module 340 configured to determine a quantized machine learning model based at least on the quantized weight parameters and the multiple scaling factors.

[0077] In some embodiments, the quantization module 330 is further configured to: determine multiple scaling factors corresponding to the multiple parameter blocks based on the distribution of parameter values ​​in the multiple parameter blocks; and perform quantization on the multiple parameter blocks using the multiple scaling factors to obtain quantized weight parameters.

[0078] In some embodiments, the quantization module 330 is further configured to: for each of a plurality of parameter blocks, determine the maximum absolute value of the parameter values ​​in the parameter block and the maximum finite value corresponding to the quantization format of the quantizable model component; and determine the scaling factor corresponding to the parameter block based on the maximum absolute value and the maximum finite value.

[0079] In some embodiments, the quantization module 330 is further configured to determine a scaling factor corresponding to the parameter block based on the ratio between the maximum absolute value and the maximum finite value.

[0080] In some embodiments, the machine learning model includes a visual language model or a visual language action model.

[0081] In some embodiments, the quantifiable model component includes at least a linear layer in the machine learning model.

[0082] In some embodiments, the partitioning module 320 is further configured to divide the weight parameters into multiple parameter blocks according to a predetermined size.

[0083] In some embodiments, the second determining module 340 is further configured to: replace the weight parameters in the machine learning model with quantized weight parameters and multiple scaling factors to obtain a quantized machine learning model; and determine the metadata of the quantized machine learning model based on the tagging information related to the quantization of the quantizable model components, wherein the tagging information indicates at least one of the following: quantizable model components, unquantized model components in the quantized machine learning model, the quantization format of the quantizable model components, or the size of the parameter block.

[0084] In some embodiments, the apparatus 300 further includes: a dequantization module configured to perform dequantization on a plurality of quantized parameter blocks in a quantized machine learning model using a plurality of scaling factors to obtain dequantized weight parameters; and to replace the plurality of quantized parameter blocks in the quantized machine learning model with the dequantized weight parameters to obtain a dequantized machine learning model.

[0085] The modules included in device 300 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 300 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0086] Figure 4 A block diagram of an electronic device 400 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 4The electronic device 400 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0087] like Figure 4 As shown, electronic device 400 is in the form of a general-purpose electronic device. Components of electronic device 400 may include, but are not limited to, one or more processors or processing units 410, memory 420, storage device 430, one or more communication units 440, one or more input devices 450, and one or more output devices 450. Processing unit 410 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 420. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 400.

[0088] Electronic device 400 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 400, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 420 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 430 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 400.

[0089] Electronic device 400 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 4 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 420 may include computer program product 425 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0090] The communication unit 440 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 400 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 400 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0091] Input device 450 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 450 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 400 can also communicate with one or more external devices (not shown) via communication unit 440 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 400, or with any device that enables electronic device 400 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0092] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transient computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the methods described above.

[0093] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0095] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0097] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A model quantization method, comprising: Identify quantifiable model components in a machine learning model; The weight parameters of the quantifiable model components are divided into multiple parameter blocks; The plurality of parameter blocks are quantized using a plurality of scaling factors corresponding to the plurality of parameter blocks to obtain quantized weight parameters, wherein the quantized weight parameters include a plurality of quantized parameter blocks corresponding to the plurality of parameter blocks; as well as The quantized machine learning model is determined based at least on the quantized weight parameters and the plurality of scaling factors.

2. The method of claim 1, wherein performing quantization on the plurality of parameter blocks comprises: Based on the distribution of parameter values ​​in the plurality of parameter blocks, determine the plurality of scaling factors corresponding to the plurality of parameter blocks; as well as The multiple parameter blocks are quantized using the multiple scaling factors to obtain the quantized weight parameters.

3. The method of claim 2, wherein determining the plurality of scaling factors comprises: For each of the plurality of parameter blocks, determine the maximum absolute value of the parameter values ​​in the parameter block and the maximum finite value corresponding to the quantization format of the quantizable model component; as well as Based on the maximum absolute value and the maximum finite value, determine the scaling factor corresponding to the parameter block.

4. The method of claim 3, wherein determining the scaling factor corresponding to the parameter block comprises: The scaling factor corresponding to the parameter block is determined based on the ratio between the maximum absolute value and the maximum finite value.

5. The method according to claim 1, wherein the machine learning model includes a visual language model or a visual language action model.

6. The method of claim 1, wherein the quantifiable model component comprises at least a linear layer in the machine learning model.

7. The method according to claim 1, wherein dividing the weight parameters into the plurality of parameter blocks comprises: The weight parameters are divided into multiple parameter blocks according to a predetermined size.

8. The method of claim 1, wherein determining the quantized machine learning model comprises: The quantized machine learning model is obtained by replacing the weight parameters in the machine learning model with the quantized weight parameters and the plurality of scaling factors. as well as Based on the tagging information related to the quantization of the quantizable model components, the metadata of the quantized machine learning model is determined, wherein the tagging information indicates at least one of the following: the quantizable model components, unquantized model components in the quantized machine learning model, the quantization format of the quantizable model components, or the size of the parameter block.

9. The method according to claim 1, further comprising: The multiple scaling factors are used to perform dequantization on the multiple quantized parameter blocks in the quantized machine learning model to obtain the dequantized weight parameters. as well as The dequantized machine learning model is obtained by replacing the multiple quantized parameter blocks in the quantized machine learning model with the dequantized weight parameters.

10. An apparatus for model quantization, comprising: The first determining module is configured to determine the quantifiable model components in the machine learning model; The partitioning module is configured to divide the weight parameters of the quantifiable model components into multiple parameter blocks; The quantization module is configured to perform quantization on the plurality of parameter blocks using a plurality of scaling factors corresponding to the plurality of parameter blocks to obtain quantized weight parameters, wherein the quantized weight parameters include a plurality of quantized parameter blocks corresponding to the plurality of parameter blocks; as well as The second determining module is configured to determine the quantized machine learning model based at least on the quantized weight parameters and the plurality of scaling factors.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory is coupled to at least one processing unit and stores instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 9.

Images