Model quantification method, device and equipment based on artificial intelligence development platform

Abstract

The invention relates to the technical field of model quantification, and discloses a model quantification method, device and equipment based on an artificial intelligence development platform, and the method comprises the steps: loading a calibration data set; extracting content of a specified column from the loaded calibration data set, and splicing texts corresponding to the extracted content into a long text; converting the spliced long text into a model input format by using a word segmentation device; intercepting fragments from the data corresponding to the model input format for multiple times, and generating corresponding attention masks for the intercepted fragments to obtain a data set meeting a preset condition; using preset parameters to construct a quantitative configuration object; and based on the quantitative configuration object, quantizing the data set meeting a preset condition to generate a quantized model. According to the method and the device, the operation of model quantification is simplified, the use difficulty is reduced, a user can complete model quantification only by setting necessary parameters without any coding, the cumbersome work that the user needs to reconfigure in each quantification task is reduced, and the efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of model quantization technology, for example, to a model quantization method, device, and equipment based on an artificial intelligence development platform. Background Art

[0002] Model quantization is a method to optimize machine learning models by reducing the model's parameter precision to reduce the model's storage usage and accelerate the inference process. With the rapid development of deep learning technology, the size and complexity of models continue to increase, resulting in tremendous pressure on computing resources and storage resources. Therefore, how to compress the model size and improve the inference efficiency without significantly reducing the model performance has become an important research direction in deep learning research. As an effective model optimization method, model quantization has achieved remarkable results in applications in many fields, especially in edge devices, embedded devices, and mobile devices, where quantization technology is particularly important.

[0003] In the context of the rapid development of deep learning technology, as the scale and complexity of models continue to increase, model optimization has become a key area in research and application. As an effective optimization method, model quantization has been widely used in various deep learning tasks, especially on resource-constrained hardware platforms (such as mobile devices, embedded devices, edge computing devices, etc.). Quantization technology can significantly improve the inference speed of the model and reduce the consumption of computing resources. However, although quantization technology has made significant progress, current quantization tools and technologies still have many shortcomings, especially in terms of ease of use, flexibility and efficiency.

[0004] Therefore, in the existing model quantization process, there are problems with ease of use, flexibility and efficiency.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application. Summary of the invention

[0006] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0007] The present disclosure provides a model quantization method based on an artificial intelligence development platform, the method comprising: Load the calibration dataset using the preset calib_data_path and calib_data_split parameters; Extract the contents of the specified column from the loaded calibration data set and concatenate the text corresponding to the extracted content into a long text, where the column name of the specified column is provided by the calib_data_split parameter; Use a tokenizer to convert the concatenated long text into the model input format; Extract fragments from the data corresponding to the model input format multiple times, generate corresponding attention masks for the extracted fragments, and concatenate the extracted fragments and attention masks into a dictionary and put them into the calibration dataset array to obtain a dataset dataset that meets the preset conditions; Use the preset calib_data_column parameter to construct a quantization configuration object, where the quantization configuration object is used to specify the detailed parameters of the model quantization; Based on the quantization configuration object, the dataset dataset that meets the preset conditions is quantized to generate a quantized model.

[0008] In some embodiments, the method may further include: Use the org_model_path parameter to pass the storage path of the initial model; Use the output_model_path parameter to pass the storage path of the quantized model; Use the calib_data_path parameter to pass the path or name of the calibration dataset to load the calibration data; Use the calib_data_split parameter to pass the split of the specified dataset; Use the calib_data_column parameter to pass the column name containing the text data in the dataset; Use the bits parameter to pass the quantization bit width of the model.

[0009] In some embodiments, the model input format is a dictionary including input_ids and fields.

[0010] In some embodiments, the above-mentioned quantization configuration object is based on quantization, and the data set dataset that meets the preset conditions is quantized to generate a quantized model, including: Based on the quantization configuration object, the quantization operation is performed on the dataset dataset that meets the preset conditions, the floating point precision weights are converted, and the quantized model is generated.

[0011] In some embodiments, the method may further include: Save the preset tokenizer to the directory passed by the output_model_path parameter, and save the tokenizer configuration, vocabulary, and files to the target directory.

[0012] In some embodiments, the method may further include: Receive quantization task information including model quantization task name, source model, quantization bit width, quantization data set, quantization data set split, and quantization data set field name input by the user based on the visualization front-end page; Based on the quantization task information, determine the CPU, GPU, and memory resources required for the quantization task, and store the quantized model.

[0013] The embodiment of the present disclosure provides a model quantization device based on an artificial intelligence development platform, the device comprising: Loading module, used to load the calibration dataset using the preset calib_data_path parameter and calib_data_split parameter; The extraction module is used to extract the content of the specified column from the loaded calibration data set and concatenate the text corresponding to the extracted content into a long text, where the column name of the specified column is provided by the calib_data_split parameter; The conversion module is used to convert the concatenated long text into the model input format based on the word segmenter; The interception module is used to intercept fragments from the data corresponding to the model input format multiple times, generate corresponding attention masks for the intercepted fragments, and concatenate the intercepted fragments and attention masks into a dictionary and put them into the calibration dataset array to obtain a dataset dataset that meets the preset conditions; A construction module is used to construct a quantization configuration object using the preset calib_data_column parameters, wherein the quantization configuration object is used to specify the detailed parameters of the model quantization; The generation module is used to quantize the dataset dataset that meets the preset conditions based on the quantization configuration object and generate a quantized model.

[0014] In some embodiments, the model input format is a dictionary including input_ids and fields.

[0015] An embodiment of the present disclosure provides an electronic device, the device comprising at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the above-mentioned model quantization method based on the artificial intelligence development platform.

[0016] An embodiment of the present disclosure provides a storage medium storing program instructions, which, when run, execute the above-mentioned model quantization method based on the artificial intelligence development platform.

[0017] The model quantization method, device, equipment and storage medium based on the artificial intelligence development platform provided by the embodiments of the present disclosure can achieve the following technical effects: The present disclosure simplifies the operation of model quantization and reduces the difficulty of use. Users do not need any coding and only need to set the necessary parameters to complete model quantization, which reduces the tedious work of users having to reconfigure in each quantization task and improves efficiency.

[0018] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] One or more embodiments are exemplarily described by corresponding drawings, which do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements, and the drawings do not constitute a scale limitation, and wherein: Figure 1 It is a flow chart of a model quantization method based on an artificial intelligence development platform provided in an embodiment of the present disclosure; Figure 2 It is a structural schematic diagram of a model quantization device based on an artificial intelligence development platform provided in an embodiment of the present disclosure; Figure 3 It is a structural schematic diagram of a model quantization device based on an artificial intelligence development platform provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In order to be able to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0021] The terms "first", "second", etc. in the embodiments of the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so as to describe the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0022] Unless otherwise stated, the term "plurality" means two or more.

[0023] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0024] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0025] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0026] In order to solve the above problems, the present disclosure provides a model quantization method, device, equipment and storage medium based on an artificial intelligence development platform.

[0027] The following describes the model quantization method, device, equipment and storage medium based on the artificial intelligence development platform provided by the embodiments of the present disclosure in conjunction with the accompanying drawings.

[0028] Figure 1 It is a flow chart of a model quantization method based on an artificial intelligence development platform provided in an embodiment of the present disclosure.

[0029] Combination Figure 1 As shown, the model quantification method based on the artificial intelligence development platform may include: S101, load the calibration data set using the preset calib_data_path parameter and calib_data_split parameter; S102, extracting the content of a specified column from the loaded calibration data set, and concatenating the text corresponding to the extracted content into a long text, wherein the column name of the specified column is provided by the calib_data_split parameter; S103, using a word segmenter to convert the concatenated long text into a model input format; S104, extracting segments from the data corresponding to the model input format multiple times, generating corresponding attention masks for the extracted segments, and concatenating the extracted segments and the attention masks into a dictionary and putting them into a calibration dataset array to obtain a dataset dataset that meets preset conditions; S105, constructing a quantization configuration object using a preset calib_data_column parameter, wherein the quantization configuration object is used to specify detailed parameters of the model quantization; S106, based on the quantization configuration object, quantize the data set dataset that meets the preset conditions to generate a quantized model.

[0030] In some embodiments, Figure 1 The method may also include: Use the org_model_path parameter to pass the storage path of the initial model; Use the output_model_path parameter to pass the storage path of the quantized model; Use the calib_data_path parameter to pass the path or name of the calibration dataset to load the calibration data; Use the calib_data_split parameter to pass the split of the specified dataset; Use the calib_data_column parameter to pass the column name containing the text data in the dataset; Use the bits parameter to pass the quantization bit width of the model.

[0031] In some embodiments, the model input format is a dictionary consisting of input_ids and fields.

[0032] In some embodiments, based on the quantization configuration object, a data set dataset that meets the preset conditions is quantized to generate a quantized model, including: Based on the quantization configuration object, the quantization operation is performed on the dataset dataset that meets the preset conditions, the floating point precision weights are converted, and the quantized model is generated.

[0033] In some embodiments, Figure 1 The method may also include: Save the preset tokenizer to the directory passed by the output_model_path parameter, and save the tokenizer configuration, vocabulary, and files to the target directory.

[0034] In some embodiments, Figure 1 The method may also include: Receive quantization task information including model quantization task name, source model, quantization bit width, quantization data set, quantization data set split, and quantization data set field name input by the user based on the visualization front-end page; Based on the quantization task information, determine the CPU, GPU, and memory resources required for the quantization task, and store the quantized model.

[0035] In a specific example, starting with making a Docker image, Figure 1 The model quantization method based on the artificial intelligence development platform is further described in .

[0036] Creating a Docker Image 1. Create a Docker image based on the Python environment, FROM python:3.10.

[0037] 2. Install necessary Python libraries, such as transformers, numpy, auto-gptq, etc., pip install transformers, numpy, auto-gptq.

[0038] 3. Write Python code to parse command line parameters.

[0039] 3.1 Use the --org_model_path parameter to pass the storage path of the initial model.

[0040] 3.2 Use the --output_model_path parameter to pass the storage path of the quantized model.

[0041] 3.3 Use the --calib_data_path parameter to pass the path or name of the calibration dataset to load the calibration data.

[0042] 3.4 Use the --calib_data_split parameter to pass the specified data set split. Common splits include train, test, validation, etc.

[0043] 3.5 Use the --calib_data_column parameter to pass the name of the column in the dataset that contains the text data. Most datasets may contain multiple columns, such as ID, label, and text. This parameter explicitly tells the program which column should be used to extract the text.

[0044] 3.6 Use the -bit parameter to pass the quantization bit width of the model.

[0045] 4. Load the calibration dataset according to the parameters provided in steps 3.3 and 3.4.

[0046] 5. Extract the contents of the specified column from the loaded dataset. The specified column name is provided by the parameter in step 3.4. And concatenate all the text in the column with two line breaks into a long text 6. Use the tokenizer to convert the concatenated long text into the model input format, which is a dictionary including input_ids and other possible fields (such as attention_mask).

[0047] 7. Extract the appropriate length from the input_ids field in the dictionary, generate the corresponding attention mask for the extracted segment, concatenate the extracted segment and attention mask into a dictionary and put it into the calibration dataset array. Repeat this process multiple times to obtain the final usable dataset dataset.

[0048] 8. Construct a quantization configuration object based on the parameters in step 3.5. The quantization configuration object is used to specify the detailed parameters of the model quantization. These parameters will directly affect the performance of the quantized model, including inference speed, model size, and accuracy.

[0049] 9. Load the pre-trained model and apply the quantization configuration, while configuring the maximum memory usage.

[0050] 10. Quantize the given dataset (dataset in step 7). Perform the quantization operation of the model to convert the floating point precision weights to low precision (such as 4-bit, 8-bit), with the aim of reducing the model size and accelerating reasoning while maintaining the smallest possible accuracy loss.

[0051] 11. Save the quantized model to the specified directory (the parameter directory passed in step 3.2), save the quantized model weights to disk, and choose to save the model file in safetensors format to improve security and efficiency.

[0052] 12. Save the tokenizer to the specified directory (the parameter directory passed in step 3.2), and save the tokenizer configuration, vocabulary, and other necessary files (such as special symbols, word segmentation rules, etc.) to the specified directory.

[0053] 13. Save the Python code as app.py and copy it to the / app directory of the Docker image.

[0054] 14. Set the image startup command to python / app / app.py.

[0055] 15.Package the Docker image and push it to the image repository for use.

[0056] Visual front end 16. The visualization front-end page receives user input, including the model quantization task name, source model, quantization bit width, quantization data set, quantization data set split, and quantization data set field name.

[0057] 17. The visualization front end sets the CPU, GPU, and memory resources required for the quantization task.

[0058] 18. The visualization front end provides the function of exporting quantization results and stores the quantized model in object storage.

[0059] K8s Scheduling 19. Assemble the k8s pod according to the user input in steps 16 and 17. The pod uses the image packaged in step 15, matches the user input in step 16 with the user parameters received by the python code, and sets the resources required by the pod according to step 17 20. Mount the quantized dataset selected by the user to the / dataset directory of the pod as a data volume, mount the source model selected by the user to the / mnt / models directory of the pod, and mount the quantized output model to the / output directory of the pod.

[0060] 21. The user creates a model quantization task in the visual front-end interface and passes in the necessary parameters. The scheduling module then creates a pod to execute the model quantization task, and the quantized model is saved in a fixed directory.

[0061] The present disclosure simplifies the operation of model quantization and reduces the difficulty of use. Users do not need any coding, and only need to set the necessary parameters to complete the model quantization. It reduces the tedious work of users to reconfigure in each quantization task and improves efficiency. It automatically manages the allocation and scheduling of computing resources to ensure that tasks obtain appropriate hardware support during execution. It provides a real-time visual interface, and users can intuitively see the progress and effects of quantization tasks and the adjustment of various parameters.

[0062] and Figure 1 Corresponding to the model quantization method based on the artificial intelligence development platform, the present disclosure also provides a model quantization device based on the artificial intelligence development platform, such as Figure 2 As shown, the device comprises: A loading module 201 is used to load a calibration data set using a preset calib_data_path parameter and a preset calib_data_split parameter; An extraction module 202 is used to extract the content of a specified column from the loaded calibration data set, and to splice the text corresponding to the extracted content into a long text, wherein the column name of the specified column is provided by the calib_data_split parameter; A conversion module 203, used for converting the concatenated long text into a model input format based on a word segmenter; The interception module 204 is used to intercept segments from the data corresponding to the model input format multiple times, generate corresponding attention masks for the intercepted segments, and splice the intercepted segments and the attention masks into a dictionary and put them into the calibration dataset array to obtain a dataset dataset that meets the preset conditions; A construction module 205 is used to construct a quantization configuration object using a preset calib_data_column parameter, wherein the quantization configuration object is used to specify detailed parameters of the model quantization; The generation module 206 is used to quantize the data set dataset that meets the preset conditions based on the quantization configuration object to generate a quantized model.

[0063] In some embodiments, Figure 2 In the device shown, the storage path of the initial model is passed using the org_model_path parameter; Use the output_model_path parameter to pass the storage path of the quantized model; Use the calib_data_path parameter to pass the path or name of the calibration dataset to load the calibration data; Use the calib_data_split parameter to pass the split of the specified dataset; Use the calib_data_column parameter to pass the column name containing the text data in the dataset; Use the bits parameter to pass the quantization bit width of the model.

[0064] In some embodiments, the model input format is a dictionary including input_ids and fields.

[0065] In some embodiments, the above-mentioned quantization configuration object is based on quantization, and the data set dataset that meets the preset conditions is quantized to generate a quantized model, including: Based on the quantization configuration object, the quantization operation is performed on the dataset dataset that meets the preset conditions, the floating point precision weights are converted, and the quantized model is generated.

[0066] In some embodiments, Figure 2 In the device shown, the preset tokenizer can also be saved to the directory passed by the output_model_path parameter, and the configuration, vocabulary and files of the tokenizer can be saved to the target directory.

[0067] In some embodiments, Figure 2 In the device shown, quantization task information including model quantization task name, source model, quantization bit width, quantization data set, quantization data set split, and quantization data set field name input by the user can also be received based on the visualization front-end page; Based on the quantization task information, determine the CPU, GPU, and memory resources required for the quantization task, and store the quantized model.

[0068] Combination Figure 3As shown, the embodiment of the present disclosure also provides a model quantization device 300 based on an artificial intelligence development platform, including a processor 304 and a memory 301. Optionally, the system may also include a communication interface 302 and a bus 303. Among them, the processor 304, the communication interface 302, and the memory 301 can communicate with each other through the bus 303. The communication interface 302 can be used for information transmission. The processor 304 can call the logic instructions in the memory 301 to execute the model quantization method based on the artificial intelligence development platform of the above embodiment.

[0069] In addition, the logic instructions in the memory 301 described above can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0070] The memory 301 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor 304 executes the function application and data processing by running the program instructions / modules stored in the memory 301, that is, the model quantization method based on the artificial intelligence development platform in the above embodiment is implemented.

[0071] The memory 301 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 301 may include a high-speed random access memory and may also include a non-volatile memory.

[0072] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as a model quantization method based on an artificial intelligence development platform.

[0073] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

Claims

1. A model quantification method based on an artificial intelligence development platform, characterized in that: The method comprises: Load the calibration dataset using the preset calib_data_path and calib_data_split parameters; Extract the content of the specified column from the loaded calibration data set, and concatenate the text corresponding to the extracted content into a long text, wherein the column name of the specified column is provided by the calib_data_split parameter; Use a tokenizer to convert the concatenated long text into the model input format; Extract fragments from the data corresponding to the model input format multiple times, generate corresponding attention masks for the extracted fragments, and concatenate the extracted fragments and attention masks into a dictionary and put them into the calibration dataset array to obtain a dataset dataset that meets the preset conditions; Use the preset calib_data_column parameter to construct a quantization configuration object, wherein the quantization configuration object is used to specify detailed parameters of model quantization; Based on the quantization configuration object, the dataset dataset that meets the preset conditions is quantized to generate a quantized model.

2. The method according to claim 1, characterized in that The method comprises: Use the org_model_path parameter to pass the storage path of the initial model; Use the output_model_path parameter to pass the storage path of the quantized model; Use the calib_data_path parameter to pass the path or name of the calibration dataset to load the calibration data; Use the calib_data_split parameter to pass the split of the specified dataset; Use the calib_data_column parameter to pass the column name containing the text data in the dataset; Use the bits parameter to pass the quantization bit width of the model.

3. The method according to claim 1, characterized in that The model input format is a dictionary including input_ids and fields.

4. The method according to claim 1, characterized in that: The method of quantizing a dataset satisfying preset conditions based on the quantization configuration object to generate a quantized model includes: Based on the quantization configuration object, the quantization operation is performed on the dataset dataset that meets the preset conditions, the floating point precision weights are converted, and the quantized model is generated.

5. The method according to claim 2, characterized in that: The method further comprises: Save the preset tokenizer to the directory passed by the output_model_path parameter, and save the tokenizer configuration, vocabulary, and files to the target directory.

6. The method according to claim 1, characterized in that The method further comprises: Receive quantization task information including model quantization task name, source model, quantization bit width, quantization data set, quantization data set split, and quantization data set field name input by the user based on the visualization front-end page; Based on the quantization task information, the CPU, GPU, and memory resources required for the quantization task are determined, and the quantized model is stored.

7. A model quantification device based on an artificial intelligence development platform, characterized in that: The device comprises: Loading module, used to load the calibration dataset using the preset calib_data_path parameter and calib_data_split parameter; An extraction module, used to extract the content of a specified column from the loaded calibration data set, and splice the text corresponding to the extracted content into a long text, wherein the column name of the specified column is provided by the calib_data_split parameter; The conversion module is used to convert the concatenated long text into the model input format based on the word segmenter; The interception module is used to intercept fragments from the data corresponding to the model input format multiple times, generate corresponding attention masks for the intercepted fragments, and concatenate the intercepted fragments and attention masks into a dictionary and put them into the calibration dataset array to obtain a dataset dataset that meets the preset conditions; A construction module, used to construct a quantization configuration object using a preset calib_data_column parameter, wherein the quantization configuration object is used to specify detailed parameters of model quantization; The generation module is used to quantize the dataset dataset that meets the preset conditions based on the quantization configuration object and generate a quantized model.

8. The device according to claim 7, characterized in that The model input format is a dictionary including input_ids and fields.

9. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.

Images