Calibration dataset generation method, model quantization method, apparatus, device, and medium
By generating calibration text and images, the problem of image data dependence in the quantization of multimodal large language models is solved, achieving efficient and low-cost quantization processing and improving the model's generalization ability and accuracy.
Patent Information
- Application Number
- CN202610686505.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-18
- Publication Date
- 2026-07-14
Smart Images
Figure CN122390033A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to model quantization techniques, and in particular to a method for generating calibration datasets, a model quantization method, apparatus, device, and medium. Background Technology
[0002] In quantization of Multimodal Large Language Models (MLLM), existing image datasets are typically used directly as quantization calibration sets. These calibration sets are then input into the model to be quantized. However, this type of calibration set is highly dependent on real-world image data, resulting in high costs for image acquisition and annotation, as well as restrictions related to copyright, privacy, and licensing. This limits the applicability and scalability of the quantization method in real-world deployment scenarios. Furthermore, existing publicly available image datasets are often constructed for specific evaluation tasks, exhibiting significant task biases in image type, text format, and semantic distribution. This can easily lead to a mismatch between the calibration input distribution and the actual inference scenario of the model to be quantized, thereby introducing additional quantization statistical errors. Summary of the Invention
[0003] To address the aforementioned technical problems, embodiments of this disclosure provide a method for generating calibration datasets, a model quantization method, an apparatus, a device, and a medium.
[0004] One aspect of this disclosure provides a method for generating a calibration dataset, comprising: generating multiple calibration texts describing multiple types of scenarios using a target model to be quantized, wherein the target model is a floating-point multimodal large language model; generating calibration images corresponding to the multiple calibration texts respectively; and constructing a calibration dataset based on each calibration image, so as to quantize the target model to be quantized based on the calibration dataset.
[0005] In another aspect of this disclosure, a model quantization method is provided, comprising: acquiring a calibration dataset, the calibration dataset including multiple calibration images, the calibration dataset being obtained based on the calibration dataset generation method described above; based on the calibration dataset, quantizing the weight matrices of each linear layer in the target model to be quantized to obtain quantized post-weight matrices corresponding to each linear layer, so as to deploy the target model to a hardware device based on the quantized post-weight matrices corresponding to each linear layer, wherein the target model is a floating-point multimodal large language model.
[0006] In another aspect of this disclosure, a calibration dataset generation apparatus is provided, comprising: a text generation module for generating multiple calibration texts describing multiple types of scenarios using a target model to be quantized, wherein the target model is a floating-point multimodal large language model; an image generation module for generating calibration images corresponding to the multiple calibration texts respectively; and a dataset construction module for constructing a calibration dataset based on each calibration image, so as to perform quantization processing on the target model to be quantized based on the calibration dataset.
[0007] In another aspect of this disclosure, a model quantization apparatus is provided, comprising: a data acquisition module for acquiring a calibration dataset, the calibration dataset including multiple calibration images, the calibration dataset being obtained based on the above-described calibration dataset generation method; and a model quantization module for quantizing the weight matrices of each linear layer in a target model to be quantized based on the calibration dataset, to obtain quantized weight matrices corresponding to each linear layer, so as to deploy the target model to a hardware device based on the quantized weight matrices corresponding to each linear layer, wherein the target model is a floating-point multimodal large language model.
[0008] In another aspect, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the calibration dataset generation method and model quantization method of the above claims.
[0009] In another aspect, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the calibration dataset generation method and model quantization method of the claims described above.
[0010] In another aspect, this disclosure provides a computer program product, including computer program instructions that, when executed by a processor, implement the calibration dataset generation method and the model quantization method described above.
[0011] Based on the calibration dataset generation method, model quantization method, apparatus, device, and medium disclosed herein, calibration text describing multiple scenarios is generated by utilizing the floating-point multimodal large language model to be quantized, and corresponding calibration images are generated accordingly to construct the calibration dataset. This achieves zero dependence on real image data in the calibration data construction process, fundamentally avoiding problems such as high cost of real image acquisition and copyright / privacy restrictions. Simultaneously, the semantics and visual distribution of the calibration text autonomously generated by the floating-point multimodal large language model to be quantized are highly consistent with the real inference scenarios of the multimodal large language model to be quantized. The generated calibration images match the semantics of the text, making the input distribution of the calibration dataset closer to the actual deployment scenario. This avoids quantization statistical bias caused by mismatch in the distribution of external datasets, effectively reduces statistical errors in the quantization process, and significantly improves the generalization ability, inference accuracy, and robustness of the quantized multimodal large language model.
[0012] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0013] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0014] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein:
[0015] Figure 1 This is a flowchart illustrating a calibration dataset generation method provided in an exemplary embodiment of this disclosure.
[0016] Figure 2 This is a flowchart illustrating step S100 provided in an exemplary embodiment of this disclosure.
[0017] Figure 3 This is a flowchart illustrating step S110 provided in an exemplary embodiment of this disclosure.
[0018] Figure 4 This is a flowchart illustrating step S113 provided in an exemplary embodiment of this disclosure.
[0019] Figure 5 This is a flowchart illustrating a calibration dataset generation method provided in another exemplary embodiment of this disclosure.
[0020] Figure 6 This is a schematic flowchart of a model quantization method provided in an exemplary embodiment of this disclosure.
[0021] Figure 7 This is a flowchart illustrating the model quantization method provided in an application example of this disclosure.
[0022] Figure 8 This is a schematic diagram of the structure of a calibration dataset generation apparatus provided in an exemplary embodiment of this disclosure.
[0023] Figure 9 This is a schematic diagram of the structure of a model quantization device provided in an exemplary embodiment of this disclosure.
[0024] Figure 10 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation
[0025] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.
[0026] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.
[0027] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.
[0028] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.
[0029] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.
[0030] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.
[0031] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.
[0032] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0033] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0034] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0035] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.
[0036] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.
[0037] In the process of implementing the technology disclosed herein, research revealed that current multimodal large language model quantization processes commonly use existing image datasets (such as CMMMU) as quantization calibration sets. This approach has significant drawbacks: on the one hand, it heavily relies on real image data, leading to high costs for data collection and annotation, copyright and privacy compliance risks, and restrictions on usage authorization, which seriously affect the applicability and scalability of quantization methods in actual deployment; on the other hand, existing image datasets are mostly constructed for specific evaluation tasks, and the image types, text formats, and semantic distributions have strong task biases, which can easily lead to a mismatch between the calibration input distribution and the model's real inference scenario, thereby introducing quantization statistical errors and affecting the accuracy and stability of the quantized model.
[0038] Figure 1 This is a flowchart illustrating a calibration dataset generation method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to any electronic device, such as a terminal device or a server. Figure 1 As shown, the calibration dataset generation method may include the following steps:
[0039] Step S100: Use the target model to be quantized to generate multiple calibration texts describing multiple types of scenarios.
[0040] The calibration texts cover a variety of scenarios. Specifically, each calibration text can describe a type of scenario, which may include natural landscapes, street scenes, people, objects, animals, and text-intensive scenarios involving formulas, code, legal provisions, etc. For example, the calibration text describing an object scenario could be "a white cat lying on a red sofa".
[0041] The target model is a floating-point multimodal large language model. The target model to be quantized is a pre-trained floating-point multimodal large language model, which can be trained using multimodal information such as text, images, video, and audio. The weight parameters of the target model to be quantized are in floating-point format, such as FP32, FP16, or BF16. The input and output of the target model to be quantized can include various forms, such as text, images, audio, and video. The target model to be quantized can include, for example, the DeepSeek-VL2 model, the qwen3-vl model, the qwen3.5 series models, the Seed1.5-VL model, or the LLaVA model, etc.
[0042] In one example, a text prompt can be constructed to generate calibration text. This prompt can include the generation conditions for the calibration text, such as the number of calibration texts to be generated, the scenarios to be covered, the ratio of calibration texts describing different scenarios, and the length of the calibration texts. The text prompt is then input into the target model to be quantized, which generates multiple calibration texts. Alternatively, several nouns can be randomly sampled from the vocabulary of the target model to be quantized. Based on these nouns, the target model generates calibration texts covering both natural landscape scenarios and text-intensive scenarios (such as content in STEM fields), ensuring that the number of both types of calibration texts is equal to guarantee the general inference performance of the quantized target model across multiple scenarios.
[0043] Step S110: Based on multiple calibration texts, generate calibration images corresponding to each calibration text.
[0044] In one embodiment, the corresponding image generation path can be selected based on the scene described by the calibration text. For example, when the scene described by the calibration text is a natural scene, the calibration text is input into the diffusion model to generate a natural scene image that is semantically consistent with the calibration text as the calibration image. When the scene described by the calibration text is a text-intensive scene such as legal provisions, the calibration text is rendered as an Optical Character Recognition (OCR) image, where the OCR image refers to an image whose content is text-based and suitable for text extraction and recognition using OCR technology.
[0045] In another embodiment, a text-based graph model can be directly used to generate corresponding natural images for all calibration texts; or, a preset number of calibration texts can be randomly selected from multiple calibration texts, and a text-based graph model can be used to generate corresponding calibration images for the selected calibration texts, while the remaining unselected calibration texts can be generated through text rendering.
[0046] Step S120: Construct a calibration dataset based on each calibration image, so as to perform quantization processing on the target model to be quantized based on the calibration dataset.
[0047] The calibration dataset includes multiple calibration images. In one embodiment, before constructing the calibration dataset based on each calibration image, image quality optimization processing can be performed on each calibration image, which may include at least one of denoising, cropping, and sharpening.
[0048] In this embodiment, by utilizing the target model to be quantized to generate calibration text describing various scenarios, and then generating corresponding calibration images to construct a calibration dataset, the calibration data construction process achieves zero dependence on real image data, fundamentally avoiding problems such as high cost of real image acquisition and copyright / privacy restrictions. Simultaneously, the calibration text generated autonomously by the target model to be quantized exhibits a high degree of semantic and visual distribution consistency with the real inference scenarios of the target model, and the generated calibration images match the text semantics. This makes the input distribution of the calibration dataset closer to the actual deployment scenario, thereby avoiding quantization statistical bias caused by mismatched external dataset distributions, effectively reducing statistical errors during the quantization process, and significantly improving the generalization ability, inference accuracy, and robustness of the quantized target model.
[0049] Figure 2 This is a flowchart illustrating step S100 provided in an exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 2 As shown, step S100 may include the following steps:
[0050] Step S101: Generate the first word unit and the score of the first word unit using the target model to be quantized.
[0051] The score for the first word unit can include the probability value of the first word unit. The target model to be quantized can output multiple first word units, as well as a score for each first word unit. For example, a start symbol (e.g., ...) can be input into the target model to be quantized. <s>The target model to be quantized outputs a score for at least one initial word and at least one initial word based on the start symbol.
[0052] Step S102: Determine the starting word sequence based on the first word element.
[0053] In one implementation, the first word element can be directly determined as the starting word element sequence; when there are multiple first word elements, each first word element can be used as an independent starting word element sequence.
[0054] Step S103, iterative execution: Input the initial word sequence into the target model to be quantized, and the target model to be quantized outputs multiple candidate words and scores of multiple candidate words. Based on the scores of multiple candidate words and the scores of each word in the initial word sequence, determine N candidate word sequences. Based on the N candidate word sequences, determine the initial word sequence, until a calibration text with complete semantics is generated.
[0055] Where N is an integer greater than or equal to 1. The score of a candidate word includes the probability value of that candidate word being the next word in the starting word sequence. The candidate word sequence is formed by concatenating the starting word sequence with one candidate word. A beam search algorithm can be used, based on the scores of multiple candidate words and the score of the starting word sequence, to select the N word sequences with the highest scores from all possible combinations of each candidate word and the starting word sequence. Beam Search is a heuristic search algorithm mainly used to find the optimal or near-optimal output sequence in sequence generation tasks. It is an improvement on greedy search, balancing the search space and computational efficiency by retaining k best candidates at each step, thus avoiding the problem of greedy search easily getting trapped in local optima. In one example, the beam width of Beam Search can be set to N.
[0056] In one implementation, the target model to be quantized adopts an autoregressive generation method, that is, one word is generated at a time, and the next word is predicted based on the generated word each time. Taking the t-th iteration as an example, t is an integer greater than 1, and the starting word sequence includes t-1 words generated by the target model to be quantized. Specifically: the starting word sequence is input into the target model to be quantized, and the target model to be quantized outputs the probability distribution of multiple candidate words based on formula (1). The probability distribution includes the probability value (score) of each candidate word in the vocabulary of the target model to be quantized as the next word in the starting word sequence; then, using the Beam Search algorithm, based on the probability distribution and the score of each word in the starting word sequence, the score of all word sequences composed of each candidate word and the starting word sequence is calculated based on formula (2), and the N words with the highest scores are selected as candidate word sequences; each candidate word sequence includes t words, that is, t-1 words of the starting word sequence plus one candidate word. Each candidate word sequence is used as an independent starting word sequence and enters the t+1-th iteration.
[0057] When there are multiple starting word sequences for the target model to be quantized, the above-described operations of determining N candidate word sequences and updating them to the starting word sequence are performed independently for each starting word sequence. During the iteration process, if the target model to be quantized determines that a certain starting word sequence has formed complete semantics, the iteration of that starting word sequence is stopped, and that starting word sequence is determined as the calibration text.
[0058] Formula (1)
[0059] Formula (2)
[0060] In formulas (1)-(2), For the initial word series, As candidate lexical units, For probability distribution, Let be the forward computation function of the target model to be quantized. These are all the weight parameters of the target model to be quantized. For normalized exponential functions, For the first A candidate word sequence, For the first The probability value (i.e., score) of a word. A sequence of lexical units consisting of candidate lexical units and the sequence of starting lexical units. The sum of the log-likelihood probabilities of the probability values of each word element. To find the lexical sequence that maximizes the sum of the logarithmic probabilities of all subsequent lexical sequences formed by each candidate lexical and the starting lexical sequence, Lexical sequence The index of the Chinese word element, with values ranging from 1 to... .
[0061] In this embodiment of the disclosure, when generating calibration text iteratively, multiple high-scoring candidate lexical sequences are retained at each step. This allows multiple semantic paths (candidate lexical sequences) to be explored in parallel from the same starting sequence, effectively ensuring the diversity of the generated calibration text in terms of content, structure, and semantics. This provides a rich text foundation for the subsequent construction of calibration datasets covering multiple scenarios.
[0062] Figure 3 This is a flowchart illustrating step S110 provided in an exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 3 As shown, step S110 may include the following steps:
[0063] Step S111: Obtain semantic discrimination prompts.
[0064] Semantic discrimination prompts are used to indicate the semantic types corresponding to different semantics. Semantic types can include, for example, natural scene types and text-dominant types. Natural scene type calibration text refers to calibration text whose content includes descriptions of visual elements in the physical world. Visual elements include, but are not limited to, natural scenery, street scenes, people scenes, object scenes, animal scenes, indoor scenes, and other elements that can be visually presented through natural images. For example, when the calibration text is "a white cat lying on a red sofa," "a city skyline at sunset," or "a cup of coffee and a book on the table," the semantic type of the calibration text can be determined as natural scene type. Text-dominant type calibration text refers to calibration text whose content is mainly text-intensive information. Its typical visual presentation is a document image dominated by text symbols. Text-intensive information includes, but is not limited to, logical reasoning, mathematical derivation, program code, legal provisions, contract terms, academic papers, etc. For example, when the calibration text includes "the principle of voluntary marriage stipulated in Article 1046 of the Civil Code" or "the calculation formula of the activation function is a+b=c," the semantic type of the calibration text can be determined as text-dominant type.
[0065] Semantic discrimination prompts may include rules for determining the semantic type of calibration text, as well as calibration text examples corresponding to each semantic type. For example, rules for determining the semantic type of calibration text may include: determining whether the calibration text belongs to the natural scene type or the text-dominant type, the definition of the natural scene type, and the definition of the text-dominant type. Calibration text examples corresponding to each semantic type may include: calibration text for the natural scene type may include "a white cat lying on a red sofa", "the city skyline under the sunset", or "a cup of coffee and a book on the table"; calibration text for the text-dominant type may include "the principle of voluntary marriage stipulated in Article 1046 of the Civil Code" or "the calculation formula of the activation function is a+b=c".
[0066] Step S112: Input the semantic discrimination prompts and multiple calibration texts into the target model to be quantized, and the target model to be quantized outputs the semantic types corresponding to the multiple calibration texts.
[0067] The target model to be quantized includes a large language model, which can input semantic discrimination prompts and multiple calibration texts into the target model to be quantized. The large language model part of the target model to be quantized determines the semantic type corresponding to each of the multiple calibration texts.
[0068] Step S113: Based on the semantic types corresponding to the multiple calibration texts, generate calibration images corresponding to the multiple calibration texts respectively.
[0069] Specifically, the appropriate image generation path can be selected based on the semantic type of the calibration text. For example, when the semantic type of the calibration text is "natural scene," the calibration text is input into the diffusion model to generate the corresponding calibration image.
[0070] In this embodiment of the disclosure, semantic discrimination prompts are used to drive the target model to be quantized to perform semantic type discrimination on the calibration text and match the corresponding image generation path, thus ensuring a high degree of consistency between the text and image semantics.
[0071] Figure 4 This is a flowchart illustrating step S113 provided in an exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 4 As shown, step S113 may include the following steps:
[0072] Step S1131: For multiple calibration texts, in response to the semantic type of the calibration text being a natural scene type, a calibration image corresponding to the calibration text is generated based on the calibration text using a text-generated image model.
[0073] The text-based image model can be, for example, a diffusion model, a stable diffusion model, or a denoising diffusion probabilistic model (DDPM). For instance, the calibration text is input into the text-based image model, and the model outputs the calibration image corresponding to the calibration text.
[0074] Step S1132: In response to the fact that the semantic type of the calibration text is text-dominant, obtain the text parameters of the calibration text.
[0075] The text parameters include at least font parameters and layout parameters. Font parameters define the visual style of the rendered calibration text and may include, but are not limited to, font type (e.g., SimSun, Heiti, Times New Roman), font size, font weight (e.g., regular, bold), font color, and whether italics or underlines are used. Layout parameters define the spatial arrangement of the calibration text within the image and may include, but are not limited to, page margins, line spacing, paragraph spacing, text alignment (e.g., left alignment, center alignment, justified alignment), whether to use columns, background color, or background texture. For example, when the calibration text is code-related content, the font parameter can be set to a monospace font (e.g., CourierNew), and the background color in the layout parameters can be set to a dark color to simulate the visual effect of a code editor.
[0076] It should be noted that there is no specific order of execution between steps S1131 and S1132 in this embodiment.
[0077] Step S1133: Based on the text parameters, render the calibration text to obtain the calibration image corresponding to the calibration text.
[0078] Specifically, the rendering function R in Python can be called, and then the calibration text T, font parameter F, and layout parameter L are input into the text rendering function R. The rendering function R performs the rendering operation, mapping the text content of the calibration text to a representation in the image pixel space, thereby generating an OCR image corresponding to the calibration text, which serves as the calibration image of the calibration text.
[0079] In this embodiment of the disclosure, the calibration text is split into two paths based on the text semantic type: natural scene image generation and text-intensive image rendering. The matching relationship between different text semantics and corresponding visual forms is explicitly established, which makes up for the shortcomings of the prior art in not distinguishing different visual distributions. This makes the calibration dataset highly matched with the model's real inference scenario, effectively improving the pertinence and accuracy of quantitative calibration.
[0080] In some alternative implementations, step S120 in this disclosure may include: constructing a calibration dataset based on each calibration image and each calibration text.
[0081] The calibration dataset includes each calibration image and each calibration text. By combining the calibration text with its corresponding calibration image to form the calibration dataset, the visual encoder, cross-modal fusion module, and language model backbone in the target model to be quantized can all participate in the statistics during the quantization calibration stage, effectively improving the accuracy and stability of the quantized target model in multimodal tasks.
[0082] Figure 5 This is a flowchart illustrating a calibration dataset generation method provided in another exemplary embodiment of this disclosure. In some alternative implementations, such as... Figure 5 As shown, the steps between step S112 and step S113 include the following:
[0083] Step S200: Determine text distribution information based on the number of calibration texts belonging to the natural scene type and the number of calibration texts belonging to the dominant text type.
[0084] The text distribution information includes a target ratio. The target ratio is the ratio of the number of calibration texts belonging to the natural scene type to the number of calibration texts belonging to the text-dominant description type.
[0085] Step S210: Determine the target semantic type based on text distribution information.
[0086] The target semantic type can be either a text-dominant type or a natural scene type. A preset reference range can be set. When the target ratio value exceeds the preset reference range, the semantic type with the smaller number of corresponding calibration texts will be determined as the target semantic type.
[0087] For example, the preset reference range can be [0.8, 1.25]. When the target ratio is 0.3, it indicates that the number of calibration texts belonging to the natural scene type is too small, and the natural scene type is determined as the target semantic type. When the target ratio is 1.5, it indicates that the number of calibration texts belonging to the text-dominant type is too small, and the text-dominant type is determined as the target semantic type.
[0088] Step S220: Obtain the target text prompt word corresponding to the target semantic type.
[0089] In one implementation, the target text cue is used to guide the target model to be quantized to generate calibration text belonging to the target semantic type. The target text cue may include at least one of the following generation conditions: the definition of the target semantic type, an example of calibration text corresponding to the target semantic type, and the number of calibration texts to be generated.
[0090] For example, when the target semantic type is natural scene type, the target text prompt can be constructed as: "Please generate 5 calibration texts, each of which must be of natural scene type (calibration text of natural scene type refers to calibration text whose content includes descriptions of visual elements in the physical world, such as descriptions of natural scenery, streets, people, objects, animals, etc., that can be visually presented through natural images). Examples of calibration text of natural scene type are as follows: 'A white cat lying on a red sofa', 'City skyline at sunset'."
[0091] Step S230: Based on the target text prompt words, generate multiple calibration texts belonging to the target semantic type using the target model to be quantized.
[0092] In one implementation, target text prompts are input into a target model to be quantized, which generates calibration text of multiple target semantic types.
[0093] After generating calibration text of the target semantic type based on text distribution information, the newly generated calibration text can be merged with the previously generated calibration texts of each semantic type, and multiple corresponding calibration images can be generated based on the merged calibration texts.
[0094] In this embodiment of the disclosure, the insufficient number of target semantic types are dynamically identified based on text distribution information, and the corresponding target text prompt words are used to guide the targeted model to be quantized to be supplemented and generated in a targeted manner. This realizes the active monitoring and on-demand adjustment of the semantic distribution of calibration data, which significantly improves the representativeness of the calibration dataset and the generalization ability of the quantized model.
[0095] Figure 6 This is a flowchart illustrating a model quantization method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to any electronic device, such as a terminal device or a server. Figure 6 As shown, the model quantization method may include the following steps:
[0096] Step S300: Obtain the calibration dataset.
[0097] The calibration dataset includes multiple calibration images, and the calibration dataset is obtained based on any of the calibration dataset generation methods described in the above embodiments.
[0098] In one example, the calibration dataset may also include multiple calibration texts, each of which is obtained based on any of the calibration dataset generation methods described in the above embodiments.
[0099] Step S310: Based on the calibration dataset, the weight matrix of each linear layer in the target model to be quantized is quantized to obtain the quantized weight matrix corresponding to each linear layer, so that the target model can be deployed to the hardware device based on the quantized weight matrix corresponding to each linear layer.
[0100] The target model is a floating-point multimodal large language model. Based on a calibration dataset, post-training quantization (PTQ) methods such as Generative Pre-trained Quantization (GPTQ), Activation-aware Weight Quantization (AWQ), and AutoRound can be used to quantize the target model.
[0101] For example, Figure 7 This is a flowchart illustrating the model quantization method provided in an application example of this disclosure. This application example uses the GPTQ quantization method as an example. GPTQ is a quantization technique for generative pre-trained models. The main purpose of GPTQ is to significantly reduce storage requirements and computational costs while maintaining model performance. Figure 7 As shown, it includes:
[0102] 1. Input each calibration image in the calibration dataset into the floating-point multimodal large language model to be quantized and perform forward propagation.
[0103] 2. During the forward propagation process, obtain the input activation values of each linear layer in the floating-point multimodal large language model to be quantized.
[0104] 3. Record the normalized activation values (bn) of each normalization layer output in the floating-point multimodal large language model to be quantized.
[0105] 4. Based on the input activation values of each linear layer, calculate the approximate Hessian matrix of the weights of each linear layer.
[0106] 5. Perform layer-by-layer quantization on each linear layer.
[0107] For each linear layer, perform the following sub-steps:
[0108] 5.1 Obtain the weight matrix of the linear layer. According to the preset column processing order, perform quantization operation on each column weight in the weight matrix in turn, and quantize each column weight into an integer representation of a preset bit width (e.g., 4 bits).
[0109] 5.2 For the column that has been quantized, calculate its quantization error vector; based on the quantization error vector and the approximate Hessian matrix of the linear layer, determine the quantization compensation amount for the next column to be quantized, apply the quantization compensation amount to the next column to be quantized, and then perform quantization operation on the column.
[0110] 5.3 Repeat step 5.2 until all columns of the weight matrix of the linear layer have been quantized and compensated.
[0111] 6. Restore the columns of the quantized weight matrix after compensation to their original column order to obtain the quantized weight matrix corresponding to each linear layer.
[0112] 7. Based on the quantized weight matrix corresponding to each linear layer, deploy the quantized multimodal large language model to the hardware device; wherein, the quantized weight matrix corresponding to each linear layer is stored in the storage space of the hardware device.
[0113] 8. Utilize the deployed multimodal large language model to perform forward inference in order to complete the target task.
[0114] Figure 8 This is a schematic diagram of the structure of a calibration dataset generation apparatus provided in an exemplary embodiment of this disclosure. Figure 8 As shown, the device includes:
[0115] The text generation module 300 is used to generate multiple calibration texts describing multiple scenarios using the target model to be quantized, wherein the target model is a floating-point multimodal large language model.
[0116] The image generation module 310 generates calibration images corresponding to the plurality of calibration texts based on the plurality of calibration texts.
[0117] The dataset construction module 320 is used to construct a calibration dataset based on each calibration image, so as to perform quantization processing on the target model to be quantized based on the calibration dataset.
[0118] In some alternative implementations, the text generation module 300 in this embodiment is specifically configured to: generate a first word unit and a score for the first word unit using the target model to be quantized; determine a starting word unit sequence based on the first word unit; iteratively execute: inputting the starting word unit sequence into the target model to be quantized, outputting multiple candidate word units and scores for the multiple candidate word units from the target model to be quantized, determining N candidate word unit sequences based on the scores of the multiple candidate word units and the scores of each word unit in the starting word unit sequence, determining a starting word unit sequence based on the N candidate word unit sequences, until a calibration text with complete semantics is generated.
[0119] In some alternative implementations, the image generation module 310 in this embodiment of the present disclosure is specifically used for: acquiring semantic discrimination prompts, the semantic discrimination prompts being used to indicate the semantic types corresponding to different semantics; inputting the semantic discrimination prompts and the plurality of calibration texts into the target model to be quantized, the target model to be quantized outputting the semantic types corresponding to the plurality of calibration texts respectively; and generating calibration images corresponding to the plurality of calibration texts based on the semantic types corresponding to the plurality of calibration texts respectively.
[0120] In some alternative implementations, the step of generating calibration images corresponding to the plurality of calibration texts based on their respective semantic types, as described in this disclosure, is further used for:
[0121] For the plurality of calibration texts, in response to the semantic type of the calibration text being a natural scene type, a calibration image corresponding to the calibration text is generated based on the calibration text using a text-generated image model; in response to the semantic type of the calibration text being a text-dominant type, the text parameters of the calibration text are obtained; based on the text parameters, the calibration text is rendered to obtain the calibration image corresponding to the calibration text.
[0122] In some alternative implementations, the dataset construction module 320 in this disclosure embodiment is also used for
[0123] The calibration dataset is constructed based on each calibration image and each calibration text.
[0124] In some alternative implementations, the calibration dataset generation apparatus in this disclosure further includes:
[0125] The semantic discrimination module is used to determine text distribution information based on the number of calibration texts belonging to the natural scene type and the number of calibration texts belonging to the dominant text type;
[0126] A semantic determination module is used to determine the target semantic type based on the text distribution information;
[0127] The prompt word generation module is used to obtain the target text prompt words corresponding to the target semantic type;
[0128] The compensation text generation module is used to generate multiple calibration texts belonging to the target semantic type based on the target text prompt words and the target model to be quantized.
[0129] The calibration dataset generation apparatus of this disclosure corresponds to the calibration dataset generation method described above, and the relevant contents can be referred to each other, which will not be repeated here.
[0130] The beneficial technical effects of the exemplary embodiments of the calibration dataset generation apparatus of this disclosure can be found in the corresponding beneficial technical effects of the exemplary methods and systems described above, and will not be repeated here.
[0131] Figure 9 This is a schematic diagram of the structure of a model quantization device provided in an exemplary embodiment of this disclosure. Figure 9 As shown, the device includes:
[0132] The data acquisition module 400 is used to acquire a calibration dataset, which includes multiple calibration images and is obtained based on the calibration dataset generation method described above.
[0133] The model quantization module 410 is used to quantize the weight matrix of each linear layer in the target model to be quantized based on the calibration dataset, so as to obtain the quantized weight matrix corresponding to each linear layer, so as to deploy the target model to the hardware device based on the quantized weight matrix corresponding to each linear layer, wherein the target model is a floating-point multimodal large language model.
[0134] The model quantization apparatus of this disclosure corresponds to the model quantization apparatus of the above disclosure, and the relevant contents can be referred to each other, which will not be repeated here.
[0135] The beneficial technical effects of the exemplary model quantization apparatus of this disclosure can be found in the corresponding beneficial technical effects of the exemplary methods and systems described above, and will not be repeated here.
[0136] In addition, this disclosure also provides an electronic device, including:
[0137] Memory, used to store computer programs;
[0138] A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the calibration dataset generation method and the model quantization method described in any of the above embodiments of the present disclosure.
[0139] Figure 10 This is a schematic diagram illustrating the structure of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 10 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.
[0140] like Figure 10 As shown, the electronic device includes one or more processors and memory.
[0141] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.
[0142] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the calibration dataset generation method and the model quantization method described in the various embodiments of this disclosure above, and / or other desired functions.
[0143] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0144] In addition, the input device may include, for example, a keyboard, a mouse, etc.
[0145] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0146] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.
[0147] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the calibration dataset generation method and the model quantization method according to various embodiments of this disclosure as described in the foregoing portion of this specification.
[0148] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this disclosure. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0149] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the calibration dataset generation method and the model quantization method according to various embodiments of this disclosure as described in the foregoing portion of this specification.
[0150] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0151] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.
[0152] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0153] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0154] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.
[0155] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.
[0156] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.
[0157] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.
[0158] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.< / s>
Claims
1. A method for generating a calibration dataset, characterized in that, include: Multiple calibration texts describing various scenarios are generated using a target model to be quantized, wherein the target model is a floating-point multimodal large language model; Based on the multiple calibration texts, generate calibration images corresponding to each of the multiple calibration texts; A calibration dataset is constructed based on each calibration image, so that the target model to be quantized can be quantized based on the calibration dataset.
2. The method according to claim 1, characterized in that, The process of generating multiple calibration texts describing various scenarios using the target model to be quantized includes: The target model to be quantized is used to generate the first word element and the score of the first word element; Based on the first word element, determine the starting word element sequence; Iterative execution: The starting word sequence is input into the target model to be quantized, and the target model to be quantized outputs multiple candidate words and their scores. Based on the scores of the multiple candidate words and the scores of each word in the starting word sequence, N candidate word sequences are determined. Based on the N candidate word sequences, the starting word sequence is determined until a calibration text with complete semantics is generated.
3. The method according to claim 2, characterized in that, The step of generating calibration images corresponding to the plurality of calibration texts includes: Obtain semantic discrimination prompts, which are used to indicate the semantic types corresponding to different semantics; The semantic discrimination prompts and the multiple calibration texts are input into the target model to be quantized, and the target model to be quantized outputs the semantic types corresponding to the multiple calibration texts respectively. Based on the semantic types corresponding to the multiple calibration texts, calibration images corresponding to the multiple calibration texts are generated.
4. The method according to claim 3, characterized in that, The step of generating calibration images corresponding to the plurality of calibration texts based on their respective semantic types includes: For the plurality of calibration texts, in response to the semantic type of the calibration text being a natural scene type, a calibration image corresponding to the calibration text is generated based on the calibration text using a text-generated image model; In response to the fact that the semantic type of the calibration text is text-dominant, the text parameters of the calibration text are obtained; Based on the text parameters, the calibration text is rendered to obtain the calibration image corresponding to the calibration text.
5. The method according to claim 1, characterized in that, The construction of the calibration dataset based on each calibration image includes: The calibration dataset is constructed based on each calibration image and each calibration text.
6. The method according to claim 4, characterized in that, Before generating the calibration images corresponding to the plurality of calibration texts based on their respective semantic types, the method further includes: Based on the number of calibration texts belonging to the natural scene type and the number of calibration texts belonging to the dominant text type, text distribution information is determined; Based on the text distribution information, the target semantic type is determined; Obtain the target text prompt word corresponding to the target semantic type; Based on the target text prompt words, multiple calibration texts belonging to the target semantic type are generated using the target model to be quantized.
7. A model quantization method, characterized in that, include: A calibration dataset is obtained, the calibration dataset comprising multiple calibration images, and the calibration dataset is obtained based on the calibration dataset generation method according to any one of claims 1-6 above; Based on the calibration dataset, the weight matrices of each linear layer in the target model to be quantized are quantized to obtain the quantized weight matrices corresponding to each linear layer, so that the target model can be deployed to the hardware device based on the quantized weight matrices corresponding to each linear layer. The target model is a floating-point multimodal large language model.
8. A calibration dataset generation apparatus, characterized in that, include: The text generation module is used to generate multiple calibration texts describing various scenarios using the target model to be quantized, wherein the target model is a floating-point multimodal large language model. The image generation module generates calibration images corresponding to the multiple calibration texts, respectively. The dataset construction module is used to construct a calibration dataset based on each calibration image, so as to perform quantization processing on the target model to be quantized based on the calibration dataset.
9. A model quantization device, characterized in that, include: The data acquisition module is used to acquire a calibration dataset, which includes multiple calibration images, and the calibration dataset is obtained based on the calibration dataset generation method described in any one of claims 1-6 above; The model quantization module is used to quantize the weight matrix of each linear layer in the target model to be quantized based on the calibration dataset, so as to obtain the quantized weight matrix corresponding to each linear layer, so as to deploy the target model to the hardware device based on the quantized weight matrix corresponding to each linear layer, wherein the target model is a floating-point multimodal large language model.
10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory, wherein when the computer program is executed, it implements the method described in any one of claims 1-7.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.
12. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.