Model operation method, device, equipment, medium and product

Abstract

The invention provides a model operation method and device, equipment, a medium and a product. The method comprises the following steps: determining residual storage resources of hardware equipment; the type of a target model capable of being operated by the hardware equipment is determined by comparing residual storage resources of the hardware equipment with storage resources required by the quantitative model, the type of the target model is a first quantitative model or a second quantitative model, and the first quantitative model and a first quantitative coefficient corresponding to the first quantitative model are stored in a memory. The first quantization parameter is used for performing dequantization on the first quantization model to obtain a second quantization model, and the bit width of the model parameter corresponding to the second quantization model is greater than that of the model parameter corresponding to the first quantization model; and obtaining the corresponding target model according to the type of the target model, and running the target model by the hardware equipment to execute the target task. The effects of small memory occupation and high reasoning accuracy can be realized.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model operation method, device, equipment, medium and product. Background Art

[0002] In recent years, large language models based on the Transformer architecture in deep learning have made significant progress and breakthroughs in the field of natural language processing (NLP). These models have shown good performance in tasks such as text generation and semantic understanding by adopting complex neural network structures. However, these high-performance models often contain a large number of parameters and computing requirements. Therefore, when deploying them on edge devices with limited computing resources (such as mobile terminals or embedded systems), they need to be quantized to ensure that they can run effectively within limited hardware resources.

[0003] The current mainstream quantization technology mainly focuses on converting high-precision (such as 32-bit floating-point representation) model parameters into low-precision representation (such as 4-bit or 2-bit). And apply the low-precision quantization model for reasoning. Since directly quantizing the floating-point model to a 2-bit model will result in a large quantization error, the current method is to quantize the floating-point model to a 4-bit representation and store the 4-bit quantized model in memory, so that the 4-bit quantized model is directly loaded from the memory during the reasoning process. However, the memory space occupied by the 4-bit quantization model is still large. Therefore, how to take into account both the small memory usage and high reasoning accuracy has become an urgent problem to be solved. Summary of the invention

[0004] Based on the above technical status, the present application proposes a model operation method, device, equipment, medium and product, which can achieve the effect of small memory usage and high reasoning accuracy.

[0005] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions:

[0006] The first aspect of the present application proposes a model operation method, including: determining the remaining storage resources of a hardware device; determining the type of target model that the hardware device can run by comparing the remaining storage resources of the hardware device with the storage resources required for a quantization model, the type of the target model being a first quantization model or a second quantization model, the first quantization model and its corresponding first quantization coefficient being stored in a memory, the first quantization parameter being used to dequantize the first quantization model to obtain the second quantization model, the bit width of the model parameter corresponding to the second quantization model being greater than the bit width of the model parameter corresponding to the first quantization model; obtaining the corresponding target model according to the type of the target model, and having the hardware device run the target model to perform a target task.

[0007] In some implementations, determining the type of target model that the hardware device can run by comparing the remaining storage resources of the hardware device with the storage resources required for the quantization model includes: comparing the remaining storage resources of the hardware device with the storage resources required for the high-priority quantization model based on the priorities of the first quantization model and the second quantization model, and determining the type of target model that the hardware device can run based on the comparison result.

[0008] In some implementations, the priority of the second quantization model is greater than that of the first quantization model; wherein, based on the priorities of the first quantization model and the second quantization model, the remaining storage resources of the hardware device are compared with the storage resources required by the high-priority quantization model, and the type of target model that the hardware device can run is determined based on the comparison result, including: when the remaining storage resources of the hardware device are greater than or equal to the storage resources required by the second quantization model, determining that the type of target model that the hardware device can run is the second quantization model; wherein, obtaining the corresponding target model based on the type of the target model, and having the hardware device run the target model to perform the target task, includes: when the type of target model that the hardware device can run is the second quantization model, loading the first quantization model and its corresponding first quantization parameter from the memory, and dequantizing the first quantization model through the first quantization parameter to obtain the second quantization model, and having the hardware device run the second quantization model to perform the target task.

[0009] In some implementations, the priority of the second quantization model is greater than that of the first quantization model; wherein, based on the priorities of the first quantization model and the second quantization model, the remaining storage resources of the hardware device and the storage resources required by the high-priority quantization model are compared, and the type of target model that the hardware device can run is determined based on the comparison result, including: when the remaining storage resources of the hardware device are less than the storage resources required by the second quantization model, determining that the type of target model that the hardware device can run is the first quantization model; wherein, obtaining the corresponding target model according to the type of the target model, and having the hardware device run the target model to perform the target task, includes: when the type of the target model that the hardware device can run is the first quantization model, loading the first quantization model from the memory, and having the hardware device run the first quantization model to perform the target task.

[0010] In some implementations, the first quantization model is obtained by quantizing the second quantization model; the second quantization model is obtained by quantizing a pre-trained floating-point model, the floating-point model includes floating-point network layers connected in sequence, and each floating-point network layer corresponds to a pseudo-quantization network layer; wherein the second quantization model is obtained by the following processing: determining the pseudo-quantization coefficients corresponding to each pseudo-quantization network layer by layer quantization; quantizing the network parameters corresponding to each floating-point network layer based on the pseudo-quantization coefficients corresponding to each pseudo-quantization network layer to obtain an initial second quantization model, and determining the second quantization model based on the initial second quantization model.

[0011] In some implementations, the method of determining the pseudo-quantization parameters corresponding to each pseudo-quantization network layer by quantization layer by layer includes performing the following steps on the i-th floating-point network layer in each floating-point network layer of the floating-point model: converting the i-1th floating-point output result into the i-th integer input result corresponding to the i-th pseudo-quantization network layer through the i-th activation value quantization coefficient of the i-th pseudo-quantization network layer corresponding to the i-th floating-point network layer and the i-1th floating-point output result of the i-1th pseudo-quantization network layer; determining the i-th floating-point output result of the i-th pseudo-quantization network layer according to the i-th integer input result and the i-th pseudo-quantization weight of the i-th pseudo-quantization network layer; and adjusting the i-th activation value quantization coefficient and the i-th pseudo-weight quantization coefficient based on the difference between the i-th floating-point output result output by the i-th floating-point network layer and the floating-point dequantization result corresponding to the i-th floating-point output result.

[0012] In some implementations, determining the second quantization model based on the initial second quantization model includes: fixing the parameters of the initial second quantization model, adding a lora module to the initial second quantization model, and fine-tuning the initial second quantization model using a lora fine-tuning training method to obtain the second quantization model.

[0013] In some implementations, the first quantization model is obtained by the following processing: based on the weight of the second quantization model, constructing the objective function of the first quantization model regarding the weight quantization error according to an asymmetric quantization scheme; solving the objective function to obtain the asymmetric quantization factor of the first quantization model; quantizing the weight of the second quantization model based on the weight of the second quantization model and the asymmetric quantization factor of the first quantization model to obtain the first quantization model.

[0014] The second aspect of the present application proposes a model running device, including: a first determination unit, used to determine the remaining storage resources of a hardware device; a second determination unit, used to determine the type of target model that the hardware device can run by comparing the remaining storage resources of the hardware device with the storage resources required for the quantization model, the type of the target model is a first quantization model or a second quantization model, the first quantization model and its corresponding first quantization coefficient are stored in a memory, the first quantization parameter is used to dequantize the first quantization model to obtain the second quantization model, and the bit width of the model parameter corresponding to the second quantization model is greater than the bit width of the model parameter corresponding to the first quantization model; a model acquisition unit, used to acquire the corresponding target model according to the type of the target model, and the hardware device runs the target model to perform the target task.

[0015] The third aspect of the present application proposes an electronic device, comprising a memory and a processor; the memory is connected to the processor and is used to store programs; the processor is used to implement the model operation method described in the first aspect and any one of the implementation methods of the first aspect by running the program in the memory.

[0016] In a fourth aspect, the present application proposes a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the model operation method described in the first aspect and any one of the implementation methods of the first aspect is implemented.

[0017] In a fifth aspect, the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the model operation method described in the first aspect and any one of the implementation methods of the first aspect.

[0018] The model operation method, device, equipment, medium and product proposed in the present application first accurately evaluate the remaining storage resources of the hardware device to determine the space available for model reasoning, so as to ensure that the actual available storage space is taken into account when selecting the model, and avoid the situation where the model cannot run normally due to insufficient storage; next, the remaining storage resources of the hardware device are compared with the storage resources required by the quantization model to determine the type of target model that can be run. The target model can be a first quantization model or a second quantization model. Among them, the first quantization model and its corresponding first quantization parameter are stored in the memory, and the first quantization parameter is used to dequantize the first quantization model into a second quantization model. The model parameter bit width of the second quantization model is greater than the model parameter bit width of the first quantization model. Finally, according to the determined target model type, the corresponding quantization model is loaded from the memory, and the model is run by the hardware device to perform the target task. Thereby ensuring that under limited resources, the model that best suits the current hardware conditions is selected for operation, which can fully utilize the hardware resources and ensure the accuracy of reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of the storage format of a 32-bit floating point number in a computer provided in an embodiment of the present application;

[0021] Figure 2 A flow chart of a model operation method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the principle of the quantization process of the second quantization model provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a flow chart of a quantization process of a second quantization model provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a quantization process of a first quantization model provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a model operation device provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solution proposed in the embodiment of the present application is applicable to any target task processing application scenario. By adopting the technical solution of the embodiment of the present application, it is possible to achieve the effect of small memory usage and high reasoning accuracy. The target task may include one or more of an image processing task, an audio processing task, and a text processing task.

[0028] Among them, image processing tasks may include image classification tasks, target detection tasks, semantic segmentation tasks, instance segmentation tasks, edge detection tasks, resolution conversion tasks, image style transfer tasks, image restoration tasks, image registration tasks, illumination estimation tasks, image denoising tasks, image retrieval tasks, dynamic background removal tasks, and image synthesis tasks, etc. The following introduces each image processing task one by one:

[0029] The image classification task is to predict the type of object in an image, which can be applied to scenarios such as automatic labeling, product recognition, and medical image diagnosis.

[0030] The target detection task is to locate and identify multiple objects in an image, which can be applied to scenarios such as autonomous driving, security monitoring, and drone target tracking.

[0031] The semantic segmentation task is to assign a category label to each pixel in an image, which can be applied to scenarios such as map making, medical image analysis, and video game scene rendering.

[0032] The instance segmentation task is to identify which category each pixel belongs to, so as to distinguish different instances. It can be applied to scenarios such as video analysis and obstacle detection in autonomous driving.

[0033] The edge detection task is to detect the boundaries or contours of objects in an image, which can be applied to scenarios such as image editing and industrial defect detection.

[0034] The resolution conversion task is to convert a low-resolution image into a high-resolution image, which can be applied to scenarios such as video restoration and old photo restoration.

[0035] The image style transfer task is to transfer the style of one image to the content of another image, which can be applied to scenes such as artistic creation and movie special effects.

[0036] The image restoration task is to restore damaged or missing parts of an image, which can be applied to scenarios such as old photo restoration and artwork restoration.

[0037] The image registration task is to align two or more images, which can be applied to scenarios such as medical image analysis and remote sensing image processing.

[0038] The lighting estimation task is to estimate the lighting direction and intensity in an image, which can be applied to scenarios such as computer graphics and virtual reality.

[0039] Image denoising is a task to remove noise from images, which can be applied to scenarios such as digital photography or video streaming.

[0040] The image retrieval task is to search the image database based on content or features, which can be applied to scenarios such as search engines or social media platforms.

[0041] The dynamic background removal task is to separate foreground objects from continuous image sequences, which can be applied to scenes such as video surveillance and virtual studios.

[0042] The image synthesis task is to combine multiple image elements into a new image, which can be applied to film and television post-production, advertising design and other scenarios.

[0043] Among them, audio processing tasks may include speech recognition tasks, speaker recognition tasks, sound event detection tasks, music transcription tasks, music information retrieval tasks, sentiment analysis tasks, speech synthesis tasks, speech enhancement tasks, speech activity detection tasks, language recognition tasks, speech wake-up tasks, machine hearing tasks, audio classification tasks, audio compression and decoding tasks, audio repair tasks, etc. The following is an introduction to each audio processing task:

[0044] The speech recognition task is to convert speech signals into text, which can be applied to scenarios such as voice assistants, telephone services, and meeting records.

[0045] The speaker recognition task is to identify or verify the identity of the speaker in the audio, which can be applied to scenarios such as security systems and biometric authentication.

[0046] The sound event detection task is to detect specific sound events in audio, which can be applied to scenarios such as environmental monitoring and smart home security systems.

[0047] The music transcription task is to convert the performed music into musical scores, which can be applied to music education, music production and other scenarios.

[0048] The task of music information retrieval is to extract metadata and features from music for search and organization, which can be applied to scenarios such as music recommendation systems and copyright management.

[0049] The sentiment analysis task is to analyze the emotions or emotional states in audio, which can be applied to scenarios such as customer service and psychological research.

[0050] The speech synthesis task is to convert text into natural language speech output, which can be applied to scenarios such as assistive reading devices and navigation systems.

[0051] The task of speech enhancement is to improve the quality of speech signals, such as noise reduction, and can be applied to scenarios such as telephone communication and speech recognition pre-processing.

[0052] The voice activity detection task is to detect whether there is voice activity in the audio, which can be applied to scenarios such as voice communication and meeting recording.

[0053] The language recognition task is to determine the language used in the audio, which can be applied to scenarios such as automated processing and translation services in multilingual environments.

[0054] The voice wake-up task is to detect specific keywords to activate the device, which can be applied to scenarios such as smart speakers and mobile phone voice assistants.

[0055] The machine hearing task is to enable machines to understand complex acoustic scenes, which can be applied to scenarios such as robot navigation and environmental perception of unmanned vehicles.

[0056] The audio classification task is to classify audio clips into predefined categories, which can be applied to scenarios such as music style recognition, environmental sound classification, and voice command recognition.

[0057] Audio compression and decoding is the process of compressing audio files for storage or transmission, which can be applied to streaming services, audio file storage and other scenarios.

[0058] Audio restoration is the process of repairing damaged or noisy audio files, which can be applied to scenarios such as archival material preservation and film restoration.

[0059] Among them, text processing tasks include text classification tasks, named entity recognition tasks, part-of-speech tagging tasks, dependency syntax analysis tasks, topic model tasks, sentiment analysis tasks, machine translation tasks, question-answering systems, text summarization tasks, text generation tasks, text completion tasks, text error correction tasks, keyword extraction tasks, text implication tasks, chatbots, document clustering tasks, information retrieval tasks, and semantic similarity tasks, etc. The following introduces each text processing task one by one:

[0060] The text classification task is to classify text into predefined categories, which can be applied to scenarios such as news classification, sentiment analysis, and topic identification.

[0061] The task of named entity recognition is to identify named entities in text, such as names of people, places, and institutions, which can be applied to scenarios such as information extraction and knowledge graph construction.

[0062] The part-of-speech tagging task is to mark the part of speech for each word in the text, which can be applied to scenarios such as grammatical analysis and machine translation.

[0063] The dependency syntactic analysis task is to analyze the dependency relationship between words in a sentence, which can be applied to question-answering systems, machine translation and other scenarios.

[0064] The task of topic modeling is to discover potential topics in a document collection, which can be applied to scenarios such as literature retrieval and market research.

[0065] The sentiment analysis task is to analyze the sentiment polarity and intensity in text, which can be applied to scenarios such as social media monitoring and product review analysis.

[0066] The task of machine translation is to translate text from one language into another, which can be applied to scenarios such as cross-language communication and international business.

[0067] The question-answering system automatically answers questions in text and can be applied to scenarios such as online customer service and knowledge base query.

[0068] The text summarization task is to generate a concise version of the text, which can be applied to scenarios such as news summarization and long document summarization.

[0069] The text generation task is to generate new text paragraphs or sentences, which can be applied to scenarios such as article writing assistance and chatbots.

[0070] The text completion task is to predict the next word or sentence of the text, which can be applied to scenarios such as intelligent input suggestions and code completion.

[0071] The text error correction task is to automatically correct spelling or grammatical errors in text, which can be applied to scenarios such as word processors and online editors.

[0072] The keyword extraction task is to extract the most important keywords or phrases from the text, which can be applied to document indexing, information retrieval and other scenarios.

[0073] The text entailment task is to determine the logical relationship between two sentences, which can be applied to scenarios such as semantic similarity detection and dialogue understanding.

[0074] Chatbots are automatic response systems that simulate human conversations and can be used in scenarios such as customer service and personal assistants.

[0075] The document clustering task is to group documents according to their similarities, which can be applied to scenarios such as information organization and document retrieval.

[0076] The information retrieval task is to retrieve relevant information from a large number of documents, which can be applied to search engines, internal enterprise document management systems and other scenarios.

[0077] The semantic similarity task is to calculate the semantic similarity between two sentences or documents, which can be applied to scenarios such as semantic search and text matching.

[0078] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0079] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0080] Before introducing this application solution, the relevant technologies are first introduced:

[0081] With the rapid development of artificial intelligence technology, large language models (hereinafter referred to as models) have been widely used in many fields. However, due to the size of hardware resources, it is challenging to deploy large language models in limited hardware resources and make them have greater advantages in answer accuracy, response speed, storage occupancy, and hardware power consumption. Common model compression methods include quantization, pruning, distillation, weight sharing, etc. Among them, quantization is a more commonly used compression method. Quantization refers to the process of using lower-precision numerical representations to replace 32-bit floating-point numbers.

[0082] Figure 1 This is a schematic diagram of the storage format of a 32-bit floating point number (fp32) in a computer provided in an embodiment of the present application. Figure 1 As shown, it contains 1 sign bit, 8 exponent bits and 23 decimal places (in the training process of the large model, the data type of fp16 or bf16 format may also be used to train the basic model, and the quantization principle is the same). The calculation formula for converting a 32-bit 0, 1 variable into a full-precision floating point number is:

[0083]

[0084] In formula (1), f represents a 32-bit full-precision floating point number; sign represents the sign bit; exponent represents the exponent bit; and fraction represents the decimal point.

[0085] Quantizing a 32-bit full-precision floating point number is the process of mapping it to a discrete value. Quantization methods include symmetric quantization and asymmetric quantization, as well as linear quantization and nonlinear quantization. Taking symmetric and linear quantization as an example, assuming that X represents a set of data consisting of 32-bit floating point numbers, a way to quantize the 32-bit floating point number into n bits can be as follows:

[0086]

[0087] q max =2 n-1 -1

[0088] q min =-2 n-1 ; (2)

[0089] In formula (2), X INTn represents an integer quantized to n bits; X FP32 Represents a 32-bit full-precision floating point number; q min Represents the minimum value of the integer corresponding to the n-bit number, q max Represents the maximum value of the integer corresponding to an n-bit number; Δ represents the quantization step; clamp represents the truncation operation; and round represents rounding.

[0090] According to the above content, the lower the number of bits after quantization, the smaller the memory required to store the model's weights and other parameters, and the lower the computational cost, but the error after quantization will also increase. At present, in the field of large models, since there are only 4 valid integer values ​​(-2, -1, 0, 1) when the quantization accuracy reaches 2 bits, the numerical accuracy varies too much, resulting in large fluctuations in the effect of the quantized model. Therefore, quantization schemes with a quantization accuracy greater than or equal to 4 bits are often used.

[0091] The quantized model is usually stored in memory and loaded into the hardware device to complete the inference process. For memory, the 4-bit quantized model still takes up a lot of space and has the possibility of being further optimized to a 2-bit model. However, the quantization error of the 2-bit model is large, which will affect the inference effect. Therefore, a quantization method that can effectively reduce the model size without excessively damaging the model performance has become a key issue that needs to be solved.

[0092] In view of this, the embodiments of the present application provide a model operation method, device, equipment, medium and product, and the process of quantizing a floating-point model into a 2-bit model is divided into two stages, namely, firstly quantizing the floating-point model into a second quantization model, such as a 4-bit model, and then quantizing the first quantization model into a first quantization model, such as a 2-bit model, and storing the first quantization model and its corresponding first quantization parameter in the memory.

[0093] During inference, the remaining storage resources of the hardware device are compared with the storage resources required by the quantization model, and the type of the target model that the hardware device can run is identified as the first quantization model or the second quantization model based on the comparison result. Then, based on the identification result, the first quantization model is directly loaded from the memory or the loaded first quantization model is dequantized to obtain the second quantization model, which is then applied to the inference process, thereby achieving the effect of small memory occupancy and high inference accuracy.

[0094] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0095] The present application embodiment first proposes a model operation method, see Figure 2 As shown, the method includes steps S101 to S103:

[0096] S101: Determine the remaining storage resources of a hardware device.

[0097] In this embodiment, the hardware device refers to various chips that can run a neural network model to achieve target task reasoning, such as general-purpose processors, specialized chips, and edge computing chips.

[0098] The general-purpose processor may be a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0099] The dedicated chip can be TPU (Tensor Processing Unit), FPGA (Field-Programmable Gate Array) or AIGC (Application-Specific Integrated Circuit).

[0100] Edge computing chips can be MCU (Microcontroller Unit) or low-power AI accelerators (Low-Power AI Accelerators).

[0101] Among them, the remaining storage resources of the hardware device can be obtained through commands or functions specifically used to obtain the remaining storage resources. For example, on a general-purpose processor or a dedicated chip, the information of the remaining storage resources can be obtained through tools provided by the operating system or APIs (Application Programming Interfaces) provided by programming libraries. For edge computing chips, the remaining storage resources can be monitored through interfaces provided by specialized management software or software development kits (SDKs).

[0102] S102: Determine the type of target model that can be run by the hardware device by comparing the remaining storage resources of the hardware device with the storage resources required by the quantization model.

[0103] The type of the target model includes a first quantization model or a second quantization model. The first quantization model and its corresponding first quantization coefficient are stored in a memory, and the first quantization parameter is used to dequantize the first quantization model to obtain a second quantization model. In addition, the bit width of the model parameter corresponding to the second quantization model is greater than the bit width of the model parameter corresponding to the first quantization model.

[0104] In some embodiments, in order to make a reasonable choice between different types of quantization models, determining the type of target model that the hardware device can run by comparing the remaining storage resources of the hardware device with the storage resources required by the quantization model may include step a1:

[0105] a1. According to the priorities of the first quantization model and the second quantization model, compare the remaining storage resources of the hardware device with the storage resources required by the high-priority quantization model, and determine the type of target model that the hardware device can run according to the comparison result.

[0106] The priority order of the first quantization model and the second quantization model can be preset manually. The priority order of the first quantization model and the second quantization model includes: the priority of the first quantization model is greater than the priority of the second quantization model, such as the first quantization model is a high priority and the second quantization model is a low priority; or, the priority of the second quantization model is greater than the priority of the first quantization model, such as the first quantization model is a low priority and the second quantization model is a high priority.

[0107] In some embodiments, when the priority of the second quantization model is greater than that of the first quantization model, step a1 includes the following steps a11 and a12:

[0108] a11. When the remaining storage resources of the hardware device are greater than or equal to the storage resources required by the second quantization model, determine that the type of the target model that can be run by the hardware device is the second quantization model.

[0109] For example, the hardware device is a GPU. The storage resource of the GPU is the video memory. When the remaining video memory of the GPU is greater than or equal to the video memory space required by the second quantization model (such as a 4-bit quantization model), it is determined that the target model that the hardware device can run is the second quantization model.

[0110] Since the 4-bit quantization model requires more video memory space than the 2-bit quantization model, when the remaining video memory of the GPU is sufficient, the 4-bit quantization model is preferred for inference.

[0111] a12. When the remaining storage resources of the hardware device are less than the storage resources required by the second quantization model, determine that the type of the target model that can be run by the hardware device is the first quantization model.

[0112] Continuing to refer to the example in step a11, when the remaining video memory of the GPU is smaller than the video memory space required by the second quantization model, it is determined that the type of the target model that the hardware device can run is the first quantization model (eg, a 2-bit quantization model).

[0113] At this time, since the remaining GPU memory is not enough to support the reasoning of the 4-bit quantization model, the 2-bit quantization model can be selected for reasoning in this case. Although the reasoning speed of the 2-bit quantization model is slower, it requires less GPU memory and can run under limited memory resources.

[0114] Through step a11 and step a12, a suitable quantization model can be selected for inference according to the remaining storage resources of the hardware device, thereby ensuring that the model can still be effectively run when resources are limited, while improving the inference speed and accuracy.

[0115] Continue reading Figure 2 , after step S102, the method may further include step S103:

[0116] S103. Obtain a corresponding target model according to the type of the target model, and execute the target task by running the target model through a hardware device.

[0117] On the basis of step a11, step S103 includes the following step a13:

[0118] a13. When the type of target model that the hardware device can run is the second quantization model, the first quantization model and its corresponding first quantization parameter are loaded from the memory, and the first quantization model is dequantized using the first quantization parameter to obtain the second quantization model, and the hardware device runs the second quantization model to perform the target task.

[0119] Among them, the quantized model and its corresponding quantization parameters are stored in the memory. During inference, the quantization model needs to be loaded from the memory. Since the first quantization model is stored in the memory, when it is determined that the target model that the hardware device (such as GPU) can run is the second quantization model (such as a 4-bit quantization model), it is first necessary to load the first quantization model (such as a 2-bit quantization model) and its corresponding first quantization parameter from the memory. Then, the first quantization model is dequantized into the second quantization model using the first quantization parameter. Finally, the hardware device runs the second quantization model to perform the target task.

[0120] On the basis of step a12, step S103 includes the following step a14:

[0121] a14. When the type of the target model that can be run by the hardware device is the first quantization model, the first quantization model is loaded from the memory, and the hardware device runs the first quantization model to perform the target task.

[0122] When it is determined that the type of target model that the hardware device can run is the first quantization model, the first quantization model is directly loaded from the memory to the hardware device, and the hardware device runs the first quantization model to perform the target task. Since the required storage resources are small, the effect of running the model for reasoning under resource-constrained conditions can be achieved.

[0123] Through step a13 and step a14, the corresponding type of model can be loaded from the memory according to the remaining storage resources of the hardware device and the selected target model type. For the model that needs to be dequantized, the dequantization process is completed using the quantization parameters, and the hardware device runs the selected model to perform the target task. This ensures that when resources are limited, the appropriate quantization model can be selected according to the actual situation for reasoning to complete the target task.

[0124] The above embodiment introduces how to select a suitable quantization model to run according to the remaining storage resources of the hardware device. Before selecting a suitable quantization model, the original floating-point model needs to be quantized. The quantization process of the floating-point model is to reduce the memory usage of the model and improve the operating efficiency on the hardware device. The following is the quantization process of the floating-point model:

[0125] The embodiment of the present application adopts a two-layer quantization scheme for the quantization process of the floating-point model, which specifically includes a first-stage quantization process and a second-stage quantization process. The first-stage quantization process includes: quantizing the original floating-point model into a 4-bit second quantization model. The second-stage quantization process includes: quantizing the 4-bit second quantization model into a 2-bit first quantization model. The 2-bit first quantization model is used for storage, and the 4-bit second quantization model obtained after 2-bit dequantization is used for reasoning.

[0126] That is to say, the first quantized model is obtained by quantizing the second quantized model; the second quantized model is obtained by quantizing the pre-trained floating-point model. Among them, pre-trained floating-point models refer to deep learning models that have been pre-trained on large-scale data sets. These models have been trained and can be used for a variety of downstream tasks, such as text classification, question-answering systems, image recognition, etc. Ways to obtain pre-trained floating-point models include: open source model libraries, official websites or third-party websites (such as specialized model database websites), links to pre-trained models included in papers and research reports, etc.

[0127] Figure 3 The schematic diagram of the principle of the quantization process of the second quantization model provided in the embodiment of the present application. Figure 3 As shown, the floating-point model includes various floating-point network layers connected in sequence, and each floating-point network layer corresponds to a pseudo-quantization network layer.

[0128] Among them, the original floating point model is quantized into a second quantization model of 4 bits, such as Figure 4 As shown, it includes the following steps S401 and S402:

[0129] S401 , determining pseudo quantization coefficients corresponding to respective pseudo quantization network layers by means of layer-by-layer quantization.

[0130] Layer-by-layer quantization refers to quantizing each layer in the network (including floating-point network layers and pseudo-quantized network layers) one by one. It helps optimize the model's storage requirements and inference speed while maintaining high inference accuracy.

[0131] Specifically, step S401 includes the following steps b1 to b3:

[0132] b1. Convert the i-1th floating-point output result into the i-th integer input result corresponding to the i-th pseudo-quantization network layer through the i-th activation value quantization coefficient of the i-th pseudo-quantization network layer corresponding to the i-th floating-point network layer and the i-1th floating-point output result of the i-1th pseudo-quantization network layer.

[0133] This step uses the activation value quantization coefficient of the current layer (layer i) to convert the floating-point output result of the previous layer (layer i-1) into the integer input result of the current layer. Taking layer 2 as an example, the activation value quantization coefficient of layer 2 is used to convert the floating-point output result of layer 1 into the integer input result of layer 2.

[0134] Continue reading Figure 2 , X represents a set of calibration data sets, Layer represents each floating-point network layer of the floating-point model, and QLayer represents each layer of the neural network including the pseudo-quantization process, hereinafter referred to as the pseudo-quantization network layer.

[0135] Taking i=2 as an example, the activation value quantization coefficient of the second pseudo-quantized network layer is recorded as s a2 , used to convert floating point input to int8 type input. The weight quantization coefficient is denoted by s 12 .

[0136] For floating point input X, convert it to int8 type data X using the following formula (3): int :

[0137]

[0138] b2. Determine the i-th floating-point output result of the i-th pseudo-quantization network layer according to the i-th integer input result and the i-th pseudo-quantization weight of the i-th pseudo-quantization network layer.

[0139] This step uses the integer input result of the current layer and the pseudo-quantization weight to perform calculations to obtain an integer output result, and dequantizes the integer output result into a floating-point output result.

[0140] Assuming that the pseudo-quantization weight of the second pseudo-quantization network layer is W2 = [2, 0; 2, 0], then X int After the second pseudo-quantization network layer, the pseudo-quantization weight W2 is first used for pseudo-quantization processing to obtain the output Y of type int8 2int :

[0141]

[0142] Furthermore, the weight quantization coefficient s of the pseudo quantization weight 12 and activation value quantization coefficient s a2 Y 2int Dequantize to floating point value Y1 and get the i-th floating point output result. This process can be expressed as the following formula:

[0143] Y2=s a2 *Y 2int *s 12 ; (5)

[0144] Among them, the formula of pseudo quantization weight W2 is as follows:

[0145]

[0146] Q s12 (W) = round (clamp (W / s 12 ))=W q ; (6)

[0147]

[0148] Where W∈R m×n ,s 12 ∈R m .

[0149] In the formula, Represents the weighted quantization coefficient s 12 Quantize the weights W of the floating-point network layer; Represents the weighted quantization coefficient s 12 The quantized weights are dequantized; the first line of formula (6) represents the weights W of the floating-point network layer are quantized and dequantized to obtain the weights of the pseudo-quantized network layer; the second line of formula (6) represents the weight quantization coefficient s 12 Quantize the weight W of the floating-point network layer to obtain the quantized weight W q ; The third line of formula (6) represents the quantized weight W q and weight quantization coefficient s 12 , which can be dequantized.

[0150] b3. Based on the difference between the i-th floating-point output result output by the i-th floating-point network layer and the floating-point dequantization result corresponding to the i-th floating-point output result, adjust the i-th activation value quantization coefficient and the i-th pseudo weight quantization coefficient.

[0151] Specifically, the quantization loss Loss is determined based on the difference between the i-th floating-point output result of the i-th floating-point network layer and the floating-point dequantization result corresponding to the i-th floating-point output result, and then back-propagation is performed through the Loss to adjust the i-th activation value quantization coefficient and the i-th pseudo-weight quantization coefficient, and training is continued until the quantization loss is less than the preset loss or the number of iterations reaches the preset number, and the i-th pseudo-quantization network training is completed. At this time, the i-th activation value quantization coefficient and the i-th weight quantization coefficient corresponding to the i-th pseudo-quantization network are stored.

[0152] The Loss may be determined based on the binary norm of the difference between the ith floating-point output result and the floating-point dequantization result corresponding to the ith floating-point output result.

[0153] That is, this step b3 is to calculate the difference between the floating-point output result of the current layer and the floating-point output result after dequantization, determine the quantization loss Loss based on the difference, adjust the activation value quantization coefficient and the weight quantization coefficient through back propagation of Loss, and continue training until the quantization loss is less than a preset threshold or reaches a preset number of iterations.

[0154] Among them, step b1 to step b3 is a training process for a pseudo-quantized network layer. After completing the training process of a pseudo-quantized network layer through steps b1 to step b3, the method of steps b1 to step b3 can also be used to continue training the next pseudo-quantized network layer, and so on, until the training of all pseudo-quantized networks is completed.

[0155] Through the above embodiments, the model can be quantized and trained layer by layer to ensure that each step of quantization reduces information loss as much as possible, thereby achieving higher inference accuracy in the final model while reducing storage requirements and improving inference speed. This process is applicable to each pseudo-quantized network layer in the model, and can be applied to all layers in turn until the quantization training of the entire model is completed.

[0156] It should be noted that during the first training process, the activation value quantization coefficients and the weight quantization coefficients are usually set to initial values, such as randomly set values, and these quantization coefficients are gradually adjusted through the above training process to reduce the quantization loss.

[0157] The above embodiment introduces that the activation value quantization coefficient is obtained by training. In some embodiments, the activation value quantization coefficient can also be obtained by dynamic calculation. The dynamic calculation method is: according to the current input X, it is calculated in a perchanel (channel by channel) manner:

[0158]

[0159] In the formula, s a1 Represents the activation value quantization coefficient; X is the current input; abs() represents the absolute value; max(dim=-1) represents the maximum value calculated along the specified dimension (usually the channel dimension); the bit in the denominator represents the quantization bit width.

[0160] The pseudo-quantization process with activation is introduced by taking matrix multiplication as an example. The floating-point matrix multiplication is Y=X*W, that is, after activation, X is output after weight W, and the corresponding pseudo-quantization process is:

[0161]

[0162] Y=X fake *W fake

[0163] In the formula, the first row of formulas represents the quantization and dequantization of the weight W through the pseudo weight quantization coefficient s1 to obtain the pseudo quantized weight; the second row of formulas represents the quantization and dequantization of the weight W through the pseudo activation value quantization coefficient to obtain the activated input; the third row of formulas represents the output result based on the pseudo quantized weight and the activated input. Here, s1 and sa represent the pseudo weight quantization coefficient and pseudo activation value quantization coefficient of any layer.

[0164] It is worth noting that the pseudo weight quantization coefficients and pseudo activation value quantization coefficients of each network layer obtained by layer-by-layer quantization may be completely different, partially the same, or even completely the same.

[0165] This embodiment can select different quantization bit widths for different layers through layer-by-layer quantization, thereby reducing the storage space required for the entire model.

[0166] Among them, the dynamic calculation method is suitable for scenarios where the input data changes greatly, because it can dynamically adjust the quantization coefficient according to the current input, which helps to reduce the quantization error. In addition, layer-by-layer quantization can flexibly adapt to the characteristics of different layers and optimize the performance of the entire model by selecting appropriate quantization bit widths for different layers.

[0167] Continue reading Figure 4 , after step S401, step S402 may also be included.

[0168] S402, quantizing the network parameters corresponding to each floating-point network layer using the pseudo-quantization coefficients corresponding to each pseudo-quantization network layer to obtain an initial second quantization model, and determining a second quantization model based on the initial second quantization model.

[0169] Among them, the pseudo quantization coefficient includes a pseudo activation value quantization coefficient and a pseudo weight quantization coefficient.

[0170] After the pseudo-quantization parameters of each pseudo-quantization network layer are trained, the weights of the full-precision floating-point model can be converted into 4-bit weights and corresponding scaling coefficients according to the pseudo-quantization parameters. Specifically, the corresponding floating-point network is quantized by the pseudo-activation value quantization coefficient and pseudo-weight quantization coefficient of each pseudo-quantization network to obtain the initial second quantization model.

[0171] After the initial second quantization model is obtained, the final second quantization model can be determined based on the initial second quantization model. There are multiple implementations of determining the final second quantization model based on the initial second quantization model, which can be specifically as follows:

[0172] In some implementations, the initial second quantization model may be directly determined as the final second quantization model. This approach is suitable for situations where no additional fine-tuning is required or a satisfactory performance level has been achieved.

[0173] In some implementations, the final second quantization model is determined based on the initial second quantization model, including: fixing the parameters of the initial second quantization model, adding a lora (Low-Rank Adaptation) module to the initial second quantization model, and fine-tuning the initial second quantization model using a lora fine-tuning training method to obtain the final second quantization model.

[0174] Specifically, the final second quantization model can be obtained by adding a lora module to the weights of each layer of the network of the initial second quantization model and fine-tuning it. Among them, the A matrix and B matrix of the lora module can be initialized by decomposing the weight error matrix before and after quantization according to the SVD. The specific formula is:

[0175]

[0176] Where, Y represents output; W lora represents the weight corresponding to the lora module; X represents the input;

[0177] It represents the dequantized first quantization model obtained by dequantizing the 2-bit first quantization model through the weight quantization coefficient s2 and the activation value quantization coefficient z2 of the first quantization model, that is, the 4-bit quantization model obtained by dequantization. Among them, the solution process of the weight quantization coefficient s2 and the activation value quantization coefficient z2 of the first quantization model will be introduced in the subsequent part.

[0178] Afterwards, the weight quantization error between the 4-bit quantization model obtained by quantizing the floating-point model and the 4-bit quantization model obtained by dequantization is expressed as the following formula (10):

[0179]

[0180] Where W represents the weight parameter of the 4-bit quantization model obtained by quantizing the floating-point model;

[0181] represents the weight parameter of the dequantized first quantization model obtained by dequantizing the 2-bit first quantization model through the weight quantization coefficient s2 of the first quantization model and the zero point value z2 in asymmetric quantization; the equal sign coordinate of formula (10) represents the quantization error, and the quantization error is decomposed by SVD to obtain the formula on the right side of the equal sign, where U on the right side of the equal sign represents the left factor matrix of the lora layer, ∑V * Represents the right factor matrix of the lora layer.

[0182] Formula (10) can be expressed as the following formula (11):

[0183] W lora=A*B; (11)

[0184] Lora fine-tuning is achieved by adding a low-rank matrix to the initial second quantization model weight matrix W. Specifically, the lora layer consists of two factor matrices A and B, both of which are low-rank, that is, their dimensions are small. The weight matrix W after lora fine-tuning is obtained by the two factor matrices A and B. lora .

[0185] In this way, LoRa fine-tuning can effectively adjust the model without significantly increasing the model size, thereby improving model performance. This approach is particularly suitable for resource-constrained devices because only the parameters of the LoRa layer need to be trained without retraining the entire model.

[0186] Through the above steps S401 and S402, the floating point model can be quantized into a 4-bit second quantization model to complete the first stage of quantization. However, the memory occupied by the 4-bit second quantization model is still large, so the 4-bit second quantization model can be further quantized into a 2-bit first quantization model.

[0187] Figure 5 The flowchart of the quantization process of the first quantization model provided in the embodiment of the present application is as follows. Figure 5 As shown, the quantization process of the first quantization model includes the following steps S501 to S503:

[0188] S501 . Based on the weight of the second quantization model, construct an objective function of the first quantization model with respect to the weight quantization error according to an asymmetric quantization scheme.

[0189] S502: Solve the objective function to obtain an asymmetric quantization factor of the first quantization model.

[0190] The asymmetric quantization factor includes a weight quantization coefficient s2 of the first quantization model and a zero point value z2 in asymmetric quantization.

[0191] The objective function is shown in formula (12):

[0192]

[0193] In formula (12), φ represents l p＜1 norm; s2 and z2 represent the weight quantization coefficient of the first quantization model and the zero value in asymmetric quantization; W bit4 Represents the weight of the second quantization model of 4 bits; It indicates that the weight of the second quantization model of 4 bits is quantized by the weight quantization coefficient s2 of the first quantization model and the zero point value z2 in the asymmetric quantization; It means that the quantized second quantization model is dequantized by using the weight quantization coefficient s2 of the first quantization model and the zero point value z2 in asymmetric quantization.

[0194] make We can get formula (13):

[0195] minφ(W e )

[0196]

[0197] Right now:

[0198] minφ(W e )

[0199]

[0200] The Lagrange multiplier method is used to transform formula (14) into:

[0201]

[0202] At this time, solving formula (15) is equivalent to solving formula (12). The following approximate solution is used to solve formula (15):

[0203] (1) Assuming z2 is 0, the grid search method is used to obtain the optimal value of s2, that is, the weight quantization coefficient, using formula (12) as the objective function.

[0204] (2) Based on s2 obtained in (1), s2 of the objective function shown in formula (15) is fixed, and then the coordinate rotation method is used to start with z2 fixed to 0 and cyclically optimize W e And z2. That is, solve formula (16) and formula (17) in turn, and finally obtain the value of the zero-point value variable z2 in asymmetric quantization, that is, the zero-point value.

[0205]

[0206] When solving formula (16), use formula (18) and (19) to calculate It is an exact solution when p=1 and an approximate solution when 0<p<1.

[0207]

[0208] When solving formula (17), and Substitute it into the formula and simplify it, and set its first-order derivative to zero. We can approximate the formula to calculate Z according to formulas (20) and (21): t+1 :

[0209]

[0210] S503 , quantize the weight of the second quantization model based on the weight of the second quantization model and the asymmetric quantization factor of the first quantization model to obtain the first quantization model.

[0211] Specifically, step S503 can be expressed as It means that the weight of the 4-bit second quantization model is quantized by the weight quantization coefficient s2 of the first quantization model and the zero point value z2 in the asymmetric quantization to obtain the first quantization model.

[0212] The embodiments of the present application accurately measure the remaining storage resources of the hardware device through commands or functions specifically used to obtain the remaining storage resources, ensuring that the actual available storage space is taken into account when selecting the model, thereby avoiding the situation where the model cannot run normally due to insufficient storage.

[0213] Furthermore, the remaining storage resources of the hardware device are compared with the storage resources required by the quantization model, and when the remaining storage resources are sufficient to support the second quantization model (such as a 4-bit quantization model), the second quantization model is preferentially selected for inference to obtain higher inference speed and better accuracy. Alternatively, when the remaining storage resources are insufficient to support the second quantization model, the first quantization model (such as a 2-bit quantization model) is selected. Although the inference speed is slower, the required storage space is smaller, which is more suitable for resource-constrained environments.

[0214] And, when it is determined that the hardware device can run the second quantization model, the first quantization model and its corresponding first quantization parameter are loaded from the memory, and the first quantization model is dequantized into the second quantization model by the first quantization parameter. Through the dequantization process, a higher precision model can be used for reasoning within a limited storage space, thereby improving the accuracy of reasoning.

[0215] By accurately evaluating the remaining storage resources of the hardware device, the first quantization model or the second quantization model is reasonably selected. Since the first quantization model occupies less memory, it is suitable for situations where storage space is limited. Therefore, even in the case of limited resources, a suitable model can be selected for efficient operation.

[0216] When hardware resources are sufficient, the second quantization model is preferred for inference to improve accuracy. The dequantization process allows even a lower-precision quantization model (first quantization model) to be dequantized into a higher-precision model (second quantization model), thereby improving inference accuracy under limited resources.

[0217] In summary, the technical solution proposed in this application can achieve the effect of small memory usage and high inference accuracy by accurately evaluating hardware resources, reasonably selecting quantization models, and adopting dequantization processes. It is especially suitable for resource-constrained edge devices and can effectively improve the operating efficiency and performance of the model.

[0218] Corresponding to the above-mentioned model operation method, the embodiment of the present application also proposes a model operation device, such as Figure 6 As shown, the device includes: a first determination unit 601, used to determine the remaining storage resources of the hardware device; a second determination unit 602, used to determine the type of target model that the hardware device can run by comparing the remaining storage resources of the hardware device with the storage resources required by the quantization model, the type of the target model is a first quantization model or a second quantization model, the first quantization model and its corresponding first quantization coefficient are stored in the memory, the first quantization parameter is used to dequantize the first quantization model to obtain the second quantization model, and the bit width of the model parameter corresponding to the second quantization model is greater than the bit width of the model parameter corresponding to the first quantization model; a model acquisition unit 603, used to acquire the corresponding target model according to the type of the target model, and the hardware device runs the target model to perform the target task.

[0219] In some embodiments, the second determination unit 602 determines the type of target model that the hardware device can run by comparing the remaining storage resources of the hardware device with the storage resources required for the quantization model, including: comparing the remaining storage resources of the hardware device with the storage resources required for the high-priority quantization model according to the priority of the first quantization model and the second quantization model, and determining the type of target model that the hardware device can run according to the comparison result.

[0220] In some embodiments, the priority of the second quantization model is greater than that of the first quantization model; wherein, the second determination unit 602 compares the remaining storage resources of the hardware device with the storage resources required by the high-priority quantization model according to the priorities of the first quantization model and the second quantization model, and determines the type of target model that the hardware device can run according to the comparison result, including: when the remaining storage resources of the hardware device are greater than or equal to the storage resources required by the second quantization model, determining that the type of target model that the hardware device can run is the second quantization model; wherein, the model acquisition unit 603 acquires the corresponding target model according to the type of the target model, and the hardware device runs the target model to perform the target task, including: when the type of target model that the hardware device can run is the second quantization model, loading the first quantization model and its corresponding first quantization parameter from the memory, and dequantizing the first quantization model through the first quantization parameter to obtain the second quantization model, and the hardware device runs the second quantization model to perform the target task.

[0221] In some embodiments, the priority of the second quantization model is greater than that of the first quantization model; wherein the second determination unit 602 compares the remaining storage resources of the hardware device with the storage resources required for the high-priority quantization model according to the priorities of the first quantization model and the second quantization model, and determines the type of target model that the hardware device can run according to the comparison result, including: when the remaining storage resources of the hardware device are less than the storage resources required by the second quantization model, determining that the type of target model that the hardware device can run is the first quantization model; wherein the model acquisition unit 603 acquires the corresponding target model according to the type of the target model, and the hardware device runs the target model to perform the target task, including: when the type of the target model that the hardware device can run is the first quantization model, loading the first quantization model from the memory, and the hardware device runs the first quantization model to perform the target task.

[0222] In some embodiments, the first quantization model is obtained by quantizing the second quantization model; the second quantization model is obtained by quantizing a pre-trained floating-point model, the floating-point model includes floating-point network layers connected in sequence, and each floating-point network layer corresponds to a pseudo-quantization network layer; wherein the second quantization model is obtained by the following processing: determining the pseudo-quantization coefficients corresponding to each pseudo-quantization network layer by layer quantization; quantizing the network parameters corresponding to each floating-point network layer based on the pseudo-quantization coefficients corresponding to each pseudo-quantization network layer to obtain an initial second quantization model, and determining the second quantization model based on the initial second quantization model.

[0223] In some embodiments, the method of determining the pseudo-quantization parameters corresponding to each pseudo-quantization network layer by quantization layer by layer includes performing the following steps on the i-th floating-point network layer in each floating-point network layer of the floating-point model: converting the i-1th floating-point output result into the i-th integer input result corresponding to the i-th pseudo-quantization network layer through the i-th activation value quantization coefficient of the i-th pseudo-quantization network layer corresponding to the i-th floating-point network layer and the i-1th floating-point output result of the i-1th pseudo-quantization network layer; determining the i-th floating-point output result of the i-th pseudo-quantization network layer according to the i-th integer input result and the i-th pseudo-quantization weight of the i-th pseudo-quantization network layer; and adjusting the i-th activation value quantization coefficient and the i-th pseudo-weight quantization coefficient based on the difference between the i-th floating-point output result output by the i-th floating-point network layer and the floating-point dequantization result corresponding to the i-th floating-point output result.

[0224] In some embodiments, determining the second quantization model based on the initial second quantization model includes: fixing the parameters of the initial second quantization model, adding a lora module to the initial second quantization model, and fine-tuning the initial second quantization model using a lora fine-tuning training method to obtain the second quantization model.

[0225] In some embodiments, the first quantization model is obtained by the following processing: based on the weight of the second quantization model, constructing the objective function of the first quantization model regarding the weight quantization error according to an asymmetric quantization scheme; solving the objective function to obtain the asymmetric quantization factor of the first quantization model; quantizing the weight of the second quantization model based on the weight of the second quantization model and the asymmetric quantization factor of the first quantization model to obtain the first quantization model.

[0226] The model operation device provided in this embodiment belongs to the same application concept as the model operation method provided in the above embodiments of this application, and can execute the model operation method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the model operation method. For technical details not fully described in this embodiment, please refer to the specific processing content of the model operation method provided in the above embodiments of this application, and will not be repeated here.

[0227] The functions implemented by each unit in the above model running device can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0228] It should be understood that the units in the above model running device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All units of the above device can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in the form of a processor calling software, and the remaining part can be implemented in the form of a hardware circuit.

[0229] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0230] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0231] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0232] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to a data input component via the interface circuit of the control device.

[0233] The data input component specifically refers to a functional component capable of inputting or collecting voice data, such as a microphone, etc.

[0234] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0235] The processor in the control device is also a circuit with signal processing capability, which executes the model operation method introduced in the above embodiment. The specific implementation of the processor can refer to the above processor implementation, and the embodiment of the present application does not strictly limit it.

[0236] Another embodiment of the present application also provides an electronic device, see Figure 7 As shown, the device includes:

[0237] Memory 200 and processor 210;

[0238] The memory 200 is connected to the processor 210 and is used to store programs;

[0239] The processor 210 is used to implement the model running method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0240] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0241] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0242] A bus may include a pathway that transfers information between components of a computer system.

[0243] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0244] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0245] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0246] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0247] Output device 240 may include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0248] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0249] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any model operation method provided in the above embodiments of the present application.

[0250] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs the program stored in the memory through the data interface to execute the model operation method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the introduction to the embodiment of the above-mentioned model operation method.

[0251] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the model operation method described in any of the above embodiments of this specification.

[0252] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0253] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the model operation method described in any of the above embodiments of this specification.

[0254] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0255] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0256] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0257] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0258] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0259] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0261] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0262] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0263] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0264] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model operation method, characterized in that: include: Determine the remaining storage resources of the hardware device; By comparing the remaining storage resources of the hardware device with the storage resources required by the quantization model, determining the type of the target model that the hardware device can run, the type of the target model is a first quantization model or a second quantization model, the first quantization model and its corresponding first quantization coefficient are stored in the memory, the first quantization parameter is used to dequantize the first quantization model to obtain the second quantization model, and the bit width of the model parameter corresponding to the second quantization model is greater than the bit width of the model parameter corresponding to the first quantization model; A corresponding target model is acquired according to the type of the target model, and the target model is run by the hardware device to execute the target task.

2. The method according to claim 1, characterized in that The determining the type of the target model that can be run by the hardware device by comparing the remaining storage resources of the hardware device with the storage resources required by the quantization model includes: According to the priorities of the first quantization model and the second quantization model, the remaining storage resources of the hardware device and the storage resources required by the high-priority quantization model are compared, and the type of the target model that the hardware device can run is determined according to the comparison result.

3. The method according to claim 2, characterized in that The priority of the second quantization model is greater than that of the first quantization model; The step of comparing the remaining storage resources of the hardware device with the storage resources required by the high-priority quantization model according to the priorities of the first quantization model and the second quantization model, and determining the type of the target model that can be run by the hardware device according to the comparison result, includes: In a case where the remaining storage resources of the hardware device are greater than or equal to the storage resources required by the second quantization model, determining that the type of the target model that can be run by the hardware device is the second quantization model; Wherein, acquiring a corresponding target model according to the type of the target model, and executing the target model by the hardware device to perform the target task, includes: When the type of target model that the hardware device can run is the second quantization model, the first quantization model and its corresponding first quantization parameter are loaded from the memory, and the first quantization model is dequantized using the first quantization parameter to obtain the second quantization model, and the hardware device runs the second quantization model to perform the target task.

4. The method according to claim 2, characterized in that: The priority of the second quantization model is greater than that of the first quantization model; The step of comparing the remaining storage resources of the hardware device with the storage resources required by the high-priority quantization model according to the priorities of the first quantization model and the second quantization model, and determining the type of the target model that can be run by the hardware device according to the comparison result, includes: In a case where the remaining storage resources of the hardware device are less than the storage resources required by the second quantization model, determining that the type of the target model that can be run by the hardware device is the first quantization model; Wherein, acquiring a corresponding target model according to the type of the target model, and executing the target model by the hardware device to perform the target task, includes: In a case where the type of the target model that can be run by the hardware device is the first quantization model, the first quantization model is loaded from the memory, and the hardware device runs the first quantization model to perform the target task.

5. The method according to any one of claims 1 to 4, characterized in that: The first quantization model is obtained by quantizing the second quantization model; The second quantization model is obtained by quantizing a pre-trained floating-point model, wherein the floating-point model includes sequentially connected floating-point network layers, and each of the floating-point network layers corresponds to a pseudo-quantization network layer; The second quantization model is obtained by the following processing: Determine the pseudo-quantization coefficients corresponding to each pseudo-quantization network layer by quantizing layer by layer; The network parameters corresponding to each floating-point network layer are quantized based on the pseudo-quantization coefficients corresponding to each pseudo-quantization network layer to obtain an initial second quantization model, and the second quantization model is determined based on the initial second quantization model.

6. The method according to claim 5, characterized in that The determining of the pseudo quantization parameters corresponding to each pseudo quantization network layer by layer quantization includes performing the following steps on the i-th floating point network layer in each floating point network layer of the floating point model: By using the i-th activation value quantization coefficient of the i-th pseudo-quantization network layer corresponding to the i-th floating-point network layer and the i-1-th floating-point output result of the i-1-th pseudo-quantization network layer, converting the i-1-th floating-point output result into the i-th integer input result corresponding to the i-th pseudo-quantization network layer; Determine an i-th floating-point output result of the i-th pseudo-quantization network layer according to the i-th integer input result and the i-th pseudo-quantization weight of the i-th pseudo-quantization network layer; Based on a difference between an i-th floating-point output result output by the i-th floating-point network layer and a floating-point dequantization result corresponding to the i-th floating-point output result, the i-th activation value quantization coefficient and the i-th pseudo weight quantization coefficient are adjusted.

7. The method according to claim 5, characterized in that The determining the second quantization model based on the initial second quantization model comprises: The parameters of the initial second quantization model are fixed, and a lora module is added to the initial second quantization model. The initial second quantization model is fine-tuned by using a lora fine-tuning training method to obtain the second quantization model.

8. The method according to any one of claims 1 to 4, characterized in that: The first quantitative model is obtained by the following processing: Based on the weights of the second quantization model, constructing an objective function of the first quantization model with respect to a weight quantization error according to an asymmetric quantization scheme; Solving the objective function to obtain an asymmetric quantization factor of the first quantization model; The weight of the second quantization model is quantized based on the weight of the second quantization model and the asymmetric quantization factor of the first quantization model to obtain a first quantization model.

9. A model operation device, characterized in that: include: A first determining unit, used to determine the remaining storage resources of the hardware device; a second determination unit, configured to determine a type of a target model that can be run by the hardware device by comparing the remaining storage resources of the hardware device with the storage resources required by the quantization model, wherein the type of the target model is a first quantization model or a second quantization model, wherein the first quantization model and its corresponding first quantization coefficient are stored in a memory, wherein the first quantization parameter is used to dequantize the first quantization model to obtain the second quantization model, and wherein a bit width of a model parameter corresponding to the second quantization model is greater than a bit width of a model parameter corresponding to the first quantization model; The model acquisition unit is used to acquire the corresponding target model according to the type of the target model, and the hardware device runs the target model to perform the target task.

10. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 8 by running the program in the memory.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method as claimed in any one of claims 1 to 8.

Images