Multi-modal large model perception quantitative training method and device, computer equipment and storage medium

Through perceptual quantization training technology, the multimodal model is optimized, which solves the problems of computational complexity and high memory usage on edge devices, and realizes efficient operation and accuracy maintenance in resource-limited environments.

CN118981604APending Publication Date: 2024-11-19STATE GRID HENAN ELECTRIC POWER ELECTRIC POWER SCI RES INST +1

Patent Information

Application Number
CN202411057371.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Multimodal large models are difficult to operate efficiently on edge devices with limited resources, and traditional processing methods cannot effectively analyze diversified power grid data, resulting in limited decision-making accuracy and timeliness.

Method used

By obtaining data of different mode grids, preprocessing and data cleaning, building a multi-modal large model, and using perceptual quantization training technology, setting the initial range of weights and activation values, inserting pseudo-quantization operators, simulating quantization operations, and quantization Errors are included in training until the model converges, and the quantized network is derived to reduce computational complexity and memory usage.

Benefits of technology

It realizes efficient operation of multimodal large models on embedded devices, reducing computational complexity and memory footprint, while maintaining model accuracy and adapting to the requirements of edge devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981604A_ABST
    Figure CN118981604A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a multi-modal large model sensing quantitative training method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring different modal power grid data; preprocessing the different modal power grid data to obtain a processing result; constructing a multi-modal large model; and utilizing the processing result to carry out perception quantitative training on the multi-modal large model. By implementing the method provided by the embodiment of the invention, the calculation complexity and memory occupation of the multi-modal large model can be effectively reduced, so that the multi-modal large model can adapt to the requirements of embedded equipment, and meanwhile, the precision of the model is kept.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a multimodal large model training method, and more specifically to a multimodal large model perceptual quantization training method, device, computer equipment and storage medium. Background Art

[0002] In the power industry, the management and maintenance of smart grids face the challenge of processing diverse data, including images, text, time series, etc. These data sources are extensive, including smart meter readings, surveillance camera images, equipment status logs, etc. Traditional processing methods are often unable to efficiently analyze these data, which affects the accuracy and timeliness of decision-making.

[0003] Multimodal large models, such as those that combine visual, language, and time series data, demonstrate great potential for processing data in this complex context. By integrating multi-source information, these models can provide more comprehensive analysis and prediction capabilities. However, multimodal large models usually have large parameters and high computational complexity, which makes them difficult to run efficiently on resource-limited edge devices. Quantization is one of the most effective ways to reduce the computation time and energy consumption of neural networks. In neural network quantization, the storage precision of weights and activation tensors is lower than the 16-bit or 32-bit precision usually used for training. When moving from 32 bits to 8 bits, the memory overhead of storing tensors is reduced by 4 times, while the computational cost of matrix multiplication is reduced by 16 times in a quadratic manner. Neural networks have been shown to be robust to quantization, which means that they can be quantized to lower bit widths with relatively little impact on the accuracy of the network. In addition, neural network quantization can often be applied together with other commonly used neural network optimization methods (such as neural architecture search, compression, and pruning). For any practical use case of deep learning, this is an essential step in the model efficiency pipeline. However, neural network quantization is not free. Low-bitwidth quantization introduces noise to the network, resulting in reduced accuracy. While some networks are robust to this noise, others require additional work to exploit the benefits of quantization.

[0004] Therefore, it is necessary to design a method to effectively reduce the computational complexity and memory usage of large multimodal models so that they can adapt to the requirements of embedded devices while maintaining the accuracy of the model. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a multi-modal large model perceptual quantization training method, device, computer equipment and storage medium.

[0006] To achieve the above object, the present invention adopts the following technical solution: a multi-modal large model perceptual quantization training method, comprising:

[0007] Obtain power grid data of different modes;

[0008] Preprocessing the different modal power grid data to obtain processing results;

[0009] Build a large multimodal model;

[0010] The processing results are used to perform perceptual quantization training on the multimodal large model.

[0011] A further technical solution is: the preprocessing of the different modal power grid data to obtain a processing result includes:

[0012] Performing data cleaning on the different modal power grid data to obtain cleaning results;

[0013] The cleaning result is converted into a data format to obtain a processing result.

[0014] Its further technical solution is: the multimodal large model includes an image processing layer for extracting image features, a text processing layer for extracting text features, and a multimodal fusion layer for fusing image features with text features.

[0015] A further technical solution is: using the processing results to perform perceptual quantization training on a multimodal large model, including:

[0016] Set the initial range of weights and activation values ​​and the initial value of activation values;

[0017] Inserting a pseudo-quantization operator into the weights and activation values ​​that need to be quantized in the multimodal large model to obtain a quantized network layer;

[0018] The processing result is used to simulate the quantization operation and utilize the pseudo quantization operator of the quantization network layer to incorporate the quantization error into the training of the multimodal large model.

[0019] A further technical solution is: using the processing result to simulate the quantization operation and using the pseudo quantization operator of the quantization network layer to incorporate the quantization error into the training of the multimodal large model, including:

[0020] Calculating the range of weights and activation values ​​of the quantized network layer;

[0021] According to the range, a quantization loss is introduced into the forward reasoning and backward parameter updating process until the network of the multimodal large model converges and a quantized network is obtained;

[0022] Export the quantized network.

[0023] A further technical solution is: introducing the quantization loss into the forward reasoning and backward parameter updating process according to the range until the network of the multimodal large model converges and a quantized network is obtained, including:

[0024] calculating a quantization parameter according to the range;

[0025] In the forward reasoning process, the quantization parameters are substituted into the quantization formula, the weights in the quantization network layer are converted into quantized integer values, and the quantization loss is calculated;

[0026] In the backward parameter update process, the gradient of quantization loss is considered and the network parameters of the multimodal large model are adjusted to minimize the overall loss.

[0027] Determining whether the network of the multimodal large model has converged;

[0028] If the network of the multimodal large model converges, deleting the pseudo quantization operator in the quantization network layer;

[0029] Inserting a quantization operator and an inverse quantization operator before and after the quantization network layer respectively to obtain a quantization network;

[0030] If the network of the multimodal large model has not converged, the step of calculating the quantization parameter according to the range is performed.

[0031] A further technical solution is: the quantization parameter includes a scale factor and a zero point.

[0032] The present invention also provides a multi-modal large model perceptual quantization training device, comprising:

[0033] A data acquisition unit, used for acquiring power grid data of different modes;

[0034] A preprocessing unit, used for preprocessing the power grid data of different modes to obtain a processing result;

[0035] A model building unit, used to build a multimodal large model;

[0036] The perceptual quantization training unit is used to perform perceptual quantization training on the multimodal large model using the processing results.

[0037] The present invention further provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program.

[0038] The present invention also provides a storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0039] Compared with the prior art, the present invention has the following advantages: the present invention collects power grid data of different modes and pre-processes them to obtain high-quality processing results. Then, the multi-modal large model is constructed by fusing the data of each mode. The model is optimized by using a suitable network structure and perceptual quantization training technology, so as to effectively reduce the computational complexity and memory usage of the multi-modal large model, adapt it to the requirements of embedded devices, and maintain the accuracy of the model.

[0040] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0042] Figure 1 A schematic diagram of an application scenario of a multimodal large model perceptual quantization training method provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a flow chart of a multimodal large model perceptual quantization training method provided by an embodiment of the present invention;

[0044] Figure 3 A schematic diagram of a sub-process of a multi-modal large model perceptual quantization training method provided in an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of a sub-process of a multi-modal large model perceptual quantization training method provided in an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of a sub-process of a multi-modal large model perceptual quantization training method provided in an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of a sub-process of a multi-modal large model perceptual quantization training method provided in an embodiment of the present invention;

[0048] Figure 7 A schematic block diagram of a multi-modal large model perceptual quantization training device provided by an embodiment of the present invention;

[0049] Figure 8 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0052] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0053] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0054] See also Figure 1 and Figure 2 , Figure 1 A schematic diagram of an application scenario of the multimodal large model perceptual quantization training method provided in an embodiment of the present invention. Figure 2 A schematic flow chart of a multi-modal large model perceptual quantization training method provided in an embodiment of the present invention. The multi-modal large model perceptual quantization training method is applied in a server. The server interacts with a terminal, obtains power grid data of different modes from the terminal, and then uses these data to perform perceptual quantization training on the constructed multi-modal large model; wherein, QAT (perceptual quantization training, Quantization-Aware Training) technology can effectively reduce the computational complexity and memory usage of the model, making it adaptable to the requirements of embedded devices while maintaining the accuracy of the model.

[0055] Neural networks are typically trained using FP32 weights and activations. If inference is performed in FP32, the processing elements and accumulators must support floating-point logic, and 32-bit data needs to be transferred from memory to the processing unit. During neural network inference, MAC operations and data transfers consume a lot of energy. Therefore, in perceptual quantization training, significant benefits can be obtained by using lower-bit fixed-point or quantized representations of these quantities. Low-bit fixed-point representations, such as INT8, not only reduce the amount of data transferred, but also reduce the size and energy consumption of MAC operations, because the cost of digital algorithms generally grows quadratically with the number of bits used, and because fixed-point addition is more efficient than floating-point addition.

[0056] Figure 2 Schematic diagram of the process of multi-modal large model perceptual quantization training method provided by an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S140.

[0057] S110, acquiring power grid data of different modes.

[0058] In this embodiment, the power grid data of different modalities include image data and text data of the power grid.

[0059] Grid operations involve complex and diverse business scenarios and multiple data sources, and the formats and types of data are also very diverse. For example, image data generated by smart meter readings and surveillance cameras can be obtained through image crawlers, API interfaces or manual collection; text data including equipment status logs, maintenance records, etc., these data can be used to extract useful information through natural language processing (NLP) tools.

[0060] The diversity of power grid data requires the collection process to be flexible and adaptable, able to process data in different formats and styles to support subsequent analysis and application.

[0061] S120: pre-process the power grid data of different modes to obtain processing results.

[0062] In this embodiment, the processing result refers to the data obtained after data cleaning and format conversion of different modal power grid data.

[0063] In one embodiment, see Figure 3 , the above-mentioned step S120 may include steps S121 to S122.

[0064] S121. Clean the power grid data of different modes to obtain a cleaning result.

[0065] In this embodiment, the cleaning result refers to the result obtained after cleaning the power grid data of different modes.

[0066] Specifically, data cleaning can include: Handling missing values: For missing data points, interpolation, mean filling, or more advanced machine learning methods (such as autoencoders) can be used to repair them. The specific repair method depends on the type and characteristics of the data, and the most suitable method needs to be selected to fill the missing values.

[0067] Outlier processing: Detect and process outliers through statistical methods (such as the Z-score method) or machine learning algorithms (such as the isolation forest algorithm). Outliers may interfere with data analysis and model training processes, so it is very important to identify and process outliers in a timely manner.

[0068] Data standardization and normalization: Standardize the data (such as z-score standardization) and normalize it (such as min-max normalization) to ensure that the data distribution of different features or variables is consistent. This helps to improve the stability and effect of model training, especially when using algorithms that need to calculate distances or compare numerical values.

[0069] These data preprocessing methods can effectively improve the quality and availability of power grid data and provide a more reliable foundation for subsequent analysis and application.

[0070] S122: convert the data format of the cleaning result to obtain a processing result.

[0071] In this embodiment, an image processing library (such as OpenCV) is used to convert the images acquired by the power grid monitoring camera into a tensor of fixed size. This processing method helps to standardize the format of image data so that subsequent machine learning models can effectively process and analyze it.

[0072] To pre-process text data such as equipment status logs and maintenance records, you can use natural language processing technology, such as BERT (Bidirectional Encoder Representations from Transformers). BERT can encode text data into word vectors, which helps understand and analyze the semantic information in the text.

[0073] These preprocessing methods can effectively prepare power grid data and make it suitable for various data analysis and machine learning tasks, thereby improving data availability and analysis efficiency.

[0074] S130. Build a large multimodal model.

[0075] In this embodiment, the multimodal large model includes an image processing layer for extracting image features, a text processing layer for extracting text features, and a multimodal fusion layer for fusing image features with text features.

[0076] The image processing layer uses convolutional neural networks (CNN) to extract image features and combines them with advanced models such as ResNet or EfficientNet to improve the accuracy and efficiency of image feature extraction.

[0077] The text processing layer uses transformers or recurrent neural networks (RNN) to process text data and uses pre-trained models such as BERT or GPT to extract text features.

[0078] The multimodal fusion layer combines the features of each modality through attention mechanism, weighted average or multimodal transformer (such as Perceiver) to form a comprehensive feature representation.

[0079] S140: Perform perceptual quantization training on the multimodal large model using the processing results.

[0080] In this embodiment, quantization-aware training simulates quantization during the training process, and uses a pseudo-quantization operator to include the accuracy loss caused by quantization into the training error, so that the optimizer can minimize the quantization error during the training process and obtain higher model accuracy.

[0081] In one embodiment, see Figure 3 , the above-mentioned step S140 may include steps S141 to S143.

[0082] S141. Set the weight, the initial range of the activation value, and the initial value of the activation value.

[0083] In this embodiment, the initial range q of weights and activation values ​​to be quantized in the multimodal large model is determined min and q max , and set their initial values. This is to ensure the stability and effectiveness of the subsequent quantization process.

[0084] Accurately setting the initial range and initial value can help control the errors that may occur during the quantization process, thereby improving the accuracy and stability of the quantized model.

[0085] S142. Insert a pseudo-quantization operator into the weights and activation values ​​that need to be quantized in the multimodal large model to obtain a quantized network layer.

[0086] In this embodiment, the quantized network layer refers to a network layer formed after the weights and activation values ​​that need to be quantized are inserted into a pseudo-quantization operator.

[0087] Pseudo-quantization operator is a technique that can preserve the representation accuracy of data in the analog quantization process while reducing the computation and storage requirements.

[0088] The use of pseudo-quantization operators can achieve high efficiency of quantization calculations while maintaining model accuracy, which is especially important for large multimodal models in resource-limited environments.

[0089] S143. Using the processing result to simulate the quantization operation and using the pseudo-quantization operator of the quantization network layer, the quantization error is incorporated into the training of the multimodal large model.

[0090] In one embodiment, see Figure 5 , the above-mentioned step S143 may include steps S1431 to S1433.

[0091] S1431. Calculate the range of weights and activation values ​​of the quantized network layer.

[0092] In this embodiment, before simulating the quantization operation, it is necessary to calculate the range q of the weights and activation values ​​of the quantized network layer. min and q max These ranges are crucial for subsequent quantization loss calculation and parameter tuning.

[0093] S1432. Introducing quantization loss into the forward reasoning and backward parameter updating process according to the range until the network of the multimodal large model converges and a quantized network is obtained.

[0094] In this embodiment, during the forward reasoning and backward parameter updating process, a quantization loss is introduced according to the calculated quantization loss range. Through this process, the network parameters of the multimodal large model are adjusted to converge to a quantized state.

[0095] Introducing quantization loss into the training process helps ensure that the quantized model can adapt to different computing resource requirements while maintaining accuracy and improving computing efficiency.

[0096] In one embodiment, see Figure 6 , the above-mentioned step S1432 may include steps S14321 to S14326.

[0097] S14321. Calculate a quantization parameter according to the range.

[0098] In this embodiment, the quantization parameters include a scale factor and a zero point. Accurately calculating the quantization parameters can ensure numerical accuracy and computational efficiency during the quantization process. Reasonable selection of the quantization parameters can minimize computational and storage requirements while maintaining model performance.

[0099] Quantization is the process of converting floating point numbers to integer representations, defined by three key parameters: scale factor (s), zero point (z), and bit width (b). These parameters vary for different types of quantization; for quantization training, asymmetric quantization and symmetric quantization are generally included. The scale factor (s) represents the scaling ratio of the floating point number to the integer grid, usually a floating point number. The zero point (z) ensures error-free quantization of actual zero values ​​and is an integer. The bit width (b) defines the range of the integer grid and is usually expressed as (2 b ), which is the number of digits in the integer.

[0100] Once the three quantization parameters are defined, the quantization operation can be performed.

[0101] First, we map a raw weight or activation value vector x to an unsigned integer grid {0,...,2 b -1},

[0102] Indicates rounding operation, the clamp(x; a, c) function is defined as follows:

[0103]

[0104] On the contrary, in order to recover the real data x from the quantized data, a dequantization operation is performed: Combining the quantization step and the dequantization step together, we get the general quantization function q(.):

[0105] Through the dequantization step, the quantization range (q min ,q max ), where q min =-sz,q max =(2 b -1-z). Any x value outside this range will be clipped to its limit, resulting in clipping errors (errors in precision). If you want to reduce clipping errors, you can increase the quantization range by increasing the scale factor s. However, increasing the scale factor will also increase rounding errors, because rounding errors are within the range

[0106] Symmetric quantization limits the zero point to 0, which simplifies the zero point offset processing in asymmetric quantization.

[0107] It is divided into signed and unsigned forms, suitable for different data distribution situations: unsigned symmetric quantization is suitable for data with a single-tail distribution, such as the activation value of ReLU; signed symmetric quantization is suitable for data with a symmetrical distribution around the zero point.

[0108] These forms of uniform quantization simplify the computational overhead of the quantizer by fixing the zeros and scaling factors, and by allowing the choice of integer grid type appropriate for the particular data distribution.

[0109] In summary, quantization achieves the conversion from floating-point numbers to integer representation by defining scale factors, zero points, and bit widths, as well as appropriate quantization and dequantization operations, thereby improving the operating efficiency and speed of deep learning models in embedded devices and resource-limited environments.

[0110] S14322. During the forward reasoning process, the quantization parameters are substituted into the quantization formula, the weights in the quantization network layer are converted into quantized integer values, and the quantization loss is calculated.

[0111] In this embodiment, during the forward reasoning process, the weights in the quantized network layer are converted into quantized integer values ​​by applying the quantization formula according to the calculated quantization parameters. At the same time, the precision loss of the quantized model is evaluated by calculating the quantization loss.

[0112] By quantizing integer values, efficient storage and calculation of weights can be achieved, thereby improving the speed and efficiency of the model on resource-limited devices. Calculating quantization loss helps evaluate the actual performance of the quantized model and provides a basis for subsequent optimization.

[0113] S14323. In the backward parameter update process, the ladder of quantization loss is considered and the network parameters of the multimodal large model are adjusted to minimize the overall loss.

[0114] In this embodiment, the gradient update considering the quantization loss can effectively optimize the parameters of the quantized model so that it can adapt to the numerical constraints and errors brought by quantization while maintaining accuracy, thereby improving the stability and generalization ability of the model.

[0115] S14324. Determine whether the network of the multimodal large model has converged.

[0116] In this embodiment, it is checked whether the training process of the multimodal large model has converged to a satisfactory state. Convergence means that the model has reached the expected performance level on a given training data set.

[0117] Confirming the convergence status of the model is an important step in the quantization training process. Only after the model converges can the quantized model be guaranteed to have acceptable performance in practical applications.

[0118] S14325. If the network of the multimodal large model converges, delete the pseudo-quantization operator in the quantization network layer.

[0119] In this embodiment, if the network of the multimodal large model has converged, it indicates that the quantization training has achieved the expected effect, and the pseudo-quantization operator in the quantization network layer can be safely deleted.

[0120] Removing pseudo-quantization operators can reduce the complexity and computational burden of the model, while simplifying the deployment and integration process of subsequent models.

[0121] S14326. Insert a quantization operator and an inverse quantization operator before and after the quantization network layer, respectively, to obtain a quantization network.

[0122] In this embodiment, a quantization operator and a dequantization operator are respectively inserted before and after the quantization network layer to ensure that the entire network structure can correctly handle quantization and dequantization operations during deployment and inference.

[0123] Inserting quantization operators and dequantization operators can ensure that the quantized model can correctly restore the original floating-point representation in actual deployment, ensuring the accuracy and consistency of the inference results.

[0124] In summary, the above steps, from calculating the quantization parameters to generating the final quantized network, ensure the effectiveness and reliability of the multimodal large model in the quantization process through strict quantization loss control and parameter adjustment, and provide an efficient quantization solution for practical applications.

[0125] If the network of the multimodal large model has not converged, execute S14321.

[0126] S1433. Export the quantization network.

[0127] In this embodiment, a quantized network is derived, that is, a quantized multimodal large model is obtained for use in practical applications.

[0128] After exporting the quantized network, the model can be deployed and run on resource-constrained devices, such as in embedded systems or mobile devices, thereby improving the practical application value of the model.

[0129] In summary, this perceptual quantization training method effectively realizes the quantization of multimodal large models by gradually processing the setting of weights and activation values, inserting pseudo-quantization operators, introducing quantization losses and other steps, thereby improving computational efficiency and application feasibility while maintaining model performance.

[0130] In addition, the method of this embodiment also designs an error compensation mechanism, and uses methods such as knowledge distillation to reduce the error introduced by quantization and improve the performance of the model after quantization. Knowledge distillation is a technique that captures the behavior and decision-making process of a complex model (teacher model) by training a simpler model. In quantization, the teacher model is usually an unquantized or high-precision model, while the student model is a quantized model. After quantization, the model may experience performance degradation or loss of accuracy, especially when processing small-range data. These problems can be improved by fine-tuning and calibrating the model, such as fine-tuning with higher-precision weights and activations, and calibrating to adjust quantization parameters; dynamic range adjustment can adjust the quantization parameters according to the statistical characteristics of the input data at runtime to adapt to different input ranges, thereby reducing quantization errors, such as adjusting quantization parameters.

[0131] Further optimize the trained quantized model, including model pruning and low-bit width operation optimization, to ensure its running efficiency and robustness on edge devices; use tools (such as TensorFlow Lite, ONNX Runtime) to convert the model into a format suitable for edge devices, compile and optimize it; deploy the optimized model to edge devices to monitor power system data in real time, provide intelligent early warning and decision support. Design an automatic update mechanism to regularly update and optimize the model based on new data.

[0132] The method of this embodiment innovatively integrates image and text data, improving the comprehensiveness and accuracy of power system data analysis. By introducing multimodal converters and attention mechanisms, efficient fusion of data of different modalities is achieved. The use of quantization-aware training technology significantly reduces the computational complexity and memory usage of the model, enabling large models to run efficiently on resource-constrained edge devices. The combination of dynamic quantization and mixed precision training further improves the performance and flexibility of the quantization model. By deploying quantization models on edge devices, real-time monitoring and intelligent early warning of the power system are achieved, improving the reliability and security of the system. An automatic update mechanism is designed to ensure continuous optimization of the model in a dynamic environment; an error compensation mechanism is introduced to reduce the error introduced by quantization through methods such as knowledge distillation, thereby improving the accuracy and stability of the model. This is particularly important in the context of high reliability requirements for power systems.

[0133] The above-mentioned multimodal large model perceptual quantization training method collects power grid data of different modes and preprocesses it to obtain high-quality processing results. Then, it constructs a multimodal large model by fusing the data of each mode. The model is optimized by using a suitable network structure and perceptual quantization training technology, thereby effectively reducing the computational complexity and memory usage of the multimodal large model, making it adaptable to the requirements of embedded devices while maintaining the accuracy of the model.

[0134] Figure 7 is a schematic block diagram of a multi-modal large model perceptual quantization training device 300 provided in an embodiment of the present invention. Figure 7 As shown, corresponding to the above multimodal large model perceptual quantization training method, the present invention also provides a multimodal large model perceptual quantization training device 300. The multimodal large model perceptual quantization training device 300 includes a unit for executing the above multimodal large model perceptual quantization training method, and the device can be configured in a server. Specifically, please refer to Figure 7 The multimodal large model perceptual quantization training device 300 includes a data acquisition unit 301, a preprocessing unit 302, a model construction unit 303 and a perceptual quantization training unit 304.

[0135] The data acquisition unit 301 is used to acquire power grid data of different modes; the preprocessing unit 302 is used to preprocess the power grid data of different modes to obtain processing results; the model construction unit 303 is used to construct a multi-modal large model; the perceptual quantization training unit 304 is used to perform perceptual quantization training on the multi-modal large model using the processing results.

[0136] In one embodiment, the pre-processing unit 302 includes a data cleaning sub-unit and a format conversion sub-unit.

[0137] The data cleaning subunit is used to clean the power grid data of different modes to obtain a cleaning result; the format conversion subunit is used to convert the data format of the cleaning result to obtain a processing result.

[0138] In one embodiment, the perceptual quantization training unit 304 includes an initial value setting subunit, an operator insertion subunit, and an analog quantization training subunit.

[0139] The initial value setting subunit is used to set the initial range of weights and activation values ​​and the initial value of activation values; the operator insertion subunit is used to insert pseudo-quantization operators into the weights and activation values ​​that need to be quantized in the multimodal large model to obtain a quantized network layer; the simulated quantization training subunit is used to use the processing results to simulate quantization operations and use the pseudo-quantization operator of the quantized network layer to incorporate the quantization error into the training of the multimodal large model.

[0140] In one embodiment, the analog quantization training subunit includes a calculation module, a loss introduction module and a derivation module.

[0141] A calculation module is used to calculate the range of weights and activation values ​​of the quantized network layer; a loss introduction module is used to introduce quantized losses into the forward reasoning and backward parameter update process according to the range until the network of the multimodal large model converges and a quantized network is obtained; an export module is used to export the quantized network.

[0142] In one embodiment, the loss introduction module includes a parameter calculation submodule, a forward reasoning submodule, a backward parameter update submodule, a judgment submodule, a deletion submodule and an operator insertion submodule.

[0143] A parameter calculation submodule is used to calculate the quantization parameters according to the range; a forward reasoning submodule is used to substitute the quantization parameters into the quantization formula during the forward reasoning process, convert the weights in the quantization network layer into quantization integer values, and calculate the quantization loss; a backward parameter update submodule is used to consider the ladder of quantization loss during the backward parameter update process, and adjust the network parameters of the multimodal large model to minimize the overall loss; a judgment submodule is used to judge whether the network of the multimodal large model converges; if the network of the multimodal large model does not converge, the calculation of the quantization parameters according to the range is executed. A deletion submodule is used to delete the pseudo-quantization operator in the quantization network layer if the network of the multimodal large model converges; an operator insertion submodule is used to insert a quantization operator and an anti-quantization operator before and after the quantization network layer to obtain a quantization network.

[0144] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned multimodal large model perceptual quantization training device 300 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and conciseness of the description, it will not be repeated here.

[0145] The multi-modal large model perceptual quantization training device 300 can be implemented in the form of a computer program. The computer program can be used in Figure 8 Runs on the computer device shown.

[0146] See also Figure 8 , Figure 8 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a server, wherein the server may be an independent server or a server cluster composed of multiple servers.

[0147] See also Figure 8 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0148] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, can enable the processor 502 to execute a multi-modal large model perceptual quantization training method.

[0149] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0150] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a multi-modal large model perceptual quantization training method.

[0151] The network interface 505 is used to communicate with other devices over the network. Figure 8 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0152] The processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps:

[0153] Acquire different modal power grid data; preprocess the different modal power grid data to obtain processing results; construct a multi-modal large model; and use the processing results to perform perceptual quantization training on the multi-modal large model.

[0154] The multimodal large model includes an image processing layer for extracting image features, a text processing layer for extracting text features, and a multimodal fusion layer for fusing image features with text features.

[0155] In one embodiment, when the processor 502 implements the step of preprocessing the power grid data of different modes to obtain a processing result, the processor 502 specifically implements the following steps:

[0156] The different modal power grid data are cleaned to obtain a cleaning result; and the cleaning result is converted into a data format to obtain a processing result.

[0157] In one embodiment, when the processor 502 implements the step of performing perceptual quantization training on the multimodal large model using the processing result, the processor 502 specifically implements the following steps:

[0158] The initial range of weights and activation values ​​and the initial value of the activation value are set; a pseudo-quantization operator is inserted into the weights and activation values ​​that need to be quantized in the multimodal large model to obtain a quantized network layer; the quantization error is incorporated into the training of the multimodal large model by simulating the quantization operation using the processing result and using the pseudo-quantization operator of the quantized network layer.

[0159] In one embodiment, when the processor 502 implements the step of using the processing result to simulate the quantization operation and using the pseudo quantization operator of the quantization network layer to incorporate the quantization error into the training step of the multimodal large model, the processor 502 specifically implements the following steps:

[0160] Calculate the range of weights and activation values ​​of the quantized network layer; introduce quantization loss into the forward reasoning and backward parameter update process according to the range until the network of the multimodal large model converges and obtains a quantized network; and derive the quantized network.

[0161] In one embodiment, when the processor 502 implements the step of introducing the quantization loss into the forward reasoning and backward parameter updating process according to the range until the network of the multimodal large model converges and a quantized network is obtained, the processor 502 specifically implements the following steps:

[0162] Calculate quantization parameters according to the range; in the forward reasoning process, substitute the quantization parameters into the quantization formula, convert the weights in the quantization network layer into quantized integer values, and calculate the quantization loss; in the backward parameter update process, consider the ladder of the quantization loss and adjust the network parameters of the multimodal large model to minimize the overall loss; determine whether the network of the multimodal large model converges; if the network of the multimodal large model converges, delete the pseudo-quantization operator in the quantization network layer; insert a quantization operator and an inverse quantization operator before and after the quantization network layer, respectively, to obtain a quantized network; if the network of the multimodal large model does not converge, execute the calculation of quantization parameters according to the range.

[0163] The quantization parameters include a scale factor and a zero point.

[0164] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0165] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0166] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0167] Acquire different modal power grid data; preprocess the different modal power grid data to obtain processing results; construct a multi-modal large model; and use the processing results to perform perceptual quantization training on the multi-modal large model.

[0168] The multimodal large model includes an image processing layer for extracting image features, a text processing layer for extracting text features, and a multimodal fusion layer for fusing image features with text features.

[0169] In one embodiment, when the processor executes the computer program to implement the step of preprocessing the power grid data of different modes to obtain a processing result, the processor specifically implements the following steps:

[0170] The different modal power grid data are cleaned to obtain a cleaning result; and the cleaning result is converted into a data format to obtain a processing result.

[0171] In one embodiment, when the processor executes the computer program to implement the step of performing perceptual quantization training on the multimodal large model using the processing result, the processor specifically implements the following steps:

[0172] The initial range of weights and activation values ​​and the initial value of the activation value are set; a pseudo-quantization operator is inserted into the weights and activation values ​​that need to be quantized in the multimodal large model to obtain a quantized network layer; the quantization error is incorporated into the training of the multimodal large model by simulating the quantization operation using the processing result and using the pseudo-quantization operator of the quantized network layer.

[0173] In one embodiment, when the processor executes the computer program to implement the step of using the processing result to simulate the quantization operation and using the pseudo quantization operator of the quantization network layer to incorporate the quantization error into the training step of the multimodal large model, the processor specifically implements the following steps:

[0174] Calculate the range of weights and activation values ​​of the quantized network layer; introduce quantization loss into the forward reasoning and backward parameter update process according to the range until the network of the multimodal large model converges and obtains a quantized network; and derive the quantized network.

[0175] In one embodiment, when the processor executes the computer program to implement the step of introducing the quantization loss into the forward reasoning and backward parameter updating process according to the range until the network of the multimodal large model converges and a quantized network is obtained, the processor specifically implements the following steps:

[0176] Calculate quantization parameters according to the range; in the forward reasoning process, substitute the quantization parameters into the quantization formula, convert the weights in the quantization network layer into quantized integer values, and calculate the quantization loss; in the backward parameter update process, consider the ladder of the quantization loss and adjust the network parameters of the multimodal large model to minimize the overall loss; determine whether the network of the multimodal large model converges; if the network of the multimodal large model converges, delete the pseudo-quantization operator in the quantization network layer; insert a quantization operator and an inverse quantization operator before and after the quantization network layer, respectively, to obtain a quantized network; if the network of the multimodal large model does not converge, execute the calculation of quantization parameters according to the range.

[0177] The quantization parameters include a scale factor and a zero point.

[0178] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0179] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0180] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0181] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0183] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-modal large model perceptual quantization training method, characterized in that: include: Obtain power grid data of different modes; Preprocessing the different modal power grid data to obtain processing results; Build a large multimodal model; The processing results are used to perform perceptual quantization training on the multimodal large model.

2. The multimodal large model perceptual quantization training method according to claim 1, characterized in that: The preprocessing of the different modal power grid data to obtain processing results includes: Performing data cleaning on the different modal power grid data to obtain cleaning results; The cleaning result is converted into a data format to obtain a processing result.

3. The multimodal large model perceptual quantization training method according to claim 1, characterized in that: The multimodal large model includes an image processing layer for extracting image features, a text processing layer for extracting text features, and a multimodal fusion layer for fusing image features with text features.

4. The multimodal large model perceptual quantization training method according to claim 1, characterized in that: The method of using the processing results to perform perceptual quantization training on the multimodal large model includes: Set the initial range of weights and activation values ​​and the initial value of activation values; Inserting a pseudo-quantization operator into the weights and activation values ​​that need to be quantized in the multimodal large model to obtain a quantized network layer; The processing result is used to simulate the quantization operation and utilize the pseudo quantization operator of the quantization network layer to incorporate the quantization error into the training of the multimodal large model.

5. The multimodal large model perceptual quantization training method according to claim 4, characterized in that: The method of utilizing the processing result to simulate the quantization operation and utilizing the pseudo quantization operator of the quantization network layer to incorporate the quantization error into the training of the multimodal large model includes: Calculating the range of weights and activation values ​​of the quantized network layer; According to the range, a quantization loss is introduced into the forward reasoning and backward parameter updating process until the network of the multimodal large model converges and a quantized network is obtained; Export the quantized network.

6. The multimodal large model perceptual quantization training method according to claim 5, characterized in that: The step of introducing the quantization loss into the forward reasoning and backward parameter updating process according to the range until the network of the multimodal large model converges and a quantized network is obtained includes: calculating a quantization parameter according to the range; In the forward reasoning process, the quantization parameters are substituted into the quantization formula, the weights in the quantization network layer are converted into quantized integer values, and the quantization loss is calculated; In the backward parameter update process, the gradient of quantization loss is considered and the network parameters of the multimodal large model are adjusted to minimize the overall loss. Determining whether the network of the multimodal large model has converged; If the network of the multimodal large model converges, deleting the pseudo quantization operator in the quantization network layer; Inserting a quantization operator and an inverse quantization operator before and after the quantization network layer respectively to obtain a quantization network; If the network of the multimodal large model has not converged, the step of calculating the quantization parameter according to the range is performed.

7. The multimodal large model perceptual quantization training method according to claim 6, characterized in that: The quantization parameters include a scale factor and a zero point.

8. A multi-modal large model perceptual quantization training device, characterized in that: include: A data acquisition unit, used for acquiring power grid data of different modes; A preprocessing unit, used for preprocessing the power grid data of different modes to obtain a processing result; A model building unit, used to build a multimodal large model; The perceptual quantization training unit is used to perform perceptual quantization training on the multimodal large model using the processing results.

9. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Image-text matching model compression and acceleration method based on orthogonal similarity distillation and system thereof

    CN112990296A

  • Neural network model quantitative deployment method supporting multiple hardware platforms

    CN116362287A

  • Multi-mode fruit sensing system and device and storage medium

    CN117095153A

  • Model compression method and device, electronic equipment and storage medium

    CN117371508A

  • Method and apparatus for extracting multi-modal POI feature

    WO2022198854A1

Cited By

  • Multi-modal large language model quantification method and device, equipment, storage medium and product

    CN119227770A

  • Model quantification method and system based on hardware perception, medium, product and terminal

    CN121168541A