A multi-modal large model training optimization method and device in the electric power vertical field
By adopting pruning and low-rank fine-tuning technologies in the multimodal large model in the vertical field of power, the problems of large number of models and high computing resources are solved, and the efficient deployment and accuracy of the model are achieved, which is suitable for power equipment defect identification and other Task.
Patent Information
- Application Number
- CN202411120027.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-15
AI Technical Summary
When training multimodal large models in the vertical field of power, the weight parameters are large, the video memory requirements are high, the training time is long and the computing resource requirements are high, which makes it difficult to implement the model.
The multimodal large model is processed by pruning method, combined with low-rank fine-tuning technology, pruning and fine-tuning the model by determining pruning criteria and pruning methods, reducing the model size and compensating for accuracy loss, which is suitable for business in the vertical field of power Scene.
It effectively reduces the demand for computing resources, reduces inference delay, and enhances model accuracy through LoRA fine-tuning module, making the model more suitable for deployment and training, and maintains the generalization ability of large models.
Smart Images

Figure CN118643470B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vertical large-scale model training in the electric power industry, and in particular to a multi-modal large-scale model training optimization method and device in the electric power vertical field. Background Art
[0002] In recent years, with the continuous growth of my country's electricity demand, the power system has become one of the most indispensable infrastructures in people's production and life. There are rich algorithm requirements in the power scenario, such as: the harm of wildfire smoke to transmission lines, the judgment of personnel operation norms, the estimation of power system status, the assessment and analysis of system potential risks, etc., all require the use of algorithms for analysis and diagnosis to improve safety, availability, and reduce costs. In the identification of defects in power equipment (blurred and damaged dials of equipment meters, cracked insulators, rusted metal parts, etc.), defective equipment poses a huge hidden danger to the operation of equipment. In the past, manpower was often unable to detect and warn in time, and the maintenance cost was high. Combined with multi-modal large model artificial intelligence algorithms, it can effectively reduce the investment of manpower and material resources, and achieve rapid response, accurate alarm and 24-hour monitoring.
[0003] With the rise of large language model technology, multimodal large models have great application potential due to their powerful cross-modal representation and generalization capabilities. Multimodal large model technology is a cutting-edge direction in the field of artificial intelligence, which aims to improve the model's understanding, reasoning and generation capabilities by combining different types of data (such as text, images, audio and video). By integrating information from multiple modalities, the model can more comprehensively understand complex queries and tasks and realize cross-modal knowledge fusion and application. Applying it to the rich scenarios of power systems can significantly improve performance.
[0004] Model implementation faces challenges of low latency, high throughput, and high efficiency. Model pruning and compression algorithms can simplify pre-trained models with large parameters and complex structures into smaller models, thereby reducing hardware storage, bandwidth, and computing requirements, and achieving the purpose of accelerating model reasoning and implementation.
[0005] However, due to the huge number of model parameters of multimodal large models, high requirements on the scale of training data and computing resources during training, high inference latency and difficult deployment, the model implementation faces the challenges of low latency, high throughput and high efficiency. In addition, for vertical field scenarios, the number of parameters and representation capabilities of large models are relatively redundant, which is a waste of hardware storage, bandwidth and computing resources. How to accelerate the deployment of multimodal large models in the power vertical field is an urgent problem to be solved. Summary of the invention
[0006] The present invention provides a multimodal large model training optimization method and device in the electric power vertical field to solve the problems of large number of weight parameters, high video memory requirements, long training time and high computing resource requirements during deployment during multimodal large model training, which are difficult to implement in the electric power vertical field.
[0007] The present invention is achieved through the following technical solutions:
[0008] A first aspect of the present invention provides a multi-modal large model training optimization method for power equipment defect identification, comprising:
[0009] Obtain multimodal data of power vertical business scenarios and compose image and text data pairs as training samples;
[0010] Determine a pre-trained large language model and a pre-trained visual model suitable for the electric power vertical business scenario according to the training sample resources and the computing resources, and fuse the pre-trained large language model and the pre-trained visual model to obtain a multimodal large model;
[0011] Determining a pruning criterion and a pruning method, and performing pruning processing on the multimodal large model according to the pruning criterion and the pruning method;
[0012] The pruned multimodal large model is subjected to low-rank fine-tuning training through the training samples to obtain a multimodal large model in the vertical field of electric power.
[0013] The present invention injects the pruning method into the multimodal large model to complete the training of business scenarios in the vertical field of electricity, solves the problems of large weight parameters, high video memory requirements, long training time and high computing resource requirements during large model training, and fine-tunes the pruned multimodal large model through a low-rank fine-tuning method to compensate for the loss of model accuracy caused by pruning. This can not only compress the model size to make it more suitable for training and deployment, but also maintain the powerful generalization ability of the large model, thereby obtaining an easy-to-deploy multimodal large model in the power field.
[0014] In one embodiment, the pruning method adopts a deep structured pruning method; the pruning process of the multimodal large model according to the pruning criterion and the pruning method includes:
[0015] A deep structured pruning method is used, with transformer block as the pruning unit;
[0016] Evaluate the importance score of the transformer block according to the pruning criteria;
[0017] Perform one-step pruning or multi-step iterative pruning based on the importance score of the transformer block.
[0018] In one implementation, the pruning criterion adopts the Taylor criterion, and the Taylor criterion expression is:
[0019] ;
[0020] In the formula, k Indicates the serial number of the computing unit in the model network, n express each The serial number of the transformer block, D To evaluate the dataset; Indicates that in the evaluation data set D The loss on Indicates k The nth transformer block in the computational unit i neurons and j The weight matrix of the neuron connections before pruning is calculated. Indicates n Taylor criterion importance score of a transformer block.
[0021] In one implementation, the pruning criterion is a perplexity criterion, and the perplexity criterion expression is:
[0022] ;
[0023] In the formula, S Represents a sequence of sentences; Indicates the token length; Indicates that the current model structure removes the The remaining part after the transformer block, Indicates removal of The loss of predicting the next token after a transformer block, Representing sentence sequences S The Tokens, Representing sentence sequences S 1st to All tokens, calculation results Indicates n The perplexity criterion importance score of a transformer block.
[0024] In one implementation, the one-step pruning based on the importance score of the transformer block includes:
[0025] Sort the transformer blocks according to their importance scores, and remove the N transformer blocks with the lowest order at one time, where N is an integer greater than 1 and less than the total number of transformer blocks.
[0026] In one implementation, the number N of pruned transformer block units is determined based on training sample resources, computing resources, image resolution, and text length.
[0027] In one embodiment, performing low-rank fine-tuning training on the pruned multimodal large model using the training samples includes:
[0028] The pruned pre-trained large language model and pre-trained visual model are injected into the LoRA fine-tuning module respectively;
[0029] During training, the pre-trained weight parameters are frozen, the gradient is not updated, and the multimodal large model is fully fine-tuned through the training samples.
[0030] In one embodiment, the pruned pre-trained large language model and pre-trained visual model are respectively injected into the LoRA fine-tuning module, including:
[0031] Inject the query layer and value layer of the attention mechanism module in all transformer blocks of the pruned pre-trained large language model into the LoRA module;
[0032] The query layer, value layer, and key layer of the attention mechanism module in all transformer blocks of the pruned pre-trained visual model are injected into the LoRA module.
[0033] In one embodiment, the fusing the pre-trained large language model and the pre-trained visual model to obtain a multimodal large model includes:
[0034] The pre-trained large language model and the pre-trained visual model are fused through the cross-attention mechanism of BERT, the language features of the pre-trained large language model and the visual features of the pre-trained visual model are aligned to obtain a multimodal large model.
[0035] In one implementation, the pre-trained large language model is LLaMA-7B, and the pre-trained visual model is ViT-G / 14.
[0036] A second aspect of the present invention provides a multimodal large model training optimization device for power equipment defect identification, comprising:
[0037] The training sample library is used to obtain multimodal data of power vertical business scenarios and form pairs of image and text data as training samples;
[0038] A model building module, used to determine a pre-trained large language model and a pre-trained visual model suitable for the power vertical business scenario according to training sample resources and computing resources, and to fuse the pre-trained large language model and the pre-trained visual model to obtain a multimodal large model;
[0039] A pruning module, used to determine a pruning criterion and a pruning method, and to perform pruning processing on the multimodal large model according to the pruning criterion and the pruning method;
[0040] The fine-tuning module is used to perform low-rank fine-tuning training on the pruned multimodal large model through the training samples to obtain a multimodal large model in the vertical field of electricity.
[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0042] The pruning method is injected into the multimodal large model to complete the training of the power vertical field scenario, and the low-rank fine-tuning method is used to improve the model accuracy of the pruned model, which accelerates the implementation of the multimodal large model in the power system while ensuring the model accuracy.
[0043] The importance of the pruning unit transformer block is sorted by pruning criteria, and the less important transformer blocks are pruned at one time, effectively reducing the inference delay.
[0044] Through BERT's cross-attention mechanism, a fusion enhancement method of language and visual modalities is introduced to fully capture the interactive characteristics of language and vision and improve the multimodal performance of the model.
[0045] By adopting deep structured pruning, the pruned structure is regular and not limited by batch size, which is conducive to improving the deployment reasoning efficiency, and the accuracy loss can be compensated by retraining.
[0046] Vertical domain task migration is completed through fine-tuning training. The LoRA module injected during training can be merged into the backbone structure of the original network after training is completed, and the output model can be directly used for deployment. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:
[0048] Figure 1 It is a flow chart of a multi-modal large model training optimization method in the electric power vertical field according to an embodiment of the present invention;
[0049] Figure 2 It is a block diagram of a multi-modal large model training and optimization device in the electric power vertical field according to an embodiment of the present invention;
[0050] Figure 3 It is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.
[0052] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to or inherent to other steps or units of the device.
[0053] The terms used in the various embodiments of the application are only used to describe the purpose of specific embodiments and are not intended to limit the various embodiments of the application. As used herein, the singular form is intended to also include the plural form, unless the context clearly indicates otherwise. Unless otherwise limited, all terms used here (including technical terms and scientific terms) have the same meaning as the meanings commonly understood by ordinary technicians in the field of the various embodiments of the application. The terms (such as the terms defined in the dictionary generally used) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning, unless clearly defined in the various embodiments of the application.
[0054] The embodiments of the present invention provide a multimodal large model training optimization method and device in the vertical field of electric power, which is suitable for the deployment of multimodal large models in the vertical field, and is conducive to reducing computing and deployment resources while ensuring model accuracy.
[0055] like Figure 1 As shown, Figure 1 This is a flow chart of a multi-modal large model training optimization method in the electric power vertical field according to an embodiment of the present invention, comprising the following steps:
[0056] S1, obtain multimodal data of power vertical business scenarios and form image and text data pairs as training samples;
[0057] S2, determining a pre-trained large language model and a pre-trained visual model suitable for the power vertical business scenario based on training sample resources and computing resources, and fusing the pre-trained large language model and the pre-trained visual model to obtain a multimodal large model;
[0058] S3, determining a pruning criterion and a pruning method, and pruning the multimodal large model according to the pruning criterion and the pruning method;
[0059] S4, through training samples, low-rank fine-tuning training is performed on the pruned multimodal large model to obtain a multimodal large model in the power vertical field.
[0060] In step S1, in response to the business needs in the power vertical field, pictures of power equipment defects in the vertical domain are obtained, and text descriptions are given to the pictures according to business needs. The description annotations need to be aligned with the current business scenario needs to form picture and text pairs. For example, in the power equipment defect identification business, pictures and corresponding text descriptions of defective equipment in the power system are obtained to form picture and text sample pairs in multimodal data as fine-tuning training resources for the multimodal large model of power equipment defect identification tasks.
[0061] During the text annotation process, the text annotation specifications for business scenarios can be summarized to form a unified and efficient annotation method and process.
[0062] In step S2, appropriate large language models and visual models are determined based on the training resources and deployed computing resources of the power business scenario. There are many open source models, and it is necessary to select them as pre-trained models based on the training resources and computing resources available.
[0063] Considering that we already have CPU computing resources, we estimate the CPU memory usage during the training phase based on the model parameters, and select the pre-trained model based on this. The estimation formula for the memory usage is:
[0064] ;
[0065] In the formula, represents the memory usage, Ψ is the number of model parameters, K It is the memory coefficient consumed by the optimizer, which is 12 (using mixed precision training and the optimizer is Adam).
[0066] In addition to processing computing resources, you can also consider training sample resources, that is, the scale of training samples that can be obtained in the power business scenario, and select a suitable pre-training model.
[0067] Furthermore, under the premise of meeting resource requirements, the criteria for selecting pre-trained models also include: A. Support capabilities for specific languages. In Chinese scenarios, it is more reasonable to use models pre-trained with a large number of Chinese corpus tokens, such as QwenLM, Baichuan2, ChatGLM, etc.; B. Requirements for input resolution. The parameter scale of the visual model can be determined according to the image resolution to select a suitable visual model; C. Deployment friendliness of the model architecture. Select models that are easy to deploy or applicable to existing architectures.
[0068] In actual business scenarios, for the power equipment defect identification business, the LLaMA-7B large language model and the ViT-G / 14 visual model were finally selected for fusion.
[0069] In one implementation, the fusion of the large language model and the visual model can align the language features of the large language model and the visual features of the visual model through the cross-attention mechanism of BERT.
[0070] The expression of the cross attention mechanism is as follows:
[0071] ;
[0072] In the formula, represents the cross attention mechanism, is the query embedding of the text, is the key embedding of the image, is the value embedding of the image, where The interval setting can be performed, that is, the embedding of text and image is fused every certain number of network layers. Query, Key and Value are three common concepts in the attention mechanism, and embedding represents its embedded features.
[0073] Through this step, the pre-trained model is flexibly selected according to business needs and model selection criteria, and the fusion enhancement method of language and visual modalities is introduced to fully capture the language and visual interaction characteristics and improve the multimodal performance of the model.
[0074] In step S3, vertical application scenarios are usually limited by training resources and deployment computing resources, and due to the verticality of the tasks, the parameters and characterization capabilities of the general large model are redundant for specific tasks. The power business scenario requirements include training resources, inference deployment computing resources, image resolution, text length, etc. These factors affect or restrict the training process and model training results of the business model. Therefore, while meeting the requirements, determine the models and the number of pruning units that need to be pruned in the multimodal large model so that the pruned model meets the resource requirements of the business.
[0075] In the multimodal architecture of language model and visual model, the language model often accounts for the majority of parameters, so the large language model can be pruned first.
[0076] First, determine the pruning criteria, pruning method, and pruning unit.
[0077] In this embodiment, the pruning method is preferably deep structured pruning (i.e., pruning and compressing the layers of the model, removing layers with weak representation capabilities, to achieve the purpose of streamlining the model), with transformer block as the pruning unit.
[0078] Compared with structured pruning and unstructured pruning, structured pruning refers to deleting some specific weights or filters at each layer level to reduce the number of parameters to be learned in the neural network. Structured pruning considers the entire network and does not prune each parameter independently. Unstructured pruning refers to gradually reducing the connections or parameters with smaller weights in the neural network to achieve the purpose of reducing the size and complexity of the model. Unstructured pruning selectively prunes each parameter without considering the specific structure or pattern of the network. Comparing structured pruning with unstructured pruning, the granularity of unstructured pruning is finer than that of structured pruning, and the accuracy is higher, but it is not conducive to deployment and difficult to accelerate; in comparison, although structured pruning has a loss of accuracy, the structure after pruning is regular, easy to deploy and accelerate, and the loss of accuracy can be compensated by retraining. Structured pruning is preferred.
[0079] In addition, compared with deep pruning and width pruning, width pruning prunes the coupled structure of multiple attention heads to streamline the network. This method is limited by the batch size of the data, and the optimization of inference speed is not obvious. However, deep pruning focuses on important blocks, and can effectively reduce the size of the model after pruning. At the same time, it is not limited by the batch size, and the efficiency gain of deploying inference is large. Therefore, the present invention preferably uses a structured deep pruning method to prune the transformer block.
[0080] Secondly, the importance score of the pruned unit is determined according to the pruning criteria, and the unit that needs to be pruned is determined based on this.
[0081] Among them, the pruning criteria can adopt Taylor criterion and perplexity level criterion (PPL).
[0082] The Taylor criterion focuses on the degree of error caused by removing the pruning unit (transformer block), that is, measuring the change in training loss before and after removing a transformer block. The measurement expression is as follows:
[0083] ;
[0084] In the formula, k Indicates the serial number of the computing unit (multi-head attention module, linear mapping layer module) in the model network, n Indicates the sequence number of each transformer block; i, j Represents the index number of the weight, i.e. i neurons and j The weights of the neuron connections, Expressing the k In the computational unit n The index number of the transformer block is ( i , j )’s weights are the weight matrix before pruning, To evaluate the dataset, Indicates k In the computational unit n The network structure of the transformer block before pruning in the evaluation dataset D The loss on Indicates k In the computational unit n The network structure of the transformer block after pruning is re-evaluated on the dataset D The loss on is the Taylor criterion expression, indicating that n The importance score of a transformer block. This formula reflects the change in loss before and after the transformer block is removed.
[0085] The expression of Taylor criterion is:
[0086] ;
[0087] In the formula, k Indicates the serial number of the computing unit (multi-head attention module, linear mapping layer module) in the model network, n Indicates ntransformer block; i , j Represents the index number of the weight, i.e. i and j The weights of the neuron connections, D To evaluate the dataset, Indicates that the network structure is evaluating data D The loss on loss ; Calculation results Indicates n Taylor criterion importance score of a transformer block. This evaluation criterion focuses on the impact of weight removal on training loss.
[0088] The expression of the perplexity criterion (PPL) is:
[0089] ;
[0090] In the formula, S Represents a sequence of sentences; Indicates the token length; Indicates that the current model structure removes the The remaining part after the transformer block, Indicates removal of The loss of predicting the next token after a transformer block, Representing sentence sequences S The Tokens, Representing sentence sequences S 1st to All tokens, calculation results Indicates n The perplexity criterion importance score of each transformer block. This method reflects the behavior of the model, has good generative performance, and has low reliance on heuristics for selecting pruning candidates.
[0091] Finally, based on the importance score, the model is pruned using either a one-step pruning method or a multi-step iterative pruning method.
[0092] In one embodiment, the pruning criterion adopts a fused Taylor criterion and perplexity criterion, that is, the Taylor criterion importance score and the perplexity criterion importance score are weighted averaged, and the average value is used as the importance score of the pruning unit transformerblock, and then subsequent pruning processing is performed, thereby integrating the respective advantages of the two criteria, avoiding the deviation of the evaluation results of a single criterion, and achieving the best pruning effect.
[0093] Among them, the one-step pruning method includes: sorting the importance of transformer blocks according to the pruning criteria (Taylor criterion, perplexity criterion or fused pruning criteria), and pruning the N transformer blocks with the lowest ranking at one time, where N is an integer greater than 1 and N is less than the total number of transformer blocks. N can be determined based on training sample resources, computing resources, image resolution and text length, for example, 15%, 20%, 30% of the total number of transformer blocks.
[0094] One-step pruning can effectively reduce inference latency. Although its accuracy is slightly lower than that of multi-step iterative pruning, it can be compensated by subsequent lightweight fine-tuning.
[0095] The multi-step iterative pruning process is as follows:
[0096] Step 1: Initialize pruning parameters. Specifically: initialize the number of pruning iterations T, the importance score threshold S, initialize model parameters, and the learning rate lr ;
[0097] Step 2: Get the validation set samples under the iteration number, input these samples into the network for forward calculation, calculate the importance score of each block according to the above PPL criterion, determine whether pruning is needed, determine the pruning unit, and the block with a score lower than the threshold S will be pruned;
[0098] Step 3: According to the pruned blocks determined in step 2, calculate the network structure before and after pruning in the test set data set D If pruning is not required, the network is directly calculated on the evaluation dataset. D losses;
[0099] Step 4: Calculate the absolute difference between the pruned loss calculated in step 3 and the loss of the original network structure (reference loss), and perform an exponential operation of e on the difference ( ), and use the obtained result as the loss back propagation of the pruned network to calculate the weight gradient value of the pruned network structure;
[0100] Step 5: Gradient descent method updates the weight parameters of the pruned network. ,in is the updated weight, is the weight before updating, lr is the learning rate, is the gradient calculated in step 4;
[0101] Step 6: Repeat steps 1 to 5 until the model converges.
[0102] In step S4, the pruned pre-trained model is injected into the LoRA fine-tuning module, that is, the pruned language model and the pruned visual model are respectively injected into the LoRA fine-tuning module, and the multimodal model integrating the language model and the visual model is fully fine-tuned through the business training set.
[0103] The expression of LoRA fine-tuning is:
[0104] ;
[0105] In the formula, is the pre-trained weight, , express dimensional real number matrix, freeze the pre-trained weight parameters during fine-tuning, and do not update the gradient; and represents the low-rank adaptation matrix, which is responsible for gradient updates during fine-tuning, and , represents the weights updated during fine-tuning training. So for normal forward operation: , after adding LoRA fine-tuning, it becomes:
[0106] ;
[0107] In the formula, h The inference output representing the original network structure, x Represents input data.
[0108] The final goal of LoRA is to optimize:
[0109]
[0110] In the formula, is the full model parameter, is the output prediction confidence of the original backbone network, For training data , t is the current data batch, Represents the total batches of current data training, Indicated in the data x and y Under the premise t The conditional probability of batches of data, represents the weight of the original backbone network structure, Model parameters fine-tuned for LoRA, where , that is, to achieve the purpose of fine-tuning a small number of parameters rather than all parameters.
[0111] Furthermore, the LoRA fine-tuning module is injected into the pruned language model and the pruned visual model respectively, specifically: the query layer and value layer of the attention mechanism module in all transformer blocks of the pruned pre-trained large language model are injected with the LoRA module; the query layer, value layer and key layer of the attention mechanism module in all transformer blocks of the pre-trained visual model are injected with the LoRA module.
[0112] During the training process, the multimodal large model injected with the LoRA fine-tuning module is fully or lightly fine-tuned through the training samples of the power vertical business scenario, that is, the image and text data pairs of the business data, so as to complete the domain migration of the task. And the pruned model is retrained using the LoRA fine-tuning method to compensate for the accuracy loss introduced by pruning.
[0113] Furthermore, the LoRA module injected during training is merged into the original backbone structure after fine-tuning training is completed, that is, the network structure of the original model before the LoRA module is inserted, and the output model can be directly used for deployment.
[0114] The second aspect of the present invention provides a multi-modal large model training optimization device in the electric power vertical field, such as Figure 2 The figure shows a block diagram of a multi-modal large model training optimization device in the power vertical field, which includes:
[0115] The training sample library 101 is used to obtain multimodal data of power vertical business scenarios and form pairs of image and text data as training samples;
[0116] A model building module 102 is used to determine a pre-trained large language model and a pre-trained visual model suitable for a power vertical business scenario according to training sample resources and computing resources, and to fuse the pre-trained large language model and the pre-trained visual model to obtain a multimodal large model;
[0117] A pruning module 103 is used to determine a pruning criterion and a pruning method, and prune the multimodal large model according to the pruning criterion and the pruning method;
[0118] The fine-tuning module 104 is used to perform low-rank fine-tuning training on the pruned multimodal large model through training samples to obtain a multimodal large model in the vertical field of electric power.
[0119] A third aspect of the present invention provides an electronic device, such as Figure 3 As shown, Figure 32 is a schematic diagram of the structure of an electronic device of the present invention, the electronic device includes a processor 21, a memory 22, an input device 23 and an output device 24; the number of processors 21 in the computer device can be one or more. Figure 3 A processor 21 is taken as an example; the processor 21, the memory 22, the input device 23 and the output device 24 in the electronic device can be connected by a bus or other means. Figure 3 The example of connecting through bus is taken in the following.
[0120] The memory 22 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules. The processor 21 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 22, that is, to implement the multi-modal large model training optimization method in the electric power vertical field of any of the above embodiments of the invention.
[0121] The memory 22 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0122] The input device 23 can be used to receive the ID and password input by the user, etc. The output device 24 is used to output the network configuration page.
[0123] In a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, a multimodal large model training optimization method for the electric power vertical field of any embodiment of the present invention is implemented. The storage medium may be ROM / RAM, a magnetic disk, an optical disk, etc.
[0124] A fifth aspect of the present invention provides a computer program product, which, when executed on a computer, enables the computer to execute the multimodal large model training optimization method in the electric power vertical field according to any one of the above-mentioned embodiments of the present invention.
[0125] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A multi-modal large model training optimization method for the power vertical field, characterized in that: include: Obtain multimodal data of power vertical business scenarios and compose image and text data pairs as training samples; Determine a pre-trained large language model and a pre-trained visual model suitable for the electric power vertical business scenario according to the training sample resources and the computing resources, and fuse the pre-trained large language model and the pre-trained visual model to obtain a multimodal large model; Determining a pruning criterion and a pruning method, and performing pruning processing on the multimodal large model according to the pruning criterion and the pruning method; Performing low-rank fine-tuning training on the pruned multimodal large model through the training samples to obtain a multimodal large model in the electric power vertical field; The pruning process of the multimodal large model according to the pruning criterion and the pruning method includes: A deep structured pruning method is used, with transformer block as the pruning unit; Evaluate the importance score of the transformer block according to the pruning criteria; Pruning based on the importance score of the transformer block: Step 1: Initialize the number of pruning iterations T, the importance score threshold S, the initial model parameters, and the learning rate lr ; Step 2: Obtain the validation set samples under the current iteration step number, input the validation set samples into the multimodal large model for forward calculation, calculate the importance score of each block according to the pruning criterion, and determine the blocks with a score lower than the score threshold S as pruning units; Step 3: According to the pruning unit determined in step 2, calculate the network structure before and after pruning in the test set data set D losses; Step 4: Calculate the absolute difference between the pruned loss calculated in step 3 and the loss of the original network structure, and perform an exponential operation of e on the absolute difference, that is, calculate , the calculation result is used as the loss of the pruned network to back propagate, and the weight gradient value of the pruned network structure is calculated; Step 5: Use the gradient descent method to update the weight parameters of the pruned network: ,in is the updated weight, is the weight before updating, lr is the learning rate, is the weight gradient calculated in step 4; Step 6: Repeat steps 1 to 5 until the model converges.
2. The multi-modal large model training optimization method for the electric power vertical field according to claim 1 is characterized in that: The pruning criterion adopts the Taylor criterion, and the Taylor criterion expression is: In the formula, k Indicates the sequence number of the computing unit in the model network, n Indicates the sequence number of each transformer block, D To evaluate the dataset; Indicates that in the evaluation data set D The loss on Expressing the k In the computational unit n The first transformer block i neurons and j The weight matrix of the neuron connections before pruning is calculated. Indicates n Taylor criterion importance score of a transformer block.
3. The multi-modal large model training optimization method for the electric power vertical field according to claim 1 is characterized in that: The pruning criterion is the perplexity criterion, and the perplexity criterion expression is: In the formula, S Represents a sequence of sentences; Indicates the token length; Indicates that the current model structure removes the The remaining part after the transformer block, Indicates removal of The loss of predicting the next token after a transformer block, Represents a sentence sequence S The Tokens, Represents a sentence sequence S 1st to All tokens, calculation results Indicates n The perplexity criterion importance score of a transformer block.
4. The multi-modal large model training optimization method for the electric power vertical field according to claim 1 is characterized in that: The one-step pruning based on the importance score of the transformer block includes: Sort the transformer blocks according to their importance scores and remove the N transformer blocks with the lowest order at one time, where N is an integer greater than 1 and less than the total number of transformer blocks.
5. The multi-modal large model training optimization method in the electric power vertical field according to claim 1 is characterized in that: The step of performing low-rank fine-tuning training on the pruned multimodal large model using the training samples includes: The pruned pre-trained large language model and pre-trained visual model are injected into the LoRA fine-tuning module respectively; During training, the pre-trained weight parameters are frozen, the gradient is not updated, and the multimodal large model is fully fine-tuned through the training samples.
6. The multi-modal large model training optimization method for the electric power vertical field according to claim 5 is characterized in that: The pruned pre-trained large language model and pre-trained visual model are respectively injected into the LoRA fine-tuning module, including: Inject the query layer and value layer of the attention mechanism module in all transformer blocks of the pruned pre-trained large language model into the LoRA module; The query layer, value layer, and key layer of the attention mechanism module in all transformer blocks of the pruned pre-trained visual model are injected into the LoRA module.
7. The multi-modal large model training optimization method for the electric power vertical field according to claim 1 is characterized in that: The fusing the pre-trained large language model and the pre-trained visual model to obtain a multimodal large model includes: The pre-trained large language model and the pre-trained visual model are fused through the cross-attention mechanism of BERT, the language features of the pre-trained large language model and the visual features of the pre-trained visual model are aligned to obtain a multimodal large model.
8. The multi-modal large model training optimization method for the electric power vertical field according to claim 1 is characterized in that: The pre-trained large language model selects LLaMA-7B, and the pre-trained visual model selects ViT-G / 14.
9. A multi-modal large model training and optimization device for the electric power vertical field, characterized in that: include: The training sample library is used to obtain multimodal data of power vertical business scenarios and form pairs of image and text data as training samples; A model building module, used to determine a pre-trained large language model and a pre-trained visual model suitable for the power vertical business scenario according to training sample resources and computing resources, and to fuse the pre-trained large language model and the pre-trained visual model to obtain a multimodal large model; A pruning module, used to determine a pruning criterion and a pruning method, and to perform pruning processing on the multimodal large model according to the pruning criterion and the pruning method; A fine-tuning module, used for performing low-rank fine-tuning training on the pruned multimodal large model through the training samples to obtain a multimodal large model in the electric power vertical field; The pruning process of the multimodal large model according to the pruning criterion and the pruning method includes: A deep structured pruning method is used, with transformer block as the pruning unit; Evaluate the importance score of the transformer block according to the pruning criteria; Pruning based on the importance score of the transformer block: Step 1: Initialize the number of pruning iterations T, the importance score threshold S, the initial model parameters, and the learning rate lr ; Step 2: Obtain the validation set samples under the current iteration step number, input the validation set samples into the multimodal large model for forward calculation, calculate the importance score of each block according to the pruning criterion, and determine the blocks with a score lower than the score threshold S as pruning units; Step 3: According to the pruning unit determined in step 2, calculate the network structure before and after pruning in the test set data set D losses; Step 4: Calculate the absolute difference between the pruned loss calculated in step 3 and the loss of the original network structure, and perform an exponential operation of e on the absolute difference, that is, calculate , the calculation result is used as the loss of the pruned network to back propagate, and the weight gradient value of the pruned network structure is calculated; Step 5: Use the gradient descent method to update the weight parameters of the pruned network: ,in is the updated weight, is the weight before updating, lr is the learning rate, is the weight gradient calculated in step 4; Step 6: Repeat steps 1 to 5 until the model converges.
Citation Information
Cited By
Two-stage electric power vertical field large model fine tuning method based on supervised fine tuning and reinforcement learning
CN121189483A
