Marketing and distribution fusion scene-oriented multi-modal large model efficient fine tuning method and system

Abstract

The invention belongs to the technical field of artificial intelligence and power system crossing, and discloses a marketing and distribution fusion scene-oriented multi-modal large model efficient fine tuning method and system, and the method comprises the steps: firstly constructing a small amount of high-quality multi-modal seed data with a thinking chain by field experts; performing domain knowledge injection and multi-modal alignment on the base model in stages by adopting a parameter efficient fine tuning technology; the method comprises the following steps of: selecting an unlabeled sample, further introducing an active learning iterative loop based on hybrid uncertainty perception, automatically screening the unlabeled sample with the most rich information amount by quantifying cognitive uncertainty and accidental uncertainty of a model, and labeling the unlabeled sample by an expert, so as to expand a data continuous optimization model and form a'fine tuning-evaluation-labeling 'closed loop. According to the method, rapid and accurate adaptation of the multi-modal large model in a marketing and distribution fusion complex scene is realized with extremely low expert labeling cost, the service reliability and safety of model output are ensured, and a long-acting mechanism of sustainable evolution of the model is established.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of artificial intelligence and power system, specifically involving a fine-tuning method for large language models and multimodal large models, and in particular a method and system for efficient iterative fine-tuning using a small amount of expert-annotated data for the integrated operation and maintenance business scenario of power system. Background Technology

[0002] With the development of multimodal large model technology, its application prospects in the field of "integrated operation and maintenance" in power systems (i.e., the integration of marketing services and distribution operation and maintenance) are broad, and it can be used in complex scenarios such as intelligent customer service, fault diagnosis, inspection report generation, and power supply scheme assistance. However, adapting general large models to the highly specialized and safety-sensitive power field faces the following core challenges:

[0003] (1) Scarcity of high-quality labeled data: Domain experts (such as senior schedulers and technical experts) have valuable time and can provide very little accurate labeled data, which is difficult to support traditional large-scale supervision and fine-tuning.

[0004] (2) Difficulty in understanding and aligning multimodal data: Operation and maintenance business involves various heterogeneous data such as work order text, equipment images, topology graphs, and time-series load curves. The model needs to deeply integrate and understand the complex relationships between these modalities.

[0005] (3) Low integration of domain knowledge: The general model lacks deep domain knowledge such as power grid topology, safety regulations, and business processes, which may lead to outputs that are technically infeasible or do not comply with safety standards.

[0006] (4) High iteration cost: Traditional fine-tuning methods lack targeted data screening mechanisms, and each iteration requires labeling a large amount of random data, which is inefficient and makes it difficult to achieve continuous model evolution.

[0007] While existing technologies employ efficient parameter fine-tuning techniques based on LoRA to reduce computational load, they still cannot fundamentally solve the problem of how to guide models to master core domain capabilities with minimal expert intervention and maximum precision. Therefore, there is an urgent need for a highly targeted, data-efficient fine-tuning method that ensures the business reliability of the output results. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide an efficient fine-tuning method for multimodal large models in the context of integrated marketing and logistics scenarios. This method constructs high-quality seed data, combines efficient parameter fine-tuning with an active learning iterative loop based on uncertainty awareness, and enables the model to quickly and accurately master domain knowledge and complex tasks with minimal expert annotation costs.

[0009] Another objective of this invention is to provide a high-efficiency fine-tuning system for multimodal large models oriented towards integrated operation and maintenance scenarios.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides an efficient fine-tuning method for a multimodal large model for a scenario involving the integration of operations and transportation, comprising the following steps:

[0012] S1. Construct a high-quality multimodal seed dataset, which includes multimodal inputs and expert output annotations with thought chains for typical scenarios of camp-distribution integration.

[0013] S2, Based on the seed dataset, the pre-trained multimodal pedestal model is progressively fine-tuned in stages using a parameter-efficient fine-tuning technique. The progressive fine-tuning in stages includes at least a domain knowledge injection stage and a multimodal alignment enhancement stage.

[0014] S3, execute an active learning iterative loop based on uncertainty awareness, the loop including: applying the fine-tuned model to the unlabeled data pool and quantifying the uncertainty; selecting high-value samples based on the quantification results and submitting them to experts for labeling; merging the newly labeled data and repeating the model fine-tuning;

[0015] S4, Model Deployment and Monitoring Steps: Deploy the final fine-tuned model to the production environment and continuously monitor its performance; when the model's performance degrades in new scenarios or specific category samples, trigger a new active learning iteration loop.

[0016] The aforementioned efficient fine-tuning method for a multimodal large model in a scenario of integrated operation and maintenance includes, in step S1, the construction of a high-quality multimodal seed dataset specifically comprising: forming a cross-disciplinary expert group and formulating unified annotation standards; constructing inputs containing at least two modalities among text, image, and time-series data for at least one of the scenarios of fault assessment, solution generation, and intelligent inspection; and the expert output with a thought chain including problem diagnosis, reasoning basis, operation steps, and risk warnings.

[0017] The above-mentioned efficient fine-tuning method for a multimodal large model in a fusion scenario, in step S2, the phased progressive fine-tuning further includes a task-specific optimization phase; step S2 specifically includes:

[0018] S2.1: Domain Knowledge Injection Phase: Using text instructions and thought chain data, fine-tune the model's ability to follow instructions and logical reasoning;

[0019] S2.2: Multimodal Alignment Enhancement Stage: Using complete multimodal input-output data, the focus is on enhancing the model's ability to fuse and understand cross-modal information;

[0020] S2.3: Task-Specific Optimization Phase: Fine-tuning is performed on specific downstream task data to optimize the model's performance on specific tasks;

[0021] Each stage employs efficient parameter fine-tuning techniques, injecting and training only a small number of adapter parameters into the base model.

[0022] In the above-mentioned efficient fine-tuning method for multimodal large models in the context of marketing and distribution integration, in step S2, the efficient parameter fine-tuning technique adopts LoRA or QLoRA to inject independent trainable adapters into the text encoder and visual encoder of the multimodal base model respectively.

[0023] The above-mentioned efficient fine-tuning method for a multimodal large model in a fusion scenario of operation and maintenance, wherein the LoRA technique reparameterizes the linear layer Y = W0X in the base model as... Where W0 represents the frozen pre-trained weights, and A and B are trainable low-rank matrices with rank r < <min(d in ,d out ), where α is the scaling factor.

[0024] The above-mentioned efficient fine-tuning method for multimodal large models in a fusion scenario includes the following sub-steps in step S3: Executing an active learning iterative loop based on uncertainty awareness.

[0025] S3.1: Model Evaluation and Uncertainty Quantification: The model fine-tuned in S2 is applied to the unlabeled business data pool to quantify the uncertainty of the model's prediction results;

[0026] S3.2: Proactive selection of high-value samples: Based on the uncertainty quantification results and combined with preset strategies, the highest-value samples to be labeled are automatically selected from the business data pool; the strategies include: selecting samples with the highest model prediction uncertainty, selecting samples with high model confidence but incorrect predictions, or selecting diverse samples based on feature clustering.

[0027] S3.3: Expert annotation and data augmentation: The selected samples are handed over to domain experts for high-quality annotation to form new data;

[0028] S3.4: Iterative fine-tuning: Merge the new data with the existing training data, repeat step S2, and perform a new round of fine-tuning on the model.

[0029] In the above-mentioned efficient fine-tuning method for a multimodal large model in a scenario of integrated operation and maintenance, step S3 involves uncertainty quantification using a hybrid uncertainty measurement method, calculated as follows:

[0030] U(x i )=β·U epistemic (x i)+(1-β)·U aleatoric (x i )

[0031] Among them, cognitive uncertainty Random uncertainty For Shannon entropy, Let be the model parameters obtained from the t-th Dropout sampling. Let be the average probability distribution, and β be the weighting coefficient.

[0032] In the above-mentioned efficient fine-tuning method for multimodal large models for integrated operation and distribution scenarios, the strategy for selecting high-value samples in step S3 includes one or more combinations of selecting samples with the highest model prediction uncertainty, selecting samples with high model confidence but contradictions after rule verification, or selecting diverse samples based on sample feature clustering.

[0033] In the above-mentioned efficient fine-tuning method for multimodal large models in the context of integrated marketing and logistics scenarios, step S3 involves connecting to an external marketing and logistics knowledge graph and procedure library through a knowledge enhancement interface during the model fine-tuning process to perform knowledge retrieval and constraints on the model generation process.

[0034] Secondly, the present invention provides an efficient fine-tuning system for a multimodal large model oriented towards a fusion of operations and transportation scenarios, which implements the above-mentioned method, comprising:

[0035] Data construction module: used to support experts in constructing and labeling multimodal data pairs, generating high-quality seed datasets with thought chains;

[0036] Fine-tuning training module: integrates a high-efficiency parameter fine-tuning algorithm, supporting the execution of the multi-stage progressive fine-tuning process;

[0037] Active learning engine: Includes uncertainty quantification unit and sample selection strategy unit, used to automatically perform model evaluation, uncertainty calculation and high-value sample screening;

[0038] Iteration Management Platform: Used to manage expert annotation task flow, versioned training datasets, and track model iteration history and performance evolution;

[0039] Knowledge Enhancement Interface: Used to connect to external operation and maintenance knowledge graphs and procedure libraries during model fine-tuning, providing the model with real-time knowledge retrieval capabilities.

[0040] The above-mentioned efficient fine-tuning system for multimodal large models in the context of integrated marketing and distribution scenarios includes an active learning engine comprising an uncertainty quantification unit and a sample selection strategy unit, and an iterative management platform integrated with an expert annotation task flow system.

[0041] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0042] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0043] The technical solution of the efficient fine-tuning method and system for multimodal large models in the context of integrated operation and maintenance scenarios of the present invention has the following beneficial effects:

[0044] (1) Extremely high data efficiency: By actively learning to accurately locate the weaknesses of the model, the expert annotation power is concentrated on the samples with the most information gain, and the model performance is rapidly improved with very little annotation cost (about 10-30% of the traditional method).

[0045] (2) Strong business reliability of output results: The model is trained by introducing expert thinking chain from the source and combined with external knowledge enhancement, which ensures the technical accuracy, procedural compliance and business feasibility of the model output.

[0046] (3) Forming a sustainable evolutionary closed loop: An automated iterative mechanism of "fine-tuning-evaluation-selection-labeling-refine-tuning" has been established, enabling the model to continuously optimize itself along with business development and technology updates, and possessing long-term vitality.

[0047] (4) Balancing performance and cost: The adoption of efficient parameter fine-tuning technology significantly reduces the computational resources and time costs of each iteration, making multi-round iterations economically feasible in industrial scenarios. Attached Figure Description

[0048] Figure 1 This is an overall flowchart of the efficient fine-tuning method for multimodal large models in the context of the present invention for the integrated operation and maintenance scenario.

[0049] Figure 2 This is a schematic diagram illustrating efficient fine-tuning of multi-stage parameters according to an embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of an active learning iterative loop based on uncertainty perception, provided as an embodiment of the present invention.

[0051] Figure 4 This is an architecture diagram of the multimodal large-model high-efficiency fine-tuning system for the integrated operation and maintenance scenario of the present invention. Detailed Implementation

[0052] To enable those skilled in the art to better understand the technical solution of the present invention, its specific embodiments are described in detail below with reference to the accompanying drawings:

[0053] Please see Figure 1 , Figure 2 and Figure 3 The present invention provides an efficient fine-tuning method for a multimodal large model in a scenario involving integrated operation and maintenance, comprising the following steps:

[0054] S1. Construct a high-quality multimodal seed dataset: Form a cross-disciplinary expert group to construct data pairs containing multimodal inputs and expert output annotations with thought chains for typical scenarios of marketing and distribution integration; the multimodal inputs include at least two of the following: text, images, time-series data, and structured data; the expert outputs with thought chains include problem diagnosis, reasoning basis, operation steps, and risk warnings.

[0055] S2, Perform efficient multi-stage parameter fine-tuning: Based on the aforementioned seed dataset, perform phased progressive fine-tuning of the pre-trained multimodal pedestal model:

[0056] S2.1: Domain Knowledge Injection Phase: Using text instructions and thought chain data, fine-tune the model's ability to follow instructions and logical reasoning;

[0057] S2.2: Multimodal Alignment Enhancement Stage: Using complete multimodal input-output data, the focus is on enhancing the model's ability to fuse and understand cross-modal information;

[0058] S2.3: Task-Specific Optimization Phase: Fine-tuning is performed on specific downstream task data to optimize the model's performance on specific tasks;

[0059] Each stage employs efficient parameter fine-tuning techniques, injecting and training only a small number of adapter parameters into the base model.

[0060] S3, Execute an iterative loop of active learning based on uncertainty awareness: This loop contains the following sub-steps:

[0061] S3.1: Model Evaluation and Uncertainty Quantification: The model fine-tuned in S2 is applied to the unlabeled business data pool to quantify the uncertainty of the model's prediction results;

[0062] S3.2: Proactive selection of high-value samples: Based on the uncertainty quantification results and combined with preset strategies, the highest-value samples to be labeled are automatically selected from the business data pool; the strategies include: selecting samples with the highest model prediction uncertainty, selecting samples with high model confidence but incorrect predictions, or selecting diverse samples based on feature clustering.

[0063] S3.3: Expert annotation and data augmentation: The selected samples are handed over to domain experts for high-quality annotation to form new data;

[0064] S3.4: Iterative fine-tuning: Merge the new data with the existing training data, repeat step S2, and perform a new round of fine-tuning on the model.

[0065] S4, Model Deployment and Monitoring: Deploy the final fine-tuned model to the production environment and continuously monitor its performance; when the model's performance degrades in new scenarios or specific class samples, trigger a new active learning iteration loop.

[0066] Example 1: Active Learning Fine-Tuning Based on Hybrid Uncertainty (Taking Intelligent Fault Judgment as an Example)

[0067] Please see Figure 1 , Figure 2 and Figure 3 This embodiment details how to apply the multimodal large model efficient fine-tuning method of the present invention to perform multi-round iterative fine-tuning in the scenario of "intelligent analysis of distribution network faults".

[0068] S1, a standardized process for constructing high-quality multimodal seed data:

[0069] (1) Modal encoding and alignment preprocessing:

[0070] Text Modality (T): Work orders and procedure texts are segmented using a domain-specific dictionary-enhanced BERT word segmenter to obtain word vector sequences.

[0071] Visual modality (V): Features are extracted from the power grid topology map and equipment infrared images using Vision Transformer (ViT) to obtain image patch features.

[0072] Temporal Mode (S): Voltage and current waveform data are first standardized, and then features are extracted using a one-dimensional convolutional neural network (1D-CNN) or a Transformer encoder to obtain sequence features.

[0073] Feature alignment: Mapping features from different modalities to a unified semantic space using a trainable projection matrix: H m =W m X m +b m Where m∈{t,v,s}, d represents a unified dimension.

[0074] (2) Structured thought chain annotation: Expert output is forced to be structured using JSON format, including:

[0075]

[0076]

[0077] S2, Detailed configuration of Multi-Stage Parameter Efficient Fine-Tuning (PEFT): Using LoRA technology, for any linear layer Y = W0X in the base model, LoRA reparameterizes it as follows:

[0078]

[0079] in, For frozen pre-trained weights, For a trainable low-rank matrix, rank r < <min(d in ,d out ), where α is the scaling factor.

[0080] Phase 1 (S2.1): Enable the LoRA adapter only for the language model portion. The loss function is the standard causal language modeling loss.

[0081] Phase Two (S2.2): Enabling the LoRA adapter and multimodal fusion machine (a lightweight Transformer) after enabling all modal encoders. The loss function is multi-task loss.

[0082]

[0083] in, For the image-text matching loss, randomly replace part of the input image and let the model judge whether it is a "match" or "not a match".

[0084] Phase 3 (S2.3): For the task of "generating a fault assessment report", based on S2.2, a reinforcement learning approximation method from human feedback (RLHF) – Direct Preference Optimization (DPO) – is used. This involves constructing preference pairs (y) using expert-annotated "good solutions" and "poor solutions" generated by the baseline model. w ,y l Training is performed to make the model output more in line with expert preferences.

[0085] S3 is a core algorithm for active learning based on mixed uncertainty. This is key to achieving efficient iteration. The model after the k-th iteration is defined as...

[0086] (1) Uncertainty quantification: For sample x in the unlabeled pool i We calculate its mixed uncertainty U(x) i ):

[0087] U(x i )=β·U epistemic (x i )+(1-β)·U aleatoric (x i )

[0088] Among them, U epistemic (x i ) represents cognitive uncertainty, U aleatoric (x i ) represents random uncertainty, β is the tradeoff coefficient, and the initial round can be set to be relatively large to prioritize exploring the unknown areas of the model.

[0089] Cognitive uncertainty (epistemic): Reflects the knowledge uncertainty of the model itself due to a lack of training data. Using an MC Dropout approximation of a Bayesian neural network, and performing T random forward propagations, the entropy of the predicted probability distribution is calculated:

[0090]

[0091] Where, x i Let be the i-th input sample; T is the number of random forward propagations in MC Dropout; These are the model parameters obtained from the t-th Dropout sampling. The parameters for the t-th Dropout sampling are Below, the model for x i The output type is a probability distribution (usually a vector); Let T be the average probability distribution of forward propagation; Shannon entropy is used to measure the uncertainty of a probability distribution pair.

[0092] Aleatoric uncertainty: reflects the inherent noise in the data. Calculate the variance of the predicted probabilities after T forward propagations:

[0093]

[0094] Where, x i Let be the i-th input sample; T is the number of random forward propagations in MC Dropout; The parameters for the t-th Dropout sampling are Below, the model for x i The output type is a probability distribution (usually a vector); It represents the average probability distribution.

[0095] (2) Diversity Assurance and Error Detection: High-uncertainty samples {x} i |U(x i )>τ u} through its feature H i Perform K-means clustering and select a sample from the vicinity of each cluster center to ensure the diversity of scene coverage.

[0096] Meanwhile, from the model's high-confidence prediction {xi |maxP(y|x i )>τ c In the process, a preset hard rule checker (such as: power must be tested before power outage operation) is used to filter out "confident but wrong" samples that violate the rules.

[0097] (3) Experts not only marked the correct output y * It can also evaluate multiple candidate predictions {y} given by the model. j Preferences are ranked. This preference data can be used for the next round of DPO training to further align with human values.

[0098] S4, Example of System Interaction and Automated Iteration:

[0099] (1) After the kth round of fine-tuning, the model is deployed to the "intelligent scheduling auxiliary system".

[0100] (2) The system records all model analysis cases.

[0101] (3) The monitoring module calculates key performance indicators (KPIs), such as the manual review and modification rate and the average processing time reduction rate. When the manual review and modification rate rises above the threshold δ for a consecutive week, an alarm is triggered.

[0102] (4) The active learning engine starts automatically, calls data from the recent case library, executes the selection algorithm in S3, generates a "List of High-Value Samples to be Annotated" containing 50-100 samples, and pushes it to the "Expert Annotation Platform".

[0103] (5) Domain experts complete annotations on the platform, and the platform automatically formats the data and adds a version tag V. k+1 .

[0104] (6) The iteration management platform automatically starts the training task, using V k Data and the newly added V k+1 Data, from The checkpoint begins the rapid fine-tuning of S2 (typically taking only a few hours), producing a new model.

[0105] (7) After the new model passes the automated test, it is launched in a grayscale manner using a blue-green release method to complete one iteration.

[0106] Example 2: Generating fine-tuned power supply schemes considering knowledge retrieval enhancement

[0107] Please see Figure 1 , Figure 2 and Figure 3 This embodiment focuses on demonstrating the combination of the knowledge enhancement interface and the fine-tuning process.

[0108] Enhanced data construction with knowledge retrieval: When constructing the "PV access scheme" data in S1, an additional [knowledge query] tag is added to the input. During training, the model learns to generate a query statement targeting the internal knowledge base when it encounters this tag. For example, given the input: "[Application] Customer applies to install 80kW PV in area A. [Knowledge Query]", the expected intermediate output of the model is: "Query: Distributed PV access capacity calculation procedure; current load rate and remaining capacity of area A." Subsequently, the system simulates the retrieval and returns relevant knowledge points, and the model then generates the final scheme based on this knowledge.

[0109] Fine-tuning objective: The loss function consists of two parts: query generation loss. And knowledge-based solution generation loss The model was trained to both ask questions and utilize the answers.

[0110] Knowledge gap identification in active learning: When selecting samples in S3, the query statements generated by the model can be specifically analyzed. If the model generates poor-quality queries or no queries at all for a frequently occurring specialized concept (such as "reverse power protection"), it indicates a knowledge gap in that area. Cases involving this concept can be prioritized for expert annotation, with detailed explanations of the concept emphasized during annotation, thereby efficiently filling the knowledge gaps in the model.

[0111] Experimental data and results

[0112] Comparative experiments based on historical data from a municipal power supply company show that, using the method and system of this invention, with expert-annotated data volume only 15% of the traditional full-scale supervision and fine-tuning method, after four rounds of iteration, the model achieves and slightly surpasses the latter in both the "accuracy rate of fault diagnosis" and "compliance rate of solutions." Simultaneously, the interpretability of the model's output "thought chain" provides a clear basis for manual review, reducing the average review time by 40%.

[0113] Please see Figure 4 The present invention provides a high-efficiency fine-tuning system for multimodal large models in the context of integrated operation and maintenance scenarios, comprising a data construction module 1, a fine-tuning training module 2, an active learning engine 3, an iterative management platform 4, and a knowledge enhancement interface 5, which are connected in sequence.

[0114] Data Construction Module 1: Supports experts in constructing and labeling multimodal data pairs, generating high-quality seed datasets with thought chains. Fine-tuning Training Module 2: Integrates efficient parameter fine-tuning algorithms, supporting the execution of the multi-stage progressive fine-tuning process. Active Learning Engine 3: Includes an uncertainty quantification unit and a sample selection strategy unit, used for automated model evaluation, uncertainty calculation, and high-value sample selection. Iteration Management Platform 4: Manages expert labeling task flows, versioned training datasets, and tracks model iteration history and performance evolution. Knowledge Enhancement Interface 5: Connects to external operational knowledge graphs and procedure libraries 6 during model fine-tuning, providing the model with real-time knowledge retrieval capabilities.

[0115] In summary, the efficient fine-tuning method and system for multimodal large models in the integrated operation and maintenance scenario of the present invention, by constructing high-quality seed data and combining efficient parameter fine-tuning with an active learning iterative loop based on uncertainty awareness, enables the model to quickly and accurately master domain knowledge and complex tasks with minimal expert annotation cost.

[0116] Those skilled in the art should recognize that the above embodiments are merely illustrative of the present invention and are not intended to limit the present invention. Any variations or modifications to the above embodiments that are within the spirit and essence of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A method for efficient fine-tuning of a multimodal large model for integrated operation and maintenance scenarios, characterized in that, include: S1. Construct a high-quality multimodal seed dataset, which includes multimodal inputs and expert output annotations with thought chains for typical scenarios of camp-distribution integration. S2, Based on the seed dataset, the pre-trained multimodal pedestal model is progressively fine-tuned in stages using a parameter-efficient fine-tuning technique. The progressive fine-tuning in stages includes at least a domain knowledge injection stage and a multimodal alignment enhancement stage. S3, execute an active learning iterative loop based on uncertainty awareness, the loop including: applying the fine-tuned model to the unlabeled data pool and quantifying the uncertainty; selecting high-value samples based on the quantification results and submitting them to experts for labeling; merging the newly labeled data and repeating the model fine-tuning; S4 will deploy and monitor the final multimodal model, triggering new iteration cycles based on performance changes.

2. The efficient fine-tuning method for a multimodal large model in a scenario of integrated operation and maintenance as described in claim 1, characterized in that, In step S1, the construction of a high-quality multimodal seed dataset specifically includes: forming a cross-disciplinary expert group and formulating unified annotation standards; for at least one of the scenarios of fault assessment, solution generation, and intelligent inspection, constructing an input containing at least two modalities of text, image, and time-series data, and a thought chain output containing problem diagnosis, reasoning basis, operation steps, and risk warnings.

3. The efficient fine-tuning method for a multimodal large model in a scenario of integrated operation and maintenance as described in claim 1, characterized in that, In step S2, the phased progressive fine-tuning also includes a task-specific optimization phase; the parameter efficient fine-tuning technique uses LoRA or QLoRA to inject independent trainable adapters into the text encoder and visual encoder of the multimodal base model respectively.

4. The efficient fine-tuning method for a multimodal large model in a scenario of integrated operation and maintenance as described in claim 3, characterized in that, The LoRA technique reparameterizes the linear layer Y = W0X in the base model as... Where W0 represents the frozen pre-trained weights, and A and B are trainable low-rank matrices with rank r < <min(d in ,d out ), where α is the scaling factor.

5. The efficient fine-tuning method for a multimodal large model in a scenario of integrated operation and maintenance as described in claim 1, characterized in that, In step S3, the uncertainty quantification adopts a hybrid uncertainty measurement method, and the calculation formula is as follows: U(x i )=β·U epistemic (x i )+(1-β)·U aleatoric (x i ) Among them, cognitive uncertainty Random uncertainty For Shannon entropy, Let be the model parameters obtained from the t-th Dropout sampling. Let be the average probability distribution, and β be the weighting coefficient.

6. The efficient fine-tuning method for a multimodal large model in a scenario of integrated operation and maintenance as described in claim 1, characterized in that, In step S3, the strategy for selecting high-value samples includes one or more combinations of selecting samples with the highest model prediction uncertainty, selecting samples with high model confidence but contradictions after rule verification, or selecting diverse samples based on sample feature clustering.

7. The efficient fine-tuning method for a multimodal large model in a scenario of integrated operation and maintenance as described in claim 1, characterized in that, In step S3, during the model fine-tuning process, an external operation and maintenance knowledge graph and procedure library are connected through a knowledge enhancement interface to perform knowledge retrieval and constraints on the model generation process.

8. A high-efficiency fine-tuning system for a multimodal large model in a scenario of integrated operation and maintenance, used to implement the method described in any one of claims 1 to 7, characterized in that, include: The data construction module is used to support experts in constructing and labeling multimodal data pairs; The fine-tuning training module is used to perform efficient fine-tuning of parameters in stages; An active learning engine is used to quantify model uncertainty and automatically select samples to be labeled; An iteration management platform for managing annotation tasks, dataset versions, and model iteration history; The knowledge enhancement interface is used to connect to external knowledge bases to enhance model fine-tuning.

9. The efficient fine-tuning system for a multimodal large model oriented towards a combined operation and maintenance scenario as described in claim 8, characterized in that, The active learning engine includes an uncertainty quantification unit and a sample selection strategy unit, which are used to automatically perform model evaluation, uncertainty calculation and high-value sample screening.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

Images