Multimodal large model optimization updating method and device, equipment and storage medium

Abstract

This application relates to a method, apparatus, device, and storage medium for optimizing and updating a multimodal large model, relating to the field of power systems. The method includes: acquiring structured task units and joint feature vectors corresponding to each time node in a power inspection system; determining the normalized difference degree of each time node based on model prediction results, annotation results, and a preset fuzziness score; assigning importance scores to each time node based on the normalized difference degree, model prediction results, fuzziness score, and a preset fluctuation regularization term, obtaining importance weights; selecting the top N time nodes with the highest importance weights as retained nodes; constructing an optimized sample set based on the joint feature vectors and annotation results corresponding to the retained nodes; and training the initial multimodal large model using the optimized sample set to obtain the optimized and updated multimodal large model. This method can improve the accuracy of state prediction in power inspection tasks using a multimodal large model.

Description

Technical Field

[0001] This application relates to the field of power system technology, and in particular to a multimodal large model optimization and updating method, apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] With the deepening of the digital transformation of power systems, the demand for intelligent production environment inspection tasks has significantly increased. However, traditional intelligent diagnostic and inspection technologies are insufficient to meet the requirements of multi-source data, strong time-series dependencies, and human-machine collaboration in the field. While existing small models have achieved a certain degree of automation in single-modal tasks such as image recognition and time-series signal analysis, they still have significant limitations in the face of complex and ever-changing scenarios in power production sites. On the one hand, small models cannot simultaneously process multi-modal data such as images, text, and time-series signals, resulting in unstable performance in tasks such as defect identification and condition assessment. On the other hand, current methods generally neglect the continuity and contextual features of inspection tasks, making the models lack an understanding of the correlation between time-series changes, and relying solely on single-frame judgments makes it difficult to form a coherent diagnostic logic. Furthermore, the experience information of humans during the inspection process cannot be effectively utilized by the model, and model optimization still relies on full retraining, which is costly and time-consuming, especially in edge deployment environments where real-time iteration is difficult to achieve, resulting in low model prediction accuracy. Therefore, how to improve the accuracy of model prediction has become an urgent technical problem to be solved. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, computer equipment, and computer-readable storage medium for optimizing and updating a multimodal large model to address the aforementioned technical problems, which can improve the accuracy of multimodal large models in predicting the state of power inspection tasks.

[0004] Firstly, this application provides a method for optimizing and updating a multimodal large model, including:

[0005] Obtain the structured task units and joint feature vectors corresponding to each time node in the power inspection system; the structured task units include model prediction results and annotation results received based on the interactive interface;

[0006] Based on the model prediction results, annotation results, and preset fuzziness scores, the normalization difference degree at each time point is determined;

[0007] Based on the normalized difference, model prediction results, fuzziness score, and preset fluctuation regularization term, the importance of each time point is scored to obtain the importance weight.

[0008] Select the top N time points with the highest importance weight as the retained nodes;

[0009] An optimized sample set is constructed based on the joint feature vectors and annotation results corresponding to the retained nodes;

[0010] The initial multimodal large model is trained based on the optimized sample set to obtain the optimized and updated multimodal large model.

[0011] In one embodiment, obtaining the structured task units and joint feature vectors corresponding to each time node in the power inspection system includes:

[0012] Acquire multimodal data, model prediction results, and annotation results at various time points in the power inspection system; multimodal data includes image data, time-series signals, and text data;

[0013] Structured task units are obtained by encoding based on multimodal data, model prediction results, and annotation results;

[0014] Features extracted from image data, time-series signals, and text data are fused to obtain a joint feature vector.

[0015] In one embodiment, after training the initial multimodal large model based on the optimized sample set to obtain the optimized and updated multimodal large model, the method further includes:

[0016] Obtain the prediction accuracy, average inference time, and maximum memory usage of the optimized and updated multimodal large model on the optimized sample set, and obtain the first prediction accuracy, the first average inference time, and the first maximum memory usage.

[0017] Obtain the prediction accuracy and maximum memory usage of the initial multimodal large model on the optimized sample set, and then obtain the second prediction accuracy and the second maximum memory usage.

[0018] If the difference between the first prediction accuracy and the second prediction accuracy is greater than a preset accuracy threshold, the first average inference time is less than a preset time threshold, and the absolute value of the difference between the first maximum memory usage and the second maximum memory usage is less than a preset content threshold, then the optimized and updated multimodal large model will be deployed to the power inspection system.

[0019] In one embodiment, after deploying the optimized and updated multimodal large model to the power inspection system, the method further includes:

[0020] Acquire updated image data, updated time-series signals, and updated text data;

[0021] The features extracted from the updated image data, the updated time-series signal, and the updated text data are fused to obtain the updated joint feature vector;

[0022] The updated joint feature vector is input into the optimized and updated multimodal large model to obtain the updated model prediction results.

[0023] In one embodiment, after inputting the updated joint feature vector into the optimized and updated multimodal large model to obtain the updated model prediction result, the method further includes:

[0024] Based on the updated model prediction results within a set time range before and after the current time point, determine the stability index value for the current time point.

[0025] If the stability index value is greater than the preset stability threshold, the current time point will be marked as a predicted unstable point.

[0026] In one embodiment, after deploying the optimized and updated multimodal large model to the power inspection system, the method further includes:

[0027] Retrieve new task nodes appearing in the risk category region of the optimized sample set;

[0028] The joint feature vector corresponding to the new task node is input into the optimized and updated multimodal large model to obtain the first confidence peak.

[0029] The joint feature vector corresponding to the new task node is input into the initial multimodal large model to obtain the second confidence peak.

[0030] Calculate the difference between the first confidence peak and the second confidence peak for each new task node;

[0031] Sum the differences between all the new task nodes to get the total gain value;

[0032] Divide the total gain value by the number of new task nodes to obtain the performance gain metric;

[0033] If the performance gain index is greater than the preset gain threshold, the optimized and updated multimodal large model will continue to run.

[0034] If the performance gain index is less than or equal to the preset gain threshold, the initial multimodal large model is restored.

[0035] In one embodiment, the normalization dissimilarity at each time point is determined based on the model prediction results, annotation results, and a preset fuzziness score, including:

[0036] Semantic differences are calculated based on model prediction results and annotation results to obtain semantic difference items;

[0037] The confidence penalty term is obtained based on the model prediction results, and the complexity correction term is obtained based on the preset fuzziness score.

[0038] The weighted difference is obtained by multiplying the semantic difference term, the confidence penalty term, and the complexity correction term.

[0039] The weighted variance is smoothed to obtain the normalized variance.

[0040] Secondly, this application also provides a multimodal large model optimization and update apparatus, comprising:

[0041] The acquisition module is used to acquire the structured task units and joint feature vectors corresponding to each time node in the power inspection system; the structured task units include model prediction results and annotation results received based on the interactive interface;

[0042] The calculation module is used to determine the normalized difference degree at each time point based on the model prediction results, annotation results and preset fuzziness scores;

[0043] The scoring module is used to score the importance of each time node based on the normalized difference, model prediction results, fuzziness score, and preset fluctuation regularization term, and obtain the importance weight; the top N time nodes with the highest importance weight are selected as retained nodes.

[0044] The update module is used to construct an optimized sample set based on the joint feature vectors and annotation results corresponding to the retained nodes; the initial multimodal large model is trained based on the optimized sample set to obtain the optimized and updated multimodal large model.

[0045] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0046] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0047] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0048] The aforementioned multimodal large-scale model optimization and update method, apparatus, computer equipment, and computer-readable storage medium acquire structured task units and joint feature vectors corresponding to each time node in the power inspection system. The structured task units include model prediction results and annotation results received via an interactive interface. Based on the model prediction results, annotation results, and a preset fuzziness score, the normalized difference degree of each time node is determined. Based on the normalized difference degree, model prediction results, fuzziness score, and a preset fluctuation regularization term, each time node is scored for importance, yielding an importance weight. The top N time nodes with the highest importance weights are selected as retained nodes. An optimized sample set is constructed based on the joint feature vectors and annotation results corresponding to the retained nodes. The initial multimodal large-scale model is trained using the optimized sample set to obtain the optimized and updated multimodal large-scale model. Through this method, by determining importance weights, samples with the most significant model cognitive biases and the most critical impact on model correction are intelligently selected from time-series data to construct an optimized sample set, achieving both data minimization and high representativeness. Multimodal large models can quickly compensate for their own defects by continuously learning and optimizing the sample set. This can improve the stability, interpretability and update efficiency of large models in complex power inspection environments, thereby improving the accuracy of the model's state prediction in power inspection tasks. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating a multimodal large model optimization and update method in one embodiment;

[0051] Figure 2 This is a structural block diagram of a multimodal large model optimization and update device in one embodiment;

[0052] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0055] In one exemplary embodiment, such as Figure 1 As shown, a multimodal large model optimization and update method is provided. Taking the application of this method to a server as an example, it is illustrated as follows:

[0056] Step 102: Obtain the structured task units and joint feature vectors corresponding to each time node in the power inspection system; the structured task units include model prediction results and annotation results received based on the interactive interface.

[0057] This process involves acquiring multimodal data, model prediction results, and annotation results for each time point in the power inspection system. The multimodal data includes image data, time-series signals, and text data. The multimodal data (image data, time-series signals, text data), model prediction results, and annotation results are uniformly encoded to obtain structured task units. All structured task units form a time-sequential structured task chain. The interactive interface refers to the human-computer interaction interface set up on the computer device, used to receive user instructions and acquire user input data. The annotation results refer to the manual judgment results received through the human-computer interaction interface.

[0058] In an exemplary embodiment, step 102 includes: acquiring multimodal data, model prediction results, and annotation results for each time node in the power inspection system; the multimodal data includes image data, time-series signals, and text data; encoding the multimodal data, model prediction results, and annotation results to obtain structured task units; and fusing the features extracted from the image data, time-series signals, and text data to obtain a joint feature vector.

[0059] For example, a production inspection task typically includes multiple inspection actions performed consecutively over time, such as drones taking aerial photographs of the production line, continuous data collection of equipment operating status at fixed monitoring points, and manual submission of task notes and diagnostic conclusions by personnel in the scheduling system. To integrate this discrete data in a time-series manner, the system uses the task ID as the primary key and time as the secondary key. To indicate the logical progression, a structured task chain is generated, with a task duration set to [value]. Each sampling time point. This represents an atomic-level inspection and judgment action. The structured task unit structure is a quintuple, and the structured task chain is also mentioned. As shown in the following formula:

[0060] ;

[0061] in, Image data represents a single frame image captured from a front-end camera at that moment. The image comes from a camera deployed on a power transmission tower or within the station. The interface standard can be the ONVIF protocol, or it can be an aerial frame captured during the flight of a drone. The acquisition can be triggered by a preset path in the task scheduling system. For example, the image is stored in JPEG compression format with a resolution of 640×480 and a three-channel tensor format. These are timing signals, which are continuous electrical or environmental parameters uploaded by equipment status acquisition devices (such as online monitoring terminals, intelligent grounding devices, and transformer temperature and humidity monitoring devices). The sampling period can be uniformly set to 5ms. It contains data within a fixed-length time window (e.g., the past 30 seconds), and the transmission protocol can be Modbus-TCP, with the format being a fixed-length vector sequence. The data is text-based, including fields such as work order number, task description, and equipment location information. It is automatically generated by the dispatch center during the task issuance phase, and the field length can be limited to no more than 100 bytes.

[0062] The model prediction results are the initial multimodal large model (the currently deployed model)'s predictions of the input. The prediction results can be directly accessed in the current step using historical prediction data, which consists of the class labels defined by the model and their corresponding confidence scores. The results can be annotations submitted by the inspection personnel (i.e., manual judgment results), and the recording method is structured text filled in on the dispatch platform or mobile operation terminal.

[0063] Optionally, to improve the consistency of subsequent calculations, the image data... and timing signals Perform feature compression. The image feature vectors are extracted using a three-layer convolutional neural network (each layer containing convolution and pooling modules) to the target dimension (e.g., 128 dimensions). ; The input is fed into a two-layer bidirectional GRU structure (64 hidden units per layer), and the output is concatenated to form a temporal feature vector of the target dimension (e.g., 128 dimensions). . The text feature vector is generated by word segmentation and embedding using the sub-word encoding-based BPE module, resulting in a fixed-length vector (maximum 32 tokens). The dimension is the target dimension (e.g., 128 dimensions). The feature vectors of the three modalities are concatenated to obtain a joint feature vector, as shown below:

[0064] ;

[0065] in, It is a 384-dimensional joint feature vector. For image feature vectors, It is a time-series feature vector. These are text feature vectors. All of them. They will be arranged in chronological order to form a joint feature vector. This is used to support subsequent model difference analysis and optimization.

[0066] Structured task chain and joint eigenvectors The data is written into a structured database system, with each record associated with a task ID and a timestamp, and indexed with second-level precision to support subsequent source tracing operations, model comparison, and sample construction processing.

[0067] Step 104: Determine the normalization difference degree at each time point based on the model prediction results, annotation results, and preset fuzziness scores.

[0068] In this process, semantic differences are calculated and smoothed for each time point based on model prediction results, annotation results, and preset fuzziness scores to obtain the normalized difference degree.

[0069] In an exemplary embodiment, step 104 includes: calculating semantic differences based on model prediction results and annotation results to obtain semantic difference terms; obtaining confidence penalty terms based on model prediction results and complexity correction terms based on preset fuzziness scores; multiplying the semantic difference terms, confidence penalty terms, and complexity correction terms to obtain weighted difference degrees; and smoothing the weighted difference degrees to obtain normalized difference degrees.

[0070] For example, power line inspection tasks exhibit significant heterogeneity and strong contextual characteristics, with vastly different image quality, temporal signal stability, and complexity of task descriptions at different time points. Simply relying on label similarity or distance calculations is clearly insufficient to support high-precision target selection optimization. Firstly, for each time point... Model prediction results and annotation results We model the semantic relationships. The form is ( ),in For category labels, The confidence level for this category; This indicates the judgment result submitted manually, in the form of ,in Consistent with the model category system, The value is fixed at 1. The category labels are mapped to a unified target dimension (e.g., 128-dimensional) semantic vector space to obtain the model's predicted semantic vectors. Compared with human judgment of semantic vectors The vectors are obtained by pre-training on historical patrol data and have a clear category semantic structure.

[0071] Considering the significant importance of model prediction confidence, input quality differences, and human judgment, semantic distance alone cannot accurately reflect these differences. Therefore, a weighted variance score is defined for each time point. as follows:

[0072] ;

[0073] The first item in parentheses represents the semantic direction difference, indicating the angle difference between the two labels in the embedding space; the second item is the confidence penalty. The first term is an adjustment coefficient used to amplify the penalty for low-confidence predictions in the model; the third term is an input complexity correction term. for The adjustment coefficient, Indicates the input mode at time t. The fuzziness score is derived from the following sources:

[0074] For image data Calculate sharpness metrics (such as image gradient variance);

[0075] For timing signals Calculate signal stability (such as first-order difference mean square);

[0076] For text data Calculate the degree of structure (e.g., sentence structure standardization scoring);

[0077] After normalizing the three items separately, a weighted average is calculated and combined to form... , A higher value indicates poorer input quality and greater processing difficulty. This formula extends the traditional semantic distance evaluation to a dynamic index that considers model internal confidence, input modal solvability, and changes in the task environment. It provides fine-grained control over signal disturbances, image blurring, and inconsistencies in text descriptions, especially in power inspection scenarios, making it highly relevant to real-world applications.

[0078] For example, in a substation inspection scenario, at a certain moment, an image shows a large area of ​​shadow due to severe backlighting. The model predicts "insulator damage" with a confidence level of only 0.62, while the human judgment is "contamination." Although semantically related, they are inconsistent. Due to the high image blur, Reaching 0.78, in the setting , Eventually The score is much higher than the unweighted difference score, which allows the system to determine that the node has significant optimization value.

[0079] In addition, to suppress the problem of excessively high scores for some nodes due to multimodal information conflicts, a regularization term is introduced to adjust the weighted difference. Smoothing control was performed to obtain the normalized difference. As shown in the following formula:

[0080] ;

[0081] in, Joint eigenvectors The L2 norm represents the total strength of the information of the current node. is the regularization strength coefficient. This term is introduced because: in nodes with dense but small modal information, it is not advisable to over-penalize the model behavior, especially when both the image and temporal signals are of high quality, tolerance should be emphasized, thereby controlling the risk of extreme nodes misleading the model training.

[0082] This step builds upon the structured task chain established in the previous step. and joint eigenvectors The goal is to maintain structural consistency for each time point. Model prediction results Comparison with annotation results The semantic differences between them are quantified, and a correction mechanism related to input complexity and model confidence is introduced to achieve difference modeling that is more in line with the characteristics of actual power inspection tasks. This determines whether the target selection for model optimization is accurate and representative.

[0083] Step 106: Based on the normalized difference, model prediction results, fuzziness score, and preset fluctuation regularization term, the importance score of each time node is obtained to obtain the importance weight.

[0084] Among them, the normalized dissimilarity sequence is the output from the previous step. Joint structured task chain With joint feature vector This enables precise screening of nodes with insufficient model performance and the construction of a high-value, low-redundancy, and reusable optimized sample set. This step involves two aspects: firstly, extracting the most representative points worthy of optimization from the entire task flow; and secondly, organizing the data structure to serve as direct input for subsequent model optimization training. Due to the challenges of long inspection cycles, sparse fault distribution, and complex human conclusions in power inspection scenarios, high variability does not necessarily indicate model errors; it could also be due to input complexity or human subjectivity. Therefore, relying solely on a fixed threshold is insufficient for effectively screening high-value nodes. To address this issue, this step proposes a dynamic screening mechanism that combines variability distribution characteristics, input modal complexity, and confidence inconsistency. This mechanism constructs a more robust node evaluation standard and includes a sample redundancy compression strategy to enhance the representativeness and uniqueness of the optimized samples, ultimately forming an optimized sample set. .

[0085] For example, define each time node Importance weight This is used to determine whether a given time point has optimization value. Taking into account three types of factors:

[0086] Normalized difference between model predictions and human judgment ;

[0087] Inconsistency in model predictions refers to the degree to which the output confidence level deviates from the current task category distribution.

[0088] Fuzzy scoring of input modalities This includes image clarity, temporal stability, and text standardization.

[0089] The importance score is constructed as follows:

[0090] ;

[0091] in, Time node Importance weights , , To adjust the parameters, This represents the model's confidence level in predicting the labels. For fuzzy scoring, This is a prediction volatility regularization term used to penalize the model for unstable predictions of similar samples in adjacent time periods. The specific calculation is as follows:

[0092] ;

[0093] in, Indicates the radius of the observation window, for example, taking =2, meaning the average confidence level of the two points before and after the current node is considered. This term effectively constrains the model from exhibiting "swinging predictions" in continuous frames or continuous signal segments. These nodes often have potential feature ambiguities and are high-risk areas for model misjudgment.

[0094] Step 108: Select the top N time nodes with the highest importance weight as the retained nodes.

[0095] Among them, the importance weight of each time point After the calculation is completed, based on the set upper limit N of the sample budget, for all Sort the nodes and select the time points corresponding to the top N importance weights as the retained nodes. All retained nodes constitute the initial selection set. When the sample size is sufficient, threshold screening can be used instead of sorting.

[0096] Step 110: Construct an optimized sample set based on the joint feature vectors and annotation results corresponding to the retained nodes.

[0097] The optimized sample set can be directly constructed based on the joint feature vectors and annotation results corresponding to the retained nodes. For example, to prevent redundant training caused by highly similar nodes in the sample set, feature space compression can be further performed. The optimized sample set is then constructed based on the compressed data, achieving both minimization and high representativeness of the optimized data. The system uses each... Centered on, calculate its relationship with The cosine similarity of other candidate samples is used; if the similarity exceeds a set threshold... (e.g., 0.96), then only retain Samples with higher scores. This operation constitutes a "non-repeating screening," and the final set of retained nodes is denoted as... Thus, the optimized sample set has been constructed. as follows:

[0098] ;

[0099] Each sample contains a joint feature vector. and annotation results ,in Alignment with the model labeling system has been completed, and structural consistency ensures that subsequent training can be directly applied.

[0100] For example, during a transformer inspection, =34, 35, and 36 correspond to the "local overheating of the fuel tank" scenario, with model prediction confidence fluctuations of 0.89, 0.65, and 0.91, respectively, and the labeled result is "slight overheating." Due to external load fluctuations in the signal, the model performance is unstable, leading to... A significant increase. The system combines fuzziness scores and normalized difference, retaining... =35 is used as the optimized sample, while the remaining redundant frames are excluded.

[0101] Step 112: Train the initial multimodal large model based on the optimized sample set to obtain the optimized and updated multimodal large model.

[0102] Among them, by optimizing the sample set For the currently deployed initial multimodal large model Perform targeted lightweight optimization to output an updated multimodal large model. This step plays a crucial role in reshaping model capabilities and enhancing local knowledge, serving as a key link in the process from problem identification, sample construction, capability improvement, to deployment optimization. Unlike traditional large-scale model fine-tuning, this step does not involve full retraining. Instead, it performs local optimization targeting specific weaknesses in power inspection scenarios (such as low-confidence samples, complex modal interference samples, and semantically offset samples). Through innovative regularization terms and update mechanisms, it ensures that the optimized model remains structurally stable and more closely reflects the characteristics of actual inspection tasks.

[0103] Optionally, the initial multimodal large model The structure consists of three parts: a feature extraction layer (a hybrid convolutional and recurrent structure), a modality fusion layer (a cross-attention structure), and a task output layer (a classification head based on linear projection). The optimization in this step only applies to the local weight matrices in the modality fusion layer and the task output layer, rather than the full model parameters, ensuring that the optimization operation can be performed quickly on edge devices.

[0104] To achieve this goal, the overall optimization objective function is defined. as follows:

[0105] ;

[0106] in, The main task loss measures the difference between model predictions and human labels. This is a confidence level self-correction term; Constraints on the structural consistency of power equipment; This is a cross-modal stability regularization term.

[0107] First item It is built based on cross-entropy and used to correct the model's class predictions on highly disparate samples. Because... The samples are mostly concentrated in the model boundary region. If standard cross-entropy is used, the gradient update direction will be affected by outliers. To avoid this problem, a dynamic weight function is introduced. Normalized variance of the samples With model confidence Joint definition:

[0108] ;

[0109] ;

[0110] This design allows samples with low confidence and high dissimilarity to receive higher weights in the optimization, thus enhancing the model's ability to correct key weaknesses. The adjustment coefficient representing the sample weights, if Then in , The time weight is 1.288, which significantly amplifies the learning gradient of this sample.

[0111] Second item This is a confidence self-correction term used to suppress overconfidence outputs caused by multimodal interference (such as noise and blurred images) during the inspection process. It is defined as:

[0112] ;

[0113] in, The local confidence mean is calculated using a sliding window with a window size of 3, representing the smoothing trend of the model's output confidence in its immediate and neighboring neighborhoods. This constraint limits the fluctuation range of the model's output, resulting in a smoother and more stable output across the time series dimension for the optimized model.

[0114] Third item The structural consistency constraint for power equipment is designed for inspection scenarios. Considering the strong structural similarity of similar equipment (such as conductors, insulators, and switchgear) in the image space, and the tendency for the model to overfit to local textures during optimization, leading to decreased generalization performance, a regularization constraint based on equipment topology consistency is introduced:

[0115] ;

[0116] in, This represents the set of node pairs belonging to the same type of device in a structured task chain. and This is the output of the intermediate layer of the image branch. This is the topological similarity decay coefficient. This factor helps keep the embedding distribution of similar devices in the semantic space close, preventing intra-class consistency from being compromised due to limited task samples during model fine-tuning. For example, in inspection images, multiple tower nodes on the same line should share visual embedding features. The introduction of this ensures that the optimized model has a stable response to these nodes.

[0117] Last item This is a cross-modal stability regularization term used to maintain the feature consistency of the model between time-series signals and image modalities, preventing the model from weakening its ability to judge time-series data when optimizing for visual anomalies. This term is defined as:

[0118] ;

[0119] in, This represents the variance of the modal output, reflecting the dispersion of the modal feature distribution. This ensures the stability of the optimized model's decisions under multimodal inputs, preventing bias towards any particular mode.

[0120] For example, the optimization process employs a hierarchical update strategy. First, the feature extraction layer of the model is frozen, and low-rank fine-tuning is performed only on the modality fusion layer and the output layer. The optimizer uses AdamW with a learning rate set to 1e−4 and regularized weight parameters. Take respectively The training batch size is 16, and the iterations are 3 to 5 rounds. After each round of training, the results are checked using a validation set. If the downward trend is not significantly improved after two consecutive rounds, the process will be stopped early. All updated parameters will be stored as independent difference files, separate from the original model structure. Only the difference module will be loaded during deployment, adapting to the limited computing power conditions in the edge computing environment of power companies.

[0121] For example, in a power transmission channel inspection optimization task, to address the issue of the model misclassifying a "minor conductor breakage" scenario as "contamination," the system... Select this type of node sample for optimization, through Maintaining line topology consistency constraints, through The balanced timing current fluctuation characteristics enable the optimized model to still output correct results under conditions such as insufficient lighting or small-angle occlusion.

[0122] We propose a lightweight fine-tuning strategy that combines multimodal feature consistency regularization terms with structural stability constraints. Low-rank parameter updates are performed only on the model fusion layer and the output layer to achieve efficient incremental optimization that can be deployed on edge devices.

[0123] The output is the optimized and updated multimodal large model. Its structure is similar to the initial multimodal large model. To maintain compatibility, only independently enableable parameter modules are added to the fusion and output layers. In power line inspection scenarios, this model demonstrates greater robustness to complex inputs, greater sensitivity to low-confidence samples, and consistent judgment across multimodal inputs.

[0124] In the aforementioned multimodal large model optimization and update method, structured task units and joint feature vectors corresponding to each time node in the power inspection system are obtained. The structured task units include model prediction results and annotation results received via an interactive interface. Based on the model prediction results, annotation results, and a preset fuzziness score, the normalized difference degree of each time node is determined. Based on the normalized difference degree, model prediction results, fuzziness score, and a preset fluctuation regularization term, the importance of each time node is scored, resulting in an importance weight. The top N time nodes with the highest importance weights are selected as retained nodes. An optimized sample set is constructed based on the joint feature vectors and annotation results corresponding to the retained nodes. The initial multimodal large model is trained using the optimized sample set to obtain the optimized and updated multimodal large model. Through this method, by determining the importance weights, samples with the most significant model cognitive bias and the most critical impact on model correction are intelligently selected from the time-series data to construct an optimized sample set, achieving both data minimization and high representativeness. Multimodal large models can quickly compensate for their own defects by continuously learning and optimizing the sample set. This can improve the stability, interpretability and update efficiency of large models in complex power inspection environments, thereby improving the accuracy of the model's state prediction in power inspection tasks.

[0125] In an exemplary embodiment, after step 112, the method further includes: obtaining the prediction accuracy, average inference time, and maximum memory usage of the optimized and updated multimodal large model on the optimized sample set, to obtain a first prediction accuracy, a first average inference time, and a first maximum memory usage; obtaining the prediction accuracy and maximum memory usage of the initial multimodal large model on the optimized sample set, to obtain a second prediction accuracy and a second maximum memory usage; if the difference between the first prediction accuracy and the second prediction accuracy is greater than a preset accuracy threshold, the first average inference time is less than a preset time threshold, and the absolute value of the difference between the first maximum memory usage and the second maximum memory usage is less than a preset content threshold, then the optimized and updated multimodal large model is deployed to the power inspection system.

[0126] Among them, the multimodal large model after output optimization and update Subsequently, its deployment in the power inspection system was completed, and its inference application in a real task environment was realized through a systematic control process, thereby generating a new round of structured task chains. This task chain will serve as the input foundation for subsequent model re-optimization, closing the loop structure of difference perception and update. This step not only addresses the model replacement issue but also incorporates post-update operational evaluation, predictive behavior monitoring, and system data recording strategies to ensure that the optimization results are controllable, effective, and traceable, ultimately forming an integrated model evolution solution for practical deployment scenarios.

[0127] Before deployment, the platform first optimized and updated the multimodal large model. Static performance evaluation and interface compatibility testing are performed on the runtime performance. This involves executing the test on an offline validation set. Calculate its in Improved recognition accuracy in the relevant category Simultaneously, its average inference time was measured. With maximum memory usage and the initial multimodal large model A comparison is performed. Specifically, the optimized and updated multimodal large model pair is obtained from the optimized sample set. The prediction accuracy, average inference time, and maximum memory usage are used to obtain the first prediction accuracy. First average inference time and the first maximum memory usage ; Obtain the prediction accuracy and maximum memory usage of the initial multimodal large model on the optimized sample set, and then obtain the second prediction accuracy. Second maximum memory usage If the accuracy of the first prediction is Compared with the second prediction accuracy The difference Greater than the preset accuracy threshold First average inference time Less than the preset time threshold And the first maximum memory usage With the second largest memory usage The absolute value of the difference is less than the preset content threshold. Then, the optimized and updated multimodal large model will be deployed to the power inspection system. The deployment conditions are:

[0128] , representing the model It provides performance improvement at the original failure point; ,Model The computing resource requirements do not exceed the capabilities of edge deployment; ,Model Structural differences are acceptable;

[0129] If the above deployment conditions are met, proceed with the deployment process. If the above deployment conditions are not met, retain the model. Replacement should be postponed. The above thresholds are configured and set through the power company's dispatch platform, for example, Generally 5%, Keep it within 300ms Typically less than 50MB.

[0130] In this embodiment, by performing model evaluation before deployment, it can be ensured that each model update is a reliable upgrade that achieves the best balance between performance, speed, and resource consumption, thereby improving the reliability of model optimization and updates.

[0131] In an exemplary embodiment, after deploying the optimized and updated multimodal large model to the power inspection system, the method further includes: acquiring updated image data, updated time-series signals, and updated text data; fusing the features extracted from the updated image data, updated time-series signals, and updated text data respectively to obtain an updated joint feature vector; and inputting the updated joint feature vector into the optimized and updated multimodal large model to obtain the updated model prediction result.

[0132] Among them, the model After deployment, obtain the updated image data. Updated timing signals and updated text data Feature extraction and concatenation are performed on the updated image data, updated time-series signal, and updated text data to obtain an updated joint feature vector. This updated joint feature vector is then input into the optimized and updated multimodal large model to obtain the updated model prediction results. .

[0133] During actual patrol missions, the model can also receive updated multimodal data in real time. And output the updated model prediction results. .

[0134] For example, image data Data is acquired via a front-end intelligent camera (such as a PTZ camera or a fixed-point high-pole camera), with the interface conforming to the RTSP protocol or ONVIF interface standard. The image resolution is controlled at 640×480, and the upload frequency is 1 frame per minute; timing signals... The data is sourced from substation environmental monitoring equipment, bus current transformers, etc., with a sampling period of 5ms and an upload frequency of once every 10 seconds; text information. Automatically generated by the scheduling task system, it includes fields such as inspection type (e.g., daily inspection, special inspection), equipment number, and task number, and is structured in JSON format; the updated model prediction results. For the model right The prediction results are generally a pair of risk categories and confidence levels, for example ("High Temperature Hazard", 0.87); Labeling results The information is filled out or corrected by the inspection personnel through a mobile work terminal or remote control console, and the structure remains consistent with the historical tasks.

[0135] The above data is automatically synchronized and recorded by the system at each time point. quintuple The tasks are stored in a structured database (such as HBase or TimescaleDB), indexed by task number and timestamp, forming a new chain of structured tasks. Its structure is similar to the initial structured task chain. Maintain consistency.

[0136] In this embodiment, after optimization, the model is put into operation in the inspection system through modular hot replacement. The system monitors the predicted stability index and performance gain index in real time, and automatically generates a new round of task chain to achieve continuous self-evolution of model, task, data and optimization.

[0137] In an exemplary embodiment, after inputting the updated joint feature vector into the optimized and updated multimodal large model to obtain the updated model prediction result, the method further includes: determining the stability index value of the current time node based on the updated model prediction results within a set time range before and after the current time node; if the stability index value is greater than a preset stability threshold, then the current time node is marked as a prediction instability point.

[0138] Among them, in order to further evaluate To assess the model's performance in real-world tasks, a stability monitoring mechanism for the model output was designed. Considering the continuity and high temporal correlation of power equipment inspection tasks, frequent oscillations in the model output within adjacent time periods may indicate insufficient temporal robustness of the optimized model. Therefore, the stability index for the current time point is calculated based on the updated model predictions before and after the current time point, yielding the stability index value. As shown below:

[0139] ;

[0140] in, The radius of the time window (3 is recommended). Representation Model At any moment The output is the confidence value for the main category. If... Greater than the preset stability threshold If the value is 0.25, the current time point is marked as a "predictive instability point," and the system will prioritize processing it in the subsequent difference analysis stage, forming a new round of optimization candidates. For example, the joint feature vector and annotation results corresponding to the current time point are added to the subsequent optimization sample set. For example, attribution analysis is performed based on the joint feature vectors and annotation results of all predictive instability points to determine the risk category of model anomalies; based on the sample data corresponding to this risk category, the multimodal large model is fine-tuned to improve the prediction accuracy of the multimodal large model.

[0141] In this embodiment, the system monitors the consistency of the model's predictions at specific points in time in real time. When the model's judgment on the same device or scenario fluctuates significantly in the vicinity of time, the system will automatically mark it, providing data support for risk warnings and subsequent optimization. This proactively prevents the risk of misjudgment caused by unstable model predictions.

[0142] In an exemplary embodiment, after deploying the optimized and updated multimodal large model to the power inspection system, the method further includes: acquiring new task nodes appearing in the risk category region to which the optimized sample set belongs; inputting the joint feature vector corresponding to the new task node into the optimized and updated multimodal large model to obtain a first confidence peak; inputting the joint feature vector corresponding to the new task node into the initial multimodal large model to obtain a second confidence peak; calculating the difference between the first confidence peak and the second confidence peak corresponding to each new task node; summing the differences of all new task nodes to obtain a total gain value; dividing the total gain value by the number of new task nodes to obtain a performance gain index; if the performance gain index is greater than a preset gain threshold, then the optimized and updated multimodal large model is kept running; if the performance gain index is less than or equal to the preset gain threshold, then the initial multimodal large model is restored.

[0143] Among them, for verification Whether the optimization brings substantial improvement in actual tasks, design task domain performance gain metrics. For example, new task nodes appearing in the risk category region of the optimized sample set are obtained. The joint feature vector corresponding to the new task node is input into the optimized and updated multimodal large model to obtain the first confidence peak. The joint feature vector corresponding to the new task node is input into the initial multimodal large model to obtain the second confidence peak. The performance gain metric is obtained by calculating the difference between the peak first confidence level and the peak second confidence level corresponding to all new task nodes, summing the sums, and then dividing by the number of new task nodes. As shown below:

[0144] ;

[0145] in, This represents the set of new task nodes appearing in the risk category region of the optimized sample set. This indicates the number of new task nodes. and These are the peak confidence scores of the predicted outputs of the new and old models under the same input, or evaluation functions such as the entropy inverse index. If the performance gain index is greater than the preset gain threshold (set to 0), then... ,Model Keep running; otherwise, trigger rollback logic to restore the original model. At the same time, mark this optimization as invalid.

[0146] For example, during a routine inspection of a 35kV substation, the model... The new model is more stable in judging scenarios such as "abnormal temperature rise in switchgear". It can still output the "potential hidden danger" category even when the background of the input image is complex and the time-series signal fluctuates greatly. In contrast, the old model frequently outputs "normal" under the same input, and the confidence level fluctuates greatly. When the value reaches 0.32, it exceeds the threshold, and the system identifies this point as the target for the next round of optimization.

[0147] It should be noted that the set of new task nodes appearing in the risk category region of the optimized sample set refers to task nodes identified by the system in a new inspection task after model optimization deployment that are of the same category or semantically similar to the previously optimized sample nodes. For example, if the optimized sample set contains high-risk categories such as "minor conductor strand breakage" and "insulator damage," then in the new round of inspection tasks, nodes predicted by the model to be in these categories or with similar semantic features will be included in this set to evaluate the actual performance improvement of the model on the original failure categories. These new task nodes are directly related to the optimization objective and can reflect whether the model has been effectively improved in the original weak areas. If time nodes are randomly selected, most of them will be normal equipment or low-risk samples unrelated to optimization, which will dilute the model optimization effect, make it impossible to accurately measure performance gains, and reduce the relevance and credibility of the model evolution evaluation.

[0148] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0149] Based on the same inventive concept, this application also provides a multimodal large model optimization and update apparatus for implementing the multimodal large model optimization and update method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the multimodal large model optimization and update apparatus provided below can be found in the limitations of the multimodal large model optimization and update method described above, and will not be repeated here.

[0150] In one exemplary embodiment, such as Figure 2 As shown, a multimodal large model optimization and update device is provided, comprising:

[0151] The acquisition module 202 is used to acquire the structured task units and joint feature vectors corresponding to each time node in the power inspection system; the structured task units include model prediction results and annotation results received based on the interactive interface;

[0152] The calculation module 204 is used to determine the normalization difference degree at each time point based on the model prediction results, annotation results and preset fuzziness scores;

[0153] The scoring module 206 is used to score the importance of each time node based on the normalized difference, model prediction results, fuzziness score, and preset fluctuation regularization term, and obtain the importance weight; the top N time nodes with the highest importance weight are selected as retained nodes.

[0154] The update module 208 is used to construct an optimized sample set based on the joint feature vectors and annotation results corresponding to the retained nodes; and to train the initial multimodal large model based on the optimized sample set to obtain the optimized and updated multimodal large model.

[0155] In the aforementioned multimodal large model optimization and update device, by determining importance weights, the most significant cognitive biases and the most critical samples for model correction are intelligently selected from time-series data to construct an optimized sample set, achieving minimization and high representativeness of the optimized data. Through continuous learning and optimization of the sample set, the multimodal large model can quickly compensate for its own deficiencies, improving the stability, interpretability, and update efficiency of the large model in complex power inspection environments, thereby increasing the accuracy of the model's state prediction in power inspection tasks.

[0156] In an exemplary embodiment, the acquisition module 202 is further configured to acquire multimodal data, model prediction results, and annotation results for each time node in the power inspection system; the multimodal data includes image data, time-series signals, and text data; the multimodal data, model prediction results, and annotation results are encoded to obtain structured task units; and the features extracted from the image data, time-series signals, and text data are fused to obtain a joint feature vector.

[0157] In an exemplary embodiment, the multimodal large model optimization and update device further includes a deployment module, which is used to obtain the prediction accuracy, average inference time, and maximum memory usage of the optimized and updated multimodal large model on the optimized sample set, to obtain a first prediction accuracy, a first average inference time, and a first maximum memory usage; obtain the prediction accuracy and maximum memory usage of the initial multimodal large model on the optimized sample set, to obtain a second prediction accuracy and a second maximum memory usage; if the difference between the first prediction accuracy and the second prediction accuracy is greater than a preset accuracy threshold, the first average inference time is less than a preset time threshold, and the absolute value of the difference between the first maximum memory usage and the second maximum memory usage is less than a preset content threshold, then the optimized and updated multimodal large model is deployed to the power inspection system.

[0158] In an exemplary embodiment, the update module 208 is further configured to acquire updated image data, updated time-series signals, and updated text data; fuse the features extracted from the updated image data, updated time-series signals, and updated text data respectively to obtain an updated joint feature vector; and input the updated joint feature vector into the optimized and updated multimodal large model to obtain the updated model prediction result.

[0159] In an exemplary embodiment, the update module 208 is further configured to determine the stability index value of the current time node based on the updated model prediction results within a set time range before and after the current time node; if the stability index value is greater than a preset stability threshold, the current time node is marked as a prediction unstable point.

[0160] In an exemplary embodiment, the deployment module is further configured to: acquire new task nodes appearing in the risk category region of the optimized sample set; input the joint feature vector corresponding to the new task node into the optimized and updated multimodal large model to obtain a first confidence peak; input the joint feature vector corresponding to the new task node into the initial multimodal large model to obtain a second confidence peak; calculate the difference between the first confidence peak and the second confidence peak corresponding to each new task node; sum the differences of all new task nodes to obtain a total gain value; divide the total gain value by the number of new task nodes to obtain a performance gain index; if the performance gain index is greater than a preset gain threshold, the optimized and updated multimodal large model is kept running; if the performance gain index is less than or equal to the preset gain threshold, the initial multimodal large model is restored.

[0161] In an exemplary embodiment, the calculation module 204 is further configured to perform semantic difference calculation based on the model prediction results and the annotation results to obtain a semantic difference item; obtain a confidence penalty item based on the model prediction results and obtain a complexity correction item based on a preset fuzziness score; multiply the semantic difference item, the confidence penalty item, and the complexity correction item to obtain a weighted difference degree; and smooth the weighted difference degree to obtain a normalized difference degree.

[0162] Each module in the aforementioned multimodal large model optimization and update device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0163] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal large model optimization update method.

[0164] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0165] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring structured task units and joint feature vectors corresponding to each time node in a power inspection system; the structured task unit includes model prediction results and annotation results received based on an interactive interface; determining the normalized difference degree of each time node based on the model prediction results, annotation results, and a preset fuzziness score; assigning importance scores to each time node based on the normalized difference degree, model prediction results, fuzziness scores, and a preset fluctuation regularization term to obtain importance weights; selecting the top N time nodes with the highest importance weights as retained nodes; constructing an optimized sample set based on the joint feature vectors and annotation results corresponding to the retained nodes; and training an initial multimodal large model based on the optimized sample set to obtain an optimized and updated multimodal large model.

[0166] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring multimodal data, model prediction results, and annotation results for each time node in the power inspection system; the multimodal data includes image data, time-series signals, and text data; encoding the multimodal data, model prediction results, and annotation results to obtain structured task units; and fusing the features extracted from the image data, time-series signals, and text data to obtain a joint feature vector.

[0167] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining the prediction accuracy, average inference time, and maximum memory usage of the optimized and updated multimodal large model on the optimized sample set, to obtain a first prediction accuracy, a first average inference time, and a first maximum memory usage; obtaining the prediction accuracy and maximum memory usage of the initial multimodal large model on the optimized sample set, to obtain a second prediction accuracy and a second maximum memory usage; if the difference between the first prediction accuracy and the second prediction accuracy is greater than a preset accuracy threshold, the first average inference time is less than a preset time threshold, and the absolute value of the difference between the first maximum memory usage and the second maximum memory usage is less than a preset content threshold, then the optimized and updated multimodal large model is deployed to the power inspection system.

[0168] In one embodiment, when the processor executes the computer program, it further performs the following steps: acquiring updated image data, updated time-series signals, and updated text data; fusing the features extracted from the updated image data, updated time-series signals, and updated text data respectively to obtain an updated joint feature vector; and inputting the updated joint feature vector into the optimized and updated multimodal large model to obtain the updated model prediction result.

[0169] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the stability index value of the current time node based on the updated model prediction results within a set time range before and after the current time node; if the stability index value is greater than a preset stability threshold, then the current time node is marked as a predicted unstable point.

[0170] In one embodiment, when the processor executes the computer program, it further implements the following steps: acquiring new task nodes appearing in the risk category region of the optimized sample set; inputting the joint feature vector corresponding to the new task node into the optimized and updated multimodal large model to obtain a first confidence peak; inputting the joint feature vector corresponding to the new task node into the initial multimodal large model to obtain a second confidence peak; calculating the difference between the first confidence peak and the second confidence peak corresponding to each new task node; summing the differences of all new task nodes to obtain a total gain value; dividing the total gain value by the number of new task nodes to obtain a performance gain index; if the performance gain index is greater than a preset gain threshold, then keeping the optimized and updated multimodal large model running; if the performance gain index is less than or equal to the preset gain threshold, then restoring the initial multimodal large model.

[0171] In one embodiment, when the processor executes the computer program, it further performs the following steps: calculating semantic differences based on model prediction results and annotation results to obtain semantic difference terms; obtaining confidence penalty terms based on model prediction results and complexity correction terms based on preset fuzziness scores; multiplying the semantic difference terms, confidence penalty terms, and complexity correction terms to obtain weighted difference degrees; and smoothing the weighted difference degrees to obtain normalized difference degrees.

[0172] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: obtaining structured task units and joint feature vectors corresponding to each time node in a power inspection system; the structured task units include model prediction results and annotation results received based on an interactive interface; determining the normalized difference degree of each time node based on the model prediction results, annotation results, and a preset fuzziness score; assigning importance scores to each time node based on the normalized difference degree, model prediction results, fuzziness score, and a preset fluctuation regularization term to obtain importance weights; selecting the top N time nodes with the highest importance weights as retained nodes; constructing an optimized sample set based on the joint feature vectors and annotation results corresponding to the retained nodes; and training the initial multimodal large model based on the optimized sample set to obtain an optimized and updated multimodal large model.

[0173] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring multimodal data, model prediction results, and annotation results for each time node in the power inspection system; the multimodal data includes image data, time-series signals, and text data; encoding the multimodal data, model prediction results, and annotation results to obtain structured task units; and fusing the features extracted from the image data, time-series signals, and text data to obtain a joint feature vector.

[0174] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the prediction accuracy, average inference time, and maximum memory usage of the optimized and updated multimodal large model on the optimized sample set, to obtain a first prediction accuracy, a first average inference time, and a first maximum memory usage; obtaining the prediction accuracy and maximum memory usage of the initial multimodal large model on the optimized sample set, to obtain a second prediction accuracy and a second maximum memory usage; if the difference between the first prediction accuracy and the second prediction accuracy is greater than a preset accuracy threshold, the first average inference time is less than a preset time threshold, and the absolute value of the difference between the first maximum memory usage and the second maximum memory usage is less than a preset content threshold, then the optimized and updated multimodal large model is deployed to the power inspection system.

[0175] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring updated image data, updated time-series signals, and updated text data; fusing the features extracted from the updated image data, updated time-series signals, and updated text data respectively to obtain an updated joint feature vector; and inputting the updated joint feature vector into the optimized and updated multimodal large model to obtain the updated model prediction result.

[0176] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: determining the stability index value of the current time node based on the updated model prediction results within a set time range before and after the current time node; if the stability index value is greater than a preset stability threshold, then marking the current time node as a predicted unstable point.

[0177] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: acquiring new task nodes appearing in the risk category region to which the optimized sample set belongs; inputting the joint feature vector corresponding to the new task node into the optimized and updated multimodal large model to obtain a first confidence peak; inputting the joint feature vector corresponding to the new task node into the initial multimodal large model to obtain a second confidence peak; calculating the difference between the first confidence peak and the second confidence peak corresponding to each new task node; summing the differences of all new task nodes to obtain a total gain value; dividing the total gain value by the number of new task nodes to obtain a performance gain index; if the performance gain index is greater than a preset gain threshold, then keeping the optimized and updated multimodal large model running; if the performance gain index is less than or equal to the preset gain threshold, then restoring the initial multimodal large model.

[0178] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: calculating semantic differences based on model prediction results and annotation results to obtain semantic difference terms; obtaining confidence penalty terms based on model prediction results and complexity correction terms based on preset fuzziness scores; multiplying the semantic difference terms, confidence penalty terms, and complexity correction terms to obtain weighted difference degrees; and smoothing the weighted difference degrees to obtain normalized difference degrees.

[0179] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0181] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0182] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for optimizing and updating a multimodal large model, characterized in that, The method includes: Obtain the structured task units and joint feature vectors corresponding to each time node in the power inspection system; the structured task units include model prediction results and annotation results received based on the interactive interface; Based on the model prediction results, the annotation results, and the preset fuzziness score, the normalization difference degree at each time point is determined; Based on the normalized difference, the model prediction results, the fuzziness score, and the preset fluctuation regularization term, the importance of each time node is scored to obtain the importance weight. Select the top N time points with the highest importance weight as the retained nodes; An optimized sample set is constructed based on the joint feature vector corresponding to the retained node and the annotation result; The initial multimodal large model is trained based on the optimized sample set to obtain the optimized and updated multimodal large model; The method further includes, after training the initial multimodal large model based on the optimized sample set to obtain the optimized and updated multimodal large model, the method further includes: Obtain the prediction accuracy, average inference time, and maximum memory usage of the optimized and updated multimodal large model for the optimized sample set, and obtain the first prediction accuracy, the first average inference time, and the first maximum memory usage; Obtain the prediction accuracy and maximum memory usage of the initial multimodal large model on the optimized sample set, and then obtain the second prediction accuracy and the second maximum memory usage. If the difference between the first prediction accuracy and the second prediction accuracy is greater than a preset accuracy threshold, the first average inference time is less than a preset time threshold, and the absolute value of the difference between the first maximum memory usage and the second maximum memory usage is less than a preset content threshold, then the optimized and updated multimodal large model will be deployed to the power inspection system. The method further includes, after deploying the optimized and updated multimodal large model to the power inspection system: Obtain new task nodes appearing in the risk category region to which the optimized sample set belongs; The joint feature vector corresponding to the new task node is input into the optimized and updated multimodal large model to obtain the first confidence peak. The joint feature vector corresponding to the new task node is input into the initial multimodal large model to obtain the second confidence peak. Calculate the difference between the first confidence peak and the second confidence peak for each new task node; The total gain value is obtained by summing the differences between all the new task nodes. Divide the total gain value by the number of new task nodes to obtain the performance gain index; If the performance gain index is greater than the preset gain threshold, then the optimized and updated multimodal large model will continue to run. If the performance gain index is less than or equal to the preset gain threshold, then the initial multimodal large model is restored.

2. The method according to claim 1, characterized in that, The acquisition of structured task units and joint feature vectors corresponding to each time node in the power inspection system includes: The system acquires multimodal data, model prediction results, and annotation results at various time points in the power inspection system; the multimodal data includes image data, time-series signals, and text data. The structured task unit is obtained by encoding the multimodal data, the model prediction results, and the annotation results. The features extracted from the image data, the time-series signal, and the text data are fused to obtain the joint feature vector.

3. The method according to claim 1, characterized in that, After deploying the optimized and updated multimodal large model to the power inspection system, the method further includes: Acquire updated image data, updated time-series signals, and updated text data; The features extracted from the updated image data, the updated time-series signal, and the updated text data are fused to obtain an updated joint feature vector; The updated joint feature vector is input into the optimized and updated multimodal large model to obtain the updated model prediction results.

4. The method according to claim 3, characterized in that, After inputting the updated joint feature vector into the optimized and updated multimodal large model to obtain the updated model prediction result, the method further includes: Based on the updated model prediction results within a set time range before and after the current time node, determine the stability index value for the current time node; If the stability index value is greater than the preset stability threshold, then the current time node is marked as a predicted unstable point.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the normalized difference degree at each time point based on the model prediction results, the annotation results, and the preset fuzziness score includes: Semantic differences are calculated based on the model prediction results and the annotation results to obtain semantic difference items; The confidence penalty term is obtained based on the model prediction results, and the complexity correction term is obtained based on the preset fuzziness score. The weighted difference is obtained by multiplying the semantic difference term, the confidence penalty term, and the complexity correction term; The weighted difference is smoothed to obtain the normalized difference.

6. A multimodal large model optimization and update device, characterized in that, The device includes: The acquisition module is used to acquire the structured task units and joint feature vectors corresponding to each time node in the power inspection system; the structured task units include model prediction results and annotation results received based on the interactive interface; The calculation module is used to determine the normalization difference degree at each time point based on the model prediction results, the annotation results, and the preset fuzziness score. The scoring module is used to score the importance of each time node based on the normalized difference, the model prediction result, the fuzziness score, and the preset fluctuation regularization term, and obtain the importance weight; the top N time nodes with the highest importance weights are selected as retained nodes. The update module is used to construct an optimized sample set based on the joint feature vector corresponding to the retained node and the annotation result; and to train the initial multimodal large model based on the optimized sample set to obtain the optimized and updated multimodal large model. The device further includes: The deployment module is used to obtain the prediction accuracy, average inference time, and maximum memory usage of the optimized and updated multimodal large model on the optimized sample set, and to obtain a first prediction accuracy, a first average inference time, and a first maximum memory usage; to obtain the prediction accuracy and maximum memory usage of the initial multimodal large model on the optimized sample set, and to obtain a second prediction accuracy and a second maximum memory usage; if the difference between the first prediction accuracy and the second prediction accuracy is greater than a preset accuracy threshold, the first average inference time is less than a preset time threshold, and the absolute value of the difference between the first maximum memory usage and the second maximum memory usage is less than a preset content threshold, then the optimized and updated multimodal large model is deployed to the power inspection system; The deployment module is further configured to: acquire new task nodes appearing in the risk category region of the optimized sample set; input the joint feature vector corresponding to the new task node into the optimized and updated multimodal large model to obtain a first confidence peak; input the joint feature vector corresponding to the new task node into the initial multimodal large model to obtain a second confidence peak; calculate the difference between the first confidence peak and the second confidence peak for each new task node; sum the differences of all the new task nodes to obtain a total gain value; divide the total gain value by the number of new task nodes to obtain a performance gain index; if the performance gain index is greater than a preset gain threshold, the optimized and updated multimodal large model is kept running; if the performance gain index is less than or equal to the preset gain threshold, the initial multimodal large model is restored.

7. The apparatus according to claim 6, characterized in that, The acquisition module is further configured to acquire multimodal data, model prediction results, and annotation results for each time node in the power inspection system; the multimodal data includes image data, time-series signals, and text data; the structured task unit is obtained by encoding the multimodal data, the model prediction results, and the annotation results; and the features extracted from the image data, the time-series signals, and the text data are fused to obtain the joint feature vector.

8. The apparatus according to claim 6, characterized in that, The update module is further configured to acquire updated image data, updated time-series signals, and updated text data; fuse the features extracted from the updated image data, the updated time-series signals, and the updated text data respectively to obtain an updated joint feature vector; and input the updated joint feature vector into the optimized and updated multimodal large model to obtain the updated model prediction result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images