Power plant station diagram-based multimodal large model post-training and inference optimization method

Abstract

The application relates to a power plant station diagram-based multimodal large model post-training and inference optimization method and device, computer equipment, a readable storage medium and a computer program product. The method comprises the following steps: acquiring a multimodal power dataset and a power professional knowledge graph constructed based on power plant station diagram paper data; configuring a plurality of power special post-training tasks for a general multimodal large model according to the multimodal power dataset and the power professional knowledge graph; performing post-training on the general multimodal large model according to a preset post-training optimization strategy and model training samples matched with each power special post-training task; performing quantization processing on the post-training multimodal large model; optimizing the model inference strategy of the quantized multimodal large model; and obtaining a target multimodal power large model used for performing a model inference task according to the model inference strategy. The method can improve the power specialization of the multimodal large model for understanding the power plant station diagram scene.

Description

Technical Field

[0001] This application relates to the field of intelligent power system technology, and in particular to a method, apparatus, computer equipment, readable storage medium, and computer program product for post-training and inference optimization of a multimodal large model based on power plant diagrams. Background Technology

[0002] With the gradual application of multimodal large models in the power field, the understanding of power plant maps, which contain complex topological relationships, dense professional symbols and fine-grained textual information, places extremely high demands on the professional accuracy, inference speed and generalization ability of the models.

[0003] However, current general-purpose multimodal large models, after pre-training, lack customized optimization to incorporate the specialized characteristics of power plant maps and fail to integrate professional knowledge of power plant maps. Even after post-training, these models still cannot accurately understand the semantics of power scenarios, resulting in insufficient fine-grained recognition accuracy. Furthermore, the inference stage often employs general acceleration strategies, which can easily lead to accuracy loss or poor acceleration effects, failing to meet the dual requirements of power scenarios for inference accuracy and speed. Multimodal large models trained using traditional post-training and inference methods suffer from low specialized accuracy. When directly applied to power plant map understanding, they exhibit problems such as biases in understanding specialized semantics, low fine-grained recognition accuracy, high inference latency, and high resource consumption, thus hindering the large-scale deployment of multimodal large models in core scenarios such as power operation and maintenance and dispatching. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, device, computer equipment, readable storage medium, and computer program product for post-training and inference optimization of a multimodal large model based on power plant map to address the above-mentioned technical problems, so as to achieve both power-specific accuracy and inference speed for multimodal large models of power plant map understanding scenarios.

[0005] In a first aspect, this application provides a method for post-training and inference optimization of a multimodal large model based on power plant diagrams, the method comprising the following steps:

[0006] Obtain a multimodal power dataset constructed based on power plant drawing data, and a power professional knowledge graph constructed based on power plant drawing data;

[0007] Based on the multimodal power dataset and the power expertise graph, at least two power-specific post-training tasks are configured for the general multimodal large model;

[0008] According to the preset post-training optimization strategy and the model training samples that match each of the power special post-training tasks, the general multimodal large model is post-trained to obtain the post-trained multimodal large model.

[0009] The post-trained multimodal large model is quantized to obtain a quantized multimodal large model; the model inference strategy of the quantized multimodal large model is optimized to obtain a target multimodal power large model; the target multimodal power large model is used to perform model inference tasks according to the model inference strategy.

[0010] In one embodiment, when the model inference task includes a drawing device recognition task, the model inference strategy includes a fine-grained region perception enhancement strategy, and performing the model inference task according to the model inference strategy includes:

[0011] The attention weights of the input power plant drawing image are extracted to obtain the attention heatmap of the power plant drawing image;

[0012] Based on the attention heatmap of the power plant drawing image, the key identification areas in the power plant drawing image are determined;

[0013] The key identification area is magnified to obtain the magnified key identification area;

[0014] A secondary inference is performed on the magnified key recognition area to obtain the inference result of the drawing device recognition task.

[0015] In one embodiment, when the model inference task includes a drawing device recognition task, the model inference strategy includes an image preprocessing optimization strategy, and executing the model inference task according to the model inference strategy includes:

[0016] Image enhancement processing is performed on the input power plant drawing image to obtain an enhanced power plant drawing image;

[0017] Key areas in the enhanced plant site drawing image are located to obtain a coarse positioning image;

[0018] Based on the standard primitive template of the power plant drawing image, the coarse positioning image is matched to determine the equipment information in the power plant drawing image in order to perform the drawing equipment identification task.

[0019] In one embodiment, the model inference strategy includes a task priority inference and scheduling strategy, and the step of executing the model inference task according to the model inference strategy includes:

[0020] In response to receiving the inference request of the model inference task, the task priority of the model inference task is determined according to the task priority preset relationship configured by the task priority inference and scheduling strategy; the task priority preset relationship records the mapping relationship between the model inference task and the task priority.

[0021] According to the model inference task, allocate computing resources that match the task priority, and execute the model inference task.

[0022] In one embodiment, the post-training optimization strategy includes a hard example sampling strategy, which configures the sampling probability of the model training samples as follows:

[0023]

[0024] in, Characterizing the training samples of the model Difficult sample in the middle, Characterizing the training samples of the model Regular samples in; Training samples for the model The basic weights, The sampling amplification factor for difficult examples; Weights for difficult examples; For regular sample weights;

[0025] The difficult example sampling strategy is also used to configure the step of obtaining the model training samples as follows:

[0026] Based on the sampling probability, obtain the difficult sample from the power plant drawing data;

[0027] Attribution analysis is performed on the difficult example samples to obtain the attribution analysis results. Based on the attribution analysis results, the power professional knowledge graph is updated to obtain an enhanced knowledge graph.

[0028] The difficult example samples are fused with the multimodal power dataset and the enhanced knowledge graph to obtain the model training samples.

[0029] In one embodiment, the post-training optimization strategy is configured to take the loss function used during the post-training of the general multimodal large model and configure it as follows:

[0030]

[0031] in, The cross-entropy loss of the general multimodal large model; The power industry loss that penalizes model predictions that violate the rules of the power knowledge graph; The comparison loss between the feature distance of correct question-answer pairs and the feature distance of incorrect question-answer pairs in the model prediction results of the general multimodal large model; The topology consistency loss between the predicted device path and the actual device path in the model prediction results of the general multimodal large model; , , This represents the weighting coefficient of each loss term.

[0032] Secondly, this application provides a multimodal large model post-training and inference optimization device based on power plant diagrams, the device comprising:

[0033] The training data acquisition module is used to acquire a multimodal power dataset constructed based on power plant drawing data, as well as a power professional knowledge graph constructed based on power plant drawing data;

[0034] The training task configuration module is used to configure at least two power-specific post-training tasks for the general multimodal large model based on the multimodal power dataset and the power professional knowledge graph.

[0035] The post-training strategy optimization module is used to post-train the general multimodal large model according to the preset post-training optimization strategy and the model training samples that match each of the power special post-training tasks, so as to obtain the post-trained multimodal large model.

[0036] The inference strategy optimization module is used to quantize the post-trained multimodal large model to obtain a quantized multimodal large model; optimize the model inference strategy of the quantized multimodal large model to obtain a target multimodal power large model; the target multimodal power large model is used to perform model inference tasks according to the model inference strategy.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described multimodal large model post-training and inference optimization method based on power plant diagrams.

[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0039] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0040] The aforementioned method, apparatus, computer equipment, readable storage medium, and computer program product for post-training and inference optimization of a multimodal large-scale model based on power plant maps, by acquiring a multimodal power dataset and power professional knowledge graph constructed from power plant map data, can cover multiple modal scenario data in the power system, thereby improving the generalization ability of the subsequent target large-scale model; based on the multimodal power dataset and power professional knowledge graph, at least two power-specific post-training tasks are configured for the general multimodal large-scale model, thereby improving the model's professional semantic understanding and fine-grained recognition ability of power plant maps through the specialized design of post-training tasks; and according to the preset post-training optimization strategy and the model training samples matching each power-specific post-training task, the general multimodal large-scale model is post-trained to obtain a post-trained multimodal large-scale model. This system enables power-specific training of a general model according to a pre-defined post-training optimization strategy, improving the power industry-specific adaptability and accuracy of the multimodal large model. Then, the post-trained multimodal large model is quantized to obtain a quantized multimodal large model. This lightweighting process simplifies the model, reducing resource consumption during inference and improving inference speed. Furthermore, the model inference strategy of the quantized multimodal large model is optimized, thus constructing a collaborative mechanism between post-training and inference optimization. This forms an integrated "post-training-inference optimization" solution for multimodal large models of power plant diagram understanding, resulting in a target multimodal power large model for performing model inference tasks according to the model inference strategy. This multimodal large model, tailored to power plant diagram understanding scenarios, achieves a balance between power industry-specific accuracy and inference speed. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is an application environment diagram of a multimodal large model post-training and inference optimization method based on power plant diagrams in one embodiment;

[0043] Figure 2 This is a schematic diagram illustrating the post-training and inference optimization process of a multimodal large model based on a power plant diagram in one embodiment.

[0044] Figure 3 This is a schematic diagram of a fine-grained region perception enhancement strategy in one embodiment;

[0045] Figure 4This is a flowchart illustrating an integrated post-training and inference optimization method for understanding multimodal large models of power plant diagrams in one embodiment.

[0046] Figure 5 This is a structural block diagram of a multimodal large model post-training and inference optimization device based on power plant diagrams in one embodiment;

[0047] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] Currently, most post-training methods for multimodal large models are designed for general scenarios, lacking specificity. They fail to be customized and optimized to suit the professional characteristics of power plant maps, and do not incorporate professional knowledge of power plant maps (such as topological relationships, equipment parameters, and professional symbols). They also fail to be customized to suit the characteristics of power plant map understanding tasks (such as the priority of key equipment identification and inference accuracy requirements). Furthermore, the post-training samples cover limited scenarios, and the inference optimization does not consider the differences between different voltage levels and different types of power plant maps, resulting in insufficient model generalization ability and a disconnect between the post-training task design and the power scenario.

[0050] Traditional power-related multimodal large models employ general post-training strategies and often use general acceleration strategies during the inference phase. They fail to optimize model parameters unrelated to the power scenario during inference, resulting in high model redundancy, consuming significant computational and storage resources, and making them unsuitable for deployment on power field terminal equipment. Furthermore, they tend to lead to low post-training efficiency, failing to optimize the training process in conjunction with the characteristics of power data, resulting in slow training convergence and low sample utilization. They also fail to improve the model's ability to recognize and understand specialized symbols and topological relationships, and do not consider the core requirement of "accuracy first, speed second" in power scenarios.

[0051] Traditional power-related model optimization techniques often focus on a single aspect (post-training only or inference acceleration only). Post-training and inference optimization are independent of each other and do not consider their synergistic effect, which can easily lead to accuracy loss or poor acceleration. They have not formed an integrated "post-training-inference optimization" solution, and cannot fundamentally solve the problems of model adaptability and practicality in power plant diagram understanding. This results in low professional accuracy, slow inference speed, and high resource consumption of the model. It cannot meet the dual requirements of power scenarios for inference accuracy and speed, and cannot meet the actual application needs of power scenarios. This restricts the large-scale deployment of multimodal large models in core scenarios such as power operation and maintenance and dispatch.

[0052] The multimodal large model post-training and inference optimization method based on power plant diagrams provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 can acquire a multimodal power dataset and a power professional knowledge graph constructed based on power plant drawing data; server 104 can configure at least two power-specific post-training tasks for the general multimodal large model based on the multimodal power dataset and power professional knowledge graph; server 104 can perform post-training on the general multimodal large model according to a preset post-training optimization strategy and model training samples matching each power-specific post-training task, obtaining a post-trained multimodal large model; server 104 can quantize the post-trained multimodal large model to obtain a quantized multimodal large model; optimize the model inference strategy of the quantized multimodal large model to obtain a target multimodal power large model used to execute model inference tasks according to the model inference strategy; target multimodal power large model. The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0053] In one exemplary embodiment, such as Figure 2 As shown, a method for post-training and inference optimization of a multimodal large model based on power plant diagrams is provided. This embodiment applies this method to... Figure 1 Taking server 104 as an example, for instance, the training phase is deployed on a dedicated training server cluster for the power system, while the inference phase can be deployed on a server or a field terminal. In this embodiment, the method includes the following steps S202 to S208. Wherein:

[0054] Step S202: Obtain a multimodal power dataset constructed based on power plant drawing data, and a power professional knowledge graph constructed based on power plant drawing data.

[0055] In this context, "power plant" can refer to power facilities in a power system, such as power plants, substations, and distribution stations. Power plant diagrams, including primary wiring diagrams, secondary wiring diagrams, power flow diagrams, and electrical plan diagrams of various power facilities within a power plant, are core power drawings. They typically contain a large number of specialized equipment symbols, complex topological connections, and detailed text annotations (such as equipment numbers and parameters), exhibiting numerous, dense, and precise elements.

[0056] In practical applications, server 104 can acquire a pre-constructed high-quality multimodal power dataset of power plant diagrams. It can then select 8,000 samples from this dataset as a post-training dataset (7,000 training samples and 1,000 validation samples), covering multiple voltage levels of 110kV, 220kV, and 500kV to ensure the professionalism and coverage of the samples. Furthermore, it can acquire a power professional knowledge graph constructed based on the power plant diagram data. This power professional knowledge graph is based on professional knowledge related to power plant diagrams, including equipment symbol specifications, topological relationship rules, and equipment parameter ranges, which can be used for post-training task design and loss function optimization. Thus, the constructed power professional knowledge graph is integrated into the selected multimodal power dataset, forming the basic conditions for post-training.

[0057] As a possible example, a multimodal power dataset can be constructed through the following steps: acquiring original power plant drawings from the power system; preprocessing the original power plant drawings to obtain basic sample images of the power plants; annotating the basic sample images to obtain paragraph-level image-text pair data of the power plants; using a large language model to perform atomic-level parsing on the paragraph-level image-text pair data to decompose the original power plant drawings into multiple minimum knowledge units representing a single device or attribute of the power plant; generating atomic-level question-and-answer pairs of the power plants based on each minimum knowledge unit; and augmenting the basic sample images according to at least one of the preset data augmentation methods: local magnification, key region segmentation, and view range expansion, to generate power plant datasets. The multi-scale augmented data is fused from paragraph-level text-image pairs, atomic-level question-answer pairs, and multi-scale augmented data to obtain the power dataset to be validated. Error samples in the power dataset to be validated are detected, and multiple different large language models are used to cross-validate the detected error samples. The cross-validated error samples are then corrected to obtain the accurate sample dataset. Bias detection is performed on the accurate sample dataset to obtain the bias detection results, which represent the data deviation between the power dataset to be validated and the application scenario of power plants. Based on the bias detection results, the accurate sample dataset is adjusted, thereby validating the quality of the power dataset to be validated and obtaining a multimodal power dataset for training a large multimodal power model.

[0058] Among them, paragraph-level image-text pairs are used to record the mapping relationship between the original power plant drawing data and its corresponding semantic text description; atomic-level question-and-answer pairs are used to record the mapping relationship between questions and answers regarding equipment information in power plants.

[0059] Step S204: Based on the multimodal power dataset and the power expertise graph, configure at least two power-specific post-training tasks for the general multimodal large model.

[0060] The power-specific post-training tasks include basic semantic alignment, fine-grained equipment recognition, topological relationship understanding, equipment parameter extraction, and comprehensive scene reasoning. Each power-specific post-training task corresponds to a different post-training level, namely, basic semantic alignment, fine-grained equipment recognition, topological relationship understanding, equipment parameter extraction, and comprehensive scene reasoning.

[0061] In the specific implementation, server 104 can configure basic semantic alignment, fine-grained equipment identification, topological relationship understanding, equipment parameter extraction, and comprehensive scenario reasoning for the general multimodal large model based on the multimodal power dataset and power professional knowledge graph. Differentiated training strategies can be designed for these power-specific tasks.

[0062] When selecting a general multimodal large model, you can choose a mainstream general multimodal large model on the market as the base model, retain the core feature extraction and cross-modal interaction capabilities of the base model, remove redundant parameters for general scenarios, and reduce the complexity of post-training.

[0063] Step S206: According to the preset post-training optimization strategy and the model training samples that match the post-training tasks of each power special project, the general multimodal large model is post-trained to obtain the post-trained multimodal large model.

[0064] In the specific implementation, the server 104 can perform mixed precision training on the general multimodal large model according to the post-training optimization strategy that matches the post-training task of each power special project, and the model training samples obtained by the adaptive sampling algorithm. This can be done using a mixed precision of FP16 (16-bit half precision) + FP32 (32-bit single precision) to improve the training speed and reduce the memory usage while ensuring the training accuracy. The training convergence speed can be improved by more than 25%, thus obtaining the post-trained multimodal large model (post-trained model).

[0065] Optionally, the learning rate during training can be set to 1e-5, the batch size to 32, and the number of training epochs to 15. A cosine annealing strategy can be used to adjust the learning rate to avoid overfitting, and then a multimodal large model can be trained.

[0066] Simultaneously, an early stopping mechanism can be set in the post-training process: using validation set accuracy as an indicator, training stops when the validation set accuracy shows no improvement for three consecutive rounds, and the optimal post-trained multimodal model is saved. Furthermore, training resources can be allocated according to the priority of different training tasks. Based on the needs of the power scenario, tasks such as topology relationship judgment and key equipment identification can be set as high-priority tasks, allocating more training resources to ensure the accuracy of core tasks. Therefore, through the above differentiated training strategies and targeted optimization methods, the recognition accuracy of the post-trained multimodal large model for subtle technical symbols and complex topological relationships can be comprehensively enhanced.

[0067] For example, ① for the basic semantic alignment task, a professional semantic alignment task can be designed based on the power industry knowledge graph and multimodal dataset to accurately align power industry terms, topological relationships, and image features, thereby improving the professional semantic understanding ability of the subsequently trained multimodal large model. For the training strategy of the basic semantic alignment task, a contrastive learning reinforcement strategy can be adopted, using InfoNCE (a loss function based on noise contrastive estimation) to construct positive sample pairs (correct symbol-name) and negative sample pairs (incorrect symbol-name), thus narrowing the distance between positive samples in the feature space and pushing away negative samples.

[0068] ② For training strategies of fine-grained device recognition tasks, a region attention focusing strategy can be adopted. A learnable spatial attention mask can be introduced into the visual encoder to give higher weight to the device region and suppress background interference. At the same time, Focal Loss can be used to solve the sample imbalance problem of small target devices.

[0069] ③ The training strategy for the topology understanding (topology judgment) task can adopt a graph structure embedding strategy, which transforms the plant graph topology into an adjacency matrix that can be processed by a graph neural network (GNN). In the cross-modal fusion layer of the post-trained multimodal large model, a graph attention mechanism (GAT) is introduced to enable the model to learn the graph representation of "node (device) - edge (connection relationship)".

[0070] ④ For training strategies of comprehensive scenario reasoning tasks, a chain thinking (CoT) supervision strategy can be adopted. The training data includes step-by-step reasoning labels such as "first identify the main transformer → then track the outgoing line → judge the switch status → draw a conclusion". Through supervision and fine-tuning, the intermediate reasoning steps are output by the subsequent training of multimodal large model, thereby improving the interpretability and accuracy of complex reasoning.

[0071] For the key optimization task of recognizing small professional symbols, the following can be further adopted: ① Multi-scale feature fusion: After extracting features from the shallow (preserving high-resolution details), medium and deep layers of the visual encoder of the post-trained multimodal large model, multi-scale fusion is performed through the feature pyramid network (FPN) of the post-trained multimodal large model, so that small targets such as grounding symbols and surge arrester symbols can be accurately located and correctly classified; ② Super-resolution auxiliary module: A lightweight super-resolution network (such as a simplified version of ESRGAN) is added to the input of the post-trained multimodal large model to perform 2× or 4× super-resolution reconstruction on the symbol-dense areas before feeding them into the main model, so as to improve the symbol recognition of the target multimodal power large model; ③ Symbol template matching enhancement: A power symbol template library based on the standard of graphic symbols for electrical diagrams can be constructed. During training, power symbol templates are randomly superimposed on power plant drawing images at different scales, angles and transparency to enhance the robustness of the target multimodal power large model to deformed and occluded symbols.

[0072] For the key optimization task of identifying complex topological relationships, the following can be further adopted: ① Topology-aware attention mechanism: Introduce topological position encoding in the cross-modal attention calculation of the post-trained multimodal large model, and consider the spatial location of visual features and the electrical distance of the equipment (such as the electrical distance of equipment connected by the same bus is 1), so that the attention is integrated with electrical topology information; ② Path reasoning loss: Add a path consistency loss function to the topology relationship understanding layer, requiring the final target multimodal power large model to minimize the edit distance between the path prediction result of equipment A to equipment B and the actual topology path, so as to enhance the end-to-end topology reasoning capability; ③ Hierarchical topology parsing: Analyze the complex power plant map in layers according to voltage level (such as 500kV layer → 220kV layer → 110kV layer), perform topology reasoning independently for each layer, and then perform inter-layer association between each voltage level, so as to effectively reduce the complexity of a single reasoning.

[0073] Step S208: Quantize the post-trained multimodal large model to obtain the quantized multimodal large model; optimize the model inference strategy of the quantized multimodal large model to obtain the target multimodal power large model.

[0074] Among them, the target multimodal power large model is used to perform model inference tasks according to the model inference strategy.

[0075] In practical applications, server 104 can perform quantization processing on the post-trained multimodal large model through quantization methods such as structured pruning, quantization optimization, and parameter distillation to obtain the quantized multimodal large model.

[0076] Optionally, (1) Structured pruning: By performing structured pruning on the post-trained multimodal model, redundant convolutional layers and fully connected layers that are irrelevant to the power scenario can be removed, while retaining parameters related to core feature extraction and professional semantic understanding, a structured pruning model can be obtained, and the total number of model parameters can be reduced by more than 40%.

[0077] (2) Quantization optimization: The INT8 quantization strategy can be used to quantize the weights and activation values ​​in the post-trained multimodal large model to obtain a quantized optimized model. Under the premise that the loss of accuracy does not exceed 2%, the model storage and computation can be reduced, and the inference speed can be improved by more than 50%.

[0078] (3) Parametric Distillation: Small-scale distillation models can be constructed, using the optimal post-trained multimodal large model as the teacher model and the distillation model as the student model. The professional knowledge and reasoning ability of the teacher model are transferred to the student model, further reducing model complexity and adapting to field terminal deployment. Lightweight distillation models can lower the hardware threshold for terminal deployment, enabling the model to run on edge computing devices; improve inference speed to meet the real-time response requirements of the field; inherit the professional capabilities of the teacher model in power-specific tasks such as symbol recognition and topology judgment, maintaining high accuracy while compressing the model; enhance the generalization robustness to complex plant and station diagram scenarios; reduce the hardware and maintenance costs of batch deployment; and support rapid fine-tuning using a small amount of field data, flexibly adapting to changes in plant and station equipment. Through parametric distillation, large models can be effectively deployed from the cloud to the power production field.

[0079] Furthermore, server 104 can optimize the model inference strategy of the quantized multimodal large model, such as task priority inference and scheduling strategies, image preprocessing optimization strategies, inference caching mechanisms, and fine-grained region awareness enhancement strategies, to obtain the target multimodal power large model. It can also optimize the inference output format of the target multimodal power large model so that it can be adapted to the data interfaces of the power system's operation and maintenance platform and scheduling platform, without requiring additional format conversion and reducing deployment costs.

[0080] As an example, regarding the inference caching mechanism, when target content (such as plant image segments and equipment symbols) is generated multiple times in the inference results of the target multimodal power model, the inference results of frequently occurring plant image segments and equipment symbols can be cached. When encountering the same model inference task scenario later, the target multimodal power model can directly call the cache to reduce repeated inference and further improve the inference speed.

[0081] The aforementioned multimodal large-scale model post-training and inference optimization method based on power plant maps acquires a multimodal power dataset constructed from power plant map data and a power professional knowledge graph constructed from the same data. This covers multiple modal scenario data within the power system, enhancing the generalization ability of the subsequent target large-scale model. Based on the multimodal power dataset and the power professional knowledge graph, at least two power-specific post-training tasks are configured for the general multimodal large-scale model. Through the specialized design of these post-training tasks, the model's professional semantic understanding and fine-grained recognition capabilities regarding power plant maps are improved. Following a pre-defined post-training optimization strategy and matching model training samples for each power-specific post-training task, the general multimodal large-scale model is post-trained to obtain a post-trained multimodal large-scale model, thereby enabling the model to perform various functions according to different scenarios. Following the pre-defined post-training optimization strategy, power-specific training of the general model was achieved, improving the power industry adaptability and professional accuracy of the multimodal large model. Then, the post-trained multimodal large model was quantized to obtain a quantized multimodal large model. This lightweighting process simplifies the model, reducing resource consumption during inference and improving inference speed. Furthermore, the model inference strategy of the quantized multimodal large model was optimized, thus constructing a collaborative mechanism between post-training and inference optimization. This resulted in an integrated "post-training-inference optimization" solution for the multimodal large model of power plant diagram understanding, yielding a target multimodal power large model for performing model inference tasks according to the model inference strategy. This multimodal large model, tailored to power plant diagram understanding scenarios, achieves a balance between power industry-specific accuracy and inference speed.

[0082] In an exemplary embodiment, when the model inference task includes a drawing and equipment recognition task, the model inference strategy includes a fine-grained region perception enhancement strategy. The model inference task is executed according to the model inference strategy, including: extracting the attention weights of the input power plant drawing image to obtain an attention heatmap of the power plant drawing image; determining key recognition regions in the power plant drawing image based on the attention heatmap; magnifying the key recognition regions to obtain magnified key recognition regions; and performing secondary inference on the magnified key recognition regions to obtain the inference result of the drawing and equipment recognition task.

[0083] Addressing the challenges posed by the dense equipment, minute symbols, and complex topology of 500kV substation maps, multimodal large-model visual encoders, limited by input resolution, suffer from insufficient perception of the textural details of minute equipment symbols, easily leading to the problem of "accurate localization but incorrect recognition." To resolve this, this application proposes an "attention heatmap-guided local pruning-magnification-re-inference" mechanism, i.e., an attention-driven fine-grained region perception enhancement strategy. For example... Figure 3As shown, fine-grained region perception enhancement strategies may include steps such as attention heatmap calculation, key region localization and cropping, local magnification and secondary inference.

[0084] As a possible example, step one, attention heatmap calculation: During the initial inference of the target multimodal power model, server 104 can simultaneously extract the cross-modal attention weights of the input power plant drawing image, perform weighted fusion of multiple layers of cross-modal attention (assigning higher weights to deeper layers), upsample to the original power plant drawing image resolution using bilinear interpolation, and generate an attention heatmap of the power plant drawing image. Step two, key region localization and cropping: Based on the attention heatmap, server 104 can use adaptive threshold segmentation to detect high-attention regions (i.e., key identification regions) in the power plant drawing image, perform connected component analysis on the key identification regions, merge them with neighboring regions, generate an outer rectangular cropping box for the key identification regions, and expand the boundaries of the outer rectangular cropping box (expanding the width and height of each region by 20%) to ensure the inclusion of the complete device context. Step 3, Local Magnification and Secondary Inference: Server 104 can adaptively determine the magnification factor based on the original area size of the power plant drawing image. LANCZOS resampling is used to locally magnify the key recognition areas, resulting in magnified key recognition areas. The local image (magnified key recognition areas) is then recombined with the original question, and secondary inference is performed on the magnified key recognition areas to obtain the inference result for the drawing equipment recognition task. If multiple high-interest areas are detected, each area can be independently magnified and inferred first, and then the local results can be fused for a global fusion inference to ensure topology consistency.

[0085] Attention-driven fine-grained region perception enhancement strategies are coordinated with task priorities: For the P0 (highest priority) security alarm identification task, attention pruning and amplification can be forcibly enabled to perform high-resolution re-inference on the alarm region; for the P1 (high priority) critical equipment status identification task, critical equipment regions can be automatically detected and amplified by 2 to 4 times; for the P2 (medium priority) topology path tracing task and the P3 (medium priority) device parameter reading task, dense regions can be selectively amplified according to the attention weight distribution; for the P4 (low priority) symbol normalization verification task and the P5 (lowest priority) full-map comprehensive report generation task, local amplification can be triggered when the global inference confidence is lower than the threshold to save computational resources.

[0086] The technical solution of this embodiment, through an attention-driven fine-grained region perception enhancement strategy, can improve the model's perception of the subtle texture details of equipment symbols in key recognition areas of power plant drawings for equipment recognition tasks. This ensures that the target multimodal large model can efficiently execute model inference tasks according to the model inference strategy.

[0087] In an exemplary embodiment, when the model inference task includes a drawing equipment recognition task, the model inference strategy includes an image preprocessing optimization strategy. The model inference task is executed according to the model inference strategy, including: performing image enhancement processing on the input power plant drawing image to obtain an enhanced power plant drawing image; locating key areas in the enhanced power plant drawing image to obtain a coarse localization image; and matching the coarse localization image based on standard primitive templates of the power plant drawing image to determine equipment information in the power plant drawing image, thereby performing the drawing equipment recognition task.

[0088] In practical implementation, the image preprocessing optimization strategy is tailored to the characteristics of power plant drawings, optimizing the image preprocessing process. Methods such as adaptive image enhancement and key area localization can be employed to reduce the processing of invalid information and improve inference efficiency. Regarding adaptive image enhancement, server 104 can perform image enhancement processing on the input power plant drawing image. It can use contrast-limited adaptive histogram equalization (CLAHE) to correct uneven illumination and low contrast issues, and adaptively adjust the image's contrast gain based on local variance. Simultaneously, it improves uneven illumination through local brightness shifting, making the symbols and lines in the power plant drawing image clearer and more distinguishable, thus obtaining an enhanced power plant drawing image. For key area localization, a two-stage strategy is adopted: In the first stage, server 104 can locate key areas in the enhanced power plant drawing image. For example, a lightweight YOLOv8 detector (a target detection model) can be used to quickly and coarsely locate key areas such as the main transformer area, busbar area, switchgear area, and labeled text area to obtain a coarse localization image. In the second stage, server 104 can use the power standard primitive template of the power plant drawing image to perform fine matching within the coarse localization image area. Through normalized cross-correlation calculation, the specific outline and type label of each device are accurately located to determine the device information in the power plant drawing image in order to perform the drawing equipment recognition task.

[0089] The technical solution of this embodiment, through an image preprocessing optimization strategy, can perform image enhancement processing and key area localization on power plant drawings for equipment identification tasks. This reduces redundant calculations across the entire power plant drawing and allows for centralized processing of key areas to determine equipment information in the power plant drawing image. This significantly improves the model's inference efficiency, thereby ensuring that the target multimodal large model can efficiently execute model inference tasks according to the model inference strategy.

[0090] In an exemplary embodiment, the model inference strategy includes a task priority inference and scheduling strategy. Executing a model inference task according to the model inference strategy includes: in response to receiving an inference request for a model inference task, determining the task priority of the model inference task according to the task priority preset relationship configured by the task priority inference and scheduling strategy; allocating computing resources matching the task priority according to the model inference task, and executing the model inference task.

[0091] Among them, the task priority preset relationship, as shown in Table 1, includes the specific classification and task priority of the reasoning tasks, and records the mapping relationship between the model reasoning tasks and task priorities.

[0092] Table 1. Specific classifications and priorities of reasoning tasks

[0093]

[0094] In the specific implementation, regarding the task priority reasoning and scheduling strategy, the server 104 can respond to the received reasoning request of the model reasoning task, parse the reasoning request in combination with the power scenario requirements, determine the task priority P of the model reasoning task according to the task priority preset relationship shown in Table 1, sort the model reasoning tasks by priority, prioritize the reasoning of key equipment identification and topology relationship judgment, and delay the reasoning of non-critical tasks to improve the response speed of core tasks.

[0095] Then, according to the model inference task, computing resources matching the task priority are allocated, and the model inference task is executed. For example, if there is a higher priority task in the current inference queue (P' < P), the inference process of the current low-priority task is paused, and computing resources (GPU memory, attention computation) are preferentially allocated to the high-priority task. For high-priority tasks of level P0 / P1, an inference caching mechanism is enabled, and the historical inference intermediate results of the plant map are retrieved from the cache first. If the cache is hit, incremental inference is only performed on the changed areas. For medium-priority tasks of level P2 / P3, block inference is adopted, and the large map is divided into multiple sub-regions for parallel processing to improve efficiency. For low-priority tasks of level P4 / P5, they are executed asynchronously in a background thread to ensure that the response speed of the foreground high-priority tasks is not affected.

[0096] The technical solution of this embodiment, through task priority reasoning and scheduling strategy, can determine the task priority of the model reasoning task according to the preset relationship of task priority for the received reasoning request, so as to allocate computing resources that match the task priority, thereby improving the response speed of the core reasoning task and ensuring that the target multimodal large model can efficiently execute the model reasoning task according to the model reasoning strategy.

[0097] In an exemplary embodiment, the post-training optimization strategy includes a hard example sampling strategy, which configures the sampling probability of model training samples as follows:

[0098]

[0099] in, Representation model training samples Difficult sample in the middle, Representation model training samples Regular samples in; Training samples for the model The basic weights, The sampling amplification factor for difficult examples; Weights for difficult examples; These are the weights for regular samples.

[0100] The difficult example sampling strategy is also used to configure the steps of obtaining model training samples as follows: according to the sampling probability, obtain difficult example samples from the power plant drawing data; perform attribution analysis on the difficult example samples to obtain the attribution analysis results; update the power professional knowledge graph according to the attribution analysis results to obtain the enhanced knowledge graph; and fuse the difficult example samples with the multimodal power dataset and the enhanced knowledge graph to obtain model training samples.

[0101] In the specific implementation, during the post-training process of the general multimodal large model, the server 104 can use an adaptive sampling algorithm based on the hard example sampling strategy (adaptive hard example sampling mechanism) to focus on sampling hard examples (such as complex topology graphs and fuzzy symbol graphs) of the model training samples. The hard examples can come from the substation multimodal dataset. The sampling ratio of hard examples can be controlled at 30%-40% to improve the sample utilization and model generalization ability.

[0102] For example, let the total number of samples in the training batch be N, where the number of hard examples is N_hard and the number of normal examples is N_normal. The hard example sampling strategy (adaptive hard example sampling mechanism) can then dynamically adjust the sampling probability of the model training samples as follows:

[0103]

[0104] in, Representation model training samples Difficult sample in the middle, Representation model training samples Regular samples in; Training samples for the model The base weight (initially 1). This is the sampling amplification factor for difficult examples (also known as the difficult example amplification factor), such as... For difficult cases, the magnification factor A dynamic adjustment strategy can be configured: i.e., in the early stages of training. It is relatively small at first, and gradually increases in size during the later stages of training. For regular sample weights; Weights for difficult examples It can decay with each training round: ,in For the attenuation factor (e.g.) =0.95), The number of training rounds is to prevent the model from overfitting to hard examples.

[0105] As a possible example, difficult examples can be automatically mined and filtered in the following ways: ① Model prediction confidence filtering: The post-trained multimodal large model obtained from the post-training process can be used to infer the training set (model training samples). Based on the confidence of the inference prediction results of the post-trained multimodal large model, samples with inference prediction confidence below a threshold θ (e.g., θ=0.5) can be selected from the model training samples; ② Loss value ranking filtering: The loss value of each model training sample on the current post-trained multimodal large model can be calculated, and samples with the top K (e.g., top 10%) loss values ​​can be selected from the model training samples; ③ Manual annotation error analysis: Power experts can be organized to classify erroneous cases in the model inference prediction results, and samples classified as "symbol confusion", "topology misjudgment", and "parameter misreading" can be selected from the model training samples; ④ Cross-scenario difference filtering: The performance of the post-trained multimodal large model on power plant diagrams at different voltage levels can be compared, and samples that perform significantly worse at a certain voltage level than at other levels can be selected from the model training samples. The obtained difficult examples can be integrated into the power professional knowledge graph and multimodal power dataset, and participate in training together with regular examples, rather than training only for difficult examples.

[0106] In addition, the difficult example sampling strategy may also include a difficult example-driven knowledge graph enhancement strategy, which configures the steps for obtaining model training samples as follows: the server 104 can obtain difficult example samples from the power plant drawing data according to the sampling probability through the above-mentioned automatic mining and filtering methods; perform attribution analysis on the mined difficult example samples to obtain the attribution analysis results of the difficult example samples; and automatically update the power professional knowledge graph according to the attribution analysis results to obtain the enhanced knowledge graph.

[0107] For example, for automatically updating the power knowledge graph, if the difficult example sample is a "symbol confusion" sample (such as confusion between the symbols of disconnector and circuit breaker), then a comparison feature node of these two symbols is added to the knowledge graph (e.g., "disconnector has no arc-extinguishing capability → symbol has no arc-extinguishing identifier"). If the difficult example sample is a "topology misjudgment" sample, then the topological constraint rules for understanding the topological relationship scenario are supplemented to the knowledge graph (e.g., "the high-voltage side of the main transformer must be connected to the busbar via a circuit breaker"). Furthermore, the updated knowledge graph can be fed back into the loss function design to strengthen the differentiation of easily confused knowledge. Then, the difficult example samples can be fused with the multimodal power dataset and the augmented knowledge graph to obtain model training samples.

[0108] The technical solution of this embodiment uses a difficult example sampling strategy to obtain difficult example samples from power plant drawing data according to the configured sampling probability. It then performs attribution analysis on the difficult example samples and updates the power professional knowledge graph based on the attribution analysis results to obtain an enhanced knowledge graph. The difficult example samples are then fused with the multimodal power dataset and the enhanced knowledge graph to obtain model training samples. This achieves knowledge graph enhancement and adaptive sampling driven by difficult example samples, which can enhance the diversity and utilization of model training samples, thereby improving the model's generalization ability.

[0109] In an exemplary embodiment, the post-training optimization strategy is used to configure the loss function employed during post-training of a general multimodal large model as follows:

[0110]

[0111] in, Cross-entropy loss for general multimodal large models; The power industry loss that penalizes model predictions that violate the rules of the power knowledge graph; The comparative loss between the feature distance of correct question-answer pairs and the feature distance of incorrect question-answer pairs in the model prediction results of a general multimodal large model; This refers to the topology consistency loss between the predicted device path and the actual device path in the model prediction results of a general multimodal large model. , , This represents the weighting coefficient of each loss term.

[0112] In practical implementation, during the post-training process of the general multimodal large model, an electrical engineering professional loss term can be introduced. Combined with cross-entropy loss and contrastive loss, a hybrid loss function can be constructed to enhance the model's learning of electrical engineering professional features and reduce professional semantic bias.

[0113] Post-training optimization strategies can be used to configure the hybrid loss function as follows:

[0114]

[0115] in, This is the cross-entropy loss of a general multimodal large model, i.e., the standard cross-entropy loss (autoregressive generation loss) of a large language model. For the power industry loss term, penalties are imposed on model prediction results that violate power knowledge graph rules (such as safety-related rules and labeling specification rules), thereby strengthening the training of multimodal large models to comply with professional specifications; To compare the loss term, the comparative loss between the feature distance of the correct question-answer pair and the feature distance of the incorrect question-answer pair in the model prediction results of the general multimodal large model is used. By narrowing the feature distance of the correct question-answer pair and widening the feature distance of the incorrect question-answer pair, the discriminative ability of the post-trained multimodal large model for electricity semantics is enhanced. To mitigate topology consistency loss, the model output conforms to power topology logic by minimizing the difference between predicted device paths (shortest paths between devices) and actual device paths (actual topology paths) in the model prediction results of the general multimodal large model, and by penalizing connectivity judgment errors. , , This represents the weighting coefficients of each loss term, such as... It serves to balance the various loss terms.

[0116] The technical solution of this embodiment introduces the cross-entropy loss, power industry loss term, contrast loss and topology consistency loss of the general multimodal large model during the post-training process, and constructs a hybrid loss function used in the post-training process. This can enhance the model's learning of power industry features and reduce the professional semantic bias of the model in the power industry scenario.

[0117] In an exemplary embodiment, after post-training and inference optimization, the target multimodal large model also needs to be validated. That is, after optimizing the model inference strategy of the quantized multimodal large model to obtain the target multimodal power large model, a post-training and inference optimization method for a multimodal large model based on power plant diagrams further includes: optimizing the model inference strategy of the general multimodal large model to obtain an inference optimized model; using the data in the multimodal power dataset that did not participate in post-training as a test set, and comparing and validating the target multimodal power large model with the general multimodal large model (basic model), the post-trained multimodal large model, and the inference optimized model based on the test set; and / or detecting the inference results output by the target multimodal power large model during the execution of the model inference task, and, if abnormal results are detected, fine-tuning the modules of the target multimodal power large model where the abnormal results are located based on the abnormal results.

[0118] For example, the validation dataset preparation can use 1156 multimodal samples of power plant maps that were not used in post-training as the test set, covering power plant maps of different voltage levels and complexities to ensure the comprehensiveness of the validation. Multi-dimensional validation metrics that may be used during collaborative validation include: accuracy metrics, such as fine-grained recognition accuracy, topology relationship judgment accuracy, and equipment parameter extraction accuracy; speed metrics, such as inference latency and frames per second (FPS); and resource metrics, such as model storage usage, GPU memory usage, and CPU utilization.

[0119] The collaborative verification process includes: (1) Comparative verification: The target multimodal power large model is compared with the general multimodal large model (basic model), the post-trained multimodal large model, and the inference optimization model in multiple dimensions to verify the superiority of the integrated scheme of "post-training-inference optimization". The judgment criteria include: If the accuracy improvement of the integrated model (i.e., the target multimodal power large model) is less than 30% or the speed improvement is less than 40% compared with the basic model, it is judged as insufficient post-training or optimization effect; If the speed improvement is less than 30% or the resource consumption reduction is less than 30% compared with the post-training model alone, it is judged as failure of the inference optimization link; If the accuracy improvement is less than 25% compared with the inference optimization model alone, it is judged as failure of the post-training link; If the accuracy difference of the integrated model across voltage levels exceeds 15%, it is judged as insufficient generalization ability. The comprehensive anomaly judgment adopts a multi-index weighted score. When the comprehensive performance is more than 15% lower than the target value, it is judged as an anomaly.

[0120] (2) Anomaly detection: Detect abnormal results (such as accuracy anomaly, speed anomaly, resource anomaly, stability anomaly) in the reasoning process of the target multimodal power large model in performing model reasoning tasks, analyze the causes and return to the corresponding module for fine-tuning.

[0121] The specific processing flow for anomaly detection is as follows: After locating the anomaly category, if the anomaly result is an accuracy anomaly, and if a certain type of error is concentrated in the accuracy anomaly (e.g., "misjudgment of disconnect switch" accounts for more than 30%), then return to the step of retraining the general multimodal large model and increase the number of difficult examples of that type; if a certain type of error is scattered in the accuracy anomaly, then check the integrity of the knowledge graph and supplement the missing rules. If the anomaly result is a speed anomaly, locate the bottleneck of the speed anomaly using performance analysis tools. If the visual encoder of the target multimodal large model is too time-consuming, then optimize the preprocessing or adopt a lighter processing method; if the inference time of the target multimodal large model is too time-consuming, then optimize the inference caching strategy or adopt speculative decoding; if the attention enhancement mechanism of the target multimodal large model is too time-consuming, then adjust the trigger threshold to reduce unnecessary secondary inference. If the anomaly result is a resource anomaly, check whether the quantization processing of the target multimodal large model is sufficient. If the quantization is insufficient, then return to the step of quantizing the retrained multimodal large model, and a more aggressive quantization strategy can be adopted. If the abnormal results indicate instability, check the sample coverage for specific voltage levels or plant types, supplement the training data, and retrain after stratification. Perform comparative verification again after retraining or optimization until no abnormalities are found.

[0122] (3) Expert review: The reasoning results of the target multimodal large model are professionally reviewed by experts in the power field to ensure that the model output complies with the power industry standards.

[0123] This application provides an integrated "post-training-inference optimization" solution for a multimodal large-scale model for power plant map understanding. It focuses on the professional adaptability, accuracy, and speed balance within the power industry context, aiming to solve the core problems of poor model adaptability, low accuracy, and slow inference in traditional technologies, thereby achieving integrated post-training and inference optimization of the multimodal large-scale model for power plant map understanding. Figure 4 The diagram shown illustrates an integrated post-training and inference optimization method for understanding multimodal large-scale power plant diagrams. The method includes the following steps S402 to S408:

[0124] Step S402, Post-training Preparation: Responsible for selecting and simplifying the general multimodal large model (basic model), screening post-training data and constructing a power professional knowledge graph to cover multiple voltage levels and multiple types of power plant diagram scenarios, thereby improving the model's generalization ability.

[0125] Step S404, Post-training for Power Industry Special Projects: Responsible for executing the stratified post-training task and optimizing the training strategy. Through the post-training strategy for power industry special projects, professional knowledge of plant and station diagrams is integrated to customize the stratified post-training task, thereby improving the model's professional semantic understanding and fine-grained recognition capabilities. The post-training process is optimized by introducing adaptive sample sampling and mixed precision training to improve training efficiency and sample utilization.

[0126] Step S406, Customized Inference Optimization: Responsible for model lightweighting, inference strategy optimization, and format adaptation. Combined with power task priorities, it aims to achieve the dual goals of "preserving accuracy and improving speed," avoiding accuracy loss caused by general optimization. It constructs a collaborative mechanism between post-training and inference optimization, ensuring that the post-training process takes into account the needs of inference optimization, and that inference optimization adapts to the characteristics of the post-trained model, forming a closed-loop optimization. It introduces a lightweight optimization strategy to simplify redundant model parameters, reduce resource consumption, and adapt to power field terminal deployment.

[0127] Step S408, Collaborative Validation: Responsible for multi-dimensional validation and anomaly fine-tuning, validating the collaborative effect of training and inference optimization after validation, and ensuring that the model meets the requirements of the power scenario.

[0128] The technical solution of this application has at least the following technical effects: 1. It solves the problem of insufficient post-training targeting: through power-specific post-training and the integration of professional knowledge, the model's accuracy in recognizing power professional symbols and topological relationships is improved by more than 35%, and the bias in understanding professional semantics is reduced by more than 60%; 2. It solves the problem of low post-training efficiency: hybrid precision training and adaptive sampling improve the training convergence speed by more than 25%, improve the sample utilization rate by 40%, and reduce training costs; 3. It solves the problem of imbalance between inference optimization and accuracy: customized inference optimization strategies improve inference speed by more than 50% without losing more than 2% of accuracy, achieving a balance between accuracy and speed; 4. It solves the problem of not forming an integrated solution: post-training and inference optimization work together to improve the overall performance (accuracy + speed) of the model by more than 45%, which is far superior to single-stage optimization; 5. It solves the problem of high resource consumption: lightweight optimization reduces model parameters by more than 40%, storage usage by 50%, and CPU utilization by 30%, making it adaptable to power field terminal deployment; 6. The optimization addresses the issue of insufficient generalization ability: post-training with samples from multiple voltage levels and scenarios improves the model's generalization ability by over 30%, making it adaptable to the needs of understanding different types of power plant diagrams; 7. Practical application value: The optimized model can be directly applied to core scenarios such as power operation and maintenance and dispatching, significantly improving the efficiency of understanding power plant diagrams, reducing manual intervention, lowering operation and maintenance costs, and promoting the large-scale deployment of multimodal large-scale models in the power field; 8. Comparative verification results: Compared with the basic model, the optimized model improves inference accuracy by 35% and inference speed by 50%; compared with the post-training model only, inference speed is improved by 45% and resource consumption is reduced by 40%; compared with the inference-only optimization model, inference accuracy is improved by 38%, fully verifying the superiority of this solution.

[0129] It should be noted that the specific limitations of the above steps can be found in the above description of the specific limitations of a multimodal large model post-training and inference optimization method based on power plant diagrams.

[0130] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0131] Based on the same inventive concept, this application also provides a device for implementing the above-mentioned multimodal large model post-training and inference optimization method based on power plant maps. The solution provided by this device is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal large model post-training and inference optimization device based on power plant maps provided below can be found in the limitations of the multimodal large model post-training and inference optimization method based on power plant maps described above, and will not be repeated here.

[0132] In one exemplary embodiment, such as Figure 5 As shown, a multimodal large model post-training and inference optimization device based on power plant diagrams is provided. The device includes:

[0133] The training data acquisition module 510 is used to acquire a multimodal power dataset constructed based on power plant drawing data, and a power professional knowledge graph constructed based on power plant drawing data.

[0134] The training task configuration module 520 is used to configure at least two power-specific post-training tasks for a general multimodal large model based on the multimodal power dataset and the power expertise graph.

[0135] The post-training strategy optimization module 530 is used to post-train the general multimodal large model according to the preset post-training optimization strategy and the model training samples that match the post-training tasks of each power special project, so as to obtain the post-trained multimodal large model.

[0136] The inference strategy optimization module 540 is used to quantize the post-trained multimodal large model to obtain the quantized multimodal large model; optimize the model inference strategy of the quantized multimodal large model to obtain the target multimodal power large model; the target multimodal power large model is used to perform model inference tasks according to the model inference strategy.

[0137] The modules in the aforementioned multimodal large-scale model post-training and inference optimization device based on power plant diagrams can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0138] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores the data required for post-training and inference optimization of a multimodal large model based on power plant diagrams. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a post-training and inference optimization method for a multimodal large model based on power plant diagrams.

[0139] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0140] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the various embodiments of the above-described multimodal large model post-training and inference optimization method based on power plant diagrams.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the various embodiments of the above-described multimodal large model post-training and inference optimization method based on power plant diagrams.

[0142] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the various embodiments of the above-described multimodal large model post-training and inference optimization method based on power plant diagrams.

[0143] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic resistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence processors, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0145] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for post-training and inference optimization of a multimodal large model based on power plant diagrams, characterized in that, The method includes: Obtain a multimodal power dataset constructed based on power plant drawing data, and a power professional knowledge graph constructed based on power plant drawing data; Based on the multimodal power dataset and the power expertise graph, at least two power-specific post-training tasks are configured for the general multimodal large model; According to the preset post-training optimization strategy and the model training samples that match each of the power special post-training tasks, the general multimodal large model is post-trained to obtain the post-trained multimodal large model. The post-trained multimodal large model is quantized to obtain a quantized multimodal large model; the model inference strategy of the quantized multimodal large model is optimized to obtain a target multimodal power large model; the target multimodal power large model is used to perform model inference tasks according to the model inference strategy.

2. The method according to claim 1, characterized in that, When the model inference task includes a drawing device recognition task, the model inference strategy includes a fine-grained region perception enhancement strategy, and executing the model inference task according to the model inference strategy includes: The attention weights of the input power plant drawing image are extracted to obtain the attention heatmap of the power plant drawing image; Based on the attention heatmap of the power plant drawing image, the key identification areas in the power plant drawing image are determined; The key identification area is magnified to obtain the magnified key identification area; A secondary inference is performed on the magnified key recognition area to obtain the inference result of the drawing device recognition task.

3. The method according to claim 1, characterized in that, When the model inference task includes a drawing device recognition task, the model inference strategy includes an image preprocessing optimization strategy, and executing the model inference task according to the model inference strategy includes: Image enhancement processing is performed on the input power plant drawing image to obtain an enhanced power plant drawing image; Key areas in the enhanced plant site drawing image are located to obtain a coarse positioning image; Based on the standard primitive template of the power plant drawing image, the coarse positioning image is matched to determine the equipment information in the power plant drawing image in order to perform the drawing equipment identification task.

4. The method according to claim 1, characterized in that, The model inference strategy includes a task priority inference and scheduling strategy. Executing the model inference task according to the model inference strategy includes: In response to receiving the inference request of the model inference task, the task priority of the model inference task is determined according to the task priority preset relationship configured by the task priority inference and scheduling strategy; the task priority preset relationship records the mapping relationship between the model inference task and the task priority. According to the model inference task, allocate computing resources that match the task priority, and execute the model inference task.

5. The method according to claim 1, characterized in that, The post-training optimization strategy includes a hard example sampling strategy, which is used to configure the sampling probability of the model training samples as follows: in, Characterizing the training samples of the model Difficult sample in the middle, Characterizing the training samples of the model Regular samples in; Training samples for the model The basic weights, The sampling amplification factor for difficult examples; Weights for difficult examples; For regular sample weights; The difficult example sampling strategy is also used to configure the step of obtaining the model training samples as follows: Based on the sampling probability, obtain the difficult sample from the power plant drawing data; Attribution analysis is performed on the difficult example samples to obtain the attribution analysis results. Based on the attribution analysis results, the power professional knowledge graph is updated to obtain an enhanced knowledge graph. The difficult example samples are fused with the multimodal power dataset and the enhanced knowledge graph to obtain the model training samples.

6. The method according to claim 1, characterized in that, The post-training optimization strategy is used to configure the loss function employed during the post-training of the general multimodal large model as follows: in, The cross-entropy loss of the general multimodal large model; The power industry loss that penalizes model predictions that violate the rules of the power knowledge graph; The comparison loss between the feature distance of correct question-answer pairs and the feature distance of incorrect question-answer pairs in the model prediction results of the general multimodal large model; The topology consistency loss between the predicted device path and the actual device path in the model prediction results of the general multimodal large model; , , This represents the weighting coefficient of each loss term.

7. A multimodal large model post-training and inference optimization device based on power plant diagrams, characterized in that, The device includes: The training data acquisition module is used to acquire a multimodal power dataset constructed based on power plant drawing data, as well as a power professional knowledge graph constructed based on power plant drawing data; The training task configuration module is used to configure at least two power-specific post-training tasks for the general multimodal large model based on the multimodal power dataset and the power professional knowledge graph. The post-training strategy optimization module is used to post-train the general multimodal large model according to the preset post-training optimization strategy and the model training samples that match each of the power special post-training tasks, so as to obtain the post-trained multimodal large model. The inference strategy optimization module is used to quantize the post-trained multimodal large model to obtain a quantized multimodal large model; optimize the model inference strategy of the quantized multimodal large model to obtain a target multimodal power large model; the target multimodal power large model is used to perform model inference tasks according to the model inference strategy.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images