Fine-tuning method for multi-modal large model in power production domain

Abstract

The application discloses a multi-modal large model fine-tuning method for the power production domain, and relates to the technical field of power artificial intelligence. Including a multi-modal large model based on the current situation of the power production domain, a feedback data set is extracted; the feedback data set is mapped to a unified semantic dimension and fused, the fusion result is stored in a structured manner, and a multi-modal unified representation is obtained; the multi-modal unified representation is subjected to four-dimensional cascade screening to obtain a target sample set, wherein the four-dimensional cascade screening includes difference degree screening, logical directionality screening, consistent coherence screening and knowledge constraint screening. First, the multi-source heterogeneous feedback data is mapped to a unified semantic space, solving the problem that images, texts and time series data are difficult to align and compare. Secondly, a four-dimensional cascade screening framework is constructed: the difference degree screening is evaluated by semantic distance and information gain, redundant data is removed and diversity samples are supplemented.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology in the power industry, specifically to a method for fine-tuning a multimodal large model for the power production domain. Background Technology

[0002] With the widespread application of multimodal large models in the power production field, continuous fine-tuning of models using actual production feedback data has become an important means to improve model performance.

[0003] However, existing fine-tuning methods still have many shortcomings: the feedback data is massive in scale but of inconsistent quality, and a large amount of redundant data increases computational costs; there is a lack of effective evaluation of the logical orientation of the data, making it difficult for the model to learn reasoning abilities that conform to the causal laws in the power industry; cross-modal contradictions and inconsistencies are prone to occur among multimodal data, and traditional methods lack a systematic verification mechanism; at the same time, power production data also need to meet industry knowledge constraints such as safety regulations and equipment physical limits, and existing methods have not established an effective constraint verification system.

[0004] Therefore, this application proposes a multimodal large model fine-tuning method that can efficiently screen high-quality, high-value, compliant and secure training samples from feedback data. Summary of the Invention

[0005] The purpose of this invention is to provide a method for fine-tuning a multimodal large model for the power production domain, in order to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal large model fine-tuning method for the power production domain, comprising: Based on the current status of the power production domain, a multimodal large model is used to extract the feedback dataset; The feedback dataset is mapped to a unified semantic dimension and fused, and the fusion result is stored in a structured manner to obtain a unified multimodal representation; The multimodal unified representation is subjected to four-dimensional cascade screening to obtain a target sample set, wherein the four-dimensional cascade screening includes difference screening, logical orientation screening, consistency and coherence screening and knowledge constraint screening. Based on the target sample set, fine-tuning training data is established for the current multimodal large model to fine-tune the model, and version management of the fine-tuned multimodal large model is carried out in conjunction with model evaluation.

[0007] As a specific solution to the technical solution of this application, the extraction of feedback dataset based on the current multimodal large model of the power production domain includes: Based on the operation logs and user interaction feedback of the existing multimodal large model, extract the multimodal feedback dataset; The multimodal feedback dataset contains data samples of at least two modalities from image modal data, text modal data, and structured time-series modal data.

[0008] As a specific solution of the technical solution of this application, the step of mapping the feedback dataset to a unified semantic dimension and fusing it, storing the fusion result in a structured manner, and obtaining a multimodal unified representation includes: Each modality in each data sample of the feedback dataset is independently encoded and projected and aligned to a unified semantic dimension. For each data sample, cross-modal semantic fusion based on dimension splicing is performed in a unified semantic dimension to obtain the corresponding multimodal unified semantic representation, which is then stored in a structured form.

[0009] As a specific solution of this application, a four-dimensional cascaded screening is performed on the multimodal unified representation to obtain a target sample set, wherein the difference screening includes: Calculate the semantic distance between the multimodal unified semantic representation of each data sample and the semantic distance between the representations of each known sample in the current training set; Based on the dimensionality integrity of each modality dimension in the multimodal unified semantic representation of each data sample, a penalty factor is calculated, wherein the penalty factor is positively correlated with the dimensionality integrity and the penalty factor is a positive real number less than 1. The product of the semantic distance and the penalty factor is taken as the effective difference. Based on the dynamic difference threshold and the effective difference, each data sample is initially screened, and data samples with an effective difference not lower than the dynamic difference threshold are retained. The dynamic difference threshold is dynamically adjusted with each iteration round and is positively correlated with the iteration round. The selection probability of each sample is determined based on the coverage of each difference level. The selection probability is then used to select samples from the initial screening results to obtain the difference screening results.

[0010] As a specific solution of the technical solution of this application, a four-dimensional cascaded screening is performed on the multimodal unified representation to obtain a target sample set, wherein the logical orientation screening further includes: Based on the pre-constructed power domain logical graph, entity relationships are extracted and logical paths are matched based on the difference screening results; Based on the logical path matching results and the power domain logic graph, the causal chain integrity, reasoning path clarity, and multimodal logic anchoring degree of each data sample are evaluated respectively. The weighted fusion of the completeness of the causal chain, the clarity of the reasoning path, and the multimodal logical anchoring degree results in a comprehensive score for logical orientation. Data samples with a comprehensive logical orientation score not lower than the logical orientation threshold are selected to obtain logical orientation selection results. The logical orientation threshold is determined based on statistical analysis of the comprehensive logical orientation score set of historical logical negative samples.

[0011] As a specific solution of this application, a four-dimensional cascaded screening is performed on the multimodal unified representation to obtain a target sample set, wherein the consistency and coherence screening includes: The logical orientation screening results are subjected to cross-modal semantic alignment verification, cross-modal numerical cross-validation, and temporal causal order verification. A consistency score is obtained by weighting the results, and the consistency verification results are selected and determined. Perform contextual coherence verification on the logical orientation filtering results, and remove incoherent samples to obtain the contextual coherence verification results. Perform multi-round dialogue coherence verification on the logical orientation screening results, and remove incoherent samples to obtain the multi-round dialogue coherence verification results. The consistency and coherence filtering results are obtained by taking the intersection of the consistency verification results, the context coherence verification results, and the multi-turn dialogue coherence verification results.

[0012] As a specific solution of the technical solution of this application, a four-dimensional cascaded screening is performed on the multimodal unified representation to obtain a target sample set, wherein the knowledge-constrained screening includes: The current knowledge constraint base and the updated knowledge constraint base of the power production domain are obtained, and differential analysis is performed to obtain a differential knowledge constraint base, wherein the differential knowledge constraint base includes newly added constraint rules and modified constraint rules. Based on the differential knowledge constraint base, the consistency and coherence screening results are respectively checked for compliance with safety procedures, compliance with equipment physical limits, and compliance with industry standards. The target sample set is obtained by filtering out data samples that fail any of the aforementioned compliance checks.

[0013] As a specific solution to the technical solution of this application, it also includes: Based on the differential knowledge constraint library, lightweight knowledge constraint rules are extracted, wherein the lightweight knowledge constraint rules can be expressed in a structured manner and include at least numerical range rules, terminology standardization rules, and operation sequence rules. Based on the lightweight knowledge constraint rules, each data sample in the feedback dataset is pre-screened using lightweight knowledge constraints to obtain the pre-screening results; The difference screening results are updated based on the intersection of the pre-screening results and the difference screening results.

[0014] As a specific solution to the technical solution of this application, it also includes: Traverse the feedback dataset and identify the fault types in the current training set whose sample count is lower than the preset minimum retention rate, as long-tail categories. A difference screening exemption is set for data samples of the long-tail category, so that they are not restricted by the difference threshold in the difference screening and can directly enter the subsequent screening stage; In the logical orientation filtering, the logical orientation threshold is lowered for data samples of the long-tail category; The data samples selected from the long-tail categories are used as long-tail protection data and combined with the target sample set to construct the fine-tuning training data.

[0015] A multimodal large-scale model fine-tuning system for the power production domain includes: The data extraction module is used to extract feedback datasets based on the current multimodal large model of the power production domain. The unified representation module is used to map the feedback dataset to a unified semantic dimension and fuse them, store the fusion result in a structured manner, and obtain a multimodal unified representation. The cascaded filtering module is used to perform four-dimensional cascaded filtering on the multimodal unified representation to obtain the target sample set. The four-dimensional cascaded filtering includes difference filtering, logical orientation filtering, consistency and coherence filtering, and knowledge constraint filtering. The fine-tuning management module is used to establish fine-tuning training data for the current multimodal large model based on the target sample set, fine-tune the model, and manage the version of the fine-tuned multimodal large model in conjunction with model evaluation.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This invention first maps multi-source heterogeneous feedback data to a unified semantic space, solving the problem of difficulty in aligning and comparing image, text, and time-series data. Second, it constructs a four-dimensional cascaded screening framework: Dissimilarity screening evaluates semantic distance and information gain to eliminate redundant data and supplement diverse samples; logical orientation screening, based on the power domain logic graph, assesses the integrity of causal chains to ensure clear reasoning direction; consistency and coherence screening verifies the semantic alignment of images and text and the causal order of time series across modalities, eliminating contradictory samples; and knowledge constraint screening, combined with a differential constraint library, checks for compliance with safety procedures, filtering out non-compliant data. The four-dimensional screening is cascaded, with dissimilarity serving as a coarse filter and the subsequent three dimensions as a fine filter, balancing efficiency and accuracy. Finally, the selected high-quality target sample set is used for incremental model fine-tuning, and version management is performed based on the evaluation results. This invention achieves efficient extraction of high-value training samples from massive feedback data, improving the quality of fine-tuning data and the efficiency of model iteration. Attached Figure Description

[0017] Figure 1This is a schematic diagram of the multimodal large model fine-tuning system structure for the power production domain of the present invention; Figure 2 This is a schematic diagram of the module interaction of the multimodal large model fine-tuning system of the present invention; Figure 3 This is a flowchart illustrating the multimodal large model fine-tuning method of the present invention; Figure 4 This is a flowchart illustrating another multimodal large model fine-tuning method of the present invention; Figure 5 This is a flowchart illustrating another multimodal large model fine-tuning method of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," etc., in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. For example, the first cluster and the second cluster mentioned below belong to different clusters. It should be understood that such names can be used interchangeably where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules in the embodiments of this application is merely a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not performed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, and the indirect coupling or communication connection between modules may be electrical or other similar forms. None of these are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separate, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0020] To improve the quality and efficiency of training data selection during the fine-tuning of multimodal large-scale models for the power production domain, and to enhance the adaptability to power domain expertise and long-tail fault scenarios, such as... Figure 1 As shown in the figure, this application provides a multimodal large model fine-tuning system for the power production domain. The fine-tuning system includes a data extraction module, a unified representation module, a cascaded filtering module, and a fine-tuning management module. Optionally, the fine-tuning system also includes a knowledge constraint base management module.

[0021] The aforementioned data extraction module is used to extract feedback datasets based on the current multimodal large model of the power production domain; the aforementioned unified representation module is used to map the feedback datasets to a unified semantic dimension and fuse them, and to store the fusion results in a structured manner to obtain a unified multimodal representation; the aforementioned cascade filtering module is used to perform four-dimensional cascade filtering on the unified multimodal representation to obtain a target sample set, wherein the four-dimensional cascade filtering includes difference filtering, logical orientation filtering, consistency and coherence filtering, and knowledge constraint filtering; the aforementioned fine-tuning management module is used to establish fine-tuning training data for the current multimodal large model based on the target sample set, to fine-tune the model, and to manage the version of the fine-tuned multimodal large model in conjunction with model evaluation.

[0022] The aforementioned data extraction module, unified representation module, cascade filtering module, and fine-tuning management module can be functional units deployed within the same computing node, or they can be distributed across multiple computing nodes and interact through network interfaces. This application does not impose specific limitations on these aspects.

[0023] Figure 2 This is a schematic diagram of the module interaction of a multimodal large model fine-tuning system provided in an embodiment of this application. Figure 2 As shown, the data extraction module interacts with the current multimodal large model to obtain operation logs and user interaction feedback; the unified representation module is connected to the data extraction module, receives the feedback dataset, and performs cross-modal semantic fusion; the cascade filtering module is connected to the unified representation module, receives the multimodal unified representation, and performs four-dimensional cascade filtering; the cascade filtering module is also connected to the knowledge constraint base management module to obtain knowledge constraint rules for the power production domain; the fine-tuning management module is connected to the cascade filtering module, receives the target sample set, and performs model fine-tuning and version management.

[0024] It should be noted that the word "module" can be omitted when describing the above modules. For example, the data extraction module can be simply referred to as the data extraction unit, the unified representation module can be simply referred to as the unified representation unit, and so on.

[0025] To facilitate understanding of the embodiments of this application, some nouns or terms involved in the embodiments of this application will be explained below.

[0026] Current multimodal large models: refers to large language models or multimodal basic models that have been deployed in the power production domain and have the capabilities of image understanding, text generation and structured data analysis.

[0027] Feedback dataset: A collection of data consisting of operation logs and user interaction feedback generated during the actual operation of the current multimodal large model, including data samples of at least two of the following modalities: image modal data, text modal data, and structured time-series modal data.

[0028] Unified semantic dimension: refers to the semantic representation space formed by mapping data of different modalities to the same vector space through independent encoding and projection alignment.

[0029] Four-dimensional cascaded screening refers to a sample screening mechanism that proceeds step by step according to four dimensions: difference screening, logical orientation screening, consistency and coherence screening, and knowledge constraint screening.

[0030] Power domain logic graph: refers to a pre-constructed knowledge graph used to describe power production equipment, fault phenomena, causal relationships and reasoning paths.

[0031] Long-tail categories: These refer to fault types or equipment anomaly types where the number of samples in the feedback dataset is lower than the preset minimum.

[0032] The following description, in conjunction with the accompanying drawings, introduces the multimodal large-scale model fine-tuning method for the power production domain provided in this application. It is understood that in the embodiments of this application, the interactive execution entities are the data extraction module, the unified representation module, the cascading filtering module, the fine-tuning management module, and the knowledge constraint base management module. For the meaning of the above modules, please refer to the preceding description.

[0033] Option 1: In the first scheme, the data extraction module is used to extract feedback datasets based on the current multimodal large model of the power production domain; the unified representation module is used to map the feedback datasets to a unified semantic dimension and fuse them, and store the fusion results in a structured manner to obtain a unified multimodal representation; the cascade filtering module is used to perform four-dimensional cascade filtering on the unified multimodal representation to obtain a target sample set; and the fine-tuning management module is used to establish fine-tuning training data for the current multimodal large model based on the target sample set, fine-tune the model, and manage the version of the fine-tuned multimodal large model in conjunction with model evaluation.

[0034] As one possible implementation method, the cascading filtering module is specifically used to sequentially perform difference filtering, logical orientation filtering, consistency filtering, and knowledge constraint filtering.

[0035] As one possible implementation method, the data extraction module is specifically used to extract a multimodal feedback dataset based on the operation logs and user interaction feedback of the current multimodal large model; the multimodal feedback dataset contains data samples of at least two modalities among image modal data, text modal data and structured time-series modal data.

[0036] As one possible implementation method, the unified representation module is specifically used to independently encode and project and align multiple modalities in each data sample in the feedback dataset, mapping them to a unified semantic dimension; for each data sample, cross-modal semantic fusion based on dimension concatenation is performed in the unified semantic dimension to obtain the corresponding multimodal unified semantic representation, and stored in a structured form.

[0037] Figure 3 This is a flowchart illustrating a multimodal large model fine-tuning method provided in an embodiment of this application. The data extraction module, unified representation module, cascaded filtering module, and fine-tuning management module in this embodiment can be... Figure 1 The corresponding module in [the document / system]. The method includes the following steps.

[0038] Step 101: The data extraction module extracts the feedback dataset based on the current multimodal large model of the power production domain.

[0039] Specifically, the data extraction module extracts a multimodal feedback dataset from the operational logs and user interaction feedback of the current multimodal large model. The operational logs include input / output records, anomaly alarm records, and performance monitoring records during the model inference process; the user interaction feedback includes instructions, correction suggestions, and annotation results entered by maintenance personnel.

[0040] The multimodal feedback dataset contains data samples of at least two of the following modalities: image modal data, text modal data, and structured time-series modal data. For example, a data sample may include: an infrared thermal imaging image of the device (image modal), fault description text entered by maintenance personnel (text modal), and time-series data of current, voltage, and temperature collected by the device in the past hour (structured time-series modal).

[0041] Based on the above approach, by extracting feedback datasets from the actual operation logs and user interaction feedback of the existing multimodal large model, it can be ensured that the data used for subsequent fine-tuning comes from real business scenarios, which helps to improve the adaptability of the fine-tuned model to actual problems in the power production domain.

[0042] Step 102: The unified representation module maps the feedback dataset to a unified semantic dimension and fuses them, stores the fusion result in a structured manner, and obtains a multimodal unified representation.

[0043] Specifically, the unified representation module independently encodes and projects multiple modalities in each data sample of the feedback dataset, mapping them to a unified semantic dimension. For example, for image modal data, a visual encoder (such as Vision Transformer) is used to encode the image feature vector; for text modal data, a text encoder (such as BERT or Transformer) is used to encode the text feature vector; and for structured temporal modal data, a temporal encoder (such as LSTM or Temporal Convolutional Network) is used to encode the temporal feature vector.

[0044] Furthermore, the unified representation module projects and aligns the feature vectors of each modality, mapping the features of each modality to a unified semantic space of the same dimension through a learnable projection matrix. Subsequently, cross-modal semantic fusion based on dimension concatenation is performed in the unified semantic dimension, concatenating the projected features of each modality of the same data sample according to the dimension to obtain a multimodal unified semantic representation corresponding to the data sample.

[0045] For example, if the image modal projection feature dimension is d1, the text modal projection feature dimension is d2, and the temporal modal projection feature dimension is d3, then the concatenated multimodal unified semantic representation dimension is D = d1 + d2 + d3. This multimodal unified semantic representation is stored in a structured form (such as key-value pairs or tensors) in the storage unit of the unified representation module.

[0046] Based on the above scheme, by mapping multimodal data to a unified semantic dimension and performing cross-modal semantic fusion, the semantic gap between different modalities can be eliminated, enabling the subsequent screening process to be measured and compared in a unified vector space, which helps to improve the accuracy and efficiency of screening.

[0047] Step 103: The cascaded screening module performs difference screening on the multimodal unified representation.

[0048] Specifically, the cascaded filtering module calculates the semantic distance between the multimodal unified semantic representation of each data sample and the representation of each known sample in the current training set. For example, this semantic distance can be calculated using Euclidean distance, cosine distance, or Mahalanobis distance.

[0049] Furthermore, the cascaded filtering module calculates a penalty factor based on the dimensionality completeness of each modality dimension in the multimodal unified semantic representation of each data sample. The penalty factor is positively correlated with dimensionality completeness and is a positive real number less than 1. For example, if a data sample contains all three modalities, the dimensionality completeness is highest, and the penalty factor is 1; if it contains only two modalities, the dimensionality completeness decreases, and the penalty factor is 0.9; if it contains only one modality, the penalty factor is 0.8.

[0050] Furthermore, the cascaded filtering module uses the product of semantic distance and penalty factor as the effective difference score. For example, if the semantic distance between a data sample and the nearest known sample in the current training set is 0.5, and the penalty factor of the sample is 0.9, then its effective difference score is 0.45.

[0051] Furthermore, the cascaded screening module performs initial screening of each data sample based on a dynamic difference threshold and an effective difference, retaining data samples whose effective difference is not lower than the dynamic difference threshold. The dynamic difference threshold is dynamically adjusted with each iteration and is positively correlated with the iteration number. For example, in the t-th iteration, the dynamic difference threshold... ,in The initial threshold is denoted by α, which is a positive stepping coefficient. Based on this method, the dynamic variability threshold gradually increases with the number of iterations, which helps to retain more samples in the early rounds to enrich diversity, and to increase the screening rigor in later rounds to ensure sample quality.

[0052] Furthermore, the cascaded screening module determines the selection probability of each sample based on the coverage of each difference level, and combines the selection probability with the probability selection from the initial sample screening results to obtain the difference screening results. For example, the effective difference is divided into three levels: high difference, medium difference, and low difference. If the coverage of high difference samples in the current training set is insufficient, the selection probability of high difference level samples is increased to ensure uniform coverage of the training set in the difference space.

[0053] Based on the above scheme, by introducing a penalty factor to reduce the dissimilarity of samples with missing modalities, and by adopting a dynamic dissimilarity threshold and a probability selection mechanism based on coverage, it is possible to ensure the novelty of samples while taking into account modal integrity and dissimilarity space coverage, which helps to improve the diversity and representativeness of fine-tuning training data.

[0054] Step 104: The cascading filtering module performs logically directed filtering on the difference filtering results.

[0055] Specifically, the cascaded filtering module, based on a pre-constructed power domain logic graph, extracts entity relationships and matches logical paths from the difference filtering results. For example, the power domain logic graph contains entity relationship chains such as "transformer—abnormal oil temperature—insulation aging—replacement of insulating oil". The cascaded filtering module extracts entities such as "transformer" and "abnormal oil temperature" from the text modality of the data samples and identifies their mapping relationships with entities in the graph.

[0056] Furthermore, the cascaded screening module evaluates the causal chain integrity, inference path clarity, and multimodal logical anchoring degree of each data sample based on the logical path matching results and the power domain logic graph. Causal chain integrity measures the completeness of the causal relationship between the fault cause and the fault phenomenon in the sample; inference path clarity measures the clarity of the inference path from the fault phenomenon to the derivation of the treatment measures; and multimodal logical anchoring degree measures the tightness of logical consistency among image, text, and time-series data.

[0057] Furthermore, the cascading filtering module weights and integrates causal chain completeness, reasoning path clarity, and multimodal logical anchoring degree into a comprehensive logical orientation score. For example, the comprehensive logical orientation score... , where w1, w2, and w3 are weighting coefficients, and w1 + w2 + w3 = 1.

[0058] Furthermore, the cascading filtering module selects data samples whose overall logical orientation score is not lower than the logical orientation threshold, and obtains the logical orientation filtering results. The logical orientation threshold is determined based on statistical analysis of the overall logical orientation score set of historical logically negative samples. For example, a batch of negative samples already marked as logical errors is collected, their overall logical orientation scores are calculated, and the upper quartile of the score distribution is taken as the logical orientation threshold.

[0059] Based on the above scheme, by introducing a power domain logic graph to perform logical path matching and multi-dimensional logical scoring on the samples, low-quality samples with logical confusion, causal breaks, or logical contradictions between modes can be effectively filtered out, which helps to improve the logical reasoning ability and power professional knowledge compliance of the fine-tuned model.

[0060] Step 105: The cascading filtering module performs consistent filtering on the logically oriented filtering results.

[0061] Specifically, the cascading filtering module performs cross-modal semantic alignment verification, cross-modal numerical cross-validation, and temporal causal order verification on the logical orientation filtering results, obtains a weighted consistency score, and filters to determine the consistency verification results.

[0062] Cross-modal semantic alignment verification is used to verify whether the semantic content of image modalities, text modalities, and time-series modalities in the same data sample points to the same fault or the same equipment state. For example, if the text description is "transformer A phase bushing discharge", while the image modal shows the transformer B phase part, then the cross-modal semantic alignment verification will fail.

[0063] Cross-modal numerical cross-validation is used to verify whether the numerical parameters mentioned in the text modality are consistent with the measured values ​​recorded in the time-series modality. For example, if the text description is "the current oil temperature is 85℃", but the time-series data shows that the oil temperature at that moment is 65℃, then the cross-modal numerical cross-validation will fail.

[0064] The temporal causality sequence verification is used to verify whether the time sequence of events in temporal modal data conforms to a causal relationship. For example, if a sample claims that "overload caused the winding temperature to rise", but the timing data shows that the temperature rise occurred before the overload occurred, then the temporal causality sequence verification will fail.

[0065] Furthermore, the cascading filtering module performs contextual coherence verification on the logically oriented filtering results and removes incoherent samples to obtain the contextual coherence verification results. For example, for samples containing multi-paragraph text, it verifies the thematic continuity and logical connection between adjacent paragraphs.

[0066] Furthermore, the cascading filtering module performs multi-round dialogue coherence verification on the logically oriented filtering results, and removes incoherent samples to obtain multi-round dialogue coherence verification results. For example, if the samples come from multi-round interaction records between the user and the model, the consistency of dialogue intent and the coherence of information transmission between adjacent rounds are verified.

[0067] Furthermore, the cascading filtering module takes the intersection of the consistency verification results, contextual coherence verification results, and multi-turn dialogue coherence verification results to obtain the consistency and coherence filtering result. That is, data samples are only retained if they pass all three verifications simultaneously.

[0068] Based on the above scheme, through a multi-verification mechanism including cross-modal semantic alignment, numerical cross-validation, temporal causal verification, and context and dialogue coherence verification, problematic samples with intermodal contradictions, data fraud, or context breaks can be effectively identified and eliminated, which helps to improve the reliability and consistency of fine-tuning training data.

[0069] Step 106: The cascading filtering module performs knowledge-constrained filtering on the consistency and coherence filtering results.

[0070] Specifically, the knowledge constraint base management module acquires the current and updated knowledge constraint bases of the power production domain and performs differential analysis to obtain a differential knowledge constraint base. This differential knowledge constraint base includes newly added and modified constraint rules. For example, if the current knowledge constraint base contains the rule "the top oil temperature of the transformer should not exceed 85℃," and the updated knowledge constraint base modifies this rule to "the top oil temperature of the transformer should not exceed 95℃," then the modified constraint rule is recorded in the differential knowledge constraint base.

[0071] Furthermore, the cascaded screening module, based on the differential knowledge constraint base, performs safety procedure compliance checks, equipment physical limit compliance checks, and industry standard compliance checks on the consistency and coherence screening results.

[0072] The safety procedure compliance check verifies whether the operational steps involved in the sample comply with electrical safety operating procedures. For example, if the sample recommends "replacing a high-voltage fuse while the circuit is energized" without mentioning necessary insulation protection measures, the safety procedure compliance check will fail.

[0073] The equipment physical limits compliance check is used to verify whether the equipment parameters involved in the sample exceed physical limits. For example, if the sample describes "increasing the rated current of this 10kV cable to 2000A", but the rated current limit of this type of cable is 800A, then the equipment physical limits compliance check will fail.

[0074] Industry standard compliance checks are used to verify whether the processes and technical parameters involved in the sample comply with current industry standards. For example, if the sample recommends using test methods from an outdated industry standard, the industry standard compliance check will fail.

[0075] Furthermore, the cascading screening module filters out data samples that fail any compliance test to obtain the target sample collection.

[0076] Based on the above scheme, by using a differential knowledge constraint library to perform triple compliance checks on the samples in terms of safety procedures, physical limits, and industry standards, it can be ensured that the fine-tuned training data meets the latest knowledge constraints and regulatory requirements of the power production domain, which helps to avoid the model outputting illegal, dangerous, or outdated suggestions after fine-tuning.

[0077] Step 107: The fine-tuning management module establishes fine-tuning training data for the current multimodal large model based on the target sample set, and performs model fine-tuning.

[0078] Specifically, the fine-tuning management module divides the target sample set into training, validation, and test sets according to a preset ratio, for example, 8:1:1. Based on the training set, the fine-tuning management module performs supervised fine-tuning (SFT) on the existing multimodal large model, updating model parameters using methods such as low-rank adaptation (LoRA) or full-parameter fine-tuning.

[0079] As an implementation method, the fine-tuning management module can adopt a hybrid precision training strategy to improve training efficiency and a gradient accumulation mechanism to adapt to large-scale training during the fine-tuning process.

[0080] Based on the above approach, fine-tuning the model using a high-quality target sample set obtained through thought cascade filtering can improve fine-tuning efficiency while ensuring data quality, and help obtain a multimodal large model of the power production domain with better performance.

[0081] Step 108: The fine-tuning management module combines model evaluation to manage the version of the fine-tuned multimodal large model.

[0082] Specifically, the fine-tuning management module evaluates the fine-tuned multimodal large model using validation and test sets. Evaluation metrics include power domain task accuracy, logical consistency score, and illusion rate. Based on the evaluation results, the fine-tuning management module archives versions of models that pass the evaluation, recording the model version number, training data version, hyperparameter configuration, and evaluation metrics; and rolls back or discards models that fail the evaluation.

[0083] As an implementation method, the fine-tuning management module can also maintain the model version lineage, record the inheritance relationship and difference data between different versions of the model, so as to support subsequent model iteration and source tracing analysis.

[0084] Based on the above scheme, by combining model evaluation with version management, it is possible to ensure that the quality of the online model is traceable and rollbackable, which helps to improve the operation and maintenance reliability and iteration efficiency of the multimodal large model in the power production domain.

[0085] The second option: In the second scheme, the cascaded filtering module is also used to traverse the feedback dataset, identify fault types whose sample count in the current training set is lower than the preset minimum retention amount, and classify them as long-tail categories; set a difference filtering exemption for data samples of long-tail categories, so that they are not restricted by the difference threshold in the difference filtering and can directly enter the subsequent filtering stage; in the logical orientation filtering, the logical orientation threshold of data samples of long-tail categories is reduced; the data samples of long-tail categories that pass the filtering are used as long-tail protection data, and merged with the target sample set to construct fine-tuning training data.

[0086] Figure 4 This is a flowchart illustrating another multimodal large model fine-tuning method provided in an embodiment of this application. The modules in this embodiment can be... Figure 1 The corresponding module in [the document / system]. The method includes the following steps.

[0087] Step 201: The data extraction module extracts the feedback dataset based on the current multimodal model of the power production domain. For specific implementation details of this step, please refer to the aforementioned... Figure 3 The description of step 101 in the method embodiment will not be repeated here.

[0088] Step 202: The unified representation module maps the feedback dataset to a unified semantic dimension and fuses them to obtain a multimodal unified representation. For specific implementation details of this step, please refer to the preceding... Figure 3 The description of step 102 in the method embodiment will not be repeated here.

[0089] Step 203: The cascaded filtering module traverses the feedback dataset and identifies long-tail categories.

[0090] Specifically, the cascading filtering module counts the number of samples for each fault type or equipment anomaly type in the feedback dataset and compares it with the number of samples of the corresponding type in the current training set. If the number of samples for a certain fault type in the current training set is lower than the preset minimum number (e.g., 100), then the fault type is marked as a long-tail category.

[0091] As one implementation method, the cascaded filtering module can also weight the fault types according to their business importance. For example, fault types that involve core main equipment (such as main transformers and GIS switchgear) but have a small sample size are preferentially labeled as long-tail categories.

[0092] Based on the above scheme, by identifying long-tail categories, we can accurately locate the weak links in the training data where samples are scarce, providing a basis for subsequent differentiated screening strategies and helping to improve the ability of the fine-tuned model to identify and handle rare faults.

[0093] Step 204: The cascaded filtering module sets a difference filtering exemption for data samples of the long-tail category.

[0094] Specifically, in the difference screening stage, the cascading screening module exempts data samples belonging to the long-tail category from the difference threshold restriction. That is, regardless of whether the effective difference of this type of sample reaches the dynamic difference threshold, it directly enters the logically directed screening stage.

[0095] As one implementation method, the cascaded screening module can also modify the penalty factor for long-tail category samples. For example, the penalty factor for long-tail category samples can be uniformly set to 1 to eliminate the adverse effect of modality loss on its difference assessment.

[0096] Based on the above scheme, by setting a difference screening exemption for long-tail category samples, it is possible to avoid being mistakenly filtered out due to excessive difference caused by sample scarcity, which helps to retain valuable long-tail fault samples.

[0097] Step 205: The cascaded filtering module performs logical orientation filtering on the difference filtering results (including exempted long-tail samples) and lowers the logical orientation threshold for data samples of the long-tail category.

[0098] Specifically, for non-long-tail category samples, the logical directivity threshold remains unchanged; for long-tail category samples, the logical directivity threshold is reduced to β times the original value, where β is a positive number less than 1 (e.g., β = 0.8). Based on this method, the logical scoring requirements for long-tail category samples can be appropriately relaxed to improve their pass rate.

[0099] Based on the above scheme, by lowering the logical orientation threshold of long-tail category samples, the screening criteria can be appropriately relaxed while ensuring the basic logical rationality, which helps to increase the selection ratio of long-tail category samples.

[0100] Step 206: The cascading filtering module performs consistent filtering on the logically oriented filtering results. For specific implementation details of this step, please refer to the aforementioned... Figure 3 The description of step 105 in the method embodiment will not be repeated here.

[0101] Step 207: The cascading filtering module performs knowledge-constrained filtering on the consistency and coherence filtering results to obtain the target sample set. For specific implementation details of this step, please refer to the aforementioned... Figure 3 The description of step 106 in the method embodiment is omitted here.

[0102] Step 208: The fine-tuning management module will use the data samples of the selected long-tail categories as long-tail protection data, merge them with the target sample set to construct fine-tuning training data, and perform model fine-tuning and version management.

[0103] Specifically, the fine-tuning management module merges the long-tail protection data with the target sample set to form the final fine-tuned training data. During the merging process, the long-tail protection data can be oversampled or weighted to further enhance its contribution to the training process. For example, in the loss function calculation, a higher sample weight is assigned to the long-tail protection data.

[0104] Based on the above scheme, by combining long-tail protection data with target sample sets to construct fine-tuning training data, the long-tail distribution problem of fault data in the power production domain can be effectively alleviated, which helps to improve the accuracy and generalization ability of the fine-tuned model in identifying rare fault types.

[0105] The third option: In the third approach, the cascaded filtering module is also used to extract lightweight knowledge constraint rules based on the differential knowledge constraint base. These lightweight knowledge constraint rules can be expressed in a structured way and include at least numerical range rules, terminology standardization rules, and operation sequence rules. Based on the lightweight knowledge constraint rules, the module performs lightweight knowledge constraint pre-screening on each data sample in the feedback dataset to obtain the pre-screening results. Based on the intersection of the pre-screening results and the difference degree filtering results, the module updates the difference degree filtering results.

[0106] Figure 5 This is a flowchart illustrating another multimodal large model fine-tuning method provided in this application embodiment. The modules in this embodiment can be... Figure 1 The corresponding module in [the document / system]. The method includes the following steps.

[0107] Step 301: The data extraction module extracts the feedback dataset based on the current multimodal model of the power production domain. For specific implementation details of this step, please refer to the aforementioned... Figure 3 The description of step 101 in the method embodiment will not be repeated here.

[0108] Step 302: The unified representation module maps the feedback dataset to a unified semantic dimension and fuses them to obtain a multimodal unified representation. For specific implementation details of this step, please refer to the preceding... Figure 3 The description of step 102 in the method embodiment will not be repeated here.

[0109] Step 303: The cascaded filtering module extracts lightweight knowledge constraint rules based on the differential knowledge constraint library.

[0110] Specifically, the cascaded filtering module selects rules that can be expressed in a structured way from the differential knowledge constraint base, which are then used as lightweight knowledge constraint rules. These lightweight knowledge constraint rules include at least numerical range rules, terminology standardization rules, and operation sequence rules.

[0111] Among them, the numerical range rule is used to limit the reasonable range of values ​​for equipment parameters and environmental indicators. For example, "the transformer load rate should be between 0% and 120%".

[0112] Terminology guidelines are used to define the standard expressions of technical terms. For example, "SF6 circuit breaker" is a standard term, "sulfur hexafluoride switch" is an acceptable synonym, and "sulfur fluoride circuit breaker" is an incorrect expression.

[0113] Operation sequence rules are used to define the order and necessary conditions of operation steps. For example, "When performing a power outage operation, the circuit breaker must be disconnected first, and then the disconnecting switch must be opened."

[0114] Based on the above approach, by extracting lightweight knowledge constraint rules from the differential knowledge constraint base, complex knowledge constraints can be transformed into structured check items that can be executed quickly, which helps to improve screening efficiency.

[0115] Step 304: The cascaded filtering module performs lightweight knowledge constraint pre-screening on each data sample in the feedback dataset according to the lightweight knowledge constraint rules, and obtains the pre-screening results.

[0116] Specifically, before performing difference-based filtering, the cascading filtering module first uses lightweight knowledge constraint rules to quickly filter each data sample in the feedback dataset. For example, for text modalities, regular expressions are used to match numerical range rules and terminology specification rules; for structured temporal modalities, threshold judgment is used to verify numerical range rules; and for samples containing operation descriptions, sequence matching is used to verify operation sequence rules.

[0117] If a data sample fails any of the lightweight knowledge constraint rules, it is directly removed; if it passes all the checks, it is included in the pre-screening results.

[0118] Based on the above scheme, by performing lightweight knowledge constraint pre-screening before difference screening, structured rules can be used to quickly remove obviously non-compliant samples, reducing the processing load of subsequent high-computational-cost screening steps and helping to improve the overall screening efficiency.

[0119] Step 305: The cascading filtering module updates the difference filtering results based on the intersection of the pre-screening results and the difference filtering results.

[0120] Specifically, the cascade filtering module performs difference filtering only on the data samples in the pre-screening results. That is, the input data for difference filtering is reduced from the entire feedback dataset to the pre-screening results. After the difference filtering is completed, the cascade filtering module takes the intersection of the difference filtering results and the pre-screening results (in fact, the pre-screening results have been used as the input for difference filtering, and the intersection here is the difference filtering result), and updates the difference filtering results.

[0121] As an implementation method, the cascaded screening module can also assign priority labels to samples that have passed the difference screening in the pre-screening results, so as to appropriately increase their selection probability in the subsequent logically directional screening.

[0122] Based on the above scheme, the intersection update mechanism of pre-screening and difference screening can filter out samples that obviously do not meet the knowledge constraints in advance while ensuring the novelty of the samples, which helps to improve the screening efficiency while ensuring the quality of the samples.

[0123] Step 306: The cascading filtering module performs logically targeted filtering on the updated difference filtering results. For specific implementation details of this step, please refer to the aforementioned... Figure 3 The description of step 104 in the method embodiment is not repeated here.

[0124] Step 307: The cascading filtering module performs consistent filtering on the logically oriented filtering results. For specific implementation details of this step, please refer to the aforementioned... Figure 3 The description of step 105 in the method embodiment will not be repeated here.

[0125] Step 308: The cascading filtering module performs knowledge-constrained filtering on the consistency and coherence filtering results to obtain the target sample set. For specific implementation details of this step, please refer to the aforementioned... Figure 3 The description of step 106 in the method embodiment is omitted here.

[0126] Step 309: The fine-tuning management module establishes fine-tuning training data based on the target sample set to perform model fine-tuning and version management. For specific implementation details of this step, please refer to the aforementioned... Figure 3 The descriptions of steps 107 to 108 in the method embodiment are not repeated here.

[0127] Based on the above scheme, by introducing a lightweight knowledge-constrained pre-screening mechanism, low-quality samples can be quickly filtered in the pre-screening stage of four-dimensional cascade screening, reducing the computational overhead of subsequent screening steps and helping to improve screening efficiency and system throughput in large-scale feedback dataset scenarios.

[0128] The above mainly describes the solutions provided by the embodiments of this application from the perspective of module interaction. It is understood that, in order to achieve the above functions, each module may include corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software.

[0129] Possible examples of the apparatus involved in the embodiments of this application when using integrated units. The communication apparatus may include: a processing unit and a communication unit. The processing unit is used to control and manage the operation of the communication apparatus. The communication unit is used to support communication between the communication apparatus and other devices. Optionally, the communication unit, also referred to as a transceiver unit, may include a receiving unit and / or a sending unit, respectively used to perform receiving and sending operations. The communication apparatus may also include a storage unit for storing the program code and / or data of the communication apparatus.

[0130] The communication device can be the cascaded filtering module in the above embodiments. The processing unit can support the communication device in executing the actions of the cascaded filtering module in the above method embodiments. Alternatively, the processing unit mainly executes the internal actions of the cascaded filtering module in the method embodiments, and the communication unit can support communication between the communication device and other devices.

[0131] In the first approach, the cascading filtering module is used to perform the aforementioned... Figure 3 The operation in the method embodiment is performed by the cascaded filtering module. Specifically, the processing unit is used to perform four-dimensional cascaded filtering on the multimodal unified representation to obtain the target sample set, wherein the four-dimensional cascaded filtering includes difference filtering, logical orientation filtering, consistency and coherence filtering, and knowledge constraint filtering.

[0132] As one possible implementation method, the processing unit is specifically used to calculate the semantic distance between the multimodal unified semantic representation of each data sample and the representation of each known sample in the current training set; calculate the penalty factor based on the dimensionality integrity of each modality dimension in the multimodal unified semantic representation of each data sample; use the product of the semantic distance and the penalty factor as the effective difference degree; perform initial screening of each data sample based on the dynamic difference degree threshold and the effective difference degree, and retain data samples whose effective difference degree is not lower than the dynamic difference degree threshold; determine the selection probability of each sample based on the coverage of each difference level, and select the sample from the initial screening results based on the selection probability to obtain the difference degree screening result.

[0133] As one possible implementation method, the processing unit is specifically used to extract entity relationships and match logical paths based on a pre-constructed power domain logic graph. Based on the logical path matching results and the power domain logic graph, it evaluates the causal chain integrity, inference path clarity, and multimodal logical anchoring degree of each data sample. It then weights and fuses these factors to obtain a comprehensive logical orientation score. Finally, it selects data samples with a comprehensive logical orientation score not lower than the logical orientation threshold to obtain the logical orientation screening results.

[0134] As one possible implementation method, the processing unit is specifically used to perform cross-modal semantic alignment verification, cross-modal numerical cross-validation, and temporal causal order verification on the logical orientation filtering results, obtain a weighted consistency score, and filter to determine the consistency verification results; perform contextual coherence verification on the logical orientation filtering results and remove incoherent samples to obtain contextual coherence verification results; perform multi-turn dialogue coherence verification on the logical orientation filtering results and remove incoherent samples to obtain multi-turn dialogue coherence verification results; and take the intersection of the consistency verification results, contextual coherence verification results, and multi-turn dialogue coherence verification results to obtain the consistent coherence filtering results.

[0135] As one possible implementation method, the processing unit is specifically used to obtain the current knowledge constraint base and the updated knowledge constraint base of the power production domain, and perform differential analysis to obtain the differential knowledge constraint base; based on the differential knowledge constraint base, the consistency and coherence screening results are respectively checked for compliance with safety procedures, compliance with equipment physical limits, and compliance with industry standards; data samples that fail any compliance check are filtered to obtain the target sample set.

[0136] In the second approach, the cascading filtering module is used to perform the aforementioned... Figure 4The operation is performed by the cascaded filtering module in the method embodiment. Specifically, the processing unit is also used to traverse the feedback dataset, identify fault types whose sample count in the current training set is lower than a preset minimum retention amount, and classify them as long-tail categories; set a difference filtering exemption for data samples of long-tail categories; and reduce the logical orientation threshold for data samples of long-tail categories in logical orientation filtering.

[0137] In the third approach, the cascading filtering module is used to perform the aforementioned... Figure 5 The operation is performed by the cascaded filtering module in the method embodiment. Specifically, the processing unit is also used to extract lightweight knowledge constraint rules based on the differential knowledge constraint library; perform lightweight knowledge constraint pre-screening on each data sample in the feedback dataset according to the lightweight knowledge constraint rules to obtain the pre-screening results; and update the difference filtering results based on the intersection of the pre-screening results and the difference filtering results.

[0138] The communication device can also be the fine-tuning management module in the above embodiments. The processing unit can support the communication device in executing the actions of the fine-tuning management module in the above method embodiments.

[0139] As one possible implementation method, the processing unit is specifically used to fine-tune the model by establishing fine-tuning training data for the current multimodal large model based on the target sample set; and to manage the version of the fine-tuned multimodal large model in conjunction with model evaluation.

[0140] Possible examples of the communication device involved in the embodiments of this application. The communication device includes a processor and interface circuitry. The processor and interface circuitry are coupled to each other. It is understood that the interface circuitry can be a transceiver or an input / output interface. Optionally, the communication device may also include a memory for storing instructions executed by the processor, or storing input data required for the processor to execute instructions, or storing data generated after the processor executes instructions.

[0141] When the communication device is used to implement the above method embodiments, the processor is used to implement the functions of the above processing unit, and the interface circuit is used to implement the functions of the above communication unit.

[0142] It is understood that the processor in the embodiments of this application can be a CPU, or other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0143] This application provides a chip (or chip system) including a processor for implementing any of the above-described method embodiments.

[0144] This application provides a computer-readable storage medium storing a computer program or instructions that, when executed, implement any of the above-described method embodiments.

[0145] This application provides a computer program product, which includes a computer program or instructions that, when executed, implement any of the above-described method embodiments.

[0146] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs or instructions.

[0147] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended embodiments and their equivalents.

Claims

1. A method for fine-tuning a multimodal large model for the power production domain, characterized in that, include: Based on the current status of the power production domain, a multimodal large model is used to extract the feedback dataset; The feedback dataset is mapped to a unified semantic dimension and fused, and the fusion result is stored in a structured manner to obtain a unified multimodal representation; The multimodal unified representation is subjected to four-dimensional cascade screening to obtain a target sample set, wherein the four-dimensional cascade screening includes difference screening, logical orientation screening, consistency and coherence screening and knowledge constraint screening. Based on the target sample set, fine-tuning training data is established for the current multimodal large model to fine-tune the model, and version management of the fine-tuned multimodal large model is carried out in conjunction with model evaluation.

2. The method for fine-tuning a multimodal large model for the power production domain according to claim 1, characterized in that, The aforementioned multimodal large model based on the current status of the power production domain extracts a feedback dataset, including: Based on the operation logs and user interaction feedback of the existing multimodal large model, extract the multimodal feedback dataset; The multimodal feedback dataset contains data samples of at least two modalities from image modal data, text modal data, and structured time-series modal data.

3. The multimodal large model fine-tuning method for the power production domain according to claim 1, characterized in that, The step of mapping the feedback dataset to a unified semantic dimension and fusing them, storing the fusion result in a structured manner, and obtaining a unified multimodal representation includes: Each modality in each data sample of the feedback dataset is independently encoded and projected and aligned to a unified semantic dimension. For each data sample, cross-modal semantic fusion based on dimension splicing is performed in a unified semantic dimension to obtain the corresponding multimodal unified semantic representation, which is then stored in a structured form.

4. The method for fine-tuning a multimodal large model for the power production domain according to claim 1, characterized in that, The multimodal unified representation is subjected to four-dimensional cascaded screening to obtain the target sample set, wherein the difference screening includes: Calculate the semantic distance between the multimodal unified semantic representation of each data sample and the semantic distance between the representations of each known sample in the current training set; Based on the dimensionality integrity of each modality dimension in the multimodal unified semantic representation of each data sample, a penalty factor is calculated, wherein the penalty factor is positively correlated with the dimensionality integrity and the penalty factor is a positive real number less than 1. The product of the semantic distance and the penalty factor is taken as the effective difference. Based on the dynamic difference threshold and the effective difference, each data sample is initially screened, and data samples with an effective difference not lower than the dynamic difference threshold are retained. The dynamic difference threshold is dynamically adjusted with each iteration round and is positively correlated with the iteration round. The selection probability of each sample is determined based on the coverage of each difference level. The selection probability is then used to select samples from the initial screening results to obtain the difference screening results.

5. The method for fine-tuning a multimodal large model for the power production domain according to claim 1, characterized in that, The multimodal unified representation is subjected to four-dimensional cascaded screening to obtain the target sample set, wherein the logical orientation screening further includes: Based on the pre-constructed power domain logical graph, entity relationships are extracted and logical paths are matched based on the difference screening results; Based on the logical path matching results and the power domain logic graph, the causal chain integrity, reasoning path clarity, and multimodal logic anchoring degree of each data sample are evaluated respectively. The weighted fusion of the completeness of the causal chain, the clarity of the reasoning path, and the multimodal logical anchoring degree results in a comprehensive score for logical orientation. Data samples with a comprehensive logical orientation score not lower than the logical orientation threshold are selected to obtain logical orientation selection results. The logical orientation threshold is determined based on statistical analysis of the comprehensive logical orientation score set of historical logical negative samples.

6. The method for fine-tuning a multimodal large model for the power production domain according to claim 1, characterized in that, The multimodal unified representation is subjected to a four-dimensional cascaded screening to obtain a target sample set, wherein the consistency and coherence screening includes: The logical orientation screening results are subjected to cross-modal semantic alignment verification, cross-modal numerical cross-validation, and temporal causal order verification. A consistency score is obtained by weighting the results, and the consistency verification results are selected and determined. Perform contextual coherence verification on the logical orientation filtering results, and remove incoherent samples to obtain the contextual coherence verification results. Perform multi-round dialogue coherence verification on the logical orientation screening results, and remove incoherent samples to obtain the multi-round dialogue coherence verification results. The consistency and coherence filtering results are obtained by taking the intersection of the consistency verification results, the context coherence verification results, and the multi-turn dialogue coherence verification results.

7. The method for fine-tuning a multimodal large model for the power production domain according to claim 1, characterized in that, The multimodal unified representation is subjected to a four-dimensional cascaded screening to obtain a target sample set, wherein the knowledge-constrained screening includes: The current knowledge constraint base and the updated knowledge constraint base of the power production domain are obtained, and differential analysis is performed to obtain a differential knowledge constraint base, wherein the differential knowledge constraint base includes newly added constraint rules and modified constraint rules. Based on the differential knowledge constraint base, the consistency and coherence screening results are respectively checked for compliance with safety procedures, compliance with equipment physical limits, and compliance with industry standards. The target sample set is obtained by filtering out data samples that fail any of the aforementioned compliance checks.

8. The method for fine-tuning a multimodal large model for the power production domain according to claim 7, characterized in that, Also includes: Based on the differential knowledge constraint library, lightweight knowledge constraint rules are extracted, wherein the lightweight knowledge constraint rules can be expressed in a structured manner and include at least numerical range rules, terminology standardization rules, and operation sequence rules. Based on the lightweight knowledge constraint rules, each data sample in the feedback dataset is pre-screened using lightweight knowledge constraints to obtain the pre-screening results; The difference screening results are updated based on the intersection of the pre-screening results and the difference screening results.

9. The method for fine-tuning a multimodal large model for the power production domain according to claim 1, characterized in that, Also includes: Traverse the feedback dataset and identify the fault types in the current training set whose sample count is lower than the preset minimum retention rate, as long-tail categories. A difference screening exemption is set for data samples of the long-tail category, so that they are not restricted by the difference threshold in the difference screening and can directly enter the subsequent screening stage; In the logical orientation filtering, the logical orientation threshold is lowered for data samples of the long-tail category; The data samples selected from the long-tail categories are used as long-tail protection data and combined with the target sample set to construct the fine-tuning training data.

10. A multimodal large-scale model fine-tuning system for the power production domain, characterized in that, include: The data extraction module is used to extract feedback datasets based on the current multimodal large model of the power production domain. The unified representation module is used to map the feedback dataset to a unified semantic dimension and fuse them, store the fusion result in a structured manner, and obtain a multimodal unified representation. The cascaded filtering module is used to perform four-dimensional cascaded filtering on the multimodal unified representation to obtain a target sample set, wherein the four-dimensional cascaded filtering includes difference filtering, logical orientation filtering, consistency and coherence filtering, and knowledge constraint filtering. The fine-tuning management module is used to establish fine-tuning training data for the current multimodal large model based on the target sample set, fine-tune the model, and manage the version of the fine-tuned multimodal large model in conjunction with model evaluation.

Images