Power system large-scale language model tuning method, system and device based on LoRA technology optimization, and medium

Abstract

The invention discloses an electric power system large-scale language model tuning method, system and device based on LoRA technology optimization and a medium, and belongs to the technical field of electric power system large-scale language models, and the method comprises the steps: obtaining historical language data formed in the operation process of an electric power system, and converting the historical language data into a training sample; constructing a task data set based on the training sample; constructing a pre-training language model; receiving a to-be-processed input text, extracting semantic features of the input text, and generating an output sequence corresponding to the input text. According to the method, the low-rank weight path is inserted in the feed-forward neural network in a parallel form, and an independent path is bound for each type of tasks, so that differential fine tuning of the electric power semantic tasks is realized on the premise of not modifying the structure and parameters of an original model, the overall tuning cost of the model is reduced, and the stability of the original model is kept.

Description

Technical Field

[0001] This invention relates to the field of large-scale language modeling technology for power systems, specifically to a method, system, equipment, and medium for optimizing large-scale language models of power systems based on LoRA technology. Background Technology

[0002] As a crucial component of the national energy infrastructure, the power system has shown a significant trend towards informatization and intelligentization in its operation and control, fault analysis, condition monitoring, and maintenance management in recent years. With the continuous networking of various power equipment and the large-scale growth of operational data, the amount of unstructured information in the power system, such as operation records, dispatch instructions, and equipment descriptions expressed in natural language or language-like formats, is increasing daily, placing higher demands on the intelligentization of related processing technologies.

[0003] Large-scale language models, due to their capabilities in natural language understanding and expression, have been explored for application in the power sector to assist in data parsing, semantic understanding, and language generation. Traditional approaches often involve domain adaptation of pre-trained language models through full-parameter fine-tuning. However, this method often faces challenges in the context of high-dimensional data in power systems, including high computational costs, complex parameter updates, and heavy storage burdens.

[0004] To reduce model tuning costs, the Low-Rank Adaptation (LoRA) method has been proposed. This method effectively reduces the number of parameters required for training and improves tuning efficiency by inserting a low-rank matrix into the partially linear transformation structure of the model. The application of LoRA technology in general domains has gradually matured, especially suitable for rapid model tuning in resource-constrained environments. However, in highly specialized scenarios such as power systems, this type of method still suffers from weak application foundations, insufficient structural matching, and low semantic parsing accuracy.

[0005] Current technologies lack LoRA tuning mechanisms tailored to the characteristics of power system linguistic data, particularly in areas such as model structure selection, parameter allocation, semantic mapping strategies, and tuning process stability, leaving room for improvement. Therefore, there is an urgent need to research LoRA optimization schemes suitable for power system linguistic data processing requirements to enhance model performance and practical applicability in such scenarios. Summary of the Invention

[0006] In view of the above-mentioned problems, the present invention is proposed.

[0007] Therefore, the technical problem solved by this invention is: how to achieve a model tuning method with low parameter count, high efficiency and strong semantic structure adaptability for power system language data while maintaining the stability of the backbone structure of the pre-trained language model, so as to improve the practical ability of large-scale language models in typical scenarios such as power dispatching, state parsing and instruction generation.

[0008] To address the aforementioned technical problems, this invention provides the following technical solution: a method for optimizing large-scale language models of power systems based on LoRA technology, comprising,

[0009] Acquire historical language data generated during the operation of the power system, and convert the historical language data into paired training samples containing input sequences and output sequences;

[0010] Task categories are divided based on semantic features in the training samples, and training samples belonging to the same task category are constructed into a task data set;

[0011] In the linear mapping sublayer of the pre-trained language model, a low-rank weighted path is configured for each task dataset, and the mapping relationship between the task category identifier and the corresponding low-rank weighted path is recorded to construct a path scheduling index table.

[0012] Keep all the original parameters in the pre-trained language model frozen, and train only the adjustable parameters in the low-rank weighted path;

[0013] The training samples from each task dataset are input into the pre-trained language model. The low-rank weighted paths bound in the path scheduling index table are called for forward propagation and parameter update, and independent callable weighted paths corresponding to the task category identifiers are trained respectively.

[0014] The system receives the input text to be processed, extracts its semantic features, and determines the corresponding task category identifier. Based on the path scheduling index table, it selects the target low-rank weight path that matches the task category identifier, performs inference in conjunction with the frozen pre-trained language model backbone structure, and generates the output sequence corresponding to the input text.

[0015] As a preferred embodiment of the LoRA-based optimization method for large-scale language models in power systems according to the present invention, the step of converting the historical language data into paired training samples containing input and output sequences includes,

[0016] Semantic units are extracted from historical language data. These semantic units include descriptive statements representing device status, instruction statements representing operational intentions, and fragments of contextual information.

[0017] The extracted semantic units are subjected to format standardization processing, which includes removing invalid punctuation, normalizing entity symbols, and unifying tense structure and subject-verb position;

[0018] Training samples are constructed based on semantic units after format standardization. The description statement representing the device status is concatenated with context information fragments to form an input sequence, and the instruction expression statement representing the operation intention is used as the corresponding output sequence.

[0019] As a preferred embodiment of the LoRA-based optimization method for large-scale language models of power systems described in this invention, the construction of the task data set includes:

[0020] The input sequences in the training samples are structurally parsed to extract semantic index tuples containing device identifiers, state parameter sets, and time series labels;

[0021] The semantic index tuples are mapped to the control action types in the corresponding output sequences to generate a pragmatic structure feature set;

[0022] Based on the combination patterns of device type, state change trend and control objective in the pragmatic structure feature set, the patterns are classified and the pragmatic structure class to which the sample belongs is marked;

[0023] Based on the preset path interference avoidance rules, training samples with the same pragmatic structure class and semantic gradient distribution divergence rate less than a set threshold are selected.

[0024] The selected training samples are assigned to the same task category and uniformly labeled according to the pragmatic structure class identifier. The input sequence and output sequence are encoded into the model input format and formed into an indexed structured sample group. The task category identifier is then bound to generate a structured task data set.

[0025] As a preferred embodiment of the power system large-scale language model tuning method based on LoRA technology described in this invention, the pre-trained language model includes multiple Transformer encoder layers, each Transformer encoder layer consisting of a multi-head attention mechanism module and a feedforward neural network module.

[0026] Each multi-head attention mechanism module includes, in sequence, a query vector linear transformation sublayer, a key vector linear transformation sublayer, and a value vector linear transformation sublayer, which are used to construct the multi-head attention representation;

[0027] Each feedforward neural network module includes, in sequence, a first linear transformation sublayer, an activation function sublayer, and a second linear transformation sublayer;

[0028] The low-rank weight path is set in the first linear transformation sub-layer of the feedforward neural network module. The low-rank weight path includes a first dimension-reducing linear matrix and a second dimension-increasing linear matrix, which are respectively used to map the original feature dimension to the low-rank feature dimension and restore it to the original feature dimension. The low-rank weight path is connected to the original parameters in the first linear transformation sub-layer in parallel.

[0029] For each task dataset, a low-rank weighted path configuration parameter group is constructed. The configuration parameter group includes a path index, a set rank value, a target insertion layer index, and a task category identifier, which are used to establish a static binding relationship between the task category identifier and the path structure.

[0030] All configuration parameter groups for low-rank weighted paths are uniformly stored in the path scheduling index table, which serves as the parameter scheduling basis for mapping and calling the pre-trained language model backbone structure and task category.

[0031] As a preferred embodiment of the power system large-scale language model tuning method based on LoRA technology described in this invention, the forward propagation and parameter update of the low-rank weighted paths bound in the call path scheduling index table includes:

[0032] Based on the task category identifier recorded in the training samples, retrieve the path configuration parameter group corresponding to the task category identifier from the path scheduling index table, and read the path index and the insertion layer index.

[0033] Based on the insertion layer index, load the low-rank weighted path associated with the path index in the corresponding linear mapping sublayer of the pre-trained language model backbone structure, and set the low-rank weighted path as the activation path of the current training state.

[0034] Based on the task category identifier, the training samples are divided into multiple training batch groups; each training batch group is input into the pre-trained language model backbone structure, and forward and backward propagation operations are performed. Only the low-rank weighted paths that match the task category identifier of the current training batch group are activated to participate in gradient calculation and parameter update.

[0035] The remaining low-rank weighted paths remain inactive during the training batch execution, and gradient propagation and update operations are not performed on all parameters in the inactive paths;

[0036] All original parameters in the backbone structure of the pre-trained language model remain frozen throughout the training process and do not participate in any gradient calculation or update operations.

[0037] Each low-rank weighted path is assigned an independent optimizer context, which contains an independent gradient cache, momentum factor, and parameter update state to isolate the training state of each low-rank weighted path.

[0038] The training rounds are executed for all training batches, and the parameter optimization process of all low-rank weighted paths is completed in sequence, generating optimized low-rank weighted path sets bound to each task category identifier.

[0039] As a preferred embodiment of the LoRA-based optimization method for large-scale language models of power systems described in this invention, the step of assigning an independent optimizer context to each low-rank weighted path includes:

[0040] The optimizer state set is initialized based on the task category identifier and bound to each low-rank weight path. The initial weight values ​​and training round indices of all adjustable parameters in the low-rank weight path are recorded in the optimizer state set.

[0041] Configure a set of backpropagation buffer structures for the optimizer state set. The buffer structures include a gradient cache, an average cache for recording the moving average gradient values, and a momentum cache for storing momentum estimates.

[0042] Before executing each training batch, the corresponding low-rank weighted path is retrieved from the path scheduling index table by task category identifier, and the associated optimizer state set and backpropagation buffer structure are loaded.

[0043] Write the backpropagation gradient of the current training batch into the gradient cache, and call the historical records in the average cache and momentum cache to calculate the updated gradient value of the current weight path.

[0044] Based on the updated gradient value, the adjustable parameters in the low-rank weighted path are iteratively updated, and the updated gradient and momentum value are synchronously written into the optimizer state set.

[0045] After each training batch, record the parameter variation magnitude, training loss value and gradient mean of the low-rank weight path, and update the corresponding optimizer state set.

[0046] When the path scheduling index table schedules to the training batch corresponding to the new task category identifier, all inactive low-rank weight paths and optimizer state sets do not participate in the training operation of the current batch, and no parameter reading, gradient calculation or state writing is performed during the round training.

[0047] When the recorded parameter variation exceeds the feature stability threshold of the training sample corresponding to the task category identifier, the learning rate is switched to the preset lower limit, and the parameter update operation for the current round is paused.

[0048] As a preferred embodiment of the LoRA-based optimization method for large-scale language models in power systems described in this invention, the following inference process is performed after the input text is input into the pre-trained language model:

[0049] The input text is parsed to extract a semantic index tuple containing power equipment identifiers, state parameter groups, and time series labels.

[0050] The semantic index tuples are mapped to control action types to generate a set of pragmatic structural features corresponding to the input text;

[0051] Based on the combined features of device type, state change trend and control objective in the pragmatic structure feature set, the matching items are queried in the pragmatic structure class index, and the task category identifier to which the input text belongs is determined.

[0052] Based on the task category identifier, retrieve the corresponding path configuration parameter group from the path scheduling index table, and read the path index and the target insertion layer index;

[0053] Based on the target insertion layer index, locate the corresponding linear mapping sub-layer in the backbone structure of the pre-trained language model, and load the low-rank weighted path associated with the path index;

[0054] Based on the currently loaded low-rank weighted path and the backbone structure of the pre-trained language model, forward inference is performed, and the output sequence corresponding to the semantic index tuple of the input text is output.

[0055] This invention provides a large-scale language model tuning system for power systems based on LoRA technology.

[0056] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a large-scale language model tuning system for power systems based on LoRA technology, comprising: a data extraction unit, which performs the collection of historical language data during the operation of the power system;

[0057] The sample construction unit performs the conversion of historical language data into paired training samples containing input and output sequences;

[0058] The task partitioning unit performs task classification based on the semantic features of the training samples and constructs a task dataset;

[0059] The path configuration unit executes the configuration of low-rank weighted paths bound to the task data set and constructs a path scheduling index table;

[0060] The training control unit executes the process of keeping the original parameters frozen in the pre-trained language model and training only low-rank weighted paths.

[0061] The weight generation unit executes the low-rank weight paths bound in the call path scheduling index table to train and form callable weight paths;

[0062] The inference execution unit extracts semantic features from the input text to be processed, determines the task category identifier, performs inference, and generates an output sequence.

[0063] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the method for optimizing a large-scale language model of a power system based on LoRA technology.

[0064] The present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the method for optimizing a large-scale language model of a power system based on LoRA technology.

[0065] The beneficial effects of this invention are as follows: This invention performs structured semantic indexing and pragmatic feature extraction on language data generated in power system operation scenarios, and combines elements such as power equipment status, control intentions and time tags to achieve high-precision structured representation of complex instructions and status statements, thereby improving the power industry's large language model's ability to recognize control semantics.

[0066] By establishing a static mapping between pragmatic structure classes and task category identifiers, and by filtering and classifying samples based on semantic gradient distribution, gradient interference between different power operation tasks is effectively suppressed, and the model's ability to distinguish and generalize in multi-task contexts is enhanced.

[0067] This invention inserts low-rank weighted paths in parallel into the feedforward neural network of the Transformer backbone and binds independent paths to each type of task. Without modifying the original model structure and parameters, it achieves differentiated fine-tuning of power semantic tasks, reduces the overall model tuning cost, and preserves the stability of the original model.

[0068] Each low-rank weighted path is configured with an independent optimizer state context during training, and independent gradients, momentum and average caches are set, which realizes the isolation control of the training state of multiple paths, avoids interference between training processes between paths, and is conducive to the independent convergence and reliable tuning of different power task paths.

[0069] During the inference process, the path scheduling index table is automatically retrieved by inputting semantic structure and pragmatic features, the corresponding task path is dynamically activated, the remaining paths are kept inactive and the main parameters are frozen, ensuring the targeted use of computing resources and the targeted matching of semantic output during the inference stage, improving the model deployment efficiency and task calling accuracy, and is suitable for real-time language understanding and response needs in various power control scenarios. Attached Figure Description

[0070] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 The above is a flowchart of a method for optimizing a large-scale language model of a power system based on LoRA technology, provided as an embodiment of the present invention. Detailed Implementation

[0072] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0073] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a method for optimizing a large-scale language model of a power system based on LoRA technology, including:

[0074] Step 1: Obtain historical language data generated during the operation of the power system, and convert the historical language data into paired training samples containing input sequences and output sequences;

[0075] In step 1, converting the historical language data into paired training samples containing input and output sequences includes,

[0076] Semantic units are extracted from historical language data. These semantic units include descriptive statements representing device status, instruction statements representing operational intentions, and fragments of contextual information.

[0077] The extracted semantic units are subjected to format standardization processing, which includes removing invalid punctuation, normalizing entity symbols, and unifying tense structure and subject-verb position;

[0078] Training samples are constructed based on semantic units after format standardization. The description statement representing the device status is concatenated with context information fragments to form an input sequence, and the instruction expression statement representing the operation intention is used as the corresponding output sequence.

[0079] In this embodiment, the historical language data generated during the operation of the power system mainly originates from various information systems deployed within the power company's production and operation system. These information systems include, but are not limited to, dispatch automation systems (SCADA / EMS), distribution network operation and maintenance management systems, relay protection and automation systems, integrated dispatching platforms, intelligent inspection systems, operation and maintenance log platforms, and on-site maintenance work order systems. This data is typically stored in text format and may also be converted into text records by a speech recognition system. The language data encompasses semantic entities such as equipment status descriptions, operation instruction content, dispatch command transmissions, and anomaly handling process records, reflecting the relationship between the status changes and control responses of electrical equipment in various operating scenarios of the power system.

[0080] After data acquisition, given the inconsistencies in the expression of language data, the significant differences in the use of professional terminology, and the complexity of contextual structures, semantic unit extraction and format standardization processing need to be performed in order to construct training sample pairs with clear structure and semantic clarity.

[0081] In this embodiment, semantic units are first extracted from the original language data. The semantic units specifically include:

[0082] Descriptions of equipment status, such as "The current temperature of main transformer No. 10 is 85 degrees Celsius, and it is in an overload operation state";

[0083] Instruction statements that express the intention of operation, such as "Please immediately reduce the load on main transformer No. 10 to avoid further temperature rise";

[0084] Contextual information fragments associated with the above statement, such as "Due to the concentrated power consumption of surrounding users, the line current continues to rise."

[0085] To improve the accuracy of semantic recognition, the extraction of the aforementioned semantic units can be based on key field templates, entity recognition dictionaries, and syntactic dependency structures. Through a domain-customized rule base, device numbers, action predicates, and state adjectives can be located and extracted, ensuring the integrity of the sentence structure.

[0086] For the extracted semantic units, a unified text normalization process is adopted in this embodiment, which specifically includes the following steps:

[0087] Remove meaningless or incorrectly formatted symbols and punctuation, such as deleting redundant characters like “…”, “——”, and “ / / / / ”;

[0088] The entity symbols are uniformly normalized, and the object names that have multiple representations are unified into standard names. For example, “#10 main transformer”, “No. 10 main transformer” and “10 transformer” are unified into “transformer-10”.

[0089] Unify grammatical structure and tense expression, standardize language structures with passive voice, inversion, and complex sentences into standard active statement expressions to facilitate model learning;

[0090] Standardize numerical units involving dimensions, such as unifying "0.4kV" to "400V" to ensure semantic consistency.

[0091] After format standardization, paired training samples are constructed based on the aforementioned semantic units. In this embodiment, the descriptive statement representing the device state is concatenated with its associated contextual information fragments to form the model's input sequence, and the instruction expression statement representing the operational intent is used as the target output of this input sequence. During concatenation, semantic tags can be used to assist the model in understanding the sentence structure, such as inserting identifiers like "[state]" and "[context]" into the input sequence to improve the model's semantic discrimination capability.

[0092] For example, construct the following training sample pairs:

[0093] Input sequence: "[Status] Transformer-10 load exceeds 90% [Context] Current power supply area users are consuming electricity centrally, and the system is operating close to its limit."

[0094] Output sequence: "Execute load shedding operation for transformer-10".

[0095] The above method can effectively establish a causal semantic relationship between input semantics and target operation, providing semantically clear structural samples for subsequent model training.

[0096] Step 2: Divide the task categories based on the semantic features in the training samples, and construct a task data set from the training samples belonging to the same task category;

[0097] In step 2, the construction of the task data set includes performing structural parsing on the input sequences in the training samples and extracting semantic index tuples containing device identifiers, state parameter groups, and time series labels;

[0098] The semantic index tuples are mapped to the control action types in the corresponding output sequences to generate a pragmatic structure feature set;

[0099] Based on the combination patterns of device type, state change trend and control objective in the pragmatic structure feature set, the patterns are classified and the pragmatic structure class to which the sample belongs is marked;

[0100] Based on the preset path interference avoidance rules, training samples with the same pragmatic structure class and semantic gradient distribution divergence rate less than a set threshold are selected.

[0101] The selected training samples are assigned to the same task category and uniformly labeled according to the pragmatic structure class identifier. The input sequence and output sequence are encoded into the model input format and formed into an indexed structured sample group. The task category identifier is then bound to generate a structured task data set.

[0102] Specifically, the steps include the following:

[0103] Structured parsing of the input sequence: First, structured parsing is performed on the input sequence of each training sample to extract semantic index tuples. These semantic index tuples include the following three core elements:

[0104] Equipment identifier: Used to characterize a specific power equipment unit that participates in operation or state change, such as "Transformer-10", "110kV Bus", "Line L123", etc.

[0105] Status parameter group: describes the operating status parameters of the device in the current scenario, usually including information such as voltage, current, active power, reactive power, temperature, frequency, power factor, etc., in the form of "temperature is 85℃" or "load rate is 92%".

[0106] Time series labels: used to describe the temporal context of a state occurrence, such as "June 5, 2024, 11:40" or "Yesterday, 5 PM", etc., to enhance the temporal consistency of semantic logic.

[0107] The aforementioned semantic elements can be extracted using methods such as regular expression templates, named entity recognition (NER), or dependency parsing. The structural parsing not only extracts lexical information but also preserves semantic dependencies, serving as the foundational feature representation for subsequent structural classification.

[0108] Mapping control action types and constructing a set of pragmatic structural features: Next, the semantic index tuples mentioned above are associated and mapped with the corresponding output sequences in the training samples to identify the control action types represented in the output statements, such as "shutdown operation", "load input", "bus connection switching", "adjusting active power output" and other control intentions, and form a set of pragmatic structural features with operational logic.

[0109] This feature set is used to characterize the semantic-logical relationship between input states and output control, and is the core basis for task category determination. For example:

[0110] Input: Device -10 High Load → Output: Reduce the load on this device;

[0111] Input: Line fault alarm → Output: Disconnect the faulty line and transfer the load path.

[0112] Categorizing pragmatic structures and generating task labels: Based on the above, and according to the typical control task characteristics of power systems, the training samples are categorized according to their pragmatic structural features. This categorization may incorporate the following elements:

[0113] Equipment types: such as power transmission equipment, transformer equipment, busbars, circuit breakers, etc.;

[0114] Trends in state changes: such as increase, decrease, fluctuation, exceeding limits, abnormality, etc.;

[0115] Control objectives include: such as loss control, load limiting, isolation, recovery, and adjustment.

[0116] By combining the above three types of elements, the classification rules for determining pragmatic structure are formed, as shown in Table 1:

[0117] Table 1 Classification Rules

[0118] Equipment type State Trend Control Target Pragmatic Structure Class Examples main transformer Temperature rise Unloading operation Category A01 power distribution lines Low voltage Pour power Category B05 Switchgear Fault alarm Removal of fault Category C03

[0119] Training samples that conform to the same pragmatic structure class are grouped into the same initial task class, and their structural feature combinations and sample index information are recorded.

[0120] Path interference avoidance and sample selection: To ensure that training samples under the same task category maintain relative consistency in semantic distribution and avoid gradient interference during optimization, this embodiment introduces a path interference avoidance rule. This rule is defined as follows:

[0121] Semantic gradient distribution divergence rate: refers to the degree to which the gradient directions of different training samples in the same pragmatic structure class are different in the feature vector space;

[0122] For samples with the same pragmatic structure class label, only samples with a gradient divergence rate lower than a set threshold (e.g., 0.2) are retained for subsequent task set construction.

[0123] This operation can remove heterogeneous samples in small sample classes, enhance the stability of path training, and improve parameter transferability.

[0124] Constructing a structured task dataset: Finally, the selected and retained training samples are labeled according to their respective pragmatic structure classes and assigned a unified task category identifier. Simultaneously, their input and output sequences are encoded into a format conforming to language model training specifications, forming a training sample group with a unified structure. This sample group constitutes the structured task dataset under the corresponding task category.

[0125] Step 3: In the linear mapping sublayer of the pre-trained language model, configure low-rank weighted paths for each task dataset, record the mapping relationship between task category identifiers and corresponding low-rank weighted paths, and construct a path scheduling index table;

[0126] In step 3, the pre-trained language model includes multiple Transformer encoder layers, each of which consists of a multi-head attention mechanism module and a feedforward neural network module.

[0127] Each multi-head attention mechanism module includes, in sequence, a query vector linear transformation sublayer, a key vector linear transformation sublayer, and a value vector linear transformation sublayer, which are used to construct the multi-head attention representation;

[0128] Each feedforward neural network module includes, in sequence, a first linear transformation sublayer, an activation function sublayer, and a second linear transformation sublayer;

[0129] The low-rank weight path is set in the first linear transformation sub-layer of the feedforward neural network module. The low-rank weight path includes a first dimension-reducing linear matrix and a second dimension-increasing linear matrix, which are respectively used to map the original feature dimension to the low-rank feature dimension and restore it to the original feature dimension. The low-rank weight path is connected to the original parameters in the first linear transformation sub-layer in parallel.

[0130] For each task dataset, a low-rank weighted path configuration parameter group is constructed. The configuration parameter group includes a path index, a set rank value, a target insertion layer index, and a task category identifier, which are used to establish a static binding relationship between the task category identifier and the path structure.

[0131] All configuration parameter groups for low-rank weighted paths are uniformly stored in the path scheduling index table, which serves as the parameter scheduling basis for mapping and calling the pre-trained language model backbone structure and task category.

[0132] In step 3, for the task data set that has been divided in step 2, and in combination with its pragmatic structure class label, low-rank weighted paths are configured in the linear mapping sublayer of the pre-trained language model to form a structural mapping relationship for multi-task semantic scheduling, and a path scheduling index table is constructed in a structured parameter configuration manner.

[0133] Specifically, the pre-trained language model adopts a multi-layer stacked structure, with each layer being a Transformer encoder layer. Each Transformer encoder layer consists of a multi-head attention mechanism module and a feedforward neural network module.

[0134] The multi-head attention mechanism module includes a query vector linear transformation sublayer, a key vector linear transformation sublayer, and a value vector linear transformation sublayer, which respectively realize the vector reconstruction and weight calculation of input features under different attention channels;

[0135] The feedforward neural network module includes a first linear transformation sublayer, an activation function sublayer, and a second linear transformation sublayer, which undertake the tasks of nonlinear feature expansion and remapping.

[0136] To avoid interfering with the original parameter structure of the model, the low-rank weight paths are placed in the first linear transformation sublayer of the feedforward neural network module and connected in parallel to the original parameters of that sublayer. Each low-rank weight path includes:

[0137] A first-dimensional reduced linear matrix, with the input dimension consistent with the input dimension of this sub-layer, and the output dimension being the set rank value;

[0138] A second linear matrix is ​​used to restore the dimensionality-reduced features to their original dimensions, maintaining consistency with the main output structure.

[0139] The above structure is introduced in parallel without changing the original parameters of the pre-trained language model backbone. All low-rank weight paths are initialized with zero-mean random distribution of weights.

[0140] For each task data set, a corresponding low-rank weighted path configuration parameter group is constructed to statically identify the path structure bound to that set. The configuration parameter group includes:

[0141] Path index: uniquely identifies each low-rank weighted path;

[0142] Task category identifier: The code from the pragmatic structure classification in step 2, such as "A01", "B05", etc.;

[0143] Set the rank value: Configure it according to the number of samples in this class, semantic complexity, or expression granularity of the control target, usually between 8 and 32;

[0144] Target insertion layer index: Defines which encoder layer's first linear transform sublayer in the Transformer backbone structure the low-rank weighted path is attached to.

[0145] For example, a training dataset from task category identifier "A01" (main transformer - temperature rise - load shedding operation) may be configured with path index P001, rank value 8, and target insertion layer index layer 4; while "C03" (switching equipment - fault alarm - fault clearing) is configured with path index P003, rank value 16, and inserted into the 10th layer encoder.

[0146] All configuration parameter groups for low-rank weighted paths are stored uniformly in the path scheduling index table. This index table is organized using a key-value structure, with the task category identifier as the primary key index and the corresponding path index and insertion position information as value structure fields. As shown in Table 2:

[0147] Table 2 Structure Fields Table

[0148] Task category identifier Path Index Set rank Insertion layer index A01 P001 8 4 B05 P002 16 7 C03 P003 16 10

[0149] This table is called during both model training and inference to implement a scheduling chain for input semantic features, task category recognition, path scheduling index lookup, and path loading.

[0150] In addition, the path scheduling index table supports incremental configuration and structural constraints, namely:

[0151] For each new pragmatic structure class, a unique low-rank weighted path must be configured;

[0152] Insertion layer indexes should avoid overlap to prevent multiple paths from competing for the same layer;

[0153] The rank configuration follows the principle that the more training samples there are and the finer the task granularity, the higher the required rank can be.

[0154] In a preferred embodiment of the present invention, during the construction of the path scheduling index table, the pre-trained language model adopts a stacked structure to form multiple Transformer encoder layers. Each Transformer encoder layer consists of a multi-head attention mechanism module and a feedforward neural network module. The multi-head attention mechanism module sequentially includes a query vector linear transformation sub-layer, a key vector linear transformation sub-layer, and a value vector linear transformation sub-layer to construct a multi-channel context-aware representation. The feedforward neural network module consists of a first linear transformation sub-layer, an activation function sub-layer, and a second linear transformation sub-layer, undertaking the tasks of feature dimension expansion and nonlinear mapping.

[0155] Each task dataset is bound to a low-rank weighted path, and the path structure is embedded in the first linear transformation sublayer of the feedforward neural network module. The low-rank path structure consists of two levels of linear transformation units, including a first dimension-reduced linear matrix. With the second-dimensional linear matrix Where R represents the set of real numbers, Indicates the core feature dimension. Indicates the set rank value, index This corresponds to the task category identifier. The structure is connected in parallel to the output path of the original transform sublayer, enabling joint modeling of feature perturbation and semantic adaptation. The corresponding path output format is as follows:

[0156]

[0157] in, This represents the current main input feature vector. This represents the characteristic perturbation term introduced by the path. The final output is obtained by adding it to the backbone parameter output:

[0158]

[0159] in, The original parameter matrix of the first linear transformation sublayer of the backbone is... This is the output of the merged sublayer.

[0160] To suppress path parameter inflation and overfitting risk, a path regularization term is set as one of the loss constraints, defined as follows:

[0161]

[0162] in, For path The parameter regularization term, Describing the Frobenius norm, The path regularization coefficient is empirically selected based on a preset path size. .

[0163] To address the potential semantic overlap risk between paths, a gradient conflict suppression term is introduced to penalize cases where multiple paths update in similar directions. It is defined as follows:

[0164]

[0165] in, The path for the current training round The parameter gradient vector, The conflict suppression coefficient is preferably within the range of [value range missing]. This was obtained through statistical fitting of the gradient angles of different pragmatic structural paths. Indicates the first The path disturbance loss value of a low-rank weighted path. Indicates the first The set of current values ​​of all adjustable parameters in a low-rank weighted path; Indicates the relationship with the first Paths coexist in other low-rank weight paths within the same pre-trained model The parameter set.

[0166] Based on the above constraints, the complete path training objective function is constructed as follows:

[0167]

[0168] in, For path The main loss function (such as cross-entropy loss or KL divergence) under the corresponding task category is combined with regularization and conflict term constraints, and the path parameters are updated through the backpropagation mechanism.

[0169] Each low-rank weighted path and its structure configuration are recorded through a configuration parameter group, including the path index. rank value Insertion layer index and the task category identifier All configuration parameter groups are stored in the path scheduling index table, organized in a key-value pair structure, with the task category identifier as the retrieval primary key, to implement the path loading and scheduling mechanism during the inference phase.

[0170] This path structure scheduling method not only ensures that the backbone model structure and parameter freezing state remain unchanged, but also achieves fine modeling of the target of various power system semantic control tasks through path structure perturbation, and decouples the path semantic space between different categories, thereby improving the model's generalization ability and modular scheduling efficiency under fine-grained tasks.

[0171] Step 4: Keep all the original parameters in the pre-trained language model frozen, and train only the adjustable parameters in the low-rank weight path;

[0172] In implementing this step, all original parameters of the pre-trained language model are set to a frozen state. Specifically, during the training phase, parameters such as the linear transformation matrix of the multi-head attention module and feedforward neural network module, the bias terms and scaling coefficients in the layer normalization structure, etc., which have already been trained in the model, are not included in the backpropagation gradient calculation, and update operations are prohibited. In the specific implementation, a parameter freeze flag is configured in the training graph computation graph to ensure that the state of these parameters remains unchanged during training.

[0173] Taking a typical scenario in a power system as an example, the semantic mapping logic required for transformer fault identification and control strategy generation differs significantly from that required for scheduling instruction optimization. Simultaneous training and updating of the language model's backbone parameters can easily lead to conflicts in gradient directions between different semantic tasks, resulting in slower model convergence and even training instability. Therefore, in this invention, to ensure the independent optimization capability of various semantic tasks, the backbone parameters are frozen, and a stable basic representation space is provided based on the existing generalization semantic capabilities of the pre-trained model.

[0174] After freezing the backbone parameters, a low-rank weight path is introduced in parallel into the first linear transformation sublayer of the feedforward neural network module. This path consists of a dimension-reduced linear mapping matrix and an dimension-increasing linear mapping matrix, which are used to map context features from the original dimension to the low-rank space and then project them back to the original dimension, respectively. In practice, the dimension of the dimension-reduced matrix is ​​set to 1 / 4 to 1 / 8 of the original dimension, adjusted according to the task complexity, while the dimension-increasing matrix maintains the same dimension as the original output.

[0175] During the training phase, while freezing the backbone parameters, only the parameters of the activated low-rank weighted paths are updated. Taking the "voltage fluctuation - recovery action" task in the equipment inspection log as an example, its corresponding low-rank weighted path is activated by the scheduling table during the model inference process, independently undertaking the responsibility of feature transformation and output generation for this semantic task. Its path parameters are gradually optimized through iterative training of multiple rounds of task samples, without affecting the structure and state of other task paths in the model.

[0176] During implementation, each low-rank path is bound to a specific task dataset and configured with an independent optimizer context to ensure the isolation of the training states of each path. The combination of freezing the backbone structure and separating training paths enables the model to flexibly invoke different task paths when facing the multi-task scheduling requirements of the power system, while maintaining efficient training convergence and structural stability.

[0177] Step 5: Input the training samples from each task dataset into the pre-trained language model, call the low-rank weighted path bound in the path scheduling index table for forward propagation and parameter update, and train to form independent callable weighted paths corresponding to the task category identifier.

[0178] In step 5, the forward propagation and parameter update of the low-rank weighted paths bound in the call path scheduling index table includes:

[0179] Based on the task category identifier recorded in the training samples, retrieve the path configuration parameter group corresponding to the task category identifier from the path scheduling index table, and read the path index and the insertion layer index.

[0180] Based on the insertion layer index, load the low-rank weighted path associated with the path index in the corresponding linear mapping sublayer of the pre-trained language model backbone structure, and set the low-rank weighted path as the activation path of the current training state.

[0181] Based on the task category identifier, the training samples are divided into multiple training batch groups; each training batch group is input into the pre-trained language model backbone structure, and forward and backward propagation operations are performed. Only the low-rank weighted paths that match the task category identifier of the current training batch group are activated to participate in gradient calculation and parameter update.

[0182] The remaining low-rank weighted paths remain inactive during the training batch execution, and gradient propagation and update operations are not performed on all parameters in the inactive paths;

[0183] All original parameters in the backbone structure of the pre-trained language model remain frozen throughout the training process and do not participate in any gradient calculation or update operations.

[0184] Each low-rank weighted path is assigned an independent optimizer context, which contains an independent gradient cache, momentum factor, and parameter update state to isolate the training state of each low-rank weighted path.

[0185] The training rounds are executed for all training batches, and the parameter optimization process of all low-rank weighted paths is completed in sequence, generating optimized low-rank weighted path sets bound to each task category identifier.

[0186] After completing the construction of the path scheduling index table, the training samples recorded in the structured task dataset are divided into multiple batches according to the task category identifier. Each batch corresponds to a task category identifier and a unique mapping relationship is established with the low-rank weighted paths configured in the scheduling index table.

[0187] Before training, the model backbone structure is initialized, all original parameters are frozen, and the state cache of all low-rank weight paths is cleared. Next, the configuration parameter groups corresponding to the task category identifier and path scheduling index table are loaded sequentially, the path index and insertion layer index information are read, the specified low-rank weight path is loaded into the target position of the Transformer backbone structure according to the insertion layer index, and it is set to the active state.

[0188] Taking the task category identifier "A01" as an example, its corresponding path index is P001, and the insertion layer index is layer 4. The low-rank weight path with path index P001 will be loaded into the feedforward neural network module of layer 4. After loading, all parameters in path P001 are activated and bound to the training samples of the current batch.

[0189] Subsequently, the training samples of class "A01" are divided into several training batches according to the order of the task data set, with each batch containing 32 samples. Each training batch is sequentially fed into the pre-trained language model backbone structure with frozen parameters to perform forward and backward propagation operations. The activation path corresponding to the current task category participates in gradient calculation, and parameter update operations are performed according to the recorded state in the optimizer context. After each training session, the weight change record, mean gradient value, and loss function value of that path are immediately updated and written to the optimizer context bound to path P001.

[0190] For all other paths that are not activated in the current training batch, such as paths with indices P002 and P003, their corresponding parameters remain static, do not participate in any calculations, and do not generate gradients or state writes. This approach ensures complete isolation of the training states between low-rank weighted paths during single-task tuning.

[0191] Before each training batch, the set of optimizer states bound to that path is loaded based on the path index, including:

[0192] Initial weights of parameters for the current path;

[0193] Training round index;

[0194] The momentum cache and average gradient cache generated in the previous iteration;

[0195] Gradient cache for the current training batch.

[0196] After inputting training samples, based on the backpropagation results, the gradients calculated for the current batch are written to the gradient cache of the corresponding path; the historical momentum estimate and moving average are used to calculate the updated gradient for that path; and this updated value is used to complete one iterative update of the adjustable parameters. Subsequently, the training management module records the magnitude of this round of parameter updates in the optimizer state set.

[0197] If, in three consecutive training rounds, the parameter update magnitude of path P001 is consistently less than 0.005 and the training loss decrease rate is less than 0.02, a parameter freeze flag will be activated for that path, and the learning rate will be decayed to 1 / 10 of its current value. Thereafter, this path will not participate in parameter update operations and will remain frozen until a change in the input data structure is detected.

[0198] After training all training batches of samples for task category “A01” is completed, the next task category identifier, such as “B05”, is automatically scheduled, and its corresponding path P002 is loaded according to the same process to perform parameter activation, sample grouping, forward propagation, gradient calculation and optimizer state recording.

[0199] Throughout the training process, the path scheduling index table serves as the basis for parameter scheduling and path localization, continuously participating in the path loading and sample scheduling processes to ensure consistency and controllability between the model structure and sample semantic classification. Ultimately, low-rank weighted paths for all task categories are trained separately, forming a set of callable optimized paths bound to task labels, which are used for inference and generation of the power language model in multi-semantic control tasks.

[0200] Furthermore, assigning an independent optimizer context to each low-rank weighted path includes,

[0201] The optimizer state set is initialized based on the task category identifier and bound to each low-rank weight path. The initial weight values ​​and training round indices of all adjustable parameters in the low-rank weight path are recorded in the optimizer state set.

[0202] Configure a set of backpropagation buffer structures for the optimizer state set. The buffer structures include a gradient cache, an average cache for recording the moving average gradient values, and a momentum cache for storing momentum estimates.

[0203] Before executing each training batch, the corresponding low-rank weighted path is retrieved from the path scheduling index table by task category identifier, and the associated optimizer state set and backpropagation buffer structure are loaded.

[0204] Write the backpropagation gradient of the current training batch into the gradient cache, and call the historical records in the average cache and momentum cache to calculate the updated gradient value of the current weight path.

[0205] Based on the updated gradient value, the adjustable parameters in the low-rank weighted path are iteratively updated, and the updated gradient and momentum value are synchronously written into the optimizer state set.

[0206] After each training batch, record the parameter variation magnitude, training loss value and gradient mean of the low-rank weight path, and update the corresponding optimizer state set.

[0207] When the path scheduling index table schedules to the training batch corresponding to the new task category identifier, all inactive low-rank weight paths and optimizer state sets do not participate in the training operation of the current batch, and no parameter reading, gradient calculation or state writing is performed during the round training.

[0208] When the recorded parameter variation exceeds the feature stability threshold of the training sample corresponding to the task category identifier, the learning rate is switched to the preset lower limit, and the parameter update operation for the current round is paused.

[0209] It should be noted that, in implementing step 5, to ensure state isolation of each low-rank weighted path during training and to avoid semantic transfer interference between tasks, this embodiment configures an independent optimizer context for each low-rank weighted path. This optimizer context includes an optimizer state set and a backpropagation buffer structure, both of which are bound to the corresponding path index in the path scheduling index table.

[0210] During the initialization phase, a path index is generated based on the task category identifier, and the optimizer state set is initialized accordingly. This set records the initial weight values ​​and round indices of all adjustable parameters in the current path, and a structured cache space is set up to store gradient, momentum, and parameter evolution data during subsequent training.

[0211] The low-rank path parameters are initialized using the industry-standard Xavier initialization method. In actual deployment, the parameter initialization range is determined through experimental fitting, taking into account the task category sample size and the Transformer network depth.

[0212]

[0213] in, These are the initial weights; These are the dimensions of the input and output of the low-rank path, respectively (e.g., if the rank is set to 16, then typically...). ); Indicates the interval A uniform distribution on the surface.

[0214] The Xavier initialization method is derived from training stability tests on a large-scale power regulation data corpus. Fitting results show that the sample size is between [missing information]. When the intervals are between the points, the above interval setting can keep the gradient variance stable and avoid gradient vanishing in the early stage.

[0215] Before a training batch starts, the target low-rank weighted path is retrieved from the path scheduling index table by task category identifier, and the associated optimizer state set and backpropagation buffer structure are loaded. After the backpropagation gradient of the current training batch is calculated, it is written to the gradient cache.

[0216] To improve the stability and directionality of gradient updates, this embodiment employs both moving average and momentum correction mechanisms. The moving average cache stores the historical average value of the current path's gradients, and its update method is as follows:

[0217]

[0218] in, This represents the moving average gradient value at the current moment; This represents the moving average value from the previous round; This represents the actual gradient value of the current training batch; The sliding coefficient is determined through small-scale training and fitting on the power equipment state samples in this embodiment, such that... The gradient change curve is smooth and converges quickly during training rounds.

[0219] Simultaneously, the momentum cache records the momentum gradient for each path to construct a more inertial parameter update direction, which is calculated as follows:

[0220]

[0221] in, This represents the momentum gradient value at the current moment; This represents the momentum value of the previous cycle; The momentum coefficient is obtained through stability analysis of power distribution line control samples and helps to enhance the inertial response of the path and reduce oscillations.

[0222] The actual update of the path parameters consists of a weighted average gradient and a momentum gradient, and the update formula is as follows:

[0223]

[0224] in, This indicates the parameter update increment for the current training step; The learning rate is set based on the initial parameter tuning optimization of the power dispatch semantic model. These are the weighting coefficients for the moving average and momentum terms, respectively, set in conjunction with the semantic gradient distribution fitting results and convergence speed evaluation.

[0225] After each training round, record the parameter update magnitude of the low-rank path in that round. Training loss With gradient mean The data is then written to the optimizer state set. When the path scheduling index table schedules to the next task category identifier, inactive paths do not participate in the current model graph construction, and do not undergo gradient propagation, parameter updates, or state writing.

[0226] To ensure the stability of path training, a feature stability threshold is set in this embodiment. When the path updates parameters in consecutive training batches If the rate of change of the loss function is greater than 0.03, the learning rate for that path will be reduced to the originally set value. ,Right now And pause parameter updates in the current round to prevent model oscillation.

[0227] Step 6: Receive the input text to be processed, extract the semantic features of the input text, and determine the corresponding task category identifier; select the target low-rank weight path that matches the task category identifier according to the path scheduling index table, perform inference in conjunction with the frozen pre-trained language model backbone structure, and generate the output sequence corresponding to the input text.

[0228] In step 6, after the input text is fed into the pre-trained language model, the following inference process is performed:

[0229] The input text is parsed to extract a semantic index tuple containing power equipment identifiers, state parameter groups, and time series labels.

[0230] The semantic index tuples are mapped to control action types to generate a set of pragmatic structural features corresponding to the input text;

[0231] Based on the combined features of device type, state change trend and control objective in the pragmatic structure feature set, the matching items are queried in the pragmatic structure class index, and the task category identifier to which the input text belongs is determined.

[0232] Based on the task category identifier, retrieve the corresponding path configuration parameter group from the path scheduling index table, and read the path index and the target insertion layer index;

[0233] Based on the target insertion layer index, locate the corresponding linear mapping sub-layer in the backbone structure of the pre-trained language model, and load the low-rank weighted path associated with the path index;

[0234] Based on the currently loaded low-rank weighted path and the backbone structure of the pre-trained language model, forward inference is performed, and the output sequence corresponding to the semantic index tuple of the input text is output.

[0235] Specifically, after receiving the input text to be processed, the system first performs standardized preprocessing operations, including removing redundant punctuation marks, standardizing terminology formats, and normalizing text encoding. Based on this, the embedded structure parsing module is invoked to perform semantic analysis on the input text, extracting semantic index tuples. The semantic index tuples consist of three parts: a device identifier, a set of state parameters, and a time series label.

[0236] Equipment identifier extraction is based on the power system standard naming vocabulary, using a combination of literal matching and alias normalization rules. If abbreviations or ambiguous terms (such as "West Station" or "Main Transformer") appear in the text, they are converted to standard names according to the mapping table provided by the power company's GIS system. For state parameter group extraction, key fields reflecting equipment operating status, such as current, voltage, temperature, frequency, and load rate, are identified through rule matching and dependency parsing techniques. Time series label processing employs a backtracking completion strategy; if the text does not explicitly include time expressions, the current system timestamp is used back 15 minutes as the state occurrence time to construct the temporal semantic chain.

[0237] After extracting the semantic index tuples, they are mapped to the built-in control action knowledge graph to generate a pragmatic structure feature set corresponding to the input text. The feature set includes three indicators: device type, state trend, and control objective. Based on these three combined features, precise matching is performed in the pragmatic structure class index established in step 2. If a match is successful, the corresponding task category identifier is determined; if multiple category identifiers have similar scores (difference less than 0.05), a fuzzy fusion mechanism is triggered to calculate the confidence fusion value of multiple paths.

[0238] The fuzzy fusion mechanism calculates the confidence scores of multiple paths by fusing them. This process first normalizes the confidence scores of each candidate task category path to ensure that the scores of different paths are within the same range. After normalization, a weighted average of the confidence scores is calculated based on the matching degree between the score of each path and its task category, yielding the fused confidence score.

[0239] If the calculated confidence fusion value exceeds the set minimum confidence threshold (0.75), then the path combination will continue to be used for inference.

[0240] If the confidence scores of all candidate category paths are below the preset confidence threshold (0.75), a confidence warning flag is output. A suboptimal path redundancy loading mode is activated; that is, when all current paths fail to reach sufficient confidence, alternative paths related to the current input text are loaded to enhance the credibility of the inference results. This ensures stable and reliable inference output even when the initially selected path has low confidence, avoiding inference failures or inaccuracies due to a single path mismatch.

[0241] After determining the task category identifier, the path scheduling index table is invoked to retrieve the path configuration parameter group corresponding to that identifier. The parameter group includes the path index and the target insertion layer index. Based on the target insertion layer index, the target linear mapping sublayer is located in the pre-trained language model backbone structure, and the low-rank weighted path associated with the path index is loaded. The loading process does not affect the parameter structure of the backbone network; it only activates the low-rank weighted path in parallel within the specified sublayer, keeping the backbone parameters frozen.

[0242] After path loading is complete, the input text is converted into the model input format by the preprocessing encoding module. The jointly activated low-rank weighted paths and the frozen backbone structure are then used for forward inference. The inference output is a natural language output sequence that semantically matches the input text. To ensure that this output is adaptable to different scenarios, the model output sequence is rewritten in a structured manner by calling a semantic output template library based on the task category identifier.

[0243] The semantic output template library stores fixed expression styles for different control task categories. For example, for task category "A01" (main transformer temperature rise - load reduction operation), its output will be formatted as: "[Command Prompt]: An abnormal rise in the main transformer temperature has been detected. It is recommended to appropriately reduce the load to [recommended value] kVA." If the output sequence contains keywords with control intent (such as "cut off", "transfer", "opening and closing", etc.), an operation risk level label will be automatically added, such as "[Level 1 Warning]" or "[For Reference Only]", to indicate the risk of operation execution.

[0244] Ultimately, the output is a natural language text sequence with structured tags, clear semantics, and conforming to scheduling operation habits. The task category identifier, path index, inference confidence level, and control recommendation level are also encapsulated to form a complete output data package for use by subsequent business systems or control personnel.

[0245] Example 2 is an embodiment of the present invention, which provides a large-scale language model tuning system for power systems based on LoRA technology, including:

[0246] The data extraction unit collects historical language data during the operation of the power system.

[0247] The sample construction unit performs the conversion of historical language data into paired training samples containing input and output sequences;

[0248] The task partitioning unit performs task classification based on the semantic features of the training samples and constructs a task dataset;

[0249] The path configuration unit executes the configuration of low-rank weighted paths bound to the task data set and constructs a path scheduling index table;

[0250] The training control unit executes the process of keeping the original parameters frozen in the pre-trained language model and training only low-rank weighted paths.

[0251] The weight generation unit executes the low-rank weight paths bound in the call path scheduling index table to train and form callable weight paths;

[0252] The inference execution unit extracts semantic features from the input text to be processed, determines the task category identifier, performs inference, and generates an output sequence.

[0253] This embodiment also provides an electronic device applicable to the optimization method of large-scale language model of power system based on LoRA technology, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the optimization method of large-scale language model of power system based on LoRA technology proposed in the above embodiment.

[0254] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the method for optimizing a large-scale language model of a power system based on LoRA technology as proposed in the above embodiments.

[0255] The storage medium proposed in this embodiment and the method for optimizing large-scale language models of power systems based on LoRA technology proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0256] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0257] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for optimizing large-scale language models of power systems based on LoRA technology, characterized by: include, Acquire historical language data generated during the operation of the power system, and convert the historical language data into paired training samples containing input sequences and output sequences; Task categories are divided based on semantic features in the training samples, and training samples belonging to the same task category are constructed into a task data set; In the linear mapping sublayer of the pre-trained language model, a low-rank weighted path is configured for each task dataset, and the mapping relationship between the task category identifier and the corresponding low-rank weighted path is recorded to construct a path scheduling index table. Keep all the original parameters in the pre-trained language model frozen, and train only the adjustable parameters in the low-rank weighted path; The training samples from each task dataset are input into the pre-trained language model. The low-rank weighted paths bound in the path scheduling index table are called for forward propagation and parameter update, and independent callable weighted paths corresponding to the task category identifiers are trained respectively. The system receives the input text to be processed, extracts its semantic features, and determines the corresponding task category identifier. Based on the path scheduling index table, it selects the target low-rank weight path that matches the task category identifier, performs inference in conjunction with the frozen pre-trained language model backbone structure, and generates the output sequence corresponding to the input text.

2. The method for optimizing a large-scale language model of a power system based on LoRA technology as described in claim 1, characterized in that: The step of converting the historical language data into paired training samples containing input and output sequences includes, Semantic units are extracted from historical language data. These semantic units include descriptive statements representing device status, instruction statements representing operational intentions, and fragments of contextual information. The extracted semantic units are subjected to format standardization processing, which includes removing invalid punctuation, normalizing entity symbols, and unifying tense structure and subject-verb position; Training samples are constructed based on semantic units after format standardization. The description statement representing the device status is concatenated with context information fragments to form an input sequence, and the instruction expression statement representing the operation intention is used as the corresponding output sequence.

3. The method for optimizing a large-scale language model of a power system based on LoRA technology as described in claim 2, characterized in that: The constructed task data set includes, The input sequences in the training samples are structurally parsed to extract semantic index tuples containing device identifiers, state parameter sets, and time series labels; The semantic index tuples are mapped to the control action types in the corresponding output sequences to generate a pragmatic structure feature set; Based on the combination patterns of device type, state change trend and control objective in the pragmatic structure feature set, the patterns are classified and the pragmatic structure class to which the sample belongs is marked; Based on the preset path interference avoidance rules, training samples with the same pragmatic structure class and semantic gradient distribution divergence rate less than a set threshold are selected. The selected training samples are assigned to the same task category and uniformly labeled according to the pragmatic structure class identifier. The input sequence and output sequence are encoded into the model input format and formed into an indexed structured sample group. The task category identifier is then bound to generate a structured task data set.

4. The method for optimizing a large-scale language model of a power system based on LoRA technology as described in claim 3, characterized in that: The pre-trained language model includes multiple Transformer encoder layers, each of which consists of a multi-head attention mechanism module and a feedforward neural network module. Each multi-head attention mechanism module includes, in sequence, a query vector linear transformation sublayer, a key vector linear transformation sublayer, and a value vector linear transformation sublayer, which are used to construct the multi-head attention representation; Each feedforward neural network module includes, in sequence, a first linear transformation sublayer, an activation function sublayer, and a second linear transformation sublayer; The low-rank weight path is set in the first linear transformation sub-layer of the feedforward neural network module. The low-rank weight path includes a first dimension-reducing linear matrix and a second dimension-increasing linear matrix, which are respectively used to map the original feature dimension to the low-rank feature dimension and restore it to the original feature dimension. The low-rank weight path is connected to the original parameters in the first linear transformation sub-layer in parallel. For each task dataset, a low-rank weighted path configuration parameter group is constructed. The configuration parameter group includes a path index, a set rank value, a target insertion layer index, and a task category identifier, which are used to establish a static binding relationship between the task category identifier and the path structure. All configuration parameter groups for low-rank weighted paths are uniformly stored in the path scheduling index table, which serves as the parameter scheduling basis for mapping and calling the pre-trained language model backbone structure and task category.

5. The method for optimizing a large-scale language model of a power system based on LoRA technology as described in claim 4, characterized in that: The forward propagation and parameter update of the low-rank weighted paths bound in the call path scheduling index table include, Based on the task category identifier recorded in the training samples, retrieve the path configuration parameter group corresponding to the task category identifier from the path scheduling index table, and read the path index and the insertion layer index. Based on the insertion layer index, load the low-rank weighted path associated with the path index in the corresponding linear mapping sublayer of the pre-trained language model backbone structure, and set the low-rank weighted path as the activation path of the current training state. Based on the task category identifier, the training samples are divided into multiple training batch groups; each training batch group is input into the pre-trained language model backbone structure, and forward and backward propagation operations are performed. Only the low-rank weighted paths that match the task category identifier of the current training batch group are activated to participate in gradient calculation and parameter update. The remaining low-rank weighted paths remain inactive during the training batch execution, and gradient propagation and update operations are not performed on all parameters in the inactive paths; All original parameters in the backbone structure of the pre-trained language model remain frozen throughout the training process and do not participate in any gradient calculation or update operations. Each low-rank weighted path is assigned an independent optimizer context, which contains an independent gradient cache, momentum factor, and parameter update state to isolate the training state of each low-rank weighted path. The training rounds are executed for all training batches, and the parameter optimization process of all low-rank weighted paths is completed in sequence, generating optimized low-rank weighted path sets bound to each task category identifier.

6. The method for optimizing a large-scale language model of a power system based on LoRA technology as described in claim 5, characterized in that: Assigning an independent optimizer context to each low-rank weighted path includes... The optimizer state set is initialized based on the task category identifier and bound to each low-rank weight path. The initial weight values ​​and training round indices of all adjustable parameters in the low-rank weight path are recorded in the optimizer state set. Configure a set of backpropagation buffer structures for the optimizer state set. The buffer structures include a gradient cache, an average cache for recording the moving average gradient values, and a momentum cache for storing momentum estimates. Before executing each training batch, the corresponding low-rank weighted path is retrieved from the path scheduling index table by task category identifier, and the associated optimizer state set and backpropagation buffer structure are loaded. Write the backpropagation gradient of the current training batch into the gradient cache, and call the historical records in the average cache and momentum cache to calculate the updated gradient value of the current weight path. Based on the updated gradient value, the adjustable parameters in the low-rank weighted path are iteratively updated, and the updated gradient and momentum value are synchronously written into the optimizer state set. After each training batch, record the parameter variation magnitude, training loss value and gradient mean of the low-rank weight path, and update the corresponding optimizer state set. When the path scheduling index table schedules to the training batch corresponding to the new task category identifier, all inactive low-rank weight paths and optimizer state sets do not participate in the training operation of the current batch, and no parameter reading, gradient calculation or state writing is performed during the round training. When the recorded parameter variation exceeds the feature stability threshold of the training sample corresponding to the task category identifier, the learning rate is switched to the preset lower limit, and the parameter update operation for the current round is paused.

7. The method for optimizing a large-scale language model of a power system based on LoRA technology as described in claim 6, characterized in that: After the input text is fed into the pre-trained language model, the following inference process is performed: The input text is parsed to extract a semantic index tuple containing power equipment identifiers, state parameter groups, and time series labels. The semantic index tuples are mapped to control action types to generate a set of pragmatic structural features corresponding to the input text; Based on the combined features of device type, state change trend and control objective in the pragmatic structure feature set, the matching items are queried in the pragmatic structure class index, and the task category identifier to which the input text belongs is determined. Based on the task category identifier, retrieve the corresponding path configuration parameter group from the path scheduling index table, and read the path index and the target insertion layer index; Based on the target insertion layer index, locate the corresponding linear mapping sub-layer in the backbone structure of the pre-trained language model, and load the low-rank weighted path associated with the path index; Based on the currently loaded low-rank weighted path and the backbone structure of the pre-trained language model, forward inference is performed, and the output sequence corresponding to the semantic index tuple of the input text is output.

8. A large-scale language model tuning system for power systems based on LoRA technology, wherein the method for tuning a large-scale language model for power systems based on LoRA technology as described in any one of claims 1 to 7 is characterized in that, include: The data extraction unit collects historical language data during the operation of the power system. The sample construction unit performs the conversion of historical language data into paired training samples containing input and output sequences; The task partitioning unit performs task classification based on the semantic features of the training samples and constructs a task dataset; The path configuration unit executes the configuration of low-rank weighted paths bound to the task data set and constructs a path scheduling index table; The training control unit executes the process of keeping the original parameters frozen in the pre-trained language model and training only low-rank weighted paths. The weight generation unit executes the low-rank weight paths bound in the call path scheduling index table to train and form callable weight paths; The inference execution unit extracts semantic features from the input text to be processed, determines the task category identifier, performs inference, and generates an output sequence.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for optimizing a large-scale language model of a power system based on LoRA technology as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for optimizing a large-scale language model of a power system based on LoRA technology as described in any one of claims 1 to 7.

Images