A method and system for updating a load identification model based on streaming reinforcement learning label query

Abstract

This invention discloses a method and system for updating a load identification model based on streaming reinforcement learning-based label querying. The method includes: collecting high-frequency voltage and current data of electrical loads and constructing a load identification dataset for pre-training; constructing a load identification model and performing offline pre-training using the pre-training dataset; constructing a load label active query agent based on streaming reinforcement learning; collecting load data online and performing event detection-triggered identification; the agent initiating / not initiating a load label query request based on the estimated confidence of the load identification model, historical query results, historical model update benefits, and the current query budget output; receiving user feedback on load labels and performing fine-tuning training on the load identification model, while simultaneously updating the agent's policy parameters. This invention enables load label querying under query budget constraints, reduces invalid labeling, and improves the online update benefits of the load identification model.

Description

Technical Field

[0001] This invention relates to the technical field of load identification, and in particular to a method and system for updating a load identification model based on streaming reinforcement learning label query. Background Technology

[0002] With the advancement of new power system construction, the types of loads on the distribution side are continuously diversifying, with a large number of distributed power sources, energy storage devices, and various power electronic loads being connected. User-side electricity consumption behavior is exhibiting high-frequency changes and strong time-varying characteristics. Against this backdrop, load identification, as a key foundational capability for refined energy consumption analysis, demand response, fault early warning, and energy efficiency management, requires long-term stable output of electrical load identification results. To meet the requirements of online applications, the load identification model must be continuously updated to adapt to aging electrical appliances, changes in user behavior, scenario migration, and fluctuations in grid operating conditions in order to maintain the stability of load identification performance.

[0003] Currently, existing load identification model update technologies based on streaming reinforcement learning label queries mainly include offline centralized training updates and fixed-strategy active label updates. Offline centralized training updates rely on periodic large-scale data collection and model retraining. The data aggregation, cleaning, labeling, and training processes are lengthy, and model parameter updates lag behind changes in load patterns. When factors such as appliance aging, changes in electricity consumption behavior, addition or replacement of loads, and grid voltage fluctuations cause sudden changes in scenario distribution, the offline update mechanism cannot promptly correct the model after the sudden change, leading to a continuous decline in online identification performance within the update interval. Fixed-strategy active label updates typically initiate load label query requests based on confidence thresholds or uncertainty ranking. The decision-making basis only covers the instantaneous uncertainty of the current sample and does not incorporate historical query results and historical model update benefits into the decision-making basis, resulting in inefficiency of query requests under budget constraints. The resulting labeling costs continue to rise, while the marginal benefit of adding labels to model parameter updates decreases, leading to a high query frequency and low performance gain. Summary of the Invention

[0004] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a load identification model update method and system based on streaming reinforcement learning label query to solve the problems of inefficient querying under cost constraints, unstable performance gains, and inability to converge to the optimal query allocation in existing methods.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, embodiments of the present invention provide a method for updating a load identification model based on streaming reinforcement learning label query, including: High-frequency voltage and current data of electrical equipment in the target power consumption scenario are collected, and a pre-training dataset for load identification is constructed by combining the electrical load labels. The initial load identification model is then trained offline based on the pre-training dataset to obtain the pre-trained load identification model. Construct a load label proactive query agent, configure a decision-making mechanism based on streaming reinforcement learning, and define the agent's state space, action space, and reward function; Based on the pre-trained load identification model, load prediction is performed on the real-time collected electricity consumption data, the current prediction state is extracted and input into the intelligent agent, and the intelligent agent outputs action instructions based on the current strategy. The action instructions include initiating a label query or not initiating a label query. In response to the action command, a tag query is initiated to obtain the real load tag reported by the user. An incremental sample is constructed by combining the current electricity consumption data. The pre-trained load identification model is then fine-tuned online using the incremental sample to obtain the updated load identification model. Based on the performance gain of model fine-tuning and the labeling cost, an immediate reward is generated. The immediate reward, the current state, the action instruction, and the next state are stored in the replay buffer to update the agent's policy parameters.

[0006] As a preferred embodiment of the load identification model update method based on streaming reinforcement learning label query described in this invention, the construction of the load identification pre-training dataset includes: The sampling frequency is set to perform high-frequency sampling of the voltage channel and the current channel with timestamp alignment, and the number of sample points in the preset period is determined based on the power frequency fundamental frequency. The high-frequency sampling sequence is divided into period slices according to the power frequency period. Set the preset event window period number, determine the preset event window sample point number, and generate event detection identifiers based on the moving root mean square difference threshold of the original current sequence. Extract the event segment voltage sequence and event segment current sequence containing load change transients with the event location as the center. Perform data quality control processing to remove event segments that meet the abnormal conditions of voltage or current upper and lower limits, obtain the load labels corresponding to the remaining valid event segments, and write the original data of the event segments, load labels and acquisition parameters into a unified data structure to form the load identification pre-training dataset.

[0007] As a preferred embodiment of the load identification model update method based on streaming reinforcement learning label query described in this invention, the structure and offline pre-training process of the initial load identification model include: The initial load identification model is composed of an input layer, a feature encoding layer, a time-series aggregation layer, and a classification output layer connected in sequence; the input layer concatenates event segment voltage and current data into an input tensor according to the channel dimension. The feature encoding layer adopts a multi-branch one-dimensional convolutional structure. Each branch is set with a different convolutional kernel length to simultaneously encode transient and steady-state waveforms, and the outputs of each branch are concatenated in the channel dimension to form an encoded representation. The temporal aggregation layer uses a bidirectional gated recurrent unit to perform temporal dependency modeling on the encoded representation and outputs an aggregation vector. The classification output layer outputs a class probability vector based on a fully connected layer and a Softmax function. The load labels in the pre-training dataset are encoded into one-bit effective vectors. The initial load identification model is pre-trained offline using a weighted cross-entropy loss function combined with a hierarchical mini-batch sampling method. After training, the model parameters are subjected to fixed-point quantization to obtain the pre-trained load identification model.

[0008] As a preferred embodiment of the load identification model update method based on streaming reinforcement learning label query described in this invention, the construction of the load label active query agent includes: Based on the streaming reinforcement learning algorithm, the agent's state space, action space, and reward function are defined. The state space includes the estimated confidence of the pre-trained load identification model, the statistics of historical query results, the statistics of historical model update revenue, and the current query budget scalar. The action space is to initiate a load label query request or not to initiate a load label query request. The reward function is a linear synthesis of the load identification model performance improvement term and the load labeling cost penalty term.

[0009] As a preferred embodiment of the load identification model update method based on streaming reinforcement learning tag query described in this invention, the load prediction based on the pre-trained load identification model for real-time collected electricity consumption data includes: calculating the moving root mean square sequence based on the real-time collected current data, and using the difference amplitude between adjacent frames as an event intensity index. Set the event trigger threshold and the event suppression duration in frames. When the event intensity index exceeds the event trigger threshold and the interval from the previous trigger frame is not less than the event suppression duration in frames, it is determined to be a load change event trigger frame. Extract the voltage and current sequences of the event window sample points associated with the trigger frame, call the pre-trained load identification model for prediction, and output the category probability vector and the predicted category identifier.

[0010] As a preferred embodiment of the load identification model update method based on streaming reinforcement learning label query described in this invention, the method includes: constructing incremental samples and using the incremental samples to fine-tune the pre-trained load identification model online to obtain the updated load identification model, which includes: When a user feedback label is received, the corresponding event fragment raw data is read and combined with the load label to form an incremental training sample and written into the incremental training buffer. When the number of samples in the incremental training buffer reaches a preset threshold, fine-tuning training is triggered. A fixed number of fine-tuning steps and a fine-tuning learning rate are used to update the model parameters with the weighted cross-entropy loss function as the target.

[0011] As a preferred embodiment of the load identification model update method based on streaming reinforcement learning label query described in this invention, the update of the agent's policy parameters includes: Each time fine-tuning training is triggered, the performance change is obtained by calculating the validation metrics of the same validation sample set before and after fine-tuning, and the actual annotation cost, which includes fixed interaction cost and user response time normalization term, is calculated based on user interaction records. The state vector, query action, performance change, and actual annotation cost are combined into a quadruple and written into the streaming experience replay buffer, and the instant reward is calculated based on the reward synthesis weight. Samples are drawn from the buffer according to the preset agent update cycle. The advantage estimate is constructed using the time difference objective. The policy network parameters are updated using the policy gradient objective with baseline. The value network parameters are updated using the mean square error objective.

[0012] Secondly, the present invention provides a load identification model update system based on streaming reinforcement learning label query, comprising: The offline pre-training module is used to collect high-frequency voltage and high-frequency current data of electrical equipment in the target power consumption scenario, and to construct a load identification pre-training dataset by combining electrical load labels. The initial load identification model is trained offline based on the pre-training dataset to obtain the pre-trained load identification model. The agent configuration module is used to construct an agent that actively queries the load label, configure a decision-making mechanism based on streaming reinforcement learning, and define the agent's state space, action space, and reward function. The real-time decision-making interaction module is used to perform load prediction on the real-time collected electricity consumption data based on the pre-trained load identification model, extract the current prediction state and input it into the intelligent agent, and the intelligent agent outputs action instructions based on the current strategy. The action instructions include initiating a label query or not initiating a label query. The online fine-tuning and update module is used to respond to the action command by initiating a tag query, obtaining the real load tags fed back by the user, constructing incremental samples in combination with the current electricity consumption data, and using the incremental samples to fine-tune the pre-trained load identification model online to obtain the updated load identification model. The policy evolution module is used to generate immediate rewards based on model fine-tuning performance gains and labeling costs. The immediate rewards, current state, action instructions, and next state are stored in the replay buffer to update the agent's policy parameters.

[0013] Thirdly, the present invention provides an electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the load identification model update method based on streaming reinforcement learning label query.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the load identification model update method based on streaming reinforcement learning label query.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention defines the state space, action space, and reward function of the load label active query agent based on a streaming reinforcement learning algorithm. It can achieve deterministic query decision output under query budget constraints, ensuring that the number of load label query requests is consistent with the budget and that the query decision is consistent with the historical update benefits. It linearly synthesizes the performance improvement term of the load identification model and the load labeling cost penalty term into an immediate reward, and writes the performance changes before and after fine-tuning and the actual labeling cost into the streaming reinforcement learning replay data to update the policy parameters of the load label active query agent. This enables the load label active query agent to continuously converge to a high-return, low-cost query behavior during online operation. Based on the load labels and corresponding event fragments fed back by users, incremental training samples are constructed and fine-tuned training is performed on the load identification model to update the model parameters. This enables closed-loop collaboration between load identification model updates and query strategy updates, eliminating invalid labeling and performance fluctuations caused by fixed strategies. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a schematic diagram of the method flow for updating a load identification model based on streaming reinforcement learning tag query, according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the overall process of a load identification model update method based on streaming reinforcement learning label query, as described in one embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0018] Example 1, referring to Figures 1-2 This is one embodiment of the present invention, which provides a method for updating a load identification model based on streaming reinforcement learning label query, including: S100: Collect high-frequency voltage and high-frequency current data of electrical equipment in the target power consumption scenario, and construct a load identification pre-training dataset by combining electrical load labels. Based on the pre-training dataset, perform offline training on the initial load identification model to obtain the pre-trained load identification model. S200: Construct a proactive query agent for load labels, configure a decision-making mechanism based on streaming reinforcement learning, and define the agent's state space, action space, and reward function; S300: Based on the pre-trained load identification model, load prediction is performed on the real-time collected electricity consumption data, the current prediction state is extracted and input into the agent, and the agent outputs action instructions based on the current strategy. The action instructions include initiating a label query or not initiating a label query. S400: In response to the action command, initiate a tag query, obtain the real load tag from the user feedback, construct incremental samples by combining the current electricity consumption data, and use the incremental samples to fine-tune the pre-trained load identification model online to obtain the updated load identification model; S500: Generates immediate rewards based on model fine-tuning performance gains and annotation costs, and stores the immediate rewards, current state, action instructions, and next state in the replay buffer to update the agent's policy parameters.

[0019] It should be noted that electrical loads exhibit different operating states and voltage fluctuation characteristics in different target power consumption scenarios, leading to engineering problems such as performance degradation of the load identification model during long-term operation. Specifically, step S100 obtains high-frequency voltage and current data through the edge terminal device acquisition module and constructs a load identification dataset for pre-training along with load tags. This provides a supervised data foundation covering the target power consumption scenario for the offline pre-training in step S200, ensuring that the pre-trained load identification model has deployable initial identification capabilities. Step S200 forms a deployable pre-trained load identification model based on preset model structure parameters, providing the estimated confidence input of the load identification model for the state space construction in step S300 and providing a stable inference subject for the event-triggered call in step S400. Step S300 uses a streaming reinforcement learning algorithm to uniformly define the state space, action space, and reward function, incorporating the estimated confidence of the load identification model, historical query results, historical model update revenue, and current query budget into the same set of decision variables, enabling the load tag-based active query agent. The output of the query action of initiating / not initiating a load label query request has budget constraint consistency and update benefit consistency; in step S400, the event detection module detects load change events and triggers the call to the load identification model, binding the query action of the load label active query agent to the load change event, thereby limiting online query and online update to event segments with significant information increments and reducing redundant samples entering the update link; in step S500, the load labels fed back by the user are obtained through the user interaction module and incremental training samples are constructed with the corresponding event segments, and fine-tuning training is performed on the load identification model to update the model parameters. At the same time, the performance changes before and after fine-tuning and the actual labeling cost are written into the streaming reinforcement learning replay data to update the policy parameters of the load label active query agent, so that the load identification model update and the query policy update converge synchronously in the same closed loop, completing the online continuous update mechanism.

[0020] This invention uses an offline pre-trained, deployable pre-trained load identification model as its foundation, an event detection module to trigger identification as the online entry point, a load label proactive query agent that outputs whether to initiate a load label query request under query budget constraints as a cost control measure, and user feedback load labels-driven incremental training samples and fine-tuning training as model update methods. Furthermore, it updates policy parameters by incorporating performance changes before and after fine-tuning and actual labeling costs into streaming reinforcement learning replay data, achieving a unified optimization of load identification model performance and load labeling cost constraints. Compared to existing technologies, this solution integrates estimated confidence, historical query results, historical model update benefits, and the current query budget into the decision-making process during online operation, avoiding invalid queries and high costs caused by fixed strategies. By performing query decisions and incremental fine-tuning under load change event triggering conditions, online updates are limited to the event segment range, ensuring high information increment in incremental training samples and reducing redundant updates.

[0021] In this embodiment of the invention, step S100, which involves constructing a pre-training dataset for load identification, includes: The sampling frequency is set to perform high-frequency sampling of the voltage channel and the current channel with timestamp alignment, and the number of sample points in the preset period is determined based on the power frequency fundamental frequency. The high-frequency sampling sequence is divided into period slices according to the power frequency period. Set the preset event window period number, determine the preset event window sample point number, and generate event detection identifiers based on the moving root mean square difference threshold of the original current sequence. Extract the event segment voltage sequence and event segment current sequence containing load change transients with the event location as the center. Perform data quality control processing to remove event segments that meet the abnormal conditions of voltage or current upper and lower limits, obtain the load labels corresponding to the remaining valid event segments, and write the original data of the event segments, load labels and acquisition parameters into a unified data structure to form a load identification pre-training dataset.

[0022] Specifically, high-frequency voltage and current data obtained by the edge terminal device's acquisition module through high-frequency sampling of electrical loads in the target power consumption scenario are used, along with load labels, to construct a load identification dataset for pre-training. The edge terminal device's acquisition module acquires single-phase or three-phase high-frequency voltage and current data at the same measurement point in the electricity meter's metering circuit, setting the sampling frequency to [missing information]. The voltage and current channels are timestamped using a unified clock; in implementation, the sampling frequency... It can be set to 12.8kHz.

[0023] Furthermore, based on the power frequency fundamental frequency identifier Determine the number of sample points in the preset period. Preset number of sample points per period This is used to divide a high-frequency sampling sequence into periodic slices according to the power frequency period, with a preset number of periodic sample points. The calculation is expressed as: in, This indicates the fundamental frequency identifier for power frequency, with a value of 50Hz. Furthermore, the preset event window cycle number is set to... And based on the preset periodic sample point number Determine the number of sample points for the preset event window. Preset number of sample points for the event window Used to extract event segments containing transient load changes, with a preset number of sample points in the event window. The calculation is expressed as: in, This represents the preset event window cycle number, which is a positive integer; Indicates the number of sample points in the preset event window; Based on event detection identifiers Event location is performed on the original sequence, and voltage segments of the event sequence are extracted centered on the event locations. With event fragment current sequence Among them, event detection identifier It is generated based on the moving root mean square difference threshold of the original current sequence, and the generation method is expressed as follows: in, Indicates the discrete sampling point number; Indicates For the end, the window length is The current sliding root mean square value; Indicates the number of sample points in the preset period; Indicates the event determination threshold; This represents the event detection identifier; in implementation, The value is 0.2A; voltage sequence of event segments With event fragment current sequence Perform data quality control processing to remove missing, saturated, and spike noise samples; among these, a voltage upper limit identifier is set. Voltage lower limit identifier Current upper limit identifier With current lower limit identifier When any sampling point in the event segment satisfies or When this happens, the corresponding event fragment is marked as invalid and excluded from the pre-training dataset; in implementation, The value is 0V. The value is 400V. The value is 0A. The value is 100A; Obtain the load label corresponding to each valid event segment and establish a sample-level mapping relationship. The load label is composed of the appliance name and switch status output by the user interaction terminal. The raw voltage data for each valid event segment Raw current data Corresponding load label, sampling frequency Power frequency fundamental frequency identifier Preset event window cycle number Preset event window sample point count The acquisition channel identifier is written into a unified data structure to form a load identification dataset for pre-training.

[0024] In this embodiment of the invention, the structure and offline pre-training process of the initial load identification model in step S100 include: The initial load identification model consists of an input layer, a feature encoding layer, a time-series aggregation layer, and a classification output layer connected sequentially. The input layer concatenates event segment voltage and current data into an input tensor according to the channel dimension. The feature encoding layer adopts a multi-branch one-dimensional convolutional structure. Each branch is set with a different convolutional kernel length to simultaneously encode transient and steady-state waveforms, and the outputs of each branch are concatenated in the channel dimension to form an encoded representation. The temporal aggregation layer uses bidirectional gated recurrent units to model the temporal dependency of the encoded representation and outputs an aggregation vector. The classification output layer outputs a class probability vector based on a fully connected layer and a Softmax function. The load labels in the pre-training dataset are encoded into one-bit effective vectors. The initial load identification model is pre-trained offline using a weighted cross-entropy loss function combined with a hierarchical mini-batch sampling method. After training, the model parameters are subjected to fixed-point quantization to obtain the pre-trained load identification model.

[0025] Specifically, a load identification model is constructed based on preset model structure parameters. The load identification model consists of an input layer, a feature encoding layer, a time-series aggregation layer, and a classification output layer connected sequentially. The input layer receives raw voltage and current data of event segments provided by the pre-training dataset and concatenates them according to the channel dimension to form the model input tensor. In implementation, The value is 2, corresponding to the voltage channel and the current channel; The feature encoding layer employs a multi-branch encoding structure composed of one-dimensional convolution operators to simultaneously encode transient and steady-state waveforms, with the set of convolution kernel lengths set as follows. With the set of convolution channel numbers as The outputs of each branch are then concatenated into an encoded representation along the channel dimension. In implementation, The value is 3. The value is 9. The value is 27. The value is 32. The value is 32. The value is 32; The temporal aggregation layer uses bidirectional gated cyclic units to represent the encoding. Perform time-series dependency modeling, and set the cyclic hidden dimension as . And output the aggregate vector In implementation, The value is 128. The value is 256; The classification output layer consists of a fully connected layer and a Softmax output, with the number of load categories set to [number missing]. and output the class probability vector. ; Encode the payload labels in the pre-training dataset into a one-bit effective vector. The load identification model is pre-trained offline using the cross-entropy loss function, which is expressed as: in, This represents the cross-entropy loss value; Indicates the number of load categories; Represents a one-bit effective vector The Middle One component; Represents the category probability vector The Middle One component; Introducing class weight vectors And construct a weighted cross-entropy loss function and a class weight vector. The weighted cross-entropy loss function, determined by the statistical values ​​of the number of samples in each class of the pre-training dataset, is expressed as: in, This represents the weighted cross-entropy loss value; Represents the category weight vector The Middle One component; Furthermore, a stratified mini-batch sampling method is used to construct training batches to maintain a consistent proportion of each category in each batch, with the batch size set to [value missing]. The initial learning rate is With training rounds And update the model parameters using the Adam optimizer; in implementation, The value is 256. Values , The value is 50; The model parameters, model structure parameters, load class mapping table, and input tensor format description file after offline pre-training are packaged into a unified deployment package. The deployment package is then subjected to fixed-point quantization to form an inference model executable by edge terminal devices. The quantization bit width is set to [value missing]. and floating-point weights Perform symmetric linear quantization to obtain integer weights , is represented as: in, Indicates the quantization bit width; Represents the floating-point weight to be quantized; Indicates the quantized integer weights; Indicates the quantization scale factor; This represents the maximum absolute value of the weight tensor to be quantized; This represents the truncation function; This represents the rounding operator; in practice, The value is 8.

[0026] In this embodiment of the invention, step S200, which involves constructing a proactive query agent for load labels, includes: Based on the streaming reinforcement learning algorithm, the agent's state space, action space, and reward function are defined. The state space includes the estimated confidence of the pre-trained load identification model, the statistics of historical query results, the statistics of historical model update revenue, and the current query budget scalar. The action space is to initiate a load label query request or not to initiate a load label query request. The reward function is a linear synthesis of the load identification model performance improvement term and the load labeling cost penalty term.

[0027] Based on the streaming reinforcement learning algorithm, the agent's state space, action space, and reward function are defined. The state space includes the estimated confidence of the load identification model, historical query results, historical model update benefits, and current query budget. The action space is to initiate or not initiate a load label query request. The reward function includes a load identification model performance improvement term and a load labeling cost penalty term. Specifically, the decision moment index is defined as... And at every decision moment State vectors are constructed from edge terminal devices State vector Estimating the confidence scalar Historical query results statistics Historical model update revenue statistics Compared with the current query budget scalar The result of splicing is represented as: in, Indicates the index of the decision-making time; Indicates the moment of decision The state vector; This represents the estimated confidence level of the load identification model; This represents a vector of historical query result statistics. This represents the historical model update return statistics; Indicates the current query budget; Estimate confidence level Category probability vector output by the load identification model The calculated confidence level, defined by a weighted fusion of the maximum class probability and the probability interval, is expressed as: in, Indicates the moment of decision The category probability vector output by the load identification model; express The One component; Indicates the number of load categories; This represents the estimated confidence level fusion coefficient; Indicates to make The category index that yields the maximum value; Historical query results statistics From the recent The validity and consistency statistics of the sub-tag query constitute the data, where validity is defined as the user's performance within the timeout threshold. Internal return of load labels, consistency is defined as the user-returned load label being the same as the load identification model's predicted category; set the most recent The number of valid returns in this query is The number of times they agree is Then the statistics of historical query results Represented as: in, Indicates the length of the historical statistics window; This indicates the user feedback timeout threshold; Indicates recent The number of valid returns in this query; Indicates recent Number of consistent occurrences in each query; Historical model update revenue statistics The validation metric is determined by the performance difference of the validation set before and after the most recent fine-tuning, and is set as follows: The updated verification metrics are Then the historical model update return statistics Represented as: in, This indicates the validation metrics prior to the most recent minor adjustment update; This indicates the validation metrics after the most recent minor adjustment and update. Current query budget scalar Budget cap identifier Counting the budget already used The calculated current queried budget scalar Represented as the percentage of remaining budget and satisfying , is represented as: in, Indicates the budget cap identifier; Indicates the time up to the decision-making moment Budget count has been used; Indicates the moment of decision The current query budget scalar; in implementation, The value is set to 20 times / day; Define the action space as a binary set. The actions This indicates that a load label query request has been initiated. This indicates that no load label query request will be initiated; in practice, the action... The user interaction module is triggered to send a query pop-up and record the feedback result. Only record the model prediction results; Define the reward function as an instant reward. Instant rewards The load identification model performance improvement term and the load labeling cost penalty term are linearly synthesized, and no labeling cost constraint is imposed for actions without queries; whereby the performance improvement estimate is set as... The labeled cost is The performance weight is Cost weighting Then an instant reward Represented as: in, Indicates the moment of decision Instant rewards; Indicates performance weight; This represents an estimate of the performance improvement; Indicates cost weight; Indicates the cost of labeling; Performance improvement estimate The marginal gain of the model update is estimated from labeled samples. The load label returned to the user when the current event segment is queried is set as... The model's cross-entropy loss for this sample before the update is: The cross-entropy loss after including the sample in fine-tuning and completing one update is: Then the performance improvement estimate Represented as: in, This indicates the payload label returned by the user; This represents the cross-entropy loss before the update; This represents the cross-entropy loss after the update; This represents an estimate of the performance improvement; Labeling cost The query action indicator variable is defined as a combination of query initiation cost and user response cost. When selecting an action hour And when selecting an action hour Set a fixed interaction cost as Set the user response time to And its upper limit is Then label the cost Represented as: in, Indicates the query action indicator variable; Indicates fixed interaction cost; Indicates user response time; Indicates the maximum user response time; This indicates the cost of labeling; in implementation, The value is 1. The value is 30s; Furthermore, a streaming reinforcement learning algorithm based on policy gradients is used to construct a load label active query agent. The load label active query agent includes a policy network and a value network, where the policy network outputs the action probability distribution. Value network output state value estimation , For policy network parameters, These are the parameters for the value network; in implementation, both the policy network and the value network adopt a two-layer fully connected structure, with the hidden layer dimension set to 128. Based on state vector Calculate actions Selection probability Action sampling is performed, with the query budget constraint as a hard constraint. Forced output action In practice, action sampling employs maximum probability selection to ensure deterministic online decision-making. Every decision moment Quadruple Write to streaming experience cache And set the cache capacity to To maintain sample freshness, a first-in, first-out (FIFO) strategy is employed; in implementation, The value is 50000; Constructing an advantage estimator using time-difference objectives And update the strategy network parameters accordingly. The discount factor is set as follows: Then the dominance estimator Represented as: in, Indicates the moment of decision The advantage estimate; Indicates the discount factor; Representing the value network in relation to state Value estimation; Update policy network parameters using a policy gradient objective with baseline. The value network parameters are updated using the mean square error objective. ; where the objective function of the policy network is loss function of value network They are represented as follows: Set the agent update cycle as And from the experience cache in each update cycle The batch size of the sample is The parameters are updated for the samples, and the rewards are adjusted accordingly. Standardization is implemented to eliminate scale differences; in practice, The value is 60s. The value is 256.

[0028] In this embodiment of the invention, step S300 involves predicting the load based on the real-time collected electricity consumption data using a pre-trained load identification model, including: calculating the moving root mean square sequence based on the real-time collected current data, and using the difference amplitude between adjacent frames as an event intensity index. Set the event trigger threshold and the event suppression duration in frames. When the event intensity index exceeds the event trigger threshold and the interval from the previous trigger frame is not less than the event suppression duration in frames, it is determined to be a load change event trigger frame. Extract the voltage and current sequences of the event window sample points associated with the trigger frame, call the pre-trained load identification model for prediction, and output the category probability vector and the predicted category identifier.

[0029] It should be noted that in practical applications, the load voltage and load current data are collected in real time by the edge terminal device; the edge terminal device collects the load voltage and load current data in real time at the same measurement point in the metering circuit.

[0030] Specifically, the event detection module detects load change events to trigger the invocation of the load identification model; the event detection module is based on current segments. Calculate the moving root mean square sequence The difference in amplitude between adjacent frames is used as an indicator of event intensity. Event intensity index Represented as: in, Indicates the frame index; Indicates the first The root mean square value of the current corresponding to the frame; Indicates the first The event intensity index of the frame; root mean square value of current Based on the Frame current segment The calculation yields the following result, which is expressed as: in, Indicates the first Intra-frame Current sample values ​​at each sampling point; Indicates the number of sample points in the frame; Indicates the index of the intra-frame sampling point; Furthermore, set the event trigger threshold to... With event suppression duration and number of frames When satisfied And the interval from the previous trigger frame is greater than or equal to When, define the first The frame is a load change event trigger frame and generates an event trigger identifier. Its determination is expressed as: in, Indicates the event trigger identifier; This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Indicates the first The event intensity index of the frame; Indicates the event trigger threshold; This indicates the frame index of the frame that triggered the previous event; Indicates the event suppression duration in frames; in implementation, The value is 0.2A. The value is 8; when At that time, construct the number of event window sample points associated with the event triggering frame. and from continuous voltage flow With continuous current flow Extracting event fragments from voltage sequences With event fragment current sequence Then, the load identification model is invoked to infer the event fragments and output the category probability vector. With prediction category identifier In implementation, The value is 2560 points; Furthermore, the load tag proactive query agent outputs a query action to determine whether to initiate a load tag query request to the user; Specifically, in each load change event trigger frame The category probability vector output by the edge terminal device based on the load identification model Calculate the estimated confidence level And combined with the statistical data of historical query results Historical model update revenue statistics Compared with the current query budget scalar Constructing state vectors Then the state vector Input load labels to actively query the agent to output query actions. In implementation, the query action Deterministic policy output on edge terminal devices; The load-label proactive query agent performs action determination under the query budget constraint and sets the budget upper limit identifier. Counting the budget already used and generate budget permission identifiers. Budget allows identifiers Represented as: in, Indicates the budget permission identifier; Indicates the frame up to the event trigger. Budget count has been used; Indicates the budget cap identifier; in implementation, The value is set to 20 times / day; when and At that time, the edge terminal device initiates a load tag query request to the user through the user interaction module, carrying the event timestamp. With prediction category identifier As a prompt message; when or At this time, the edge terminal device does not initiate a load label query request and writes the event fragment to the unlabeled cache for subsequent offline verification.

[0031] In this embodiment of the invention, step S400 involves constructing incremental samples and using these incremental samples to fine-tune the pre-trained load identification model online, resulting in an updated load identification model. When a user feedback label is received, the corresponding event fragment raw data is read and combined with the load label to form an incremental training sample and written into the incremental training buffer. When the number of samples in the incremental training buffer reaches a preset threshold, fine-tuning training is triggered. A fixed number of fine-tuning steps and a fine-tuning learning rate are used to update the model parameters with the weighted cross-entropy loss function as the target.

[0032] The user interaction module sends load tag query requests to users and receives load tags from users. Specifically, when the load tag actively queries the intelligent agent, it outputs a query action. At that time, the edge terminal device generates a load tag query request message through the user interaction module. Load tag query request message At least include the event timestamp Estimated category identifier With candidate load label list identifier And send a load tag query request message to the user-side terminal. ; Set the user feedback timeout threshold to And start a timer to receive user feedback on the timeout threshold. User feedback messages returned internally User feedback messages Includes user-selected load labels With confirmation identifier In implementation, The value is 30s. A value of 1 indicates that the user has completed the confirmation. A value of 0 indicates that the user has abandoned the confirmation. Furthermore, for each payload label Read the load tag from the event cache of the edge terminal device. corresponding event segment voltage raw data Raw data of current from event fragments and with load label Event timestamp Together they form incremental training samples ; Load label Mapped to category index And generate a one-bit valid vector ,in Satisfy the first One component is 1 and the rest are 0, and... Write incremental training samples To support oversight and fine-tuning; Set the incremental training buffer size to... Incremental training samples Write to incremental training buffer When the incremental training buffer The sample size reaches A fine-tuning training is triggered periodically; in practice, The value is 64; Fine-tuning training uses a fixed fine-tuning step identifier. With fine-tuning the learning rate identifier The parameters of the load identification model are updated, and the weighted cross-entropy loss function is used as the fine-tuning target; wherein, the fine-tuning batch index is set as... , set the first The loss of the batch sample is The average loss during a single fine-tuning training session is... Represented as: in, This represents the average loss during a single fine-tuning training session; Indicates a fixed number of fine-tuning steps identifier; Indicates the first Weighted cross-entropy loss for batch samples; in implementation, The value is 20. Values ; To avoid catastrophic forgetting caused by fine-tuning, a replay sample pool is constructed from the pre-trained dataset. And adjust according to the playback ratio identifier. Mixed sampling of replay samples and incremental samples, with the number of samples set for each fine-tuning batch. Then the number of incremental samples With the number of replay samples Represented as: in, Indicates the playback ratio identifier; This indicates a fine-tuning of the batch sample size; This indicates the number of incremental samples in the fine-tuning batch; This indicates the number of replay samples in the fine-tuning batch; in practice, The value is 0.5. The value is 128.

[0033] In this embodiment of the invention, updating the agent's policy parameters in step S500 includes: Each time fine-tuning training is triggered, the performance change is obtained by calculating the validation metrics of the same validation sample set before and after fine-tuning, and the actual annotation cost, which includes fixed interaction cost and user response time normalization term, is calculated based on user interaction records. The state vector, query action, performance change, and actual annotation cost are combined into a quadruple and written into the streaming experience replay buffer, and the instant reward is calculated based on the reward synthesis weight. Samples are drawn from the buffer according to the preset agent update cycle. The advantage estimate is constructed using the time difference objective. The policy network parameters are updated using the policy gradient objective with baseline. The value network parameters are updated using the mean square error objective.

[0034] Specifically, at each fine-tuning training trigger, the same set of validation samples is tested both before and after fine-tuning training. Calculate the validation index, and set the validation index before fine-tuning as follows: The validation metrics after fine-tuning are: And calculate the performance change. Performance change Represented as: in, This indicates the amount of performance change before and after fine-tuning; This indicates the validation metrics before fine-tuning; This indicates the validation metric after fine-tuning; in implementation, the macro-average F1 score is selected as the validation metric. Calculate the actual annotation cost based on the user interaction records of this query. Actual labeling cost Due to fixed interaction cost Composed of a normalized term for user response time, the user response time is set as... And its upper limit is And combined with query action indicator variables Obtain the actual annotation cost , is represented as: in, Indicates the query action indicator variable; Indicates fixed interaction cost; Indicates user response time; Indicates the maximum user response time; This indicates the actual labeling cost; in implementation, The value is 1. The value is 30s; Furthermore, the state vector Query action Performance change Compared to actual annotation costs Write streaming reinforcement learning replay data And based on the reward synthesis weight and Calculate the immediate reward used for agent updates Instant rewards Represented as: in, This represents the immediate reward used for agent updates; Indicates performance weight; Indicates cost weight; Construct the next state vector and quadruple Write streaming reinforcement learning replay data ,in Includes updated historical model revenue statistics Compared with the updated current query budget scalar ; According to the preset agent update cycle From streaming reinforcement learning replay data The batch size of the sample is Based on the sample, the policy network parameters of the active query agent with load label are updated. With value network parameters The updated parameters are then written to the intelligent agent parameter area of ​​the edge terminal device to complete the online policy parameter update; in implementation, The value is 60s. The value is 256.

[0035] It should be noted that existing methods lack an online decision-making mechanism that unifies the estimated confidence of the load identification model, historical query results, historical model update benefits, and the current query budget, resulting in inefficient load label queries; they lack an adaptive optimization path that incorporates the performance improvement items of the load identification model and the load labeling cost penalty items into the same reward framework, resulting in the inability to obtain stable performance gains in scenarios with limited labeling costs; and they lack a closed-loop update process that writes the performance changes before and after fine-tuning and the actual labeling costs into the replay data and updates the strategy parameters, resulting in a disconnect between the query strategy and the model update benefits, and the inability to converge to the optimal query allocation under budget constraints during the running process. This invention defines the state space, action space, and reward function of a load label proactive query agent based on a streaming reinforcement learning algorithm. The state space includes the estimated confidence of the load identification model, historical query results, historical model update benefits, and the current query budget. The action space is limited to initiating or not initiating load label query requests, achieving deterministic query decision output under query budget constraints, ensuring that the number of load label query requests is consistent with the budget, and that the query decision is consistent with historical update benefits. The performance improvement term of the load identification model and the load labeling cost penalty term are linearly synthesized into an immediate reward, and the performance changes before and after fine-tuning and the actual labeling cost are written into the streaming reinforcement learning replay data to update the policy parameters of the load label proactive query agent, enabling the load label proactive query agent to continuously converge to a high-return, low-cost query behavior during online operation. Incremental training samples are constructed based on the load labels and corresponding event fragments fed back by users, and fine-tuning training is performed on the load identification model to update the model parameters, realizing closed-loop collaboration between load identification model updates and query strategy updates, eliminating invalid labeling and performance fluctuations caused by fixed strategies.

[0036] Example 2: The above example is an illustrative scheme of a load identification model update method based on streaming reinforcement learning label query. It should be noted that the technical solution of this load identification model update system based on streaming reinforcement learning label query belongs to the same concept as the technical solution of the load identification model update method based on streaming reinforcement learning label query described above. Details not described in detail in the technical solution of the load identification model update system based on streaming reinforcement learning label query in this example can be found in the description of the technical solution of the load identification model update method based on streaming reinforcement learning label query described above.

[0037] This embodiment presents a load identification model update system based on streaming reinforcement learning label query, comprising: The offline pre-training module is used to collect high-frequency voltage and high-frequency current data of electrical equipment in the target power consumption scenario, and to build a load identification pre-training dataset in combination with electrical load labels. The initial load identification model is trained offline based on the pre-training dataset to obtain the pre-trained load identification model. The agent configuration module is used to build an agent that actively queries the load label, configure a decision-making mechanism based on streaming reinforcement learning, and define the agent's state space, action space, and reward function. The real-time decision-making interaction module is used to perform load prediction on real-time collected electricity data based on a pre-trained load identification model, extract the current prediction state and input it into the agent, and the agent outputs action instructions based on the current strategy. The action instructions include initiating a label query or not initiating a label query. The online fine-tuning and update module is used to respond to action commands by initiating tag queries, obtaining the real load tags fed back by users, constructing incremental samples by combining current electricity consumption data, and using the incremental samples to fine-tune the pre-trained load identification model online to obtain the updated load identification model. The policy evolution module is used to generate immediate rewards based on model fine-tuning performance gains and annotation costs. The immediate rewards, current state, action instructions, and next state are stored in the replay buffer to update the agent's policy parameters.

[0038] This embodiment also provides an electronic device applicable to the load identification model update method based on streaming reinforcement learning label query, including: The system includes a memory and a processor. The memory stores computer-executable instructions, and the processor executes these instructions to implement the load identification model update method based on streaming reinforcement learning label query, as proposed in the above embodiments.

[0039] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the load identification model update method based on streaming reinforcement learning label query as proposed in the above embodiments.

[0040] The storage medium proposed in this embodiment and the load identification model update method based on streaming reinforcement learning tag query proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0041] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0042] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for updating a load identification model based on streaming reinforcement learning label query, characterized in that, include: High-frequency voltage and current data of electrical equipment in the target power consumption scenario are collected, and a pre-training dataset for load identification is constructed by combining the electrical load labels. The initial load identification model is then trained offline based on the pre-training dataset to obtain the pre-trained load identification model. Construct a load label proactive query agent, configure a decision-making mechanism based on streaming reinforcement learning, and define the agent's state space, action space, and reward function; Based on the pre-trained load identification model, load prediction is performed on the real-time collected electricity consumption data, the current prediction state is extracted and input into the intelligent agent, and the intelligent agent outputs action instructions based on the current strategy. The action instructions include initiating a label query or not initiating a label query. In response to the action command, a tag query is initiated to obtain the real load tag reported by the user. An incremental sample is constructed by combining the current electricity consumption data. The pre-trained load identification model is then fine-tuned online using the incremental sample to obtain the updated load identification model. Based on the performance gain of model fine-tuning and the labeling cost, an immediate reward is generated. The immediate reward, the current state, the action instruction, and the next state are stored in the replay buffer to update the agent's policy parameters.

2. The load identification model update method based on streaming reinforcement learning label query as described in claim 1, characterized in that, The pre-trained dataset for constructing the load identification includes: The sampling frequency is set to perform high-frequency sampling of the voltage channel and the current channel with timestamp alignment, and the number of sample points in the preset period is determined based on the power frequency fundamental frequency. The high-frequency sampling sequence is divided into period slices according to the power frequency period. Set the preset event window period number, determine the preset event window sample point number, and generate event detection identifiers based on the moving root mean square difference threshold of the original current sequence. Extract the event segment voltage sequence and event segment current sequence containing load change transients with the event location as the center. Perform data quality control processing to remove event segments that meet the abnormal conditions of voltage or current upper and lower limits, obtain the load labels corresponding to the remaining valid event segments, and write the original data of the event segments, load labels and acquisition parameters into a unified data structure to form the load identification pre-training dataset.

3. The load identification model update method based on streaming reinforcement learning label query as described in claim 2, characterized in that, The structure and offline pre-training process of the initial load identification model include: The initial load identification model is composed of an input layer, a feature encoding layer, a time-series aggregation layer, and a classification output layer connected in sequence; the input layer concatenates event segment voltage and current data into an input tensor according to the channel dimension. The feature encoding layer adopts a multi-branch one-dimensional convolutional structure. Each branch is set with a different convolutional kernel length to simultaneously encode transient and steady-state waveforms, and the outputs of each branch are concatenated in the channel dimension to form an encoded representation. The temporal aggregation layer uses a bidirectional gated recurrent unit to perform temporal dependency modeling on the encoded representation and outputs an aggregation vector. The classification output layer outputs a class probability vector based on a fully connected layer and a Softmax function. The load labels in the pre-training dataset are encoded into one-bit effective vectors. The initial load identification model is pre-trained offline using a weighted cross-entropy loss function combined with a hierarchical mini-batch sampling method. After training, the model parameters are subjected to fixed-point quantization to obtain the pre-trained load identification model.

4. The load identification model update method based on streaming reinforcement learning label query as described in claim 3, characterized in that, Constructing a proactive query agent for load labels includes: Based on the streaming reinforcement learning algorithm, the agent's state space, action space, and reward function are defined. The state space includes the estimated confidence of the pre-trained load identification model, the statistics of historical query results, the statistics of historical model update revenue, and the current query budget scalar. The action space is to initiate a load label query request or not to initiate a load label query request. The reward function is a linear synthesis of the load identification model performance improvement term and the load labeling cost penalty term.

5. The load identification model update method based on streaming reinforcement learning label query as described in claim 4, characterized in that, Based on the pre-trained load identification model, load prediction is performed on the real-time collected electricity data, including: calculating the moving root mean square sequence based on the real-time collected current data, and using the difference amplitude between adjacent frames as an event intensity index. Set the event trigger threshold and the event suppression duration in frames. When the event intensity index exceeds the event trigger threshold and the interval from the previous trigger frame is not less than the event suppression duration in frames, it is determined to be a load change event trigger frame. Extract the voltage and current sequences of the event window sample points associated with the trigger frame, call the pre-trained load identification model for prediction, and output the category probability vector and the predicted category identifier.

6. The load identification model update method based on streaming reinforcement learning label query as described in claim 5, characterized in that, Construct incremental samples and use these incremental samples to fine-tune the pre-trained load identification model online, resulting in an updated load identification model including: When a user feedback label is received, the corresponding event fragment raw data is read and combined with the load label to form an incremental training sample and written into the incremental training buffer. When the number of samples in the incremental training buffer reaches a preset threshold, fine-tuning training is triggered. A fixed number of fine-tuning steps and a fine-tuning learning rate are used to update the model parameters with the weighted cross-entropy loss function as the target.

7. The load identification model update method based on streaming reinforcement learning label query as described in claim 6, characterized in that, Updating the agent's policy parameters includes: Each time fine-tuning training is triggered, the performance change is obtained by calculating the validation metrics of the same validation sample set before and after fine-tuning, and the actual annotation cost, which includes fixed interaction cost and user response time normalization term, is calculated based on user interaction records. The state vector, query action, performance change, and actual annotation cost are combined into a quadruple and written into the streaming experience replay buffer, and the instant reward is calculated based on the reward synthesis weight. Samples are drawn from the buffer according to the preset agent update cycle. The advantage estimate is constructed using the time difference objective. The policy network parameters are updated using the policy gradient objective with baseline. The value network parameters are updated using the mean square error objective.

8. A load identification model update system based on streaming reinforcement learning label query, applied to the method described in any one of claims 1-7, characterized in that, include: The offline pre-training module is used to collect high-frequency voltage and high-frequency current data of electrical equipment in the target power consumption scenario, and to construct a load identification pre-training dataset by combining electrical load labels. The initial load identification model is trained offline based on the pre-training dataset to obtain the pre-trained load identification model. The agent configuration module is used to construct an agent that actively queries the load label, configure a decision-making mechanism based on streaming reinforcement learning, and define the agent's state space, action space, and reward function. The real-time decision-making interaction module is used to perform load prediction on the real-time collected electricity consumption data based on the pre-trained load identification model, extract the current prediction state and input it into the intelligent agent, and the intelligent agent outputs action instructions based on the current strategy. The action instructions include initiating a label query or not initiating a label query. The online fine-tuning and update module is used to respond to the action command by initiating a tag query, obtaining the real load tags fed back by the user, constructing incremental samples in combination with the current electricity consumption data, and using the incremental samples to fine-tune the pre-trained load identification model online to obtain the updated load identification model. The policy evolution module is used to generate immediate rewards based on model fine-tuning performance gains and labeling costs. The immediate rewards, current state, action instructions, and next state are stored in the replay buffer to update the agent's policy parameters.

9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the load identification model update method based on streaming reinforcement learning label query as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the load identification model update method based on streaming reinforcement learning label query as described in any one of claims 1 to 7.

Images