Power system data processing method and system based on dynamic period detection

Abstract

The invention relates to the technical field of power system data processing, in particular to a power system data processing method and system based on dynamic period detection, and the method comprises the steps: collecting a historical query log of a power business system, and extracting time sequence data of query features through a sliding window; training a recurrent neural network model by using the time sequence data, and predicting a current optimal query period length; on the basis of the optimal query period length, predicting an index combination required by query through a classification model, and generating a candidate index set; optimizing an execution strategy of the candidate index set by taking the minimum index construction cost and the maximum query coverage rate as targets; and obtaining query data of the power business system in real time, and updating the candidate index set and the execution strategy when the change of the business period characteristic is detected. The objective of the invention is to match the periodic change of the power system data, improve the indexing efficiency and reduce the maintenance cost.

Description

Technical Field

[0001] The present invention relates to the technical field of power system data processing, and in particular to a power system data processing method and system based on dynamic period detection. Background Art

[0002] Core business scenarios in the power industry, such as power trading analysis, load forecasting, and real-time dispatch, require processing massive amounts of high-dimensional, time-sensitive time-series data, placing stringent demands on the real-time and concurrency capabilities of data processing systems. For example, grid load forecasting requires model training on terabytes of historical data within minutes and online inference within milliseconds. Power trading platforms must also support tens of thousands of concurrent queries per second. With the marketization of power markets and the large-scale integration of new energy sources, data volumes are growing exponentially, posing severe performance bottlenecks for traditional relational databases and static index architectures.

[0003] The power industry exhibits multi-scale cyclical characteristics, spanning daily, weekly, monthly, and seasonal periods, and the distribution of query hotspots varies significantly across different periods. Existing static indexing strategies struggle to adapt to the cyclical nature of the power industry, requiring the creation of indexes for all potential query combinations. This leads to a surge in storage overhead, while a single data update triggers the rebuilding of multiple indexes, increasing write latency. Ultimately, maintaining a large number of low-frequency indexes consumes significant system resources and reduces query acceleration, hindering the power system's ability to conduct real-time analysis and optimize decision-making. Summary of the Invention

[0004] In order to match the periodic changes of power system data, improve indexing efficiency and reduce maintenance costs, the present invention provides a power system data processing method and system based on dynamic periodic detection. The technical solutions adopted are as follows:

[0005] The technical solution of the first aspect of the present invention provides a power system data processing method based on dynamic cycle detection, the method comprising:

[0006] Collect historical query logs from the power business system and extract time series data of query features through a sliding window;

[0007] Use time series data to train a recurrent neural network model to predict the current optimal query cycle length;

[0008] Based on the optimal query cycle length, predict the index combination required for the query through a classification model to generate a candidate index set;

[0009] Optimizing the execution strategy of the candidate index set with the goal of minimizing index construction cost and maximizing query coverage;

[0010] Acquire query data from the power business system in real time, and update the candidate index set and execution strategy when changes in business cycle characteristics are detected.

[0011] Furthermore, historical query logs of the power business system are collected, and time series data of query features are extracted through a sliding window, including:

[0012] Split historical query logs based on preset window length and sliding step size;

[0013] Perform structural analysis on the query statements in each window to extract target fields and filter conditions;

[0014] The quantized field uses the frequency distribution to generate a frequency feature vector;

[0015] Quantify the query density feature of the time dimension to generate a time distribution feature vector;

[0016] The frequency feature vector and the time distribution feature vector are combined into a multidimensional time series feature vector.

[0017] Furthermore, we use time series data to train a recurrent neural network model to predict the current optimal query cycle length, including:

[0018] Constructing a recurrent neural network model and inputting the time series feature vector;

[0019] The time dependency is passed through the hidden state vector to calculate the period prediction value:

[0020] Calculate the mean square error between the predicted period value and the marked period value;

[0021] The gradient descent algorithm is used to back-propagate the error and update the network weight parameters to output the optimal query cycle length.

[0022] Furthermore, based on the optimal query cycle length, the index combination required for the query is predicted by the classification model to generate a candidate index set, including:

[0023] Extract all the field combinations involved in the query within the time interval corresponding to the optimal query cycle length;

[0024] Sort by frequency and intercept high-frequency field combinations that meet the preset coverage rate;

[0025] Train a multi-label classification model using query features as input and high-frequency field combinations as labels;

[0026] The trained multi-label classification model is used to predict query index requirements and generate a candidate index set based on the index combination probability distribution.

[0027] Furthermore, with the goal of minimizing index construction cost and maximizing query coverage, the execution strategy of the candidate index set is optimized, including:

[0028] Evaluate the index construction cost of each index in the candidate index set;

[0029] Simulate index hit scenarios based on historical query logs and calculate query acceleration coverage;

[0030] Construct a comprehensive evaluation function to balance index construction cost and query acceleration coverage;

[0031] The execution sequence is iteratively optimized based on the state transfer mechanism to output the optimal execution strategy sequence.

[0032] Furthermore, the execution process of the optimal execution strategy sequence includes:

[0033] Monitor the query hit rate of each index;

[0034] Inefficient indexes are marked based on the preset query hit rate dynamic threshold and removed in the next update cycle.

[0035] Furthermore, query data of the power business system is obtained in real time, and when changes in business cycle characteristics are detected, the candidate index set and execution strategy are updated, including:

[0036] Obtain query data in real time and input it into the updated recurrent neural network;

[0037] When the absolute difference of the change in the length of the prediction period exceeds the preset tolerance threshold, the candidate index set is regenerated;

[0038] Updates the index execution strategy based on the regenerated set of candidate indexes.

[0039] The technical solution of the second aspect of the present invention provides a power system data processing system based on dynamic cycle detection, which adopts the power system data processing method based on dynamic cycle detection described in the technical solution of the first aspect of the present invention, and the system includes:

[0040] A data collection and preprocessing module is configured to collect historical query logs of the power business system and extract time series data of query features through a sliding window;

[0041] A cycle prediction module is configured to train a recurrent neural network model using time series data to predict the current optimal query cycle length;

[0042] a classification module configured to predict the index combination required for the query through a classification model based on the optimal query cycle length and generate a candidate index set;

[0043] a strategy optimization module configured to optimize the execution strategy of the candidate index set with the goal of minimizing index construction cost and maximizing query coverage;

[0044] The sliding update module is configured to obtain query data of the power business system in real time and update the candidate index set and execution strategy when changes in business cycle characteristics are detected.

[0045] The technical solution of the third aspect of the present invention provides an electronic device, which includes: a processor and a memory communicatively connected to the processor; wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor so that the processor can execute the steps of the power system data processing method based on dynamic cycle detection described in the technical solution of the first aspect of the present invention.

[0046] The technical solution of the fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a program for implementing the power system data processing method based on dynamic cycle detection. The program for implementing the power system data processing method based on dynamic cycle detection is executed by a processor to implement the steps of the power system data processing method based on dynamic cycle detection described in the technical solution of the first aspect of the present invention.

[0047] The present invention has the following beneficial effects:

[0048] The power system data processing method based on dynamic cycle detection provided by the present invention utilizes time series data to train a recurrent neural network model, predicts the current optimal query cycle length, and then generates a candidate index set through a classification model based on the cycle characteristics, and then determines the index execution strategy with cost and coverage as the optimization target, and updates the index system in real time when a cycle change is detected. This method can dynamically adapt to the periodic query needs of the power system business, avoid redundant construction and invalid maintenance of static indexes, and thus significantly improve the index's ability to accurately cover the current query hotspots. At the same time, through strategy optimization, it balances the index construction cost and query performance, and realizes the adaptive adjustment of the database index system as the business cycle changes, thereby improving data processing efficiency, reducing storage and maintenance overhead, and enhancing the system's responsiveness to the dynamic needs of the power business. This method can be applied to scenarios such as power trading, load forecasting, and real-time scheduling, and provides a general technical solution for power data storage and analysis in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A flow chart of a method for processing power system data based on dynamic cycle detection provided by one embodiment of the present invention;

[0051] Figure 2 A schematic structural diagram of a power system data processing system based on dynamic cycle detection provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0052] To further illustrate the technical means and effects employed by the present invention to achieve the intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail a method and system for processing power system data based on dynamic cycle detection according to the present invention, including its specific implementation, structure, features, and effects. In the following description, different references to "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics of one or more embodiments may be combined in any suitable manner.

[0053] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0054] The following describes in detail a method and system for processing power system data based on dynamic period detection provided by the present invention with reference to the accompanying drawings.

[0055] See also Figure 1 , which shows a method flow chart of a power system data processing method based on dynamic cycle detection provided by an embodiment of the present invention, the method comprising:

[0056] Step S100: Collect historical query logs of the power business system and extract time series data of query features through a sliding window. Specifically, extract query logs of the past period of time, such as the past 90 days, from the power business system, including query time, fields involved, query frequency and other information. Then, segment the query logs according to the preset sliding window size, which can be expressed as x j , i.e., the query log in the jth sliding window, including the query feature vector in this window;

[0057] Step S100 specifically includes:

[0058] Step S110: Segment the historical query log based on the preset window length and sliding step size. Set the initial window length based on the prior period of the power business, such as day, week, or month, preferably 7 days, and the sliding step size can be set to 1 day to ensure the continuity of the time series data and capture the periodic characteristics. Then, sort the logs in ascending order by timestamp to generate a window set, where each window corresponds to a time series analysis unit.

[0059] Step S120: Perform structured parsing on the query statements in each window to extract target fields and filter conditions. Specifically, a SQL parsing engine may be used to parse the query statements, identifying table names, field lists, and filter fields in clauses. Field names are then standardized to the standard names in the database table structure, excluding irrelevant system fields and retaining core business fields such as "transaction date" and "user ID" to form a field set.

[0060] Step S130: Quantify the frequency distribution of field usage and generate a frequency vector. For each query statement in each window, count the number of times each field appears in the clause to generate a frequency array. Then, normalize the frequency array to highlight the dominance of high-frequency fields and form a frequency vector.

[0061] Step S140: Count the query density in the time dimension and construct a time distribution vector. Specifically, the time axis within the window can be divided into smaller granularities based on hours, weekdays, etc., and the number of queries within each time granularity is counted to generate a time distribution array to reflect the concentration trend of queries in the time dimension.

[0062] Step S150: Merge the frequency vector and the time distribution vector to output a multidimensional time series feature vector; splice the normalized frequency vector and the time distribution vector into a multidimensional feature vector x j ,completely characterize the field usage pattern and time period characteristics within the window;

[0063] For example: set an initial window size W0, such as 7 days, 14 days or 30 days, which can be based on business needs or prior experience. In the initial stage of the present invention, a relatively short period, such as 7 days, can be selected first to quickly obtain the initial value of the model, and all logs are segmented in chronological order, which can be expressed as: Window1 = [t1, t1+W0), Window2 = [t1+Δ, t1+Δ+W0)..., where Window1 represents the first sliding window and Window2 represents the second sliding window; Δ represents the sliding step size, which can be configured to be the same as W0, or it can be set to a smaller size, such as 1 day to form an overlapping window. For each window, the following features are counted: the total number of queries in the window Q count 、The frequency vector f=[f1,f2,…,f n], average execution time, peak execution time and other optional indicators; time series characteristics within the window, such as hourly / daily / weekly query distribution, etc.; these statistical results are combined into a feature vector, recorded as x j , which will serve as one of the inputs of the RNN model; this embodiment segments the historical query logs and combines structured parsing, field frequency quantification, time density statistics and feature fusion operations to capture and structure the periodic features of the power business query pattern, providing multi-dimensional feature input for subsequent periodic prediction and index optimization.

[0064] Step S200: Use a recurrent neural network model to train the time series data and predict the current optimal query cycle length; specifically, for each sliding window x j Label the real cycle The cycle length can be estimated through prior knowledge or experience, such as 7 days, 14 days, 30 days, etc.; RNN or LSTM or GRU variants can be used to train the sequence (x1, x2, ..., x n ) for training and establish the loss function:

[0065]

[0066] Where y j Represents the predicted value of the j-th window period length; represents the true value of the j-th window period length; λ represents the regularization coefficient; λ||θ|| 2 represents the L2 regularization term, θ is the model trainable parameter; through back propagation and iterative optimization, the optimal RNN parameters are obtained, and then the query feature stream x of the real-time input is used during deployment. t Given the period prediction output y t , and finally determine the current optimal period T * =round(y t ), round(y t ) is the rounding function.

[0067] Step S200 specifically includes:

[0068] Step S210: constructing a recurrent neural network model and inputting the time series feature vector;

[0069] Step S220: Calculate the period prediction value by transferring the time dependency through the hidden state vector:

[0070] Step S230: Calculate the mean square error between the predicted period value and the marked period value;

[0071] Step S240: Use the gradient descent algorithm to back-propagate the error and update the network weight parameters, and output the optimal query cycle length.

[0072] Specifically, we first determine the training set and target: by labeling a long period of history, such as the log window data of the past 180 days, we can obtain the most typical cycle length of the business in history; in the power business, common cycles can be given by statistical methods or business personnel experience, such as 7 days, 14 days, 30 days, etc. It is defined as the most frequently occurring period length, or based on logarithmic periodic harmonic analysis, the peak value of the autocorrelation function of the query pattern over a period of time is detected to estimate the optimal period. j Each received one Forming a training set x j Represents the j-th multidimensional feature vector;

[0073] In view of the long time series dependency characteristics of power data, LSTM (Long Short-Term Memory Network) or GRU (Gated Recurrent Unit) is preferred to solve the gradient vanishing problem of traditional RNN. The input sequence is {x1, x2, ..., x n For LSTM, each unit contains a forget gate f t , input gate i t , output gate o t and cell state C t , which can be described by the following formula:

[0074] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0075] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0076]

[0077] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0078]

[0079] Where W f is the weight matrix of the forget gate; h t-1 is the hidden state vector at the previous moment; x t is the current input feature vector; bf is the forget gate bias term; σ is the Sigmoid activation function; is the candidate cell state; W i is the weight matrix of the input gate; W C is the candidate cell state weight matrix; b i is the input gate bias term; b C is the candidate cell state bias term; tanh is the hyperbolic tangent activation function; C t-1 is the cell state at the previous moment; h t is the hidden state at the current moment, which is passed to the LSTM unit at the next moment and serves as the input of the output layer; W o is the output gate weight matrix; b o is the output gate bias term;

[0080] The output layer can be set as a regression layer to predict the cycle length y t , or consider the candidate period (such as {7,14,30}) as a classification label for classification; if the regression form is used, the output layer can be expressed as:

[0081] y t =W hy ·h t +b y

[0082] Where W hy is the output layer weight matrix; b y is the output layer bias term; y t is a continuous prediction value;

[0083] Define the loss function:

[0084]

[0085] Where N is the total number of training samples, that is, the total number of sliding windows; is the square loss of the jth sample, which measures the deviation between the predicted period and the true period; it is iteratively updated using stochastic gradient descent (SGD) or adaptive methods (such as Adam, RMSProp). After training is completed, the real-time (or batch) window data x is deployed. t Input the model and get the predicted output y t The optimal period T can be obtained by round(·) or selecting the maximum probability category * ;

[0086] In actual operation, every period of time (such as every day, every hour), the latest window feature x is collected again t+1, and input the trained RNN. If there is a significant deviation from the previous prediction, for example, the prediction period jumps from 7 days to 14 days, a comprehensive index strategy reconstruction or adjustment is triggered.

[0087] This embodiment realizes the dynamic perception and accurate prediction of the power business cycle through the time series modeling of recurrent neural networks. First, the gating mechanism of LSTM / GRU is used to effectively capture the long-term and short-term cycle dependencies in the power query pattern, avoiding the misjudgment of complex cycles by traditional statistical methods. Then, through the mean square error loss function and regularization constraints, the model automatically learns the mapping relationship between historical query features and cycles, reduces manual labeling deviations, and improves prediction accuracy; finally, through the rolling prediction mechanism, the system can track business cycle changes in real time, maintain prediction accuracy through iterative parameter updates, and avoid index strategy lags caused by static cycle settings. It provides a dynamic time benchmark for subsequent index optimization, so that the database index can accurately match the cycle characteristics of the current query hotspot, thereby improving data processing efficiency and reducing invalid index overhead.

[0088] Step S300: Based on the optimal query cycle length, the classification model is used to predict the index combination required for the query and generate a candidate index set;

[0089] Step S300 specifically includes:

[0090] Step S310: extract all the field combinations involved in the query within the time interval corresponding to the optimal query cycle length; specifically, according to the optimal cycle T output in step S200 * , determine the time range of the current cycle, such as the query logs of the last 7 days, and filter out all query records within this range through the database time range query statement; for each query record, use the SQL parsing engine ANTLR to extract the fields in the clause and generate field combinations; for example, common query fields in power trading may include: transaction date (TRADE_DATE), user ID (USER_ID), electricity price range (PRICE_RANGE), power contract ID (CONTRACT_ID), etc., exclude duplicate combinations and record the number of occurrences.

[0091] Step S320: Sort by frequency and intercept high-frequency field combinations that meet the preset coverage rate; specifically, sort all field combinations from high to low by the number of occurrences, and calculate the cumulative coverage query volume ratio of each combination; the preferred coverage rate threshold is 80%, and the minimum interception number k is determined by prefix sum calculation to obtain the set {I1, I2, ..., I k}, where I i represents the i-th high frequency field combination;

[0092] Step S330: Train a multi-label classification model with query features as input and high-frequency field combinations as labels; extract multi-dimensional features of the query as input: Time features: convert the query time into discrete features such as hours (0-23), weeks (1-7), whether it is a working day (0 / 1), and generate a binary vector through One-Hot Encoding; Business features: parse the business module to which the query belongs, such as "load forecast", "transaction settlement", and user types such as "ordinary user" and "dispatching center", and convert them into category codes; Field features: convert the set of fields involved in the query into a Boolean vector. For each query record, its label is the actual triggered field combination, and supports multi-label output. The model architecture uses a fully connected neural network or a convolutional neural network (CNN). The input layer dimension is the feature vector length. The hidden layer captures nonlinear relationships through the ReLU activation function. The output layer uses the Sigmoid activation function to output the number of high-frequency combinations. Each node represents the predicted probability of the corresponding index combination. The loss function uses multi-label cross entropy loss.

[0093] Step S340: Use the trained multi-label classification model to predict the query index requirements and generate a candidate index set based on the probability distribution of index combinations; specifically, for a new query, extract its feature vector and input it into the trained classification model to obtain the predicted probability vector of each index combination; then set a probability threshold to filter out the index combinations with a predicted probability higher than the threshold, or sort by probability to select the top M max combinations to ensure that the candidate set covers both high-frequency requirements and the number of indexes is controlled. By presetting the maximum number of indexes M max , the candidate set is truncated, and the combination with the highest probability is retained first. For example, if the cumulative coverage of the top three combinations with predicted probability has reached 90%, only the top three combinations are retained and subsequent low-probability combinations are discarded.

[0094] This embodiment achieves the mapping of power query requirements to index combinations through high-frequency field combination statistics and a multi-label classification model: based on the optimal period, it locates the time period when the current business is active, and accurately captures high-frequency query patterns through frequency statistics, avoiding the redundant construction of the full index; using the multi-label classification model, it maps query features to index combination predictions, enabling the system to respond to the index requirements of new queries in real time; this step converts the periodic characteristics of the power business into a specific index optimization strategy, improves the index's ability to accurately adapt to the current query hotspots, and provides an accurate candidate set for subsequent index sequence optimization.

[0095] Step S400: Optimizing the execution strategy of the candidate index set with the goal of minimizing construction cost and maximizing query coverage;

[0096] Step S400 specifically includes:

[0097] Step S410: Evaluate the index construction cost of each index in the candidate index set; specifically, the process includes:

[0098] Time cost build-time : The time required to create an index can be estimated through the database execution plan, including the time required to lock the table and the time required to scan the data;

[0099] Resource Cost build-resource : It can calculate the peak CPU utilization and IO throughput during indexing.

[0100] Comprehensive cost builb : The time and resource costs can be normalized and then weighted summed;

[0101] Step S420: Simulate index hit scenarios based on historical query logs and calculate query acceleration coverage; m , traverse each query in the historical query log, determine whether its field combination matches the field set of any index, if it matches, mark it as "hit", record the acceleration ratio, and then calculate the acceleration coverage;

[0102] Step S430: Construct a comprehensive evaluation function to balance the index construction cost and query acceleration coverage; the objective function can be expressed as:

[0103]

[0104] Where σ is the order of index establishment; Cov(σ) is the cumulative weighted coverage after indexing in the order σ; α is the trade-off coefficient. If α is large, the cost of establishment will be reduced; if α is small, more emphasis will be placed on coverage. build (σ) is the cumulative cost related to the order. For example, when creating the kth index, since the first k-1 indexes already exist, the change in data structure may lead to an increase in the index creation cost;

[0105] Step S440: Iteratively optimize the execution sequence based on the state transition mechanism and output an optimal execution strategy sequence. The execution process of the optimal execution strategy sequence includes:

[0106] Step S441: monitoring the query hit rate of each index;

[0107] Step S442: Mark inefficient indexes according to a preset dynamic threshold of query hit rate, and remove inefficient indexes in the next update cycle; this embodiment can be implemented by using a heuristic algorithm or reinforcement learning. The heuristic algorithm can adopt a greedy strategy to prioritize the establishment of the index with the highest "benefit-cost ratio", where benefit refers to the improvement of coverage and cost refers to the establishment cost; a genetic algorithm can also be used to encode the index order σ as a chromosome, and perform cross-mutation and iteration on the initial population until convergence to obtain the optimal or approximately optimal order; reinforcement learning regards index establishment as a series of actions a k After each action is executed (index is established), the system performance feedback (benefits and costs) is observed and the strategy is updated according to the accumulated rewards; methods such as Q-Learning or Actor-Critic can be used to gradually learn the optimal strategy in an offline or online environment.

[0108] This embodiment optimizes the indexing strategy of the power database through a dynamic balance mechanism between quantitative index construction cost and query coverage. First, based on the cost evaluation and coverage simulation of candidate indexes, a comprehensive objective function with sequence dependency is constructed. The optimal index establishment order is solved through greedy strategies or reinforcement learning to avoid lock conflicts and performance jitter caused by improper order. Secondly, a dynamic monitoring mechanism eliminates inefficient indexes in real time to ensure that the index set always focuses on the high-frequency query requirements of the current cycle. Ultimately, the goal of minimizing resource consumption in exchange for maximizing query performance is achieved, providing reliable support for the high concurrency and low latency requirements of the power business.

[0109] Step S500: Acquire query data of the power business system in real time, and update the index strategy when changes in business cycle characteristics are detected;

[0110] Step S500 specifically includes:

[0111] Step S510: Acquire query data in real time and input it into the updated recurrent neural network; specifically, capture the newly generated query log in real time through the database log monitoring interface or message queue, and parse out key information such as query time, involved fields, and execution time. A new window feature vector is generated using a fixed window length and sliding step size. The feature extraction process is consistent with step S100, including field frequency statistics, time distribution vector construction, etc. The new window feature is input into the RNN model trained in step S200, and the latest cycle prediction value y is output. t+1 ;

[0112] Step S520: When the absolute difference in the change in the length of the forecast period exceeds a preset tolerance threshold, the candidate index set is regenerated; the absolute difference between the current forecast period and the previous period is calculated, with the preset tolerance threshold preferably being 2 days. If the absolute difference is greater than the preset tolerance threshold, it is determined that a significant change has occurred in the business period; when a significant change is detected, the process of step S300 is initiated to regenerate the candidate index set;

[0113] Step S530: Update the index execution strategy based on the newly generated candidate index set. Specifically, the new candidate index set is input into the optimization process of step S400, the construction cost and query coverage of each index are recalculated, and a new optimal construction order is solved through a greedy algorithm or reinforcement learning. The management of the index lifecycle includes:

[0114] Eliminate old indexes: Compare the new and old candidate sets and evaluate the utilization of old indexes that are not included in the new set. If the hit rate is lower than the threshold for multiple consecutive cycles, the index is set to invisible state.

[0115] New index deployment: Create indexes in the new order, using a batch-based indexing strategy. For example, create 1-2 indexes at a time, with an interval of one hour to avoid impacting online services. During indexing, use database monitoring tools to monitor CPU and I / O loads in real time.

[0116] This embodiment ensures that the power data system always evolves in sync with business needs through real-time periodic detection and a dynamic update mechanism for indexing strategies. It upgrades static index management to a dynamic closed loop, enabling database indexes to automatically adjust with periodic fluctuations in the power business. This improves the system's adaptability and long-term operational efficiency, and effectively avoids query performance degradation or resource waste caused by delayed manual intervention.

[0117] In summary, the power system data processing method based on dynamic cycle detection provided by the present invention uses time series data to train a recurrent neural network model to predict the current optimal query cycle length, and then generates a candidate index set through a classification model based on the cycle characteristics, and then determines the index execution strategy with cost and coverage as the optimization target, and updates the index system in real time when the cycle change is detected. This method can dynamically adapt to the periodic query needs of the power system business, avoid the redundant construction and invalid maintenance of static indexes, and thus significantly improve the index's ability to accurately cover the current query hotspots. At the same time, through strategy optimization, it balances the index construction cost and query performance, and realizes the adaptive adjustment of the database index system as the business cycle changes, thereby improving data processing efficiency, reducing storage and maintenance overhead, and enhancing the system's responsiveness to the dynamic needs of the power business. This method is applicable to scenarios such as power trading, load forecasting, and real-time scheduling, and provides a general technical solution for power data storage and analysis in different scenarios.

[0118] See also Figure 2 , which shows a schematic structural diagram of a power system data processing system based on dynamic cycle detection provided by one embodiment of the present invention. The technical solution of the second aspect of the present invention provides a power system data processing system based on dynamic cycle detection, which adopts the power system data processing method based on dynamic cycle detection described in the technical solution of the first aspect of the present invention. The system includes:

[0119] A data collection and preprocessing module is configured to collect historical query logs of the power business system and extract time series data of query features through a sliding window;

[0120] A cycle prediction module is configured to train a recurrent neural network model using time series data to predict the current optimal query cycle length;

[0121] a classification module configured to predict the index combination required for the query through a classification model based on the optimal query cycle length and generate a candidate index set;

[0122] a strategy optimization module configured to optimize the execution strategy of the candidate index set with the goal of minimizing index construction cost and maximizing query coverage;

[0123] The sliding update module is configured to obtain query data of the power business system in real time and update the candidate index set and execution strategy when changes in business cycle characteristics are detected.

[0124] The technical solution of the third aspect of the present invention provides an electronic device, which includes: a processor and a memory communicatively connected to the processor; wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor so that the processor can execute the steps of the power system data processing method based on dynamic cycle detection described in the technical solution of the first aspect of the present invention.

[0125] The technical solution of the fourth aspect of the present invention provides a computer-readable storage medium, on which is stored a program for implementing the power system data processing method based on dynamic cycle detection. The program for implementing the power system data processing method based on dynamic cycle detection is executed by a processor to implement the steps of the power system data processing method based on dynamic cycle detection described in the technical solution of the first aspect of the present invention.

[0126] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0127] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

Claims

1. A power system data processing method based on dynamic period detection, characterized in that: The method comprises: Collect historical query logs from the power business system and extract time series data of query features through a sliding window; Use time series data to train a recurrent neural network model to predict the current optimal query cycle length; Based on the optimal query cycle length, predict the index combination required for the query through a classification model to generate a candidate index set; Optimizing the execution strategy of the candidate index set with the goal of minimizing index construction cost and maximizing query coverage; Acquire query data from the power business system in real time, and update the candidate index set and execution strategy when changes in business cycle characteristics are detected.

2. The power system data processing method based on dynamic period detection according to claim 1, characterized in that: Collect historical query logs from the power business system and extract time series data of query features through a sliding window, including: Split historical query logs based on preset window length and sliding step size; Perform structural analysis on the query statements in each window to extract target fields and filter conditions; The quantized field uses the frequency distribution to generate a frequency feature vector; Quantify the query density feature of the time dimension to generate a time distribution feature vector; The frequency feature vector and the time distribution feature vector are combined into a multidimensional time series feature vector.

3. The power system data processing method based on dynamic period detection according to claim 1, characterized in that: Use time series data to train a recurrent neural network model to predict the current optimal query cycle length, including: Constructing a recurrent neural network model and inputting the time series feature vector; The time dependency is passed through the hidden state vector to calculate the period prediction value: Calculate the mean square error between the predicted period value and the marked period value; The gradient descent algorithm is used to back-propagate the error and update the network weight parameters to output the optimal query cycle length.

4. The power system data processing method based on dynamic period detection according to claim 1, characterized in that: Based on the optimal query cycle length, the classification model is used to predict the index combination required for the query and generate a candidate index set, including: Extract all the field combinations involved in the query within the time interval corresponding to the optimal query cycle length; Sort by frequency and intercept high-frequency field combinations that meet the preset coverage rate; Train a multi-label classification model using query features as input and high-frequency field combinations as labels; The trained multi-label classification model is used to predict query index requirements and generate a candidate index set based on the index combination probability distribution.

5. The power system data processing method based on dynamic period detection according to claim 1, characterized in that: With the goal of minimizing index construction cost and maximizing query coverage, the execution strategy of the candidate index set is optimized, including: Evaluate the index construction cost of each index in the candidate index set; Simulate index hit scenarios based on historical query logs and calculate query acceleration coverage; Construct a comprehensive evaluation function to balance index construction cost and query acceleration coverage; The execution sequence is iteratively optimized based on the state transfer mechanism to output the optimal execution strategy sequence.

6. The power system data processing method based on dynamic period detection according to claim 5, characterized in that: The execution process of the optimal execution strategy sequence includes: Monitor the query hit rate of each index; Inefficient indexes are marked based on the preset query hit rate dynamic threshold and removed in the next update cycle.

7. The power system data processing method based on dynamic period detection according to any one of claims 1 to 6, characterized in that: Acquire query data from the power business system in real time, and update the candidate index set and execution strategy when changes in business cycle characteristics are detected, including: Obtain query data in real time and input it into the updated recurrent neural network; When the absolute difference of the change in the length of the prediction period exceeds the preset tolerance threshold, the candidate index set is regenerated; Updates the index execution strategy based on the regenerated set of candidate indexes.

8. The power system data processing system based on dynamic cycle detection is characterized in that: The power system data processing method based on dynamic period detection according to any one of claims 1 to 7 is adopted, wherein the system comprises: A data collection and preprocessing module is configured to collect historical query logs of the power business system and extract time series data of query features through a sliding window; A cycle prediction module is configured to train a recurrent neural network model using time series data to predict the current optimal query cycle length; a classification module configured to predict the index combination required for the query through a classification model based on the optimal query cycle length and generate a candidate index set; a strategy optimization module configured to optimize the execution strategy of the candidate index set with the goal of minimizing index construction cost and maximizing query coverage; The sliding update module is configured to obtain query data of the power business system in real time and update the candidate index set and execution strategy when changes in business cycle characteristics are detected.

9. An electronic device, characterized in that: The electronic device includes: a processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can perform the steps of the power system data processing method based on dynamic cycle detection as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program for implementing the power system data processing method based on dynamic cycle detection, and the program for implementing the power system data processing method based on dynamic cycle detection is executed by a processor to implement the steps of the power system data processing method based on dynamic cycle detection as described in any one of claims 1 to 7.

Images