A method and apparatus for data verification and backup in a power control center

Abstract

This invention provides a method and apparatus for data verification and backup in a power control center, comprising: acquiring hardware resource information corresponding to the operating system and constructing a thread resource pool; connecting to the power control center's data source, classifying and rating the data source, performing real-time requirement analysis using a parameterized hybrid time algorithm, and obtaining a data task queue with priority marking; monitoring the operating system load status and task queue length, applying an online distributed queue length estimation algorithm for resource evaluation, dynamically creating worker threads and allocating resources; performing parallel data verification using a sub-exponential algorithm on independent sets based on the thread resource allocation scheme; evaluating backup requirements based on the data verification result set, determining the backup strategy, and performing parallel data backup. This invention fully utilizes the characteristics of domestically produced operating systems, achieving efficient verification and backup of power control center data, improving system resource utilization and data security.

Description

Technical Field

[0001] This invention relates to the field of power system automation technology, and in particular to a method and apparatus for data verification and backup in a power control center. Background Technology

[0002] Data verification and backup technology in power control centers is a crucial component of power system automation and information security, directly impacting the safe and stable operation of the power system and data security. With the continuous expansion of power system scale and increasing intelligence, the amount of data that power control centers need to process is growing exponentially, placing higher demands on data verification and backup efficiency.

[0003] Currently, power control centers generally use scheduled tasks or single-threaded processing modes for data verification and backup. For example, they employ a full backup strategy with a fixed time window, or a data integrity verification method based on a simple checksum. These techniques perform reasonably well when processing small amounts of data, but are inefficient when dealing with massive amounts of power data.

[0004] One existing optimization technique employs a priority-based task scheduling mechanism, allocating processing resources according to data importance and combining it with an incremental backup strategy to reduce redundant data transmission. This type of technique uses a single scheduler to manage all checksum and backup tasks, introducing the concepts of task priority and resource quotas into the scheduling logic. However, it fails to fully utilize the characteristics of modern multi-core processor architectures and has insufficient compatibility with domestic operating systems.

[0005] However, this technology has obvious shortcomings: First, a single scheduler is prone to becoming a performance bottleneck, especially under high load conditions; second, the resource allocation strategy is too static and cannot be dynamically adjusted according to the real-time load of the system; third, it fails to make full use of the unique resource management interface provided by the domestic operating system, resulting in poor performance on the domestic platform; and finally, it lacks an effective distributed queue length estimation mechanism, making it difficult to achieve globally optimal task allocation. Summary of the Invention

[0006] The purpose of this invention is to provide a method and apparatus for data verification and backup in a power control center, aiming to solve one or more of the technical problems in the prior art, such as a single scheduler becoming a performance bottleneck, resource allocation strategies being too static, failure to fully utilize the characteristics of domestic operating systems, and lack of an effective distributed queue length estimation mechanism.

[0007] To achieve the above objectives, the present invention provides a method for data verification and backup in a power control center, comprising the following steps:

[0008] Obtain the hardware resource information corresponding to the operating system, calculate the effective resource parameters based on the hardware resource information, construct the thread resource pool, and obtain the system resource capability assessment report;

[0009] Connect to the data source of the power control center, classify the data source by data type and rate its importance, and use a parameterized hybrid time algorithm to perform real-time demand analysis to obtain a data task queue with priority marking;

[0010] Monitor the load status and task queue length of the operating system, apply an online distributed queue length estimation algorithm to evaluate resources, dynamically create worker threads based on the load threshold of the operating system and the backlog of the task queue, and allocate CPU time slices, memory quotas and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation scheme.

[0011] Based on the aforementioned thread resource allocation scheme, the verification task is assigned to the worker thread, and the sub-exponential algorithm on the independent set is used to perform parallel data verification. The verification results of each thread are then summarized to obtain the data verification result set.

[0012] Based on the data verification result set, the backup requirements are assessed, the backup strategy is determined, parallel data backup is performed, and the data backup execution status and results are obtained.

[0013] The present invention also provides a data verification and backup device for a power control center, comprising:

[0014] The system resource assessment module is used to obtain hardware resource information corresponding to the operating system, calculate effective resource parameters based on the hardware resource information, construct a thread resource pool, and obtain a system resource capability assessment report.

[0015] The data priority classification module is connected to the system resource assessment module. It is used to connect to the power control center data source, classify the data source by data type and rate its importance, and use a parameterized hybrid time algorithm to perform real-time demand analysis to obtain a data task queue with priority tags.

[0016] The thread resource allocation module, connected to the data priority division module, is used to monitor the load status and task queue length of the operating system, apply an online distributed queue length estimation algorithm to evaluate resources, dynamically create worker threads based on the load threshold of the operating system and the backlog of the task queue, and allocate CPU time slices, memory quotas and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation scheme.

[0017] The data verification module, connected to the thread resource allocation module, is used to allocate verification tasks to worker threads based on the thread resource allocation scheme, perform parallel data verification using a sub-exponential algorithm on independent sets, and summarize the verification results of each thread to obtain a data verification result set.

[0018] The data backup module, connected to the data verification module, is used to evaluate backup requirements based on the data verification result set, determine backup strategies, perform parallel data backups, and obtain the data backup execution status and results.

[0019] The beneficial effects of this invention are:

[0020] On the one hand, by performing parallel data verification using a sub-exponential algorithm on independent sets, combined with priority-marked task queues and dynamic thread resource allocation, the verification efficiency and backup accuracy of power control center data are significantly improved. On the other hand, by applying an online distributed queue length estimation algorithm to dynamically evaluate system resources, the optimal allocation of CPU time slices, memory quotas, and I / O bandwidth is achieved, effectively solving the resource contention problem in multi-threaded parallel processing. Therefore, this invention fully utilizes the characteristics of domestic operating systems to achieve efficient verification and backup of power control center data, improving system resource utilization and data security. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart of a power control center data verification and backup method provided in an embodiment of the present invention;

[0023] Figure 2 A flowchart for data classification and task prioritization provided in this embodiment of the invention;

[0024] Figure 3 This is a structural diagram of a power control center data verification and backup device provided in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0026] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for data verification and backup in a power control center, including the following steps:

[0027] Step S1: Obtain the hardware resource information corresponding to the operating system, calculate the effective resource parameters based on the hardware resource information, construct the thread resource pool, and obtain the system resource capability assessment report;

[0028] In the implementation of this invention, the system initialization phase first requires acquiring and evaluating available hardware resources. When booting from a domestically developed operating system, hardware resource information is obtained by calling underlying API interfaces, including the GetSystemInfo() function, the sysconf() function, and reading relevant files in the / proc virtual file system. These interfaces accurately obtain key hardware parameters such as the number of processor cores, the clock speed of each core, the total physical memory capacity, available memory space, disk I / O bandwidth, and network interface bandwidth. After obtaining these raw hardware resource parameters, the actual amount of resources available for data verification and backup tasks can be calculated by combining them with the resource usage of other processes in the current operating environment. For example, by analyzing the current average CPU load, memory usage, and I / O wait time, available CPU time slices, allocable memory space, and available I / O bandwidth can be determined. Based on these effective resource parameters, a thread pool manager is pre-created, the minimum and maximum thread count thresholds of the thread pool are set, and inter-thread communication mechanisms and synchronization primitives, such as mutexes, condition variables, and semaphores, are initialized. Finally, the maximum number of concurrent verification and backup tasks under the current hardware conditions is calculated using a resource assessment algorithm, and a system resource capacity assessment report is generated to provide a basis for decision-making in subsequent task scheduling and resource allocation.

[0029] Step S2: Connect to the power control center data source, classify the data source by data type and importance rating, and use a parameterized hybrid time algorithm to perform real-time requirement analysis to obtain a data task queue with priority marking;

[0030] After obtaining the system resource capability assessment report, it is necessary to connect to various data sources of the power control center and classify and rate these data. First, by configuring database connection parameters, API interfaces, or file system paths, connections are made to various data sources such as the SCADA real-time database, historical database, configuration information repository, and waveform recording file system. For each data source, metadata information such as data structure description, update frequency, and access mode is obtained. Subsequently, the data is classified in multiple dimensions according to preset classification rules, including classification by data type (real-time data, historical data, configuration data, etc.), data structure (structured, semi-structured, unstructured), update frequency (high-frequency, medium-frequency, low-frequency), and storage location. After completing the data classification, a sub-index algorithm on independent sets is applied to rate the importance of each type of data. This algorithm first constructs a data dependency graph, analyzes the referencing and being referenced relationships between data, calculates graph theory indicators such as in-degree, out-degree, and centrality of each data node, and then identifies key data nodes. During the rating process, factors such as the data's impact on business, regulatory requirements, data recovery difficulty, and historical query frequency can also be considered. Meanwhile, a parameterized hybrid time algorithm is employed to analyze the timeliness requirements of various data types. This algorithm, based on three core parameters—time window, data decay rate, and business time sensitivity—establishes a timeliness model for each type of data and calculates the maximum permissible delay time. Finally, a weighted algorithm is used to comprehensively calculate the data importance and timeliness requirements, resulting in a comprehensive priority index for each data task. Based on this, a multi-level task queue is constructed, outputting a data task queue with priority tags.

[0031] The power control center uses various data sources, including but not limited to SCADA real-time databases, historical databases, configuration information databases, and waveform recording file systems. First, the configuration database interface is invoked to obtain a list of all registered data sources in the control center, acquiring metadata information such as connection parameters, data structure descriptions, access permissions, and update frequencies. These data sources are then categorized in multiple dimensions according to preset classification rules: by data type (real-time data streams, static configuration data, periodically acquired data, and event-triggered data); by data structure (structured data, semi-structured data, and unstructured data); by update frequency (high-frequency update data, medium-frequency update data, and low-frequency update data); and by data storage location (primary system storage, backup system storage, and remote site storage). For newly discovered unclassified data sources, machine learning algorithms can be applied for feature analysis to attempt to classify them into existing categories or create new categories.

[0032] The parameterized hybrid time algorithm is a method that combines deterministic and stochastic time analysis to more accurately describe the timeliness characteristics of power system data. Based on three core parameters—time window, data decay rate, and business time sensitivity—the algorithm generates a timeliness model through hybrid analysis. The mathematical model constructed by the algorithm can be expressed as:

[0033] E(t) = α·W(t) + β·D(t) + γ·S(t)

[0034] Where E(t) is the timeliness score of the data at time t, W(t) is the time window function, D(t) is the data decay function, S(t) is the business sensitivity function, and α, β, and γ are weight coefficients with α+β+γ=1.

[0035] Different timeliness evaluation standards can be selected for different types of data: For SCADA real-time data, strict time window constraints (usually at the second level) are adopted; for metering data, the corresponding time window (from minutes to hours) is set according to its collection cycle; for configuration data, attention is paid to the synchronization timeliness requirements after data changes; for historical and statistical data, the time constraints of business query needs are mainly considered.

[0036] Step S3: Monitor the load status and task queue length of the operating system, apply an online distributed queue length estimation algorithm to evaluate resources, dynamically create worker threads based on the load threshold of the operating system and the backlog of the task queue, and allocate CPU time slices, memory quotas and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation scheme.

[0037] After obtaining the priority-marked task queues, thread resource allocation needs to be dynamically adjusted based on the current operating system load and task queue status. Key performance indicators such as CPU utilization, memory usage, and I / O wait time are obtained by periodically calling the operating system monitoring API to calculate the current system load level. Simultaneously, an online distributed queue length estimation algorithm is applied to analyze the current length and backlog of each priority task queue, and based on historical data and current trends, predicts the task growth trend over a future period. This algorithm can accurately estimate the queue length of each node in a distributed environment, avoiding the overhead of global synchronization required in traditional methods. Based on the system load status and task queue length estimation results, the expected waiting time of tasks and the remaining system resources are calculated. According to a preset thread creation strategy, it is determined whether to create new worker threads and the number to create. Thread creation is triggered when a high-priority task queue backlog is detected or the task waiting time exceeds a threshold; conversely, the number of active threads may be reduced when the system load approaches its limit or the task queue length decreases significantly. Verification and backup threads are dynamically created by calling thread management interfaces provided by the domestic operating system, such as pthread_create() or specific thread pool management APIs. For threads of different priorities, the operating system's resource control mechanisms, such as cgroups, process / thread priority settings (adjusted by nice values), and CPU affinity settings, allocate different CPU time slices, memory quotas, and I / O bandwidth to them. High-priority threads receive more resource quotas and higher execution priority, ensuring that critical tasks can be processed in a timely manner. Finally, the optimized thread resource allocation scheme is output, including information such as the ID, priority, resource quota, and task allocation for each thread.

[0038] The online distributed queue length estimation algorithm can accurately estimate the current length and growth trend of task queues of various priorities through local observation and statistical models without requiring global synchronization. The algorithm is based on random sampling and Markov models, and its core steps include:

[0039] 1. Randomly sample the local queue length at each node;

[0040] 2. Construct a time series model of the queue length;

[0041] 3. Apply Bayesian inference to estimate the global queue state;

[0042] 4. Combine historical data to predict future cohort growth trends.

[0043] The algorithm has a complexity of O(k·log n), where k is the number of sampling points and n is the queue length, which is significantly better than the O(n) method that requires global synchronization.

[0044] The algorithm is used to estimate the backlog of task queues of each priority level and predict the task growth trend in the future (e.g., the next 30 minutes to 2 hours). Considering business model factors, such as peak and off-peak periods in the power system and regular maintenance periods, the prediction model is adjusted to ensure the rationality of resource allocation.

[0045] Step S4: Based on the thread resource allocation scheme, the verification task is assigned to the worker thread, and the sub-exponential algorithm on the independent set is used to perform parallel data verification. The verification results of each thread are summarized to obtain the data verification result set.

[0046] After obtaining the optimized thread resource allocation scheme, parallel data verification begins. First, verification tasks are assigned to corresponding worker threads based on task priority and thread resource availability, ensuring that high-priority data is processed by high-priority threads. A task-thread mapping table can be maintained to record which worker thread each verification task is assigned to, as well as information such as the task's status, start time, and estimated completion time. For each data verification task, an appropriate verification algorithm is selected based on the data type and importance. For example, a lightweight CRC32 algorithm is used for real-time SCADA data, a high-strength hash algorithm such as SHA-256 is used for critical configuration data, and block hashing and tree hashing structures are used for large historical data files. When processing large datasets, a sub-exponential algorithm on independent sets is applied for optimization. This algorithm first analyzes the logical structure and dependencies of the data, decomposing the large dataset into multiple logical subsets, and then identifies the independent sets within these subsets—that is, sets of data blocks that have no dependencies on each other and can be processed in parallel. Based on the identified independent sets, the verification task is reorganized, splitting the original single large verification task into multiple smaller-granularity subtasks that can be executed in parallel without interfering with each other. Streaming and chunking techniques can be used to control the amount of data loaded into memory each time, reducing memory requirements. Each worker thread executes its assigned verification task in parallel, writing the verification results to a shared result pool. To avoid excessive memory pressure, intermediate result caching and persistence mechanisms can be used; when intermediate verification results reach a certain scale, they are written to temporary storage. After all subtasks are completed, result aggregation processing is initiated, employing a layered merging strategy: first merging within thread groups, then merging across thread groups, and finally, the main thread performs the final merge. The merging process involves not only simple result concatenation but also consistency checks to ensure correct data references across independent sets. Ultimately, a data verification result set is generated, containing overall verification statistics, detailed verification records, a problem list, and performance metrics.

[0047] During the task allocation process, the thread resource allocation scheme is first analyzed to obtain the number of threads available for data verification, the priority of each thread, and resource quota information. A "priority matching" strategy is adopted to assign high-priority tasks to high-priority threads, ensuring that critical data verification tasks receive sufficient computing resources and execution priority. Simultaneously, the current load of each thread is considered to avoid some threads being overloaded while others are idle. This is achieved by calculating metrics such as the current task queue length and the estimated remaining time for currently processed tasks for each thread, maintaining a load balance among threads. Furthermore, the physical storage location of the data and the CPU core location of the processing threads can be considered to optimize data locality and reduce data transmission latency.

[0048] Step S5: Evaluate backup requirements based on the data verification result set, determine backup strategy, perform parallel data backup, and obtain data backup execution status and results.

[0049] After obtaining the data verification result set, the intelligent backup strategy execution phase begins. First, the data integrity status, last backup time, and data change rate within the verification result set are analyzed. Combined with the data importance level, the backup priority and method for each dataset are evaluated. Data found to have problems during verification, data that hasn't been backed up for a long time, or important data with a high change rate can be assigned a higher backup priority. Based on the backup requirement assessment results, a parameterized hybrid timing algorithm is applied to calculate the optimal backup time window. This algorithm considers the timeliness characteristics of the data, system load changes, and storage resource status to determine the best backup execution time for different types of data. Simultaneously, the decision to use a full backup or incremental backup strategy is made based on the data change situation. For data with a high change rate, a full backup may be chosen; while for data with a low change rate but a large data volume, incremental backup is preferred to reduce network transmission and storage overhead. Parameters such as backup compression level, encryption requirements, and storage location can also be determined, generating a detailed backup strategy execution plan. During the backup execution phase, backup tasks are assigned to the corresponding backup threads according to the backup strategy execution plan and thread resource allocation scheme. Establish a backup progress monitoring mechanism to track the execution status of each backup task in real time, including the amount of data processed, the current backup rate, and the estimated completion time. Each backup thread performs data compression, encryption, and transmission operations in parallel, backing up the data to the designated storage location. During the backup process, various anomalies are detected and handled, such as data transmission interruptions, insufficient storage space, or data verification failures. For recoverable anomalies, a retry process is automatically initiated. After backup completion, verification checks are performed on the backup data to ensure its integrity and recoverability. Metadata information such as backup completion time, backup content summary, storage location, and checksum is recorded. Finally, a data backup execution status and result report is output to provide a basis for subsequent data recovery and auditing.

[0050] The following embodiments of the present invention provide examples illustrating the specific implementation methods of the above steps.

[0051] In step S1, the hardware resource information corresponding to the operating system is obtained, and the effective resource parameters are evaluated based on the hardware resource information, including:

[0052] Step S1.1: Obtain the number of processor cores, memory capacity, and I / O bandwidth information through the system API to obtain the original hardware resource parameter set;

[0053] Step S1.2: Based on the original hardware resource parameter set, read the current process usage data, calculate the allocable CPU time slice, available memory space and I / O bandwidth, and obtain the effective resource parameter set;

[0054] Step S1.3: Based on the set of effective resource parameters, set the minimum and maximum thresholds of the thread pool, initialize the thread communication mechanism, and obtain the initial state of the thread resource pool;

[0055] Step S1.4: Based on the initial state of the thread resource pool and the set of effective resource parameters, calculate the number of verification and backup tasks that can be processed concurrently, and output a system resource capacity assessment report.

[0056] In step S1.1, hardware resource information is obtained by calling the API interfaces provided by the domestic operating system. Specifically, the GetSystemInfo() or sysconf() functions are used to obtain CPU-related information, such as the number of processor cores, the clock speed of each core, and the cache size; the total physical memory capacity and currently available memory are obtained through the memory management API; and disk I / O and network interface bandwidth information are obtained by reading files such as / proc / diskstats and / proc / net / dev in the / proc virtual file system. The existence and performance parameters of specific hardware accelerators (such as encryption chips and compression cards) are detected. These raw hardware resource parameters are integrated into a structured parameter set to provide basic data for subsequent resource evaluation.

[0057] In step S1.2, based on the acquired original hardware resource parameter set and the current system operating status, the actual amount of resources available for allocation to the data verification and backup tasks is calculated. By reading files such as / proc / stat and / proc / meminfo, current CPU utilization, memory usage, and I / O wait time are obtained to analyze the resource consumption of other processes in the system. Considering that the power control center typically also runs critical business software such as SCADA systems and historical databases, a certain proportion of resources can be reserved for these core businesses to ensure that data verification and backup tasks do not affect normal business operations. After calculation, the allocable CPU time slice percentage, available memory space size, and available I / O bandwidth are obtained, forming an effective resource parameter set.

[0058] In step S1.3, the thread resource pool is configured and initialized based on the effective resource parameter set. The minimum and maximum thread count thresholds are set according to the number of available CPU cores and the load. The minimum thread count ensures the system always maintains a certain level of parallel processing capability, while the maximum thread count prevents excessive thread creation from leading to excessive thread switching overhead. Inter-thread communication mechanisms, including synchronization primitives such as mutexes, condition variables, semaphores, and message queues, can also be initialized to provide the infrastructure for subsequent parallel task processing. Furthermore, parameters such as thread stack size, thread priority strategy, and thread affinity can be configured to optimize thread execution efficiency. After these configurations are completed, a thread pool manager is created, responsible for thread creation, destruction, and status monitoring, and outputting the initial state of the thread resource pool.

[0059] In step S1.4, the system's task processing capacity is evaluated based on the initial state of the thread resource pool and the effective resource parameter set. By analyzing historical task execution data, a relationship model between resource consumption and task type and data volume is established to predict the resource requirements of different types of tasks. Combined with available resources, the maximum number of concurrently processed verification tasks and backup tasks, as well as the expected task execution efficiency, are calculated under the current conditions. Resource bottlenecks can also be assessed, identifying key resources that may limit system performance (such as CPU, memory, or I / O bandwidth). Finally, a detailed system resource capacity assessment report is generated, including a hardware configuration summary, available resource statistics, concurrent processing capacity assessment, and potential bottleneck analysis, providing a basis for subsequent task scheduling and resource allocation decisions.

[0060] like Figure 2 As shown, in step S2, the data source is classified by data type and its importance is rated, including:

[0061] Step S2.1: Receive data from the power control center, obtain the data structure description and update frequency information, classify the data by type, and obtain the data source classification results;

[0062] Step S2.2: Based on the data source classification results, obtain data access frequency and data dependency information, calculate the data importance index, and obtain the data importance rating result;

[0063] Step S2.3: Based on the data importance rating results, construct a directed dependency graph, perform criticality index calculation, and obtain the criticality of data nodes;

[0064] Step S2.4: Based on the criticality of the data nodes, apply the sub-index algorithm on independent sets to identify key data nodes and output a data classification and rating report.

[0065] In step S2.1, various data sources are connected to the power control center, and their metadata information is obtained. First, connections are established with multiple data sources, such as the SCADA real-time database, historical database, configuration information database, and waveform recording file system, using connection parameters stored in configuration files or databases. For each data source, metadata information such as its data structure description (e.g., database table structure, XML schema definition, file format specifications), data update frequency, data volume, and access interface is obtained. The parallel processing interface provided by the domestic operating system is used to simultaneously process the metadata analysis tasks of multiple data sources, improving classification efficiency. According to preset classification rules, the data is classified in multiple dimensions: by data type, it is divided into real-time data streams (e.g., SCADA real-time measurement point data), static configuration data (e.g., protection device settings), periodically collected data (e.g., metering data), and event-triggered data (e.g., alarm records); by data structure, it is divided into structured data, semi-structured data, and unstructured data; by data update frequency, it is divided into high-frequency update data, medium-frequency update data, and low-frequency update data; and by data storage location, it is divided into main system storage, backup system storage, and remote site storage. For newly discovered unclassified data sources, machine learning algorithms can be applied for feature analysis to attempt to classify them into existing categories or create new ones. Ultimately, data source classification results are generated, including each data source's category identifier, connectivity information, estimated data volume, and access method.

[0066] In step S2.2, based on the data source classification results, the importance of the data is further analyzed. Access frequency statistics for each type of data are obtained, including query counts, modification frequency, and export frequency. This information reflects the degree of data usage in the business. Simultaneously, the dependencies between data are analyzed to identify which data is referenced or depended upon by other data, constructing a data dependency network. Several factors can also be considered to determine data importance: business impact coefficient, assessing the impact of data loss or errors on power business operations; regulatory compliance requirements, determining whether it belongs to a data type that is legally required to be stored; data recovery difficulty, assessing the difficulty of retrieving or reconstructing data if it is lost; historical query frequency, analyzing the frequency with which this type of data has been queried or used historically; and data link criticality, determining whether it is located on a critical path in the entire data flow. By weighting these factors, an importance index between 0 and 100 is calculated for each type of data, and the data is divided into four levels based on the index value: critical (80-100), important (60-79), normal (40-59), and general (0-39). The rating results are output in the form of a data importance rating result table, which includes information such as data source ID, data name, importance index, level classification and rating basis.

[0067] In step S2.3, based on the data importance rating results, a directed graph model of data dependencies is constructed, and criticality metrics are calculated. In this directed graph, nodes represent data entities, and edges represent dependencies or references between data, with the direction of the edges pointing from the dependent to the dependent. Multiple centrality metrics are calculated for each node in the graph: Degree Centrality, which measures the number of direct connections to the node; Closeness Centrality, which measures the average distance from the node to all other nodes; Betweenness Centrality, which measures the frequency with which the node acts as a mediator in the shortest path between other node pairs; and Eigenvector Centrality, which considers the importance of node connections rather than just the number of connections. Global properties of the graph, such as clustering coefficients, connectivity, and diameter, can also be calculated to understand the overall structure of the data dependency network. By synthesizing these metrics, a criticality score is calculated for each data node, identifying key data nodes that occupy a central position in the data dependency network.

[0068] In step S2.4, based on the criticality of data nodes, a sub-exponential algorithm on independent sets is applied to identify the most critical set of data nodes. An independent set refers to a group of non-adjacent nodes in a graph, while the sub-exponential algorithm is an algorithm with time complexity between polynomial and exponential, particularly suitable for finding specific patterns in large-scale graph structures. First, the maximum weight independent set is found in the data dependency graph; that is, the set of nodes with the highest total criticality without violating the independence constraint (no direct connections between nodes). This problem is NP-hard and is solved using approximate algorithms with sub-exponential time complexity, such as branch and bound, local search, or genetic algorithms. By identifying these critical data nodes, the verification and backup of this data can be prioritized, thereby maximizing data security with limited resources. Finally, a detailed data classification and rating report is output, including data classification results, importance rating, criticality analysis, and critical node identification results, providing a scientific basis for subsequent task prioritization.

[0069] In step S3, an online distributed queue length estimation algorithm is applied to perform resource assessment, including:

[0070] Step S3.1: Periodically acquire CPU utilization, memory usage and I / O wait time, calculate the current system load level, and obtain the load status report of the operating system;

[0071] Step S3.2: Based on the load status report of the operating system, analyze the backlog of tasks of each priority, predict the task growth trend, and obtain the task queue length estimation result;

[0072] Step S3.3: Based on the task queue length estimation result, calculate the task waiting time and system resource reserve, determine the thread creation strategy, and obtain the thread creation decision result;

[0073] Step S3.4: Based on the thread creation decision result, call the thread management interface to create worker threads, allocate resource quotas, and output the optimized thread resource allocation scheme.

[0074] In step S3.1, system resource usage is obtained and the current load level is assessed by periodically calling the operating system monitoring API. At preset time intervals (typically a few seconds to tens of seconds), the system-provided performance monitoring interfaces are called, such as obtaining CPU utilization by reading ` / proc / stat`, memory usage by reading ` / proc / meminfo`, and I / O device usage status by reading ` / proc / diskstats` and ` / proc / net / dev`. Several key performance indicators are calculated: CPU utilization, including the percentage of user mode, system mode, I / O wait time, and idle time; memory utilization, including physical memory utilization, swap space utilization, and page scheduling frequency; I / O wait time, including disk read / write latency and network transmission latency; and the average system load, reflecting the overall system busyness. The resource usage of each thread in the thread pool can also be monitored, including CPU time, memory allocation, and the number of I / O operations. Based on this real-time monitoring data, the current system load level is calculated and compared with preset load thresholds (such as 70% CPU utilization and 80% memory utilization) to assess whether the system is in a high-load, normal-load, or low-load state. This information is integrated into an operating system load status report, providing a basis for subsequent resource allocation decisions.

[0075] In step S3.2, based on the operating system load status report and the current task queue status, an online distributed queue length estimation algorithm is applied to analyze the task backlog and predict future trends. In a distributed environment, accurately obtaining the global task queue length usually requires expensive synchronization operations. However, the online distributed queue length estimation algorithm used in this invention can accurately estimate the current length and growth trend of each priority task queue through local observation and statistical models without requiring global synchronization. First, local observation data of each priority queue is collected, including the enqueue rate, dequeue rate, and current queue length. Then, time series analysis methods, such as exponential smoothing, ARIMA models, or machine learning algorithms, are applied to predict the task growth trend over a future period (e.g., the next 30 minutes to 2 hours) based on historical data. Business model factors, such as peak and off-peak periods in the power system and regular maintenance periods, can also be considered to adjust the prediction model. Through this method, the current backlog and future growth trend of each priority task queue can be accurately estimated, and the task queue length estimation results, including indicators such as the current queue length, predicted growth rate, and queue saturation, can be output.

[0076] In step S3.3, based on the task queue length estimation results and system load status, key performance indicators are calculated and a thread creation strategy is determined. First, the expected waiting time for each priority task is calculated, i.e., the estimated time from when a task enters the queue to when it begins execution; this indicator directly affects the timeliness of data processing. Simultaneously, the current system resource availability is assessed, i.e., the ratio of the remaining available resources (CPU, memory, I / O) to the total capacity. According to the preset thread management strategy, a balance is found between task waiting time and system resource availability. When the expected waiting time for a high-priority task exceeds a threshold (e.g., critical tasks should not wait more than 5 seconds), and system resources are sufficient, a new worker thread is created; when the task queue is empty or system resources are close to saturation, the number of active threads is reduced. The overhead of thread creation and destruction can also be considered to avoid frequent changes in the number of threads. Based on these analyses, a thread creation decision is generated, including the number of threads to be created or destroyed, thread priority settings, and resource quota allocation recommendations.

[0077] In step S3.4, based on the thread creation decision, the operating system's thread management interface is invoked to dynamically adjust the number of worker threads and allocate resources. New worker threads are created using the `pthread_create()` function or a specific thread pool management API, or the number of threads is reduced by sending termination signals to existing threads. For newly created threads, their priority, CPU affinity, and resource limits are set. Utilizing resource control mechanisms provided by the domestic operating system, such as cgroups (control groups), process / thread priority settings (via nice values ​​or real-time priorities), and CPU affinity settings (binding threads to specific CPU cores), different resource quotas are allocated to threads of different priorities. High-priority threads typically receive higher CPU time slice allocations, larger memory quotas, and higher I / O bandwidth priorities. Load balancing among worker threads ensures that tasks are evenly distributed across threads, preventing some threads from being overloaded while others are idle. Finally, the optimized thread resource allocation scheme is output, including detailed information such as each thread's ID, priority, resource quota, CPU affinity settings, and current task allocation status, providing a foundation for subsequent parallel data verification and backup task execution.

[0078] In step S4, a sub-exponential algorithm on independent sets is used to perform parallel data verification, including:

[0079] Step S4.1: Analyze the logical structure and dependencies of the data, decompose the dataset into multiple logical subsets, construct a data dependency graph based on the reference relationships and update sequence between data, and obtain a data decomposition scheme;

[0080] Step S4.2: Based on the data decomposition scheme, identify the set of data blocks that have no mutual dependencies, determine the independent set partitioning strategy, and obtain the independent set partitioning result;

[0081] Step S4.3: Based on the independent set partitioning results, perform verification task reorganization on each independent set, set parallel processing parameters, and obtain the subtask partitioning scheme;

[0082] Step S4.4: Based on the subtask partitioning scheme, establish a streaming processing mechanism, control the data loading granularity, and obtain the verification execution plan;

[0083] Step S4.5: According to the verification execution plan, call the verification algorithm to perform parallel data verification, record the verification process data, and output the independent set verification results.

[0084] In step S4.1, the logical structure and dependencies of the data to be verified are first analyzed, decomposing the large dataset into smaller logical subsets. The internal structure and organization of the data are understood by analyzing the data schema definition, database table structure, or file format specifications. For structured data (such as relational databases), foreign key relationships, trigger dependencies, and view dependencies between tables are identified; for semi-structured data (such as XML or JSON files), nesting and referencing relationships between elements are analyzed; for unstructured data (such as waveform recordings), the organization of internal data blocks is identified according to file format specifications. The update sequence of the data can also be considered to analyze which data needs to be processed in a specific order to ensure consistency. Based on these analyses, a data dependency graph is constructed, where nodes represent data entities (such as database tables, configuration files, or data blocks), edges represent dependencies, and the direction of the edges points from the dependent to the dependent. Graph partitioning algorithms, such as the minimum cut set algorithm or spectral clustering algorithm, are applied to decompose the dataset into multiple logical subsets, minimizing dependencies between subsets. Finally, a data decomposition scheme is generated, including subset partitioning results, dependencies between subsets, and suggested processing order.

[0085] In step S4.2, based on the data decomposition scheme, independent datasets that can be processed in parallel are further identified. In graph theory, an independent set is a group of non-adjacent nodes in a graph, meaning there are no direct edges connecting these nodes. In a data dependency graph, an independent set represents a group of data entities that have no dependencies on each other, and these data can be processed in parallel without causing consistency issues. Finding the maximum independent set is an NP-hard problem. Heuristic or approximate algorithms, such as greedy algorithms, local search, or genetic algorithms, are used to find a better independent set partition within an acceptable time. The physical storage location of the data can also be considered, and data stored on the same physical device should be allocated to the same independent set as much as possible to reduce data transmission overhead. Different independent set partitioning strategies are used for different types of power data: for large historical databases, independent sets can be partitioned by time dimension (year, month, day) or spatial dimension (region, substation, equipment); for configuration databases, independent sets can be partitioned by functional module or configuration object type; for large waveform file sets, independent sets can be partitioned by equipment type or time window. The independent set partitioning results are output, including a member list, size estimate, and processing priority suggestions for each independent set.

[0086] In step S4.3, based on the independent set partitioning results, the verification tasks within each independent set are reorganized and their parameters are set. First, the data characteristics of each independent set are evaluated, including data size, data type complexity, and verification algorithm requirements. Then, the verification tasks for each independent set are further decomposed into multiple smaller-granularity subtasks, which are the basic units actually allocated to worker threads. The granularity setting of subtasks needs to balance parallelism and scheduling overhead: too high granularity limits parallel processing capabilities, while too low granularity increases task scheduling overhead. The granularity of subtasks is adaptively adjusted according to data characteristics and system resources, typically controlling the execution time of subtasks to between a few seconds and tens of seconds. For each subtask, parallel processing parameters are set, including the number of threads, memory allocation limits, I / O buffer size, and verification algorithm selection. Priorities can also be assigned to subtasks to ensure that verification tasks for important data are executed first. Finally, a subtask partitioning scheme is generated, including the data range, resource requirements, execution parameters, and priority settings for each subtask.

[0087] In step S4.4, a streaming processing mechanism is established based on the subtask partitioning scheme to control the data loading granularity and generate a detailed verification execution plan. For large datasets, loading all data into memory at once for processing is usually impractical; a streaming processing method is needed to load and process data in batches. By adopting a data pipeline mechanism that includes stages such as data reading, preprocessing, verification calculation, and result output, each stage can be executed in parallel, improving processing efficiency. The amount of data loaded in each batch (data block size) is determined based on the available memory size and data characteristics, typically controlled between a few MB and several hundred MB. For extremely large datasets, disk-based external sorting and verification algorithms can be used to further reduce memory requirements. A data prefetching mechanism can also be used, i.e., preloading the next data block while processing the current data block, reducing I / O wait time. For data that needs to be accessed multiple times, a caching mechanism is used to avoid repeated readings. Based on these optimizations, a detailed verification execution plan is generated, including information such as data loading strategy, processing flow, resource allocation, and exception handling mechanisms.

[0088] In step S4.5, according to the verification execution plan, the corresponding verification algorithm is invoked to perform parallel data verification, and the verification process data is recorded. First, based on the task-thread mapping relationship, subtasks are assigned to worker threads. Each worker thread performs verification operations on its assigned data subset according to the specified verification algorithm and parameters. Different verification algorithms are selected based on data type and importance: for real-time SCADA data, lightweight CRC32 or Adler-32 algorithms are typically used; for critical configuration data, SHA-256 or higher-strength hash algorithms are used; for large historical data files, block hashing and tree hash structures are used; for structured databases, in addition to file-level verification, record-level and field-level consistency verification are also performed. During execution, threads dynamically adjust their processing strategies based on the actual data situation, such as performing finer-grained verification on data areas where anomalies are detected. Threads also periodically report execution progress and resource usage, facilitating system monitoring and adjustments. A robust exception handling mechanism can capture and classify various abnormal situations, such as data access errors, verification algorithm execution anomalies, or insufficient resources. For recoverable exceptions, a retry process is automatically initiated; for unrecoverable severe exceptions, detailed information is recorded and the upper-level management module is notified. Each thread writes the verification results to a shared result pool, including checksum / hash value, processing time, and exception records. Finally, independent set verification results are output, providing input for subsequent result aggregation.

[0089] In step S4, the verification results of each thread are summarized, including:

[0090] Step S4.6: Monitor the verification progress of each worker thread, collect intermediate verification results, establish a result cache, and obtain a thread verification status report;

[0091] Step S4.7: Based on the thread verification status report, perform result merging within the thread group, verify data consistency, and obtain the merged result within the group;

[0092] Step S4.8: Based on the intra-group merging result, perform cross-thread group result merging, check the integrity of reference relationships, and obtain the global merging result;

[0093] Step S4.9: Based on the global merging results, generate verification statistics and a problem list, evaluate verification efficiency, and obtain a performance analysis report;

[0094] Step S4.10: Based on the performance analysis report, store the execution results hierarchically, establish a verification problem index, and output a data verification result set.

[0095] In step S4.6, the verification progress of each worker thread is monitored in real time, and intermediate verification results are collected. Through inter-thread communication mechanisms, such as shared memory, message queues, or condition variables, the execution status of each thread is periodically obtained, including the amount of data processed, the current processing rate, the estimated completion time, and resource usage. Threads are checked for abnormal states, such as prolonged unresponsiveness, abnormal resource usage, or frequent errors. For each thread's intermediate verification results, they are stored in a result cache, a temporary in-memory storage area for storing verification results that have not yet been finalized. To avoid excessive memory pressure, when the cache reaches a preset threshold, some results are written to a temporary file or database for later processing. The efficient IPC (Inter-Process Communication) mechanism and memory-mapped file technology provided by the domestic operating system are used to optimize the transmission and storage efficiency of intermediate results. Based on the collected information, a thread verification status report is generated, containing detailed information such as each thread's ID, current task, progress percentage, processing rate, resource usage, and exception records.

[0096] In step S4.7, based on the thread verification status report, the results within the thread group are merged. Worker threads are organized into multiple thread groups, each responsible for processing a related subset of data. Within each thread group, it is first checked whether all threads have completed their assigned tasks or whether the preset merge trigger conditions have been met (e.g., more than 80% of tasks have been completed). Then, all intermediate results for that thread group are read from the result cache, and the group-wide merge operation is performed. The merge process is not just a simple result concatenation; it also includes data consistency verification to ensure correct data reference relationships and consistent data versions within the group, and to resolve potential boundary conflicts. For problems discovered during the verification process, detailed information is recorded, including the problem type, location, severity, and possible causes. Statistical indicators for the group-wide verification are calculated, such as verification pass rate, problem density, and processing efficiency. After completing the group-wide merge, the group-wide merge result is generated, containing the merged verification data, consistency verification results, and statistical indicators.

[0097] In step S4.8, based on the intra-group merging results of each thread group, cross-thread group result merging is performed to form a globally consistent verification result. First, the merging order between thread groups is determined, usually based on data dependencies, ensuring that dependent data is merged first. Then, cross-group merging is performed progressively, integrating multiple intra-group merging results into a larger result set. During cross-group merging, the integrity of cross-group data reference relationships is carefully checked to ensure that the reference relationships between data processed by different groups are correct. For example, checking whether foreign key references are valid, whether dependencies between configuration items are satisfied, and whether the continuity of time-series data is maintained. Cross-group data conflict issues can also be resolved, such as the handling strategy when different groups produce different verification results for the same data item. For any discovered cross-group issues, detailed information is recorded and their impact is assessed. Finally, a global merge result is generated, containing verification results for all data, cross-group reference relationship verification results, and global consistency assessment information.

[0098] In step S4.9, based on the global merge results, detailed verification statistics and a problem list are generated, and verification efficiency is evaluated. Several statistical indicators are calculated: total amount of data processed, including the number of records, files, or data size; verification pass rate, i.e., the percentage of data items that pass verification out of the total; problem distribution, statistically analyzing the number of problems by problem type, severity, and data category; and verification coverage, i.e., the proportion of data items actually verified out of the total number that should be verified. A detailed problem list can also be generated, containing detailed information for each discovered problem: problem ID, problem type (e.g., data corruption, inconsistency, missing data), problem location (e.g., database table name and record ID, file path and offset), severity (usually divided into four levels: fatal, severe, warning, and alert), discovery time, and possible cause analysis. For verification efficiency, several performance indicators are evaluated: total processing time, average processing rate (data volume processed per second), resource utilization (average utilization of CPU, memory, and I / O), parallel efficiency (the ratio of actual speedup to theoretical speedup), and bottleneck analysis (identifying the main factors limiting performance). Based on these analyses, a performance analysis report is generated, containing detailed statistics, performance indicators, and optimization suggestions.

[0099] In step S4.10, based on the performance analysis report, the execution results are stored hierarchically, a verification issue index is established, and a data verification result set is finally output. A hierarchical storage strategy is adopted, storing verification results in different storage tiers according to the importance and access frequency of the data: key summary information is stored in memory for immediate access; detailed verification results are stored in local high-speed storage (such as SSDs) to support efficient querying; historical verification data and original verification logs are stored in high-capacity storage devices for long-term archiving. According to preset rules, the verification results are compressed and encrypted to reduce storage space usage and protect sensitive information. For discovered data issues, a multi-dimensional index is established to support quick querying and filtering by issue type, severity, data category, and time range. Based on the configuration, corresponding processing flows are automatically triggered: for critical and severe issues, alarms may be generated to notify the administrator; for issues that can be automatically repaired, a repair process may be initiated; for issues requiring manual intervention, work orders may be generated and assigned to relevant personnel. Finally, a complete data verification result set is output, including overall verification statistics, detailed verification records, an issue list, performance analysis, and follow-up suggestions, providing key input for the subsequent execution of intelligent backup strategies.

[0100] In step S5, the backup requirements are assessed and the backup strategy is determined, including:

[0101] Step S5.1: Analyze the data verification result set, obtain the data integrity status and change information, calculate the data change rate, and obtain the data status assessment result;

[0102] Step S5.2: Based on the data status evaluation results, obtain the last backup time and storage space information, evaluate the backup priority, and obtain a backup priority list;

[0103] Step S5.3: Based on the backup priority list, apply the parameterized hybrid time algorithm to calculate the backup time window, select the backup method, and obtain the initial backup plan;

[0104] Step S5.4: Based on the initial backup scheme, assess the storage resource status, optimize the compression parameter settings, and obtain a storage strategy scheme;

[0105] Step S5.5: Based on the storage strategy scheme, generate a backup task description, allocate storage locations, and output a backup strategy execution plan.

[0106] In step S5.1, the data verification result set is analyzed to assess the data integrity status and changes. First, integrity status information for each type of data is extracted from the verification result set, including whether the data is complete, whether there are any corruption or inconsistencies, and the severity of the problems. For data with identified problems, the impact on data availability is assessed to determine if emergency backup or recovery is necessary. Change information is analyzed by comparing the current data with the data state at the time of the last verification or backup to identify which data has changed and the extent of the change. The data change rate is calculated, which is the proportion of changed data items to the total number of data items. This metric is crucial for choosing between a full backup or incremental backup strategy. Different change detection methods are used for different types of data: for structured data, modification timestamps or version numbers can be compared; for file system data, file modification times and sizes can be compared; for content-sensitive data, hash values ​​or checksums can be compared. Change patterns and trends are analyzed, such as whether changes are concentrated in a specific area or scattered throughout the entire dataset, and whether the change frequency is stable or fluctuating. Based on these analyses, a data status assessment result is generated, including information such as the integrity status, change rate, change pattern, and problem impact assessment for each type of data.

[0107] In step S5.2, based on the data status assessment results, combined with historical backup information and storage resource status, the backup priority of each dataset is evaluated. First, information such as the last backup time, backup type (full or incremental), and backup result for each data type is obtained from the backup metadata database. The backup interval for the data is calculated, i.e., the difference between the current time and the last successful backup time, and compared with the recommended backup cycle for that data type. The status of the storage system is queried to obtain information such as available storage space, storage device performance, and backup media type, assessing the adequacy of storage resources. Based on this information, a multi-factor scoring model is used to calculate the backup priority score for each data type, considering the following factors: data importance level (from the data importance rating results in step S2); data integrity status (from the assessment results in step S5.1, data with problems receives higher priority); data change rate (data with a high change rate typically requires more frequent backups); the ratio of the backup interval to the recommended cycle (data exceeding the recommended cycle receives higher priority); and storage resource status (when storage resources are scarce, the backup priority of low-importance data may be reduced). Based on priority scores, all datasets are sorted to generate a backup priority list, which includes information such as the ID, name, priority score, recommended backup type, and recommended backup time for each dataset.

[0108] In step S5.3, based on the backup priority list, the parameterized hybrid time algorithm is applied to calculate the optimal backup time window and select an appropriate backup method. The parameterized hybrid time algorithm is a method that combines deterministic and stochastic time analysis, enabling a more accurate description of the timing characteristics of power system data backups. This algorithm is based on three core parameters: a time window parameter, defining the time period for which backups can be performed; a data decay parameter, describing the pattern of data value change over time; and a system load parameter, reflecting the load changes of the system in different time periods. First, historical system load data is analyzed to identify time periods with lower loads, which are suitable for performing resource-intensive backup tasks. Then, the data's business usage patterns are considered to avoid performing backups during periods of intensive data access or updates. The time-varying characteristics of network bandwidth and storage system performance can also be considered to select the time window with the optimal resource conditions. The selection of the backup method is mainly based on the data change rate and the type of the last backup: for data with a high change rate (e.g., exceeding 30%), a full backup is usually chosen; for data with a low change rate but a large data volume, incremental backups are preferred; for data that has not undergone a full backup for a long time, even with a low change rate, a full backup may be chosen to prevent the complexity of recovery caused by an excessively long incremental chain. Data consistency requirements can also be considered. For correlated datasets with strong consistency requirements, simultaneous backups may be chosen to ensure consistency. Based on these analyses, a backup time window and backup method are determined for each dataset, generating an initial backup plan.

[0109] In step S5.4, based on the initial backup plan, the storage resource status is further evaluated, compression parameter settings are optimized, and a detailed storage strategy plan is formed. First, the current status of the backup storage system is analyzed, including the available space, I / O performance, reliability level, and cost characteristics of each storage device. The storage requirements for each backup task are estimated, considering factors such as the original data size, expected compression ratio, and backup type (full or incremental). Then, based on data importance and access frequency, an appropriate storage tier is selected for each backup task: critical data may be stored on high-performance, high-reliability storage devices; historical data may be stored on large-capacity but lower-performance storage devices; and rarely accessed archived data may be stored on offline or near-line storage media. The data compression strategy can also be optimized by selecting appropriate compression algorithms and compression levels based on data type, importance, and system resource status: for situations with sufficient CPU resources but limited storage space, a high compression ratio but CPU-intensive algorithm may be selected; for situations with limited CPU resources, an algorithm with lower compression efficiency but lower CPU consumption may be selected; for critical data requiring rapid recovery, a lower compression level or even no compression may be selected. Data security requirements can also be considered, with encryption parameters configured for sensitive data, and appropriate encryption algorithms and key management schemes selected. Based on these analyses, a storage strategy plan is generated, which includes detailed information such as storage device allocation, compression parameter settings, encryption configuration, and expected storage space requirements for each backup task.

[0110] In step S5.5, based on the storage strategy, a detailed backup task description is generated, specific storage locations are allocated, and the final backup strategy execution plan is output. First, a unique task ID and description are created for each backup task, including data source information, backup type, expected execution time, and expected completion time. Then, based on the storage strategy, a specific storage location is allocated for each backup task, including storage device identifier, directory path, or object storage bucket name. Detailed backup parameter configurations are generated, including data source connection parameters, backup scope definition, filtering conditions, compression and encryption settings, and verification options. For incremental backup tasks, the location and identifier of the baseline backup are specified to ensure correct application of incremental backups. Resource limitation parameters for backup tasks are set, such as maximum CPU utilization, memory limits, and I / O bandwidth limits, to prevent backup tasks from affecting normal system operation. For backups of related data, dependencies and execution order between tasks are set to ensure data consistency. Monitoring and reporting parameters for backup tasks are configured, defining the metrics to be recorded and the types of reports to be generated. Finally, a complete backup strategy execution plan is output, including detailed descriptions of all backup tasks, execution parameters, resource limits, dependencies, and monitoring configurations, providing detailed guidance for subsequent parallel data backup execution.

[0111] In step S5, parallel data backup is performed, including:

[0112] Step S5.6: Based on the backup strategy execution plan, establish a backup progress monitoring mechanism, initialize the backup thread pool, and obtain the backup environment configuration;

[0113] Step S5.7: Based on the backup environment configuration, assign backup tasks to worker threads, set resource usage limits, and obtain a task allocation scheme;

[0114] Step S5.8: Based on the task allocation scheme, perform data compression and transmission operations, monitor the backup rate, and obtain the real-time backup status;

[0115] Step S5.9: Based on the real-time backup status, detect backup anomalies such as data transmission interruption, insufficient storage space, and data verification failure, execute failed task retry, and obtain an anomaly handling log;

[0116] Step S5.10: Based on the exception handling log, verify the integrity of the backup data, record the backup metadata, and output the data backup execution status and results.

[0117] In step S5.6, based on the backup strategy execution plan, a backup progress monitoring mechanism is established, and the backup thread pool is initialized. First, a backup task monitor is created to track the execution status and progress of all backup tasks. The monitor sets multiple monitoring points to periodically collect task execution data, including the amount of data processed, the current processing rate, the estimated remaining time, and resource usage. An alarm mechanism is established to trigger alarms of the appropriate level when task execution is abnormal or progress deviates significantly from expectations. Simultaneously, the backup thread pool is initialized, determining the initial number of threads and resource quotas based on resource allocation recommendations in the backup strategy execution plan and the current system load. Thread creation policies, task queue management policies, and exception handling policies are configured for the backup thread pool to ensure efficient and stable operation. Shared resources required for backup, such as buffer pools, connection pools, and temporary storage areas, are initialized, and resource allocation and reclamation policies are set. The backup data stream processing pipeline is configured, including data reading, filtering, transformation, compression, encryption, and writing stages, with appropriate parameters set for each stage. Finally, a backup environment configuration is generated, including information such as monitoring mechanism settings, thread pool configuration, resource initialization status, and processing pipeline parameters.

[0118] In step S5.7, based on the backup environment configuration, backup tasks are assigned to worker threads, and resource usage limits are set. First, the execution order of tasks is determined according to their priority and dependencies. For tasks without dependencies, high-priority tasks are processed first; for tasks with dependencies, the dependent task is executed first. A task scheduling algorithm, such as priority queue scheduling or fair shared scheduling, is used to assign backup tasks to worker threads in the backup thread pool. The allocation process considers load balancing to avoid some threads being overloaded while others are idle. Resource usage limits are set for each backup task, including CPU utilization limits, memory allocation limits, and I / O bandwidth limits, to ensure that backup tasks do not excessively consume system resources and affect other services. More resource quotas are allocated to high-priority tasks; stricter resource limits are set for low-priority tasks. Considering the principle of data locality, tasks accessing the same or similar storage devices are assigned to the same or similar threads as much as possible to reduce data transfer overhead. For large backup tasks, they are broken down into multiple subtasks and assigned to multiple threads for parallel processing to improve execution efficiency. Finally, a task allocation scheme is generated, which includes detailed information such as thread allocation, execution order, resource limits, and subtask division (if any) for each backup task.

[0119] In step S5.8, based on the task allocation scheme, data compression and transmission operations are performed, and the backup rate is monitored in real time. Each backup thread begins data backup operations according to its assigned task. For full backup tasks, the thread first establishes a connection with the data source and reads all target data; for incremental backup tasks, the thread needs to identify the data that has changed since the last backup and only backs up the changed parts. The thread performs a series of processing operations on the data according to the configured processing pipeline: first, data filtering is performed to exclude data items that do not need to be backed up; then, data transformation is performed to convert the data into a format suitable for backup; next, compression algorithms are applied to reduce the data volume; if encryption requirements are configured, the data will also be encrypted; finally, the processed data is written to the target storage location. During execution, the backup rate and resource usage are monitored in real time, including data read rate, processing rate, transmission rate, and write rate. Processing parameters, such as buffer size, batch size, and parallelism, are dynamically adjusted to optimize backup performance. The backup progress percentage and estimated completion time are calculated periodically, and task status information is updated. For tasks whose execution time exceeds the expected time, their resource allocation or priority is adjusted to ensure that critical tasks can be completed in a timely manner. This monitoring data is integrated into a real-time backup status, including information such as the progress, rate, resource usage, and estimated completion time of each task.

[0120] In step S5.9, various backup anomalies are detected and handled based on the real-time backup status. A comprehensive anomaly detection mechanism is implemented, capable of identifying various problems that may occur during the backup process: data source connection anomalies, such as connection interruption, authentication failure, or insufficient permissions; data read anomalies, such as non-existent files, corrupted data, or incorrect formatting; processing anomalies, such as compression failure, encryption errors, or insufficient memory; transmission anomalies, such as network interruption, timeout, or insufficient bandwidth; storage anomalies, such as insufficient storage space, write failure, or device failure; and consistency anomalies, such as inconsistencies caused by data modification during the backup process. For detected anomalies, corresponding handling strategies are adopted according to the anomaly type and severity: for temporary anomalies (such as network fluctuations), automatic retry operations are performed, typically with a maximum number of retries and a backoff strategy set; for resource-related anomalies (such as insufficient storage space), attempts are made to release resources or find alternative resources; for unrecoverable severe anomalies, the current task is terminated, detailed error information is recorded, and the administrator is notified. A task recovery mechanism can also be used, allowing interrupted backup tasks to resume execution from the breakpoint, avoiding resource waste caused by restarting. All abnormal situations and handling operations are recorded in the exception handling log, which includes information such as exception time, type, detailed description, scope of impact, handling operations and results.

[0121] In step S5.10, based on the exception handling log, the integrity of the backup data is verified, backup metadata is recorded, and the final backup execution status and results are output. For completed backup tasks, a backup verification operation is performed to ensure the integrity and recoverability of the backup data. The verification process includes: checksum verification, calculating the checksum of the backup data and comparing it with the checksum of the original data; structural verification, ensuring the structural integrity of the backup data, such as the database table structure and file directory structure; sampled content verification, randomly selecting some data items and comparing the backup content with the original content; and recovery testing, performing simulated recovery operations on a portion of the backup data to verify the feasibility of the recovery process. For backups that pass verification, detailed backup metadata is recorded, including: backup identification information, such as backup ID, name, and description; time information, such as start time, completion time, and duration; content summary, such as the data type, range, and quantity of the backup; storage information, such as storage location, file name, and size; verification information, such as checksum, digital signature, and verification result; and association information, such as the base backup ID (for incremental backups) and dependencies. This metadata is stored in the backup directory, and the backup metadata database is updated simultaneously for subsequent querying and management. For backups that fail verification, mark their status as "failed" and record the reason for the failure and the scope of impact. Finally, generate a complete data backup execution status and result report, including the execution status of all backup tasks, success rate, anomaly statistics, performance indicators, and recommended actions. This report serves as a summary of this backup operation and provides a basis for subsequent backup strategy optimization and troubleshooting.

[0122] like Figure 3 As shown in the figure, this embodiment of the invention also provides a data verification and backup device for a power control center, comprising:

[0123] System resource assessment module 10 is used to obtain hardware resource information corresponding to the operating system, calculate effective resource parameters based on the hardware resource information, construct a thread resource pool, and obtain a system resource capability assessment report.

[0124] The data priority classification module 20 is connected to the system resource evaluation module 10. It is used to connect to the power control center data source, classify the data source by data type and rate its importance, and use a parameterized hybrid time algorithm to perform real-time demand analysis to obtain a data task queue with priority marking.

[0125] The thread resource allocation module 30, connected to the data priority division module 20, is used to monitor the load status and task queue length of the operating system, apply an online distributed queue length estimation algorithm to evaluate resources, dynamically create worker threads according to the load threshold of the operating system and the backlog of the task queue, and allocate CPU time slices, memory quotas and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation scheme.

[0126] The data verification module 40 is connected to the thread resource allocation module 30. It is used to allocate the verification task to the working thread based on the thread resource allocation scheme, perform parallel data verification using a sub-exponential algorithm on an independent set, and summarize the verification results of each thread to obtain a data verification result set.

[0127] The data backup module 50 is connected to the data verification module 40 and is used to evaluate backup requirements based on the data verification result set, determine backup strategies, perform parallel data backup, and obtain data backup execution status and results.

[0128] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for data verification and backup in a power control center, characterized in that, Includes the following steps: Obtain the hardware resource information corresponding to the operating system, calculate the effective resource parameters based on the hardware resource information, construct the thread resource pool, and obtain the system resource capability assessment report; Connect to the data source of the power control center, classify the data source by data type and rate its importance, and use a parameterized hybrid time algorithm to perform real-time demand analysis to obtain a data task queue with priority marking; Monitor the load status and task queue length of the operating system, apply an online distributed queue length estimation algorithm to evaluate resources, dynamically create worker threads based on the load threshold of the operating system and the backlog of the task queue, and allocate CPU time slices, memory quotas and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation scheme. Based on the aforementioned thread resource allocation scheme, the verification task is assigned to the worker thread, and the sub-exponential algorithm on the independent set is used to perform parallel data verification. The verification results of each thread are then summarized to obtain the data verification result set. Based on the data verification result set, assess backup requirements, determine backup strategies, execute parallel data backup, and obtain data backup execution status and results; Obtain hardware resource information corresponding to the operating system, and evaluate effective resource parameters based on the hardware resource information, including: The system API is used to obtain the number of processor cores, memory capacity, and I / O bandwidth information to obtain the original hardware resource parameter set. Based on the original hardware resource parameter set, the current process usage data is read, and the allocable CPU time slice, available memory space, and I / O bandwidth are calculated to obtain the effective resource parameter set. Based on the set of effective resource parameters, set the minimum and maximum thresholds of the thread pool, initialize the thread communication mechanism, and obtain the initial state of the thread resource pool. Based on the initial state of the thread resource pool and the set of effective resource parameters, calculate the number of verification and backup tasks that can be processed concurrently, and output a system resource capacity assessment report; The data sources are categorized by data type and ranked by importance, including: It receives data from the power control center, obtains data structure description and update frequency information, classifies the data by type, and obtains the data source classification results. Based on the data source classification results, data access frequency and data dependency information are obtained, data importance index is calculated, and data importance rating results are obtained. Based on the data importance rating results, a directed dependency graph is constructed, and the criticality index is calculated to obtain the criticality of data nodes; Based on the criticality of the data nodes, a sub-index algorithm on independent sets is applied to identify key data nodes, and a data classification and rating report is output.

2. The method according to claim 1, characterized in that, A parameterized hybrid time algorithm is used for real-time requirement analysis, including: Obtain data type and business requirement information, determine the effective time window for data, and obtain the time window parameter set; Based on the time window parameter set, a data timeliness model is established, the data decay rate is calculated, and the decay curve parameters are obtained. Based on the attenuation curve parameters, the service's tolerance for data latency is assessed, the maximum allowable latency time is calculated, and the timeliness assessment result is obtained. Based on the timeliness assessment results and the data importance rating results, a comprehensive priority index is calculated, a multi-level task queue is constructed, and a data task queue with priority tags is output.

3. The method according to claim 2, characterized in that, Resource assessment is performed using an online distributed queue length estimation algorithm, including: The system periodically acquires CPU utilization, memory usage, and I / O wait time, calculates the current system load level, and obtains a load status report of the operating system. Based on the load status report of the operating system, the backlog of tasks of each priority is analyzed, the task growth trend is predicted, and the task queue length is estimated. Based on the task queue length estimation results, the task waiting time and system resource reserves are calculated, the thread creation strategy is determined, and the thread creation decision results are obtained. Based on the thread creation decision, the thread management interface is called to create worker threads, allocate resource quotas, and output the optimized thread resource allocation scheme.

4. The method according to claim 3, characterized in that, Parallel data verification is performed using a sub-exponential algorithm on independent sets, including: Analyze the logical structure and dependencies of the data, decompose the dataset into multiple logical subsets, construct a data dependency graph based on the reference relationships and update sequence between data, and obtain a data decomposition scheme. Based on the data decomposition scheme, a set of data blocks that are independent of each other is identified, an independent set partitioning strategy is determined, and an independent set partitioning result is obtained. Based on the independent set partitioning results, a verification task reorganization is performed on each independent set, and parallel processing parameters are set to obtain a subtask partitioning scheme. Based on the aforementioned subtask partitioning scheme, a streaming processing mechanism is established to control the data loading granularity and obtain the verification execution plan; According to the verification execution plan, the verification algorithm is invoked to perform parallel data verification, the verification process data is recorded, and the independent set verification results are output.

5. The method according to claim 4, characterized in that, Summarize the verification results of each thread, including: Monitor the verification progress of each worker thread, collect intermediate verification results, establish a result cache, and obtain a thread verification status report; Based on the thread verification status report, the results within the thread group are merged to verify data consistency and obtain the merged results within the group. Based on the intra-group merging results, cross-thread group result merging is performed, the integrity of reference relationships is checked, and a global merging result is obtained. Based on the global merging results, verification statistics and a problem list are generated, verification efficiency is evaluated, and a performance analysis report is obtained. Based on the performance analysis report, the execution results are stored hierarchically, a verification issue index is established, and a data verification result set is output.

6. The method according to claim 1, characterized in that, Assess backup needs and determine backup strategies, including: Analyze the data verification result set, obtain the data integrity status and change information, calculate the data change rate, and obtain the data status assessment result; Based on the data status assessment results, the last backup time and storage space information are obtained, the backup priority is assessed, and a backup priority list is obtained. Based on the backup priority list, a parameterized hybrid time algorithm is applied to calculate the backup time window, and a backup method is selected to obtain the initial backup plan; Based on the initial backup scheme, the storage resource status is evaluated, compression parameter settings are optimized, and a storage strategy scheme is obtained. Based on the storage strategy scheme, a backup task description is generated, storage locations are allocated, and a backup strategy execution plan is output.

7. The method according to claim 6, characterized in that, Perform parallel data backups, including: Based on the backup strategy execution plan, a backup progress monitoring mechanism is established, the backup thread pool is initialized, and the backup environment configuration is obtained. Based on the backup environment configuration, backup tasks are assigned to worker threads, resource usage limits are set, and a task allocation scheme is obtained. Based on the task allocation scheme, data compression and transmission operations are performed, backup rate is monitored, and real-time backup status is obtained. Based on the real-time backup status, backup anomalies such as data transmission interruption, insufficient storage space, and data verification failure are detected, and failed tasks are retried to obtain an anomaly handling log. Based on the exception handling log, verify the integrity of the backup data, record the backup metadata, and output the data backup execution status and results.

8. A data verification and backup device for a power control center, characterized in that, include: The system resource assessment module is used to obtain hardware resource information corresponding to the operating system, calculate effective resource parameters based on the hardware resource information, construct a thread resource pool, and obtain a system resource capability assessment report. The data priority classification module is connected to the system resource assessment module. It is used to connect to the power control center data source, classify the data source by data type and rate its importance, and use a parameterized hybrid time algorithm to perform real-time demand analysis to obtain a data task queue with priority tags. The thread resource allocation module, connected to the data priority division module, is used to monitor the load status and task queue length of the operating system, apply an online distributed queue length estimation algorithm to evaluate resources, dynamically create worker threads based on the load threshold of the operating system and the backlog of the task queue, and allocate CPU time slices, memory quotas and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation scheme. The data verification module, connected to the thread resource allocation module, is used to allocate verification tasks to worker threads based on the thread resource allocation scheme, perform parallel data verification using a sub-exponential algorithm on independent sets, and summarize the verification results of each thread to obtain a data verification result set. A data backup module, connected to the data verification module, is used to evaluate backup requirements based on the data verification result set, determine backup strategies, perform parallel data backup, and obtain data backup execution status and results. Obtain hardware resource information corresponding to the operating system, and evaluate effective resource parameters based on the hardware resource information, including: The system API is used to obtain the number of processor cores, memory capacity, and I / O bandwidth information to obtain the original hardware resource parameter set. Based on the original hardware resource parameter set, the current process usage data is read, and the allocable CPU time slice, available memory space, and I / O bandwidth are calculated to obtain the effective resource parameter set. Based on the set of effective resource parameters, set the minimum and maximum thresholds of the thread pool, initialize the thread communication mechanism, and obtain the initial state of the thread resource pool. Based on the initial state of the thread resource pool and the set of effective resource parameters, calculate the number of verification and backup tasks that can be processed concurrently, and output a system resource capacity assessment report; The data sources are categorized by data type and ranked by importance, including: It receives data from the power control center, obtains data structure description and update frequency information, classifies the data by type, and obtains the data source classification results. Based on the data source classification results, data access frequency and data dependency information are obtained, data importance index is calculated, and data importance rating results are obtained. Based on the data importance rating results, a directed dependency graph is constructed, and the criticality index is calculated to obtain the criticality of data nodes; Based on the criticality of the data nodes, a sub-index algorithm on independent sets is applied to identify key data nodes, and a data classification and rating report is output.

Images